Fiona: Circular imports for Python 3.8 on Windows if DLLs are already loaded

[rbuffat]

WIth fiona 1.8.15, the issue is fixed now in the conda-forge builds, only the Windows py3.8 build is still failing: https://github.com/conda-forge/fiona-feedstock/pull/167

The failure is with the general error message of partially initialized module, so doesn't give much pointers (at least to me):

running run_test.py
Traceback (most recent call last):
  File "D:\bld\fiona_1599200296823\test_tmp\run_test.py", line 6, in <module>
    import fiona
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\__init__.py", line 84, in <module>
    import fiona._loading
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\_loading.py", line 44, in <module>
    import fiona.ogrext
  File "fiona/ogrext.pyx", line 29, in init fiona.ogrext
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\env.py", line 14, in <module>
    with fiona._loading.add_gdal_dll_directories():
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)

_Originally posted by @jorisvandenbossche in https://github.com/Toblerity/Fiona/issues/941#issuecomment-686982973_

[rbuffat]

The issue is with https://github.com/Toblerity/Fiona/blob/52ad006172bdb0d9c240bd827b958a58c63e5303/fiona/_loading.py#L43-L46

Starting with Python 3.8 dll directories must be added with add_dll_directory() on Windows. (https://github.com/Toblerity/Fiona/issues/851)

On Fionas appveyor, the import of import fiona.ogrext will fail as there are no DLL presents at the time of import, or fiona.ogrext is already loaded. But in the case of conda, the DLLs are already loaded. Thus the import of fiona.ogrext does not raise an ImportError and fiona.ogrext will import fiona.env, that again imports fiona._loading, which leads to a circular import.

A quick fix could be to change the test import to something that does not import anything else from Fiona. E.g.

 try: 
     import fiona._err 
 except ImportError as e: 
     search_gdal_dll_directories() 

But this will be quite dangerous because with adding additional imports the problem can arise again.

[sgillies]

@rbuffat as we solve this, let's also move this code

if platform.system() == 'Windows' and sys.version_info >= (3, 8):
    # if loading of extension modules fails, search for gdal dll directories
    try:
        import fiona.ogrext
    except ImportError as e:
        search_gdal_dll_directories()

into add_gdal_dll_directories(). Code that runs on import only is difficult to test.

[rbuffat]

I created a PR in https://github.com/Toblerity/Fiona/pull/945. If you have already implemented something feel free to close it.

[rbuffat]

Fixed in Fiona 1.8.16

[iboates]

I'm leaving a comment here because it's the only Google result which matched my error exactly. Hopefully it can save someone else a similar headache.

I am using a conda environment when getting this error and it was solved by starting a new environment where I made sure that every package was being taken from the conda-forge channel (except for some libraries only available from the esri channel.

[janhurst]

I'm leaving a comment here because it's the only Google result which matched my error exactly. Hopefully it can save someone else a similar headache.

I am using a conda environment when getting this error and it was solved by starting a new environment where I made sure that every package was being taken from the conda-forge channel (except for some libraries only available from the esri channel.

I am seeing the same behaviour with conda 4.9.2 on Windows 10 trying to get geopandas up and running. Creating an environment and forcing only conda-forge packages. As best as I can tell everything is "correctly" coming from conda-forge.

various versions are:
gdal 3.2.2
fiona 1.8.18
geopandas 0.9.0

[jorisvandenbossche]

@janhurst can you post the output of conda list in the environment?

[jjohnarios]

@jorisvandenbossche I'm getting the same error with the same conda and packages versions as @janhurst . I have some non conda-forge packages which i believe have nothing to do with geopandas. I'm posting the exact error i'm getting and my conda list output.

AttributeError                            Traceback (most recent call last)
<ipython-input-7-a62d01c1d62e> in <module>
----> 1 import geopandas as gpd

~\Anaconda3\envs\nilmtk-env\lib\site-packages\geopandas\__init__.py in <module>
      5 from geopandas.array import points_from_xy  # noqa
      6 
----> 7 from geopandas.io.file import _read_file as read_file  # noqa
      8 from geopandas.io.arrow import _read_parquet as read_parquet  # noqa
      9 from geopandas.io.arrow import _read_feather as read_feather  # noqa

~\Anaconda3\envs\nilmtk-env\lib\site-packages\geopandas\io\file.py in <module>
     18 
     19 try:
---> 20     from fiona import Env as fiona_env
     21 except ImportError:
     22     try:

~\Anaconda3\envs\nilmtk-env\lib\site-packages\fiona\__init__.py in <module>
     83 
     84 import fiona._loading
---> 85 with fiona._loading.add_gdal_dll_directories():
     86     from fiona.collection import BytesCollection, Collection
     87     from fiona.drvsupport import supported_drivers

AttributeError: module 'fiona' has no attribute '_loading'

conda_list.txt

[janhurst]

I noticed that the environment seems to work ok when i try to read a file in a notebook when using jupyter lab, yet the same notebook will throw the error I posted when running inside vscode.

@jjohnarios do you happen to be using vscode?

I dug a bit further. If I start vscode from a Powershell prompt (eg code . and note using the base conda environment), then open a notebook, and set the notebook kernel to my Python interpreter I am seeing the issue. I confirmed that sys.version and sys.executable seem to be pointing to the "geo" conda environment correctly, just the import geopandas call failing as indicated above.

However, If I first activate the "geo" environment (eg conda activate geo) in Powershell, and then do the same process of starting vscode and set the notebook kernel etc the error does not occur!

I'm pretty confused now :) ... I don't know if this is a vscode issue, or perhaps something screwy in conda and its Powershell integration etc.

@jorisvandenbossche for reference my conda list output is attached, as well as the environment yaml.

geo.yml.txt
conda_list.txt

[jjohnarios]

@janhurst I've tried vscode and import works fine. I was using jupyter notebook and spyder and the error kept occurring there. I made sure i was using the same kernel and environment. Consequently, i realised that : import geopandas works for me in python Anaconda prompt, in VScode, in Sublime Text and NOT IN jupyter notebook and spyder.

[andrewnell]

I have the exact same issue, where import geopandas works for me in the anaconda prompt and sublime, however, I get the same error as above in Jupyter Notebook, Jupyter Lab or Spyder.

It should also be noted that this is only for an environment I have created using Python 3.6, however, if I create an environment with Python 3.8 I do not have this issue. @jorisvandenbossche any ideas what may be causing this issue?

[rbuffat]

@jorisvandenbossche do you have an idea how to debug this? I don't have access to a Windows machine at the moment, so the only possibility would be to create a CI workflow that would reproduce this. But for that, we would require the exact steps to reproduce this issue.

Maybe placing print(dir(fiona)) in front of from fiona import Env as fiona_env in https://github.com/geopandas/geopandas/blob/master/geopandas/io/file.py#L20
might reveal some more useful information.

[jorisvandenbossche]

I also don't have a Windows machine readily available.
Since the issue seem to be triggered by using an environment that is not directly activated, but used as a kernel from some UI (notebook, spyder), one possible issue could be that fiona is not finding/looking for the correct GDAL location (although I still find it a strange error message for that ..).

@janhurst @jjohnarios @andrewnell can you try if setting the following environment variable before importing geopandas/fiona inside the notebook / spyder helps? It should be the equivalent of:

@set "GDAL_DATA=%CONDA_PREFIX%\Library\share\gdal"

so in terms of python code you can run in the notebook before importing geopandas/fiona, I think it would be something like

import os
os.environ["GDAL_DATA"] = os.environ["CONDA_PREFIX"] + "\Library\share\gdal"

[jjohnarios]

I also don't have a Windows machine readily available.
Since the issue seem to be triggered by using an environment that is not directly activated, but used as a kernel from some UI (notebook, spyder), one possible issue could be that fiona is not finding/looking for the correct GDAL location (although I still find it a strange error message for that ..).

@janhurst @jjohnarios @andrewnell can you try if setting the following environment variable _before_ importing geopandas/fiona inside the notebook / spyder helps? It should be the equivalent of:

@set "GDAL_DATA=%CONDA_PREFIX%\Library\share\gdal"

so in terms of python code you can run in the notebook before importing geopandas/fiona, I _think_ it would be something like

import os
os.environ["GDAL_DATA"] = os.environ["CONDA_PREFIX"] + "\Library\share\gdal"

I've tried that but doesn't seem to work. Same error message.

[jorisvandenbossche]

@jjohnarios OK, thanks for trying!

Aonther question: you posted above your conda list output, and indeed I don't directly see anything suspicious (the packages no from conda-forge but from pypi shouldn't have an impact on fiona). But could you try if you can reproduce it with a minimal, newly created environment as well?
Eg like the sequence of commands at https://geopandas.readthedocs.io/en/latest/getting_started/install.html#creating-a-new-environment

[janhurst]

Also didn't see any difference setting GDAL_DATA.

[jorisvandenbossche]

Another question (sorry for the many questions, but that's at the moment the only way for me to help debug it): do you only see it with the latest fiona 1.8.18? If you explicitly install fiona 1.8.17 in the environment, do you still see the error?

Or if you explicitly ask for an older GDAL? (you both seem to have 3.2.2 installed, which is relatively new (which might explain that it's a new issue)) Do you also have the error when creating the environment with fiona=1.8.18 gdal=3.1?

[janhurst]

But could you try if you can reproduce it with a minimal, newly created environment as well?
Eg like the sequence of commands at https://geopandas.readthedocs.io/en/latest/getting_started/install.html#creating-a-new-environment

I tried to follow the getting started linked as exactly as possible. The only addition is that the ipykernel package is needed to actually switch to the kernel in vscode's notebook environment.

Unfortunately I am still seeing the same behaviour - starting vscode from the base environment and trying to select the geo_env kernel results in the module 'fiona' has no attribute '_loading' message in an AttributeError exception.

For reference the conda list output for the environment as described in the geopandas getting started plus conda install ipykernel is attached.
geo_env_conda_list.txt

[janhurst]

Do you also have the error when creating the environment with fiona=1.8.18 gdal=3.1?

Missed this, its late here but I'll give this a shot in the next day or two.

[RicardoSQueen]

Hey, there,

Having the same issue as @janhurst.

Another question (sorry for the many questions, but that's at the moment the only way for me to help debug it): do you only see it with the latest fiona 1.8.18? If you explicitly install fiona 1.8.17 in the environment, do you still see the error?

Tried with fiona 1.8.17 and 1.8.16 and still no success (same error)

Or if you explicitly ask for an older GDAL? (you both seem to have 3.2.2 installed, which is relatively new (which might explain that it's a new issue)) Do you also have the error when creating the environment with fiona=1.8.18 gdal=3.1?

Tried that too and still no success. Here is my current conda list output:
conda_list.txt

Just reforcing: on anaconda prompt the import occurs successfully, on vscode it does not work.

[fargiolas]

Probably a different bug, but I'm seeing a similar circular import error when I import geopandas (fiona 1.18.18, geopandas 0.9.0, gdal 3.2.2, installed with pip and brew on macos). The error goes away if I import fiona before geopandas.

[jorisvandenbossche]

OK, still a mystery .. (and thanks all for the testing!)
Some other things to verify / try:

1. Does this only happen with geopandas 0.9, or also with older versions (0.8)?

2. Something that changed in the latest geopandas version, is that we delayed the fiona import (https://github.com/geopandas/geopandas/pull/1775), but which might have as concrete effect that the order of imports of the different geo-reletad packages changed.
So maybe something to try, does one of the following import sequences also trigger the error? (in the notebook/spyder/vscode environment)

import pyproj
import fiona

import shapely
import fiona

3. Another possible "pattern" in the posted conda list outputs, is that you all have pyproj 3.0.1. Could someone try creating an environment but with an older version of pyproj? (3.0.0 or 2.6.1)?
(not directly an idea how that would influence it, but you never know ... and to eliminate possibilities)

[ogelin]

Hi, on MacOS using pip installs I also had this error when importing geopandas. Python 3.9.2.

I changed pyproj from 3.0.1 to 3.0.0 and now the error goes away as long as I import fiona before geopandas.

If I only import geopandas without fiona first, I still get the "Circular import" error message that others reported.

[LSgeo]

I noticed that the environment seems to work ok when I try to read a file in a notebook when using jupyter lab, yet the same notebook will throw the error I posted when running inside vscode.

@jjohnarios do you happen to be using vscode?

I dug a bit further. If I start vscode from a Powershell prompt (eg code . and note using the base conda environment), then open a notebook, and set the notebook kernel to my Python interpreter I am seeing the issue. I confirmed that sys.version and sys.executable seem to be pointing to the "geo" conda environment correctly, just the import geopandas call failing as indicated above.

However, If I first activate the "geo" environment (eg conda activate geo) in Powershell, and then do the same process of starting vscode and set the notebook kernel etc the error _does not occur_!

I'm pretty confused now :) ... I don't know if this is a vscode issue, or perhaps something screwy in conda and its Powershell integration etc.

@jorisvandenbossche for reference my conda list output is attached, as well as the environment yaml.

geo.yml.txt
conda_list.txt

Hi, I've stumbled across here too, and wanted to add my two cents. I'm running a script in VScode on windows 10, and only encounter the issue when executing via a Jupyter Interactive notebook session.

I've encountered a lot of other dll issues with other libraries in the Jupyter Interactive window, that don't occur when running the script normally via the command line. There are a lot (of vaguely similar problems) spread across various github issues, but e.g. , vscode-jupyter and numpy

As you mention in the above comment, activating an environment and calling code to start vscode works around the issue for me. I'd love to learn why this is, and if there is a better solution.

[jorisvandenbossche]

Some general background context: conda has the concept of "environment activation" (which happens if you manually activate an environment in the console with conda activate env_name). This step sets up the environment to work properly, and for example sets the appropriate environment variables (eg it adds the env directory and the env's /bin directory to the PATH).

Specifically for fiona, we rely on GDAL, and the libgdal package from conda-forge has a specific activation script (see https://github.com/conda-forge/gdal-feedstock/tree/master/recipe/scripts), which sets the GDAL_DATA environment variable. In addition, I think fiona relies on being able to find the GDAL binary based on PATH.

If those environment variables are not properly set, packages will have problems finding the correct binaries / libraries.

And I suppose this is the problem that causes the issues you all encounter when using jupyter notebook or vscode.

For Jupyter Notebook with using kernels based on conda environments, this is a known problem that this usage pattern doesn't automatically activate the environment when starting the kernel (https://github.com/jupyter/notebook/issues/3588). One workaround is to install notebook in the environment you want to use, first activate the environment, and then start the notebook from that environment (this defeats the purpose of having multiple kernels a bit, though). Another solution is to use a tool like https://github.com/Anaconda-Platform/nb_conda_kernels that ensures kernels are activated properly. Finally, you could also prepend the appropriate ..\\envs\\env_name\\Library\\bin to your PATH manually in the notebook before importing fiona/geopandas.

For VSCode, I have less experience with this, but according to https://code.visualstudio.com/docs/python/environments#_select-and-activate-an-environment it seems that it should be possible to adjust your settings to ensure the environment is activated.

---

Now, what I am less sure about is why it seems that there now new issues with this (given that multiple people posted here in the last weeks), and if something changed in fiona / gdal / the conda packages to make this problem worse / easier to run into.
(unless this is just coincidental)

[fargiolas]

Now, what I am less sure about is why it seems that there now _new_ issues with this (given that multiple people posted here in the last weeks), and if something changed in fiona / gdal / the conda packages to make this problem worse / easier to run into.
(unless this is just coincidental)

I've been trying to track this on my setup (macos, arm64, packages installed with brew and pip3, no conda) with little success. The first fiona import geopandas/io/file.py fails with:

>>> import geopandas Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/homebrew/lib/python3.9/site-packages/geopandas/__init__.py", line 7, in <module> from geopandas.io.file import _read_file as read_file # noqa File "/opt/homebrew/lib/python3.9/site-packages/geopandas/io/file.py", line 20, in <module> from fiona import Env as fiona_env File "/opt/homebrew/lib/python3.9/site-packages/fiona/__init__.py", line 85, in <module> with fiona._loading.add_gdal_dll_directories(): AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)

But if I wrap a try/except around all the fiona imports in fiona/__init__.py after with fiona._loading.add_gdal_dll_directories(): the exception I get is the following:

>>> import geopandas exception: dlopen(/opt/homebrew/lib/python3.9/site-packages/fiona/ogrext.cpython-39-darwin.so, 2): Symbol not found: _GEOSArea Referenced from: /opt/homebrew/opt/libspatialite/lib/libspatialite.7.dylib Expected in: flat namespace in /opt/homebrew/opt/libspatialite/lib/libspatialite.7.dylib Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/homebrew/lib/python3.9/site-packages/geopandas/__init__.py", line 7, in <module> from geopandas.io.file import _read_file as read_file # noqa File "/opt/homebrew/lib/python3.9/site-packages/geopandas/io/file.py", line 12, in <module> import fiona File "/opt/homebrew/lib/python3.9/site-packages/fiona/__init__.py", line 112, in <module> __gdal_version__ = get_gdal_release_name() NameError: name 'get_gdal_release_name' is not defined

The same doesn't happen if I import fiona directly, only from geopandas. I tried setting gdal datadir with export GDAL_DATA=$(gdal-config --datadir) but it does not help.

There are some old mac specific bugs about the missing symbol in libspatialite, see e.g. https://github.com/Homebrew/homebrew-core/issues/5161 maybe some recent change triggered them again.

Still digging, I'll comment if I find anything new, if you have some test you'd like me to run let me know.

[907Resident]

Hi, on MacOS using pip installs I also had this error when importing geopandas. Python 3.9.2.

I changed pyproj from 3.0.1 to 3.0.0 and now the error goes away as long as I import fiona before geopandas.

If I only import geopandas without fiona first, I still get the "Circular import" error message that others reported.

This worked for me using the same Python interpreter but on a Windows. However, I can only make it work using a script in vscode and not in notebook

[rbuffat]

I just got the AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import) with geopandas on Linux having a broken local install of Fiona.

The way geopandas imports Fiona (or how Fiona uses the _loading module) seems to suppress error messages when importing Fiona.

Steps to reproduce: Install fiona using pip. Remove /usr/lib/python3.9/site-packages/Fiona.libs/libgdal-d962753e.so.28.0.2

Python 3.9.4 (default, Apr 20 2021, 15:51:38) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fiona
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/fiona/__init__.py", line 86, in <module>
    from fiona.collection import BytesCollection, Collection
  File "/usr/lib/python3.9/site-packages/fiona/collection.py", line 11, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: libgdal-d962753e.so.28.0.2: cannot open shared object file: No such file or directory
>>> import geopandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/geopandas/__init__.py", line 7, in <module>
    from geopandas.io.file import _read_file as read_file  # noqa
  File "/usr/lib/python3.9/site-packages/geopandas/io/file.py", line 12, in <module>
    import fiona
  File "/usr/lib/python3.9/site-packages/fiona/__init__.py", line 85, in <module>
    with fiona._loading.add_gdal_dll_directories():
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)

[amylastuka]

Probably a different bug, but I'm seeing a similar circular import error when I import geopandas (fiona 1.18.18, geopandas 0.9.0, gdal 3.2.2, installed with pip and brew on macos). The error goes away if I import fiona before geopandas.

Thank you, I had this same problem and importing fiona first fixed it for me!