Archivebox: DJango branch: circular import problem: crontab and logging

[terxw]

Describe the bug

After updating to newest django branch I no longer can launch app

Steps to reproduce

1. Run ArchiveBox e.g. ./bin/archivebox add

Screenshots or log output

Traceback (most recent call last):                                                                                                                                                                                                            
  File "/home/kangus/src/ArchiveBox/bin/archivebox", line 7, in <module>                                                                                                                                                                      
    from .cli import main                                                                                                                                                                                                                     
  File "/home/kangus/src/ArchiveBox/archivebox/cli/__init__.py", line 65, in <module>                                                                                                                                                         
    SUBCOMMANDS = list_subcommands()                                                                                                                                                                                                          
  File "/home/kangus/src/ArchiveBox/archivebox/cli/__init__.py", line 41, in list_subcommands                                                                                                                                                 
    module = import_module('.archivebox_{}'.format(subcommand), __package__)                                                                                                                                                                  
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module                                                                                                                                                                 
    return _bootstrap._gcd_import(name[level:], package, level)                                                                                                                                                                               
  File "/home/kangus/src/ArchiveBox/archivebox/cli/archivebox_init.py", line 11, in <module>                                                                                                                                                  
    from ..main import init, docstring                                                                                                                                                                                                        
  File "/home/kangus/src/ArchiveBox/archivebox/main.py", line 8, in <module>                                                                                                                                                                  
    from crontab import CronTab, CronSlices                                                                                                                                                                                                   
  File "/usr/local/lib/python3.8/dist-packages/crontab.py", line 91, in <module>                                                                                                                                                              
    import logging                                                                                                                                                                                                                            
  File "/home/kangus/src/ArchiveBox/archivebox/logging.py", line 16, in <module>                                                                                                                                                              
    from .index.schema import Link, ArchiveResult                                                                                                                                                                                             
  File "/home/kangus/src/ArchiveBox/archivebox/index/__init__.py", line 13, in <module>                                                                                                                                                       
    from ..system import atomic_write                                                                                                                                                                                                         
  File "/home/kangus/src/ArchiveBox/archivebox/system.py", line 12, in <module>                                                                                                                                                               
    from crontab import CronTab                                                                                                                                                                                                               
ImportError: cannot import name 'CronTab' from partially initialized module 'crontab' (most likely due to a circular import) (/usr/local/lib/python3.8/dist-packages/crontab.py)                                                              
Error in sys.excepthook: 
Traceback (most recent call last):                                                                                                                                                                                                    [0/6631]
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 12, in <module>
    import os, glob, subprocess, os.path, time, pwd, sys, requests_unixsocket
  File "/usr/lib/python3/dist-packages/requests_unixsocket/__init__.py", line 1, in <module>
    import requests
  File "/usr/lib/python3/dist-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/usr/lib/python3/dist-packages/urllib3/__init__.py", line 7, in <module>
    from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 3, in <module>
    import logging
  File "/home/kangus/src/ArchiveBox/archivebox/logging.py", line 16, in <module>
    from .index.schema import Link, ArchiveResult
  File "/home/kangus/src/ArchiveBox/archivebox/index/__init__.py", line 13, in <module>
    from ..system import atomic_write
  File "/home/kangus/src/ArchiveBox/archivebox/system.py", line 12, in <module>
    from crontab import CronTab
  File "/usr/local/lib/python3.8/dist-packages/crontab.py", line 154, in <module>
    LOG = logging.getLogger('crontab')
AttributeError: partially initialized module 'logging' has no attribute 'getLogger' (most likely due to a circular import)

Original exception was:
Traceback (most recent call last):
  File "/home/kangus/src/ArchiveBox/bin/archivebox", line 7, in <module>
    from .cli import main
  File "/home/kangus/src/ArchiveBox/archivebox/cli/__init__.py", line 65, in <module>
    SUBCOMMANDS = list_subcommands()
  File "/home/kangus/src/ArchiveBox/archivebox/cli/__init__.py", line 41, in list_subcommands
    module = import_module('.archivebox_{}'.format(subcommand), __package__)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/kangus/src/ArchiveBox/archivebox/cli/archivebox_init.py", line 11, in <module>
    from ..main import init, docstring
  File "/home/kangus/src/ArchiveBox/archivebox/main.py", line 8, in <module>
    from crontab import CronTab, CronSlices
  File "/usr/local/lib/python3.8/dist-packages/crontab.py", line 91, in <module>
    import logging
  File "/home/kangus/src/ArchiveBox/archivebox/logging.py", line 16, in <module>
    from .index.schema import Link, ArchiveResult
  File "/home/kangus/src/ArchiveBox/archivebox/index/__init__.py", line 13, in <module>
    from ..system import atomic_write
  File "/home/kangus/src/ArchiveBox/archivebox/system.py", line 12, in <module>
    from crontab import CronTab
ImportError: cannot import name 'CronTab' from partially initialized module 'crontab' (most likely due to a circular import) (/usr/local/lib/python3.8/dist-packages/crontab.py)

Software versions

• OS: Linux Mint 20/ubuntu 20.04
• ArchiveBox version: 848977e
• Python version: python 3.8.2

[cdvv7788]

@terxw what page are you trying to index? I ran archivebox add https://example.com and it worked.
I will review it anyway, but if you can provide a little more information to reproduce the issue, that would be helpful.

[cdvv7788]

I was able to reproduce it!

[cdvv7788]

@terxw I had the issue with circular imports, but not from crontab. Can you please pull the latest django branch and try again.

[terxw]

The error appeared at any command e.g. archivebox add, update, and with any url.
I tried updating to the new version 0965031 after fix by cdvv7788 and I got this error:

most recent call last):
  File "./bin/archivebox", line 7, in <module>
    from .cli import main
  File "/home/kangus/src/ArchiveBox/archivebox/cli/__init__.py", line 65, in <module>
    SUBCOMMANDS = list_subcommands()
  File "/home/kangus/src/ArchiveBox/archivebox/cli/__init__.py", line 41, in list_subcommands
    module = import_module('.archivebox_{}'.format(subcommand), __package__)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/kangus/src/ArchiveBox/archivebox/cli/archivebox_help.py", line 11, in <module>
    from ..main import help, docstring
  File "/home/kangus/src/ArchiveBox/archivebox/main.py", line 8, in <module>
    from crontab import CronTab, CronSlices
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/home/kangus/.local/share/virtualenvs/archivebox_django/lib/python3.8/site-packages/python_crontab-2.5.1-py3.8.egg/crontab.py", line 91, in <module>
  File "/home/kangus/src/ArchiveBox/archivebox/logging.py", line 19, in <module>
    from .util import enforce_types
  File "/home/kangus/src/ArchiveBox/archivebox/util.py", line 14, in <module>
    import requests
  File "/home/kangus/.local/share/virtualenvs/archivebox_django/lib/python3.8/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/home/kangus/.local/share/virtualenvs/archivebox_django/lib/python3.8/site-packages/urllib3/__init__.py", line 7, in <module>
    from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
  File "/home/kangus/.local/share/virtualenvs/archivebox_django/lib/python3.8/site-packages/urllib3/connectionpool.py", line 29, in <module>
    from .connection import (
  File "/home/kangus/.local/share/virtualenvs/archivebox_django/lib/python3.8/site-packages/urllib3/connection.py", line 40, in <module>
    from .util.ssl_ import (
  File "/home/kangus/.local/share/virtualenvs/archivebox_django/lib/python3.8/site-packages/urllib3/util/__init__.py", line 20, in <module>
    from .retry import Retry
  File "/home/kangus/.local/share/virtualenvs/archivebox_django/lib/python3.8/site-packages/urllib3/util/retry.py", line 20, in <module>
    log = logging.getLogger(__name__)
AttributeError: partially initialized module 'logging' has no attribute 'getLogger' (most likely due to a circular import)

i used pew for venv:

cd /home/kangus/src/ArchiveBox
pew new -p /usr/local/bin/python3.8 -a $(pwd) archivebox_django
pip install -U setuptools
python3 ./setup.,py install

python -V
Python 3.8.2
python3 -V
Python 3.8.2

It looks to my like urllib3 is trying to run getLogger() from ArchiveBox/archivebox/logging.py instead of python lgger module

[cdvv7788]

@terxw How are you running the commands?
Can you try installing with pip install -e . in the folder, and then running the commands with archivebox add http://example.com?
I am having the same issue you are having when running directly with bin/archivebox (checking that atm).

[cdvv7788]

@terxw We renamed the module to prevent the issue. Once #379 is merged, you should be able to use it as you normally do.

[terxw]

Tried latest commit, installed in venv with pip3 install -e . and no import problem!