Boto: Don't load `/etc/boto.cfg` by default when used as a library

[rkrzr]

The Problem

When doing import boto in a Python program, boto will by default look for the global /etc/boto.cfg file.
This file may or may not exist and it may or may not have the right configuration settings.
It it does not have the correct settings the whole Python program will fail with an obscure error.

Concrete example:

On a new GCE box with Ubuntu 16.04 this file will exist and contain the following line:

[Plugin]
plugin_directory = /usr/lib/python3/dist-packages/google_compute_engine/boto

This hard-codes a Python 3 path, that boto will try to load then (even if your program is in Python 2).
This e.g. also happens when your program is run in a virtualenv. Boto thus "escapes" the virtualenv.

Possible solution

I would argue that it's a safer default to not load any config files when boto is used as a library.
This could just be an explicit flag that can be turned on (--load-global-config e.g.).

If that is not an option, then it would be good to document this prominently (I ended up digging through the source to find out what's happening).

[EmSyndio]

not sure if this is connected, but I've started having issues on Travis where boto would not load because it was missing the google-compute-engine requirement.
It worked on my local machine without it, but not on Travis.

django-compress that depends on boto was throwing: Could not load Boto's S3 bindings.

Just putting it out here, since when googling this issue I could not find any answer.

the fix for me was to
pip install google-compute-engine

[rkrzr]

Travis is running on GCE, and GCE boxes come with a hard-coded /etc/boto.cfg by default, like I described above. So your problem is very likely caused by this exact issue. If you don't want to have to pip install google-compute-engine you can probably work around this problem by setting the environment variable BOTO_CONFIG=/tmp/nowhere in your .travis.yml.
This way it will not use the global config with the wrong settings.

[EmSyndio]

@rkrzr Thanks.
What should there be in the default boto.cfg then?
I never had to worry about this before, and I don't seem to have a boto.cfg on my local machine, I don't see one in the repo, so not sure what's supposed to be in it.
Or do you mean that pointing at a folder that doesn't have the file will do?
Thanks

[rkrzr]

Or do you mean that pointing at a folder that doesn't have the file will do?

Exactly: Pointing at a file that doesn't exist solves the problem, since it will then use the default settings, which are fine for most use cases.

[timabbott]

[Edited since I figured out how to override the GCE behavior]

This issue problem basically broke installing Zulip in Google Compute Engine; I think it's entirely Google's fault. We were able to work around it with the BOTO_CONFIG environment variable in https://github.com/zulip/zulip/commit/88bb6c6cadd91a9811f55e214496d9d701b58ac0 in case anyone else runs into the same problem. See https://github.com/zulip/zulip/issues/4839 for details.

[nackjicholson]

I rely on a python library named moto for mocking aws in some integration tests. My builds run on TravisCI, they were breaking on import of moto.

before_install:
# Because travis runs on Google Cloud and has a /etc/boto.cfg it breaks my
# dependency on moto in our integration tests.
# https://github.com/spulec/moto/issues/1771
# https://github.com/boto/boto/issues/3741
# This overrides travis and tells it to look nowhere.
- export BOTO_CONFIG=/dev/null

Added that to my .travis.yml per @rkrzr above. Works!