Pipenv: Default encoding for .env files

Created on 12 Apr 2018 · 10Comments · Source: pypa/pipenv

I had some Russian symbols in my .env file while launching pipenv shell from Windows 10 and got the following result:

$ pipenv shell
Loading .env environment variables…
Traceback (most recent call last):
  File "c:\program files (x86)\python36-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files (x86)\python36-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files (x86)\Python36-32\Scripts\pipenv.exe\__main__.py", line 9, in <module>
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\cli.py", line 564, in shell
    core.load_dot_env()
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\core.py", line 155, in load_dot_env
    dotenv.load_dotenv(denv, override=True)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 250, in load_dotenv
    return DotEnv(f, verbose=verbose).set_as_environment_variables(override=override)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 94, in set_as_environment_variables
    for k, v in self.dict().items():
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 73, in dict
    values = OrderedDict(self.parse())
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 80, in parse
    for line in f:
  File "c:\program files (x86)\python36-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 426: character maps to <undefined>

If it helps, here's the help output

$ python -m pipenv.help

$ python -m pipenv.help output

Pipenv version: '11.10.0'

Pipenv location: 'C:\\Program Files (x86)\\Python36-32\\lib\\site-packages\\pipenv'

Python location: 'C:\\Program Files (x86)\\Python36-32\\python.exe'

Other Python installations in PATH:

3.6: C:\Program Files (x86)\Python36-32\python.exe
3.6.4: C:\Program Files (x86)\Python36-32\python.exe
3.6.4: C:\Windows\py.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.4',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.16299',
 'python_full_version': '3.6.4',
 'python_version': '3.6',
 'sys_platform': 'win32'}

System environment variables:

ACLOCAL_PATH
ALLUSERSPROFILE
APPDATA
COMMONPROGRAMFILES
COMPUTERNAME
COMSPEC
CONFIG_SITE
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
DISPLAY
EXEPATH
FPS_BROWSER_APP_PROFILE_STRING
FPS_BROWSER_USER_PROFILE_STRING
FSHARPINSTALLDIR
HOME
HOMEDRIVE
HOMEPATH
HOSTNAME
INFOPATH
LANG
LEIN_JAVA_CMD
LOCALAPPDATA
LOGONSERVER
MANPATH
MINGW_CHOST
MINGW_PACKAGE_PREFIX
MINGW_PREFIX
MSYSTEM
MSYSTEM_CARCH
MSYSTEM_CHOST
MSYSTEM_PREFIX
NUMBER_OF_PROCESSORS
ORIGINAL_PATH
ORIGINAL_TEMP
ORIGINAL_TMP
OS
ONEDRIVE
PATH
PATHEXT
PKG_CONFIG_PATH
PLINK_PROTOCOL
PROCESSOR_ARCHITECTURE
PROCESSOR_ARCHITEW6432
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMFILES
PS1
PSMODULEPATH
PUBLIC
PWD
PROGRAMDATA
PROGRAMFILES(X86)
PROGRAMW6432
SESSIONNAME
SHELL
SHLVL
SSH_ASKPASS
SYSTEMDRIVE
SYSTEMROOT
TEMP
TERM
TMP
TMPDIR
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
WINDIR
_
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
PYTHONUNBUFFERED

Pipenv▒specific environment variables:

Debug▒specific environment variables:

PATH: C:\Users\feana\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\feana\bin;C:\Program Files\Docker\Docker\Resources\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Python36-32\Scripts;C:\Program Files (x86)\Python36-32;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files\Git\cmd;C:\Users\feana\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft VS Code\bin;C:\Users\feana\.lein\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
SHELL: C:\Program Files\Git\usr\bin\bash.exe
LANG: en_US.UTF-8
PWD: C:/Users/feana

Expected result

Correct .env file parsing.

Actual result

Somehow, it thinks the file is encoded in CP1252. Why?..

Steps to replicate

Be on Win10
Have cyrillic in .env
Run pipenv shell

Behavior Change Bugfix Vendored Dependencies

Source

feakuru

Most helpful comment

@feakuru Yes, this should not happen. Post the traceback and I’ll work on it. Regarding the env file, what encoding do you expect it to be decoded with? UTF-8? (And for your question, it does not know what the file is encoded in, so it simply uses the system default, i.e. the code page encoding on Windows.)

uranusjr on 12 Apr 2018

👍4

All 10 comments

Actually, now that I've removed the cyrillic symbols, I'm getting an AttributeError: module 'pexpect' has no attribute 'spawn'. Should I open another issue?..

feakuru on 12 Apr 2018

uranusjr on 12 Apr 2018

👍4

What about using io.open or whatever?

techalchemy on 12 Apr 2018

Dotenv already uses io.open but does not specify an encoding 🤷‍♂️ See the linked python-dotenv issue above. I fixed that without touching dotenv however, by passing it a pre-decoded StringIO instead.

uranusjr on 12 Apr 2018

@uranusjr so are we good to close this?

techalchemy on 29 Apr 2018

@techalchemy Urrgh, I never got to work on this in the end… This is still not fixed, sorry.

uranusjr on 29 Apr 2018

😕1

meet same bug with utf-8 encoded .env file on windows

this is a bug of python-dotenv, they just open .env file and try to readline.

Their solution is that they will add a arguments encoding(default 'utf-8'), which means pipenv need to find .env file's corrent encoding first and pass it to python-dotenv to fix issue like this.

https://github.com/theskumar/python-dotenv/pull/161

Trim21 on 1 Apr 2019

Pull request is welcomed. Don’t wait for people; take on it yourself if you already know what to do :)

uranusjr on 1 Apr 2019

@uranusjr I do want to start a pull request, but https://github.com/theskumar/python-dotenv/pull/161 is not merged yet. Unless I can merge it into vendor before python-dotenv merge this pr.

Trim21 on 1 Apr 2019

cc @techalchemy to maybe revisit this when we’re about to release the next version.

uranusjr on 2 Apr 2019

Was this page helpful?

0 / 5 - 0 ratings