Virtualenv: microsoft store python support

[JoDoX]

1. Install Python 3.7 from Microsoft Store
2. get error on trying to create new virtualenv

Microsoft Windows [Version 10.0.18908.1000]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\jodox>mkdir \temp\example

C:\Users\jodox>cd \temp\example

C:\temp\example>virtualenv .virtualenv
Using base prefix 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0'
New python executable in C:\temp\example\.virtualenv\Scripts\python.exe
Command C:\temp\example\.vir...v\Scripts\python.exe -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command C:\temp\example\.vir...v\Scripts\python.exe - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_internal\__init__.py", line 19, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\__init__.py", line 8, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\connectionpool.py", line 7, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\socket.py", line 49, in <module>
    import _socket
ImportError: DLL load failed: Access is denied.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
ImportError: cannot import name 'main' from 'pip' (C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\__init__.py)
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\Scripts\virtualenv.exe\__main__.py", line 9, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 831, in main
    symlink=options.symlink,
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1123, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 973, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1060, in _install_wheel_with_search_dir
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 924, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command C:\temp\example\.vir...v\Scripts\python.exe - setuptools pip wheel failed with error code 1

C:\temp\example>pip list
Package          Version
---------------- ----------
certifi          2019.3.9
pipenv           2018.11.26
virtualenv       16.6.0
virtualenv-clone 0.5.3

C:\temp\example>

[gaborbernat]

@zooba this feels like bad packaging on Windows store side 🤔?

[JoDoX]

@gaborbernat I think bug in virtualenv, because venv module work fine

Microsoft Windows [Version 10.0.18908.1000]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\jodox>cd \temp\example

(.venv) C:\temp\example>python3 -m venv .venv

C:\temp\example>.\.venv\Scripts\activate.bat

(.venv) C:\temp\example>python -V
Python 3.7.3

(.venv) C:\temp\example>python -c "import sys;print(sys.executable)"
C:\temp\example\.venv\Scripts\python.exe

(.venv) C:\temp\example>python -c "import socket"

(.venv) C:\temp\example>

[gaborbernat]

Not really man. venv provisions the created venv with whatever you have bundled. Here importing the network stack fails (which virtualenv does to install latest pip/setuptools). Pass in a --no-download and virtualenv should also work.

[JoDoX]

@gaborbernat I don't want virtualenv, but pipenv use it and no have option to pass args

C:\temp\example>pipenv --help
Usage: pipenv [OPTIONS] COMMAND [ARGS]...

Options:
  --where             Output project home information.
  --venv              Output virtualenv information.
  --py                Output Python interpreter information.
  --envs              Output Environment Variable options.
  --rm                Remove the virtualenv.
  --bare              Minimal output.
  --completion        Output completion (to be eval'd).
  --man               Display manpage.
  --support           Output diagnostic information for use in GitHub issues.
  --site-packages     Enable site-packages for the virtualenv.  [env var:
                      PIPENV_SITE_PACKAGES]
  --python TEXT       Specify which version of Python virtualenv should use.
  --three / --two     Use Python 3/2 when creating virtualenv.
  --clear             Clears caches (pipenv, pip, and pip-tools).  [env var:
                      PIPENV_CLEAR]
  -v, --verbose       Verbose mode.
  --pypi-mirror TEXT  Specify a PyPI mirror.
  --version           Show the version and exit.
  -h, --help          Show this message and exit.


Usage Examples:
   Create a new project using Python 3.7, specifically:
   $ pipenv --python 3.7

   Remove project virtualenv (inferred from current directory):
   $ pipenv --rm

   Install all dependencies for a project (including dev):
   $ pipenv install --dev

   Create a lockfile containing pre-releases:
   $ pipenv lock --pre

   Show a graph of your installed dependencies:
   $ pipenv graph

   Check your installed dependencies for security vulnerabilities:
   $ pipenv check

   Install a local setup.py into your virtual environment/Pipfile:
   $ pipenv install -e .

   Use a lower-level pip command:
   $ pipenv run pip freeze

Commands:
  check      Checks for security vulnerabilities and against PEP 508 markers
             provided in Pipfile.
  clean      Uninstalls all packages not specified in Pipfile.lock.
  graph      Displays currently-installed dependency graph information.
  install    Installs provided packages and adds them to Pipfile, or (if no
             packages are given), installs all packages from Pipfile.
  lock       Generates Pipfile.lock.
  open       View a given module in your editor.
  run        Spawns a command installed into the virtualenv.
  shell      Spawns a shell within the virtualenv.
  sync       Installs all packages specified in Pipfile.lock.
  uninstall  Un-installs a provided package and removes it from Pipfile.
  update     Runs lock, then sync.


C:\temp\example>pipenv install
Creating a virtualenv for this project…
Pipfile: C:\temp\example\Pipfile
Using C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\python.exe (3.7.3) to create virtualenv…
[==  ] Creating virtual environment...Already using interpreter C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\python.exe
Using base prefix 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0'
New python executable in C:\Users\jodox\.virtualenvs\example-Q6QloNNM\Scripts\python.exe
Command C:\Users\jodox\.virt...M\Scripts\python.exe -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command C:\Users\jodox\.virt...M\Scripts\python.exe - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_internal\__init__.py", line 19, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\__init__.py", line 8, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\connectionpool.py", line 7, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\socket.py", line 49, in <module>
    import _socket
ImportError: DLL load failed: Access is denied.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
ImportError: cannot import name 'main' from 'pip' (C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\__init__.py)
----------------------------------------
...Installing setuptools, pip, wheel...done.

Failed creating virtual environment
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\cli\command.py", line 254, in install
[pipenv.exceptions.VirtualenvCreationException]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 1741, in do_install
[pipenv.exceptions.VirtualenvCreationException]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 574, in ensure_project
[pipenv.exceptions.VirtualenvCreationException]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 506, in ensure_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       python=python, site_packages=site_packages, pypi_mirror=pypi_mirror
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 935, in do_create_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       extra=[crayons.blue("{0}".format(c.err)),]
[pipenv.exceptions.VirtualenvCreationException]: Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 2580, in <module>
    main()
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 831, in main
    symlink=options.symlink,
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1123, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 973, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1060, in _install_wheel_with_search_dir
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 924, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command C:\Users\jodox\.virt...M\Scripts\python.exe - setuptools pip wheel failed with error code 1

Failed to create virtual environment.

C:\temp\example>

[gaborbernat]

It still boils down to a packaging issue of core, as _socket is not importable 🤔 Will need to validate this though.

[gaborbernat]

@pfmoore this seems pip related 🤔 but strangely manifests when called from virtualenv by putting the wheel on the python path 👍

Command C:\Users\traveler\gi...v\Scripts\python.exe -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command C:\Users\traveler\gi...v\Scripts\python.exe - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_internal\__init__.py", line 19, in <module>
  File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\__init__.py", line 8, in <module>
  File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\connectionpool.py", line 7, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\socket.py", line 49, in <module>
    import _socket
ImportError: DLL load failed: Access is denied.

Edit: NM, seems our new virtualenv doesn't have access to _socket, my bad 🤦‍♂

[gaborbernat]

So trying to import _socket on a freshly created virtualenv fails with:

# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.py
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.pyw
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.pyc
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.pyd
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: Access is denied.

While doing the same from the installed python

# trying C:\Users\traveler\git\bre\env\Scripts\_socket.cp37-win_amd64.pyd
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.pyd
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.py
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.pyw
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.pyc
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.pyd
# extension module '_socket' loaded from 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\\DLLs\\_socket.pyd'
# extension module '_socket' executed from 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\\DLLs\\_socket.pyd'
import '_socket' # <_frozen_importlib_external.ExtensionFileLoader object at 0x0000022BDBDF2390>

Now inspecting the access data for this file:

PS C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs> Get-Acl _socket.pyd | fl


Path   : Microsoft.PowerShell.Core\FileSystem::C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.pyd
Owner  : NT AUTHORITY\SYSTEM
Group  : NT AUTHORITY\SYSTEM
Access : BUILTIN\Users Allow  ReadAndExecute, Synchronize
         S-1-15-3-1715986413-2752779232-2586321702-1327295686-2285127808-2907678250-160211910 Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  Read, Synchronize
         NT SERVICE\TrustedInstaller Allow  FullControl
         S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204 Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\SYSTEM Allow  FullControl
         NT AUTHORITY\LOCAL SERVICE Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\NETWORK SERVICE Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\RESTRICTED Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:SYG:SYD:AI(XA;ID;0x1200a9;;;BU;(WIN://SYSAPPID Contains "PYTHONSOFTWAREFOUNDATION.PYTHON.3.7_QBZ5N2KFRA8P0"))(A;ID;0x1200a9;;;S-1-15-3-1715986413-2752779232-2586321702-1327295686-2285127808-2907678250-160211910)(A;ID;FR;;;BU)(A;ID;FA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)
         (A;ID;0x1200a9;;;S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204)(A;ID;FA;;;SY)(A;ID;0x1200a9;;;LS)(A;ID;0x1200a9;;;NS)(A;ID;0x1200a9;;;RC)

Looking at this I'm not sure what's this missing 🤔

[pfmoore]

I've never used the Store distribution of Python. I'd suggest asking @zooba for help here, as he's the expert on that distribution.

@JoDoX Can you do a simple test? Start up Python, and run

import _socket
import socket

That should work with no errors. If you get an error, then the issue is with something in Python, rather than with virtualenv (or pip).

[gaborbernat]

@pfmoore see my post above. Those commands work if starting it from the installed python. However, they fail from within the virtual environment python. Seems to be related somehow with the fact that the DLLs can be only accessed if run from the installed Python. Once you copy the executable it seems to lose the ability to access the upstream DLLs. Not sure given this the issue is virtualenv or Windows store packaging related.

Edit:

Creating a venv and looking at the startup log shows somehow venvs actually try loading the pyd from the original location. It actually loads everything from the original location. Both Include and Lib are empty. In virtualenv we copy these over for now.... Maybe there's a better way to create venvs than what we have now 🤔 Will need to investigate.

# C:\Users\traveler\git\venv\lib\encodings\__pycache__\__init__.cpython-37.pyc matches C:\Users\traveler\git\venv\lib\encodings\__init__.py
# code object from 'C:\\Users\\traveler\\git\\venv\\lib\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying C:\Users\traveler\git\venv\lib\codecs.cp37-win_amd64.pyd
# trying C:\Users\traveler\git\venv\lib\codecs.pyd
# trying C:\Users\traveler\git\venv\lib\codecs.py
# C:\Users\traveler\git\venv\lib\__pycache__\codecs.cpython-37.pyc matches C:\Users\traveler\git\venv\lib\codecs.py
# code object from 'C:\\Users\\traveler\\git\\venv\\lib\\__pycache__\\codecs.cpython-37.pyc'

vs

# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.py
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.pyw
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.pyc
# C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\encodings\__pycache__\__init__.

[pfmoore]

@gaborbernat Ah, sorry - I skimmed your post and misunderstood what you were saying. My bad.

I certainly wouldn't rule out that copying the Python executable might cause issues. We really need @zooba here, as I'm not aware of any developer-level docs on how the Store build works, and I'm not sufficiently familiar with the code to go diving into it.

[zooba]

Yeah, you can't copy the executable. Once it's outside the app package it won't have its context properly activated by the operating system.

You can either symlink it or copy the python.exe that's in venv/scripts/nt. It's a special copy of the py.exe launcher that will look for pyvenv.cfg to find the actual one to launch. (Or if you don't want to use that file, you can build another version of it that looks for your preferred config settings.)

[pfmoore]

Once it's outside the app package it won't have its context properly activated by the operating system.

Somewhat offtopic, but can you point me at the code that makes this be the case (and/or the relevant MS docs on this)? I'd like to understand the store app better than I currently do (I basically understand nothing about MS Store apps at the moment), so I don't have to keep pinging you on this stuff ;-)

[zooba]

I'm not sure that this part is even documented, and if it is, it'll probably be deep inside the MSIX docs or possibly in some AppV specifications if you can find that (which is the older version of the isolation technology used).

The packages are meant to be used as apps, primarily, which means a single entry point (and prevent other apps from using our DLLs). So we can't LoadLibrary files in the package until we've verified that we belong to the package, and in the general case the only way to do that is to launch the original python.exe.

Oh, and to be clear, these are all deliberate, by-design OS restrictions. Python did not choose to make any of this happen, so there's no code in CPython for it. Though the parts of launcher.c hidden behind the VENV_REDIRECT define may be interesting (and unsupported, internal details...)

[pfmoore]

Thanks. I did some brief digging, and https://docs.microsoft.com/en-us/windows/msix/desktop/desktop-to-uwp-behind-the-scenes looks like it might be relevant (although I haven't read it yet, I'm mostly posting this so I can find the link again later ;-))

@gaborbernat I suspect the "right" way for virtualenv to support store Python will be to copy the redirector fromvenv/scripts/nt, as @zooba suggested. But that's extremely close to the idea of using venv to do the environment creation, that I suggested a while back, and you weren't keen on because of the compatibility implications. I suspect that it would be even worse if we did it for the Store application, but not the "standard" Python with the same version. (Plus, I'm not sure that mixing the redirector with things like virtualenv's custom site.py would even work).

I'm happy to help out with coding, but I'll need your guidance on the approach you want to take.

(Oh, and @zooba, while I have your attention, how well does the store Python co-exist with a "normal" Python install? If I want to test this stuff, will I be better with a clean VM, to avoid interfering with my main Python install? Or can I install and uninstall the store Python with no side-effects?)

[gaborbernat]

So I gave this a good thought. In the end, I think it makes sense to switch the core logic to:

• if possible create the venv with the stdlib provided venv, patch stdlib created venv to offer the same guarantees our own,
• otherwise, use our own logic.

A few places where stdlib provided might venv might not be present:

• python 2,
• distributions where the venv is packaged separately and not installed at the moment (such as ubuntu python3.7-venv). @pfmoore taught? We'll need to come up with a list of guarantees we offer at the moment to implement this.

[pfmoore]

I'd be a strong +1 on this. With Python 2 reaching EOL very soon, we're going to be in a situation where basically all supported versions of Python have venv (and those that don't, still have the mechanisms - pyvenv.cfg). So if we can avoid the hacks we currently use as much as possible, that would be a huge win (and it would also make compatibility between virtualenv and venv a non-issue :-))

The hard part is deciding what constitutes the list of guarantees we offer. I'm not actually sure that there are many - the only difficult one is the sys.real_prefix vs sys.base_prefix issue, and honestly I believe we should move to follow the stdlib on this in any case (with a suitable transition period).

[pradyunsg]

I concur with @pfmoore on this.

[gaborbernat]

Just to be clear, Python 2 support is not planned to be dropped at least for the next 3 years. The new solution will just prefer to use stdlib venv where possible and fallback to builtin logic otherwise.

[pfmoore]

Understood. The existing process needs to remain so that Python 2 is supported, but if we can avoid it where core venv is available that's great.

[gaborbernat]

Started a separate discussion in https://github.com/pypa/virtualenv/issues/1366 👍

[zooba]

(Oh, and @zooba, while I have your attention, how well does the store Python co-exist with a "normal" Python install? If I want to test this stuff, will I be better with a clean VM, to avoid interfering with my main Python install? Or can I install and uninstall the store Python with no side-effects?)

Sorry I missed this, @pfmoore (Github notifications don't always get my attention...).

You can install/uninstall the store Python with no side-effects - this is actually one of it's biggest features. (Though a downside of this is that it can't update the real user registry, and so PEP 514 isn't quite sufficient, since overwriting values would break the clean install/uninstall cycle.)

[gaborbernat]

I think we talked over Twitter that it would not be an issue of overwrite if per pep 514 we use a different company (e.g. Microsoft Store) instead of PythonCore. And then we just need to update py.exe to accept that too.

[neoOpus]

using venv instead of virtualenv solves the issue for me

[gaborbernat]

The old virtualenv cannot and will not support this. virtualenv 20, the rewrite, should. We need to validate, sadly AFAIK it's not possible to add the store python to the Azure CI. @zooba can you confirm that this still is the case, and will be going ahead?

[gaborbernat]

In first iteration we can just mark the builtin/self-do method not support the store; to check if the python is store pyhton @zooba suggested:

You probably want to try to open sys.executable for reading then, as I think that's your biggest issue (it'll fail)
Or you could check whether pythonXY.dll isn't in the same directory as sys.executable, which I think is another assumption you'll need to fix
That one can happen in a few different ways (embedding, freezing, static build) and I suspect virtualenv won't work with any of them right now
Though the old (pre 3.5) installers could put it into the system folder, which would still work, so not a perfect test.

[gaborbernat]

This now has been achieved as part of #1502. Virtualenv will correctly mark the store python as not compatible with its builtin strategy and fallback to delegate the task to the venv module of the store python. The test suite of the project passes. Furthermore, now we will use the symlink instead of the copy method if the Windows OS supports it (we test this by trying to create a symlink inside a temp folder).

Built-in support for Microsoft Store within the virtualenv may be done in a future subsequent ticket, probably by improving our cpython3windows class to use the new techniques needed for this.