Wrye-bash: Use scandir for speeding up os walk operations

[Utumno]

Relevant branch: https://github.com/wrye-bash/wrye-bash/commits/utumno-scandir

https://pypi.python.org/pypi/scandir/1.5

This speeds stuff _considerably_ but we need to feed the pyd to py2exe somehow

Unhandled exception at 0x740B8FAC (msvcr90.dll) in Wrye Bash.exe: 0xC0000005: Access violation reading location 0x0023FFFF.

I suspect that is because I have the 64 dlls - or cause py2exe can't play with the pyd (see #329)? Installed scandir from:

https://pypi.python.org/packages/2.7/s/scandir/scandir-1.2.win-amd64-py2.7.exe#md5=896625409dc758be28ce52e27abeeb27

But there is a much newer version we need to experiment with

[lojack5]

I could probably figure this one out fairly quickly. py2exe isn't too hard to pull in additional dependencies.

[D4id4los]

I could successfully build your branch without any changes using scandir 1.5 installed with pip install scandir. The standalone runs on my dev machine and on a virtual machine without python installed and the attached bugdump says it is using scandir.

(The standalone was repackaged as zip because github does not allow .7z files)

Wrye Bash 307.2017.08.26b - Standalone Executable.zip
Wrye Bash 307.2017.08.26b - Installer.exe.zip

build.log.txt

BashBugDump.txt

[Utumno]

Excellent - then this could be made a documentation issue (wiki/readme) ? Maybe dependencies on ms runtimes ? Work on packaging scripts needed (did you build using scripts\package_for_release.py ? Give it to people in threads (AFK mods) and see if it runs for them.

[D4id4los]

Yes I did use scripts\package_for_release.py without making any changes to the scripts. I'll go post it on AFK mods.

[D4id4los]

Here is a packaged version with this branched rebased on utumno-wip 8a635e28a71f58d594dddad76f66f65ea4ad7a7c:

Wrye Bash 307.2017.08.27.1938 - Installer.exe.zip
Wrye Bash 307.2017.08.27.1938 - Standalone Executable.7z.zip

Loot api is not working however. I tried including the .py file and .dll from https://github.com/loot/loot-api-python but it throwed casting exceptions and I could not get it to work.

[Utumno]

Loot api is not working however. I tried including the .py file and .dll from https://github.com/loot/loot-api-python but it throwed casting exceptions and I could not get it to work.

Hmm we do not use latest currently, see #329 - @WrinklyNinja could you give us a rundown (it's maybe time to bundle a new loot API for beta2 so let's edit also release docs for that) ?

[Ortham]

Bash uses the v0.10.x API, right? The v0.11.0 changelog is here, the Python wrapper was also updated to v2.0.0, with changes summarised in this commit.

In short, there were a whole bunch of LOOT API changes to allow the LOOT application to be built on top of the API, but I attempted to minimise changes to the wrapper's API, so while upgrading isn't just a straight drag 'n' drop replacement, it shouldn't be painful either.

[Utumno]

Hrumph:

C:\Dropbox\eclipse_workspaces\python\wrye-bash>c:\_\Python27\python.exe -m pip install --upgrade scandir
Collecting scandir
  Using cached scandir-1.5.tar.gz
Building wheels for collected packages: scandir
  Running setup.py bdist_wheel for scandir ... error
  Complete output from command c:\_\Python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\mrd\\appdata\\local\\temp\\pip-build-gsc1ev\\scandir\\setu
p.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d c:\users\mrd\appdata
\local\temp\tmpx1gudlpip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win32-2.7
  copying scandir.py -> build\lib.win32-2.7
  running build_ext
  building '_scandir' extension
  error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27

  ----------------------------------------
  Failed building wheel for scandir
  Running setup.py clean for scandir
Failed to build scandir
Installing collected packages: scandir
  Found existing installation: scandir 1.2
    DEPRECATION: Uninstalling a distutils installed project (scandir) has been deprecated and will be removed in a future version. This is due to the fact that uninstalli
ng a distutils project will only partially uninstall the project.
    Uninstalling scandir-1.2:
      Successfully uninstalled scandir-1.2
  Running setup.py install for scandir ... error
    Complete output from command c:\_\Python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\mrd\\appdata\\local\\temp\\pip-build-gsc1ev\\scandir\\se
tup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\mrd\app
data\local\temp\pip-htlx3g-record\install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build\lib.win32-2.7
    copying scandir.py -> build\lib.win32-2.7
    running build_ext
    building '_scandir' extension
    error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27

    ----------------------------------------
  Rolling back uninstall of scandir
Command "c:\_\Python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\mrd\\appdata\\local\\temp\\pip-build-gsc1ev\\scandir\\setup.py';f=getattr(tokeni
ze, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\mrd\appdata\local\temp\pip-htlx
3g-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\mrd\appdata\local\temp\pip-build-gsc1ev\scandir\

So I installed "Microsoft Visual C++ 9.0" "from http://aka.ms/vcpython27":

C:\Dropbox\eclipse_workspaces\python\wrye-bash>c:\_\Python27\python.exe -m pip install --upgrade scandir
Collecting scandir
  Using cached scandir-1.5.tar.gz
Building wheels for collected packages: scandir
  Running setup.py bdist_wheel for scandir ... done
  Stored in directory: C:\Users\MrD\AppData\Local\pip\Cache\wheels\47\af\a2\eb591a17e9709a17d8b53696f6ad89914a05eaf9c091f36e18
Successfully built scandir
Installing collected packages: scandir
  Found existing installation: scandir 1.2
    DEPRECATION: Uninstalling a distutils installed project (scandir) has been deprecated and will be removed in a future version. This is due to the fact that uninstalli
ng a distutils project will only partially uninstall the project.
    Uninstalling scandir-1.2:
      Successfully uninstalled scandir-1.2
Successfully installed scandir-1.5

so:

Wrye Bash starting
Using Wrye Bash Version 307
OS info: Windows-7-6.1.7601-SP1
Python version: 2.7.12
wxPython version: 2.8.12.1 (msw-unicode)
Using scandir 1.5
input encoding: UTF8; output encoding: None; locale: ('en_US', 'cp1253')
filesystem encoding: mbcs 
bash.py  324 main: Searching for game to manage:
bush.py   76 _supportedGames: Detected the following supported games via Windows Registry:
bush.py   78 _supportedGames:  Oblivion: C:\GAMES\TESIV\Oblivion
bush.py   78 _supportedGames:  Skyrim Special Edition: F:\GAMES\SteamLibrary\steamapps\common\Skyrim Special Edition
bush.py   78 _supportedGames:  Skyrim: C:\GAMES\Skyrim
bush.py  136 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.py  142 _detectGames: Set game mode to Skyrim based on sOblivionPath setting in bash.ini:  C:\GAMES\Skyrim
bush.py  156 __setGame:  Using Skyrim game: C:\GAMES\Skyrim
testing UAC
mods_metadata.py  228 __init__: Using LOOT API version: 0.10.2
.....

Now let's see if it builds for me

[Utumno]

@BeermotorWB - here's one for you - I need to merge but first I need to document the msvc dependency, or indeed how should one set this up - could you add this to the readme (in one commit on a branch over 371-scandir branch) ? And also try with scandir 1.6 (should be ok)

[BeermotorWB]

@Utumno I think I know what you mean. I'll see what I can do.

For documentation scope and purpose: would you prefer to have people use a scandir wheel file or grab it from pip? I used the pip method but that requires the user to install Microsoft VC for Python 2.7. Not a big deal as long as Microsoft doesn't move or remove the download link.

If the wheel method is preferred, my laptop is still running a virgin Python 2.7.14 install and I could document using the 'official' wheel from here.

Also is scandir going to be pre-packaged/frozen in the Standalone executable versions?

[Utumno]

Thanks @BeermotorWB !

For documentation scope and purpose: would you prefer to have people use a scandir wheel file or grab it from pip?

Well both I guess - I used pip too as seen above - should be the preferred method ? I leave it up to you

Also is scandir going to be pre-packaged/frozen in the Standalone executable versions?

Yep - btw if you come round to it it would be nice you try to run the packaging scripts - I am currently the only one who runs them and this is not good. See: https://github.com/wrye-bash/wrye-bash/wiki/[github]-Making-A-Release

It would be an opportunity to refresh/simplify (much needed) our release procedure. Eventually we should be able to upload the nightlies to nexus...

[BeermotorWB]

@Utumno pushed doc update to 371-scandir.

I documented both methods. If anyone thinks it is too confusing or verbose I don't have a problem changing it.

Now on to building a release-building environment. :)