Wrye-bash: Reworking build scripts

[Utumno]

_Reusing old issue: Drop git/github files from python packages_

Anyone modify package_for_release.py to skip those files when packing python versions ? probably the scripts folder too

---

Build scripts need to be revamped - most of it is using outdated code and practices.
Plus, having it streamlined and completely automated allows us to later use it for some interesting things (cough ci servers cough).

• [x] Have gitPython imported AFTER ensuring git is on PATH (see: efdaba2 5ad4f9c).
• [x] Look into replacing py2exe - new issue (#491)
• [x] Look into replacing NSIS - new issue (#477)
• [x] Update LOOT API to 4.0.2:
  
  
  
  • [x] Create setup.py for LOOT API to make it installable via pip (see: pull-request).
  
  
  • [x] Py2exe apparently can't load loot binaries - needs more testing?
  
  

[MacSplody]

@Utumno Is this not already done?
Or am I missing what it is you need changed?

[Utumno]

The python source we package contains files it shouldn't - among wich the scripts/ folder and .gitignore IIRC

[MacSplody]

@Utumno
This is with the build on wrye-code-collection.
So it must be fixed, I'll have a look and see if there are any differences.

[Utumno]

We should probably close this @GandaG or hijack for your build scripts

[GandaG]

@Utumno "good first issue" :P Sure, having a permanent place for discussion would be nice. Am I allowed to also hijack the OP if needed?

[Utumno]

You can hijack all you want :P

[Infernio]

We should upgrade our bundled LOOT to loot-api-python v4.0.2 ASAP:

1. Some people are reporting an error related to LOOT. This is because they're using newer versions of the masterlist, which have some syntax changes that our current bundled loot-api-python (v3.1.1) can't handle.
2. A few days ago, I investigated why the dirty plugin highlighting in WB is broken. It turned out that this was on LOOT's side: https://github.com/loot/loot-api-python/issues/10. 4.0.2 fixes the bug, restoring the 'dirty mods checker' functionality in WB.

That obviously also means switching to the v0.14 branches for the masterlists.

[GandaG]

@Infernio the new loot install script allows changing version/revision by argument. Try running
python scripts/install_loot_api.py -lv 4.0.2 -lr d356ac2 -lm URL_TO_MSVC_2017. For the masterlists
python scripts/update_taglist.py -mv 0.14. Then build like normal.

In the meantime I'll start working on making a setup.py for the loot api so we can install them via pip instead of these hacks.

[Utumno]

Dear Nazgul,

Thanks for taking care of that - LOOT integration has caused me a _lot_ of pain, would love to have a robust solution

Ut :P

[GandaG]

@Sharlikran raised a valid point in discord - while it seems replacing the loot local files with a pip package has no issues for running WB, packaging is another matter entirely. The extra package files are not picked up by py2exe and I don't use outdated software enough to know how to make hooks. Since loot installation is currently handled I'll put this on the backburner until we replace py2exe.

[Sharlikran]

With LOOT v0.14.x and MSVC 2017 x86 redistributable, Wrye Bash doesn't use the LOOT api 4.0.2 after Wrye Bash is compiled into an EXE.

[Utumno]

Let's discuss this at #431

[GandaG]

@Utumno @Infernio would it be better if the nightlies used the commit's author date instead of the current timestamp? It would reduce confusion for which builds correspond to which commits.

[Utumno]

You mean the commit with the date?
It shouldn't make a difference then IIUC so why not? Btw as you may have
noticed I removed that commit from the nightly branch. So the idea is the
nightly branch that we build has that commit at Head so the bugdumps have
that one. but once we push more stuff on nightly we remove it so the (few)
people that use the python nightly have a plain 307 version in their
bugdumps

On Thu, Aug 1, 2019, 05:02 Daniel Nunes [e-mail] wrote:

@Utumno https://github.com/Utumno @Infernio
https://github.com/Infernio would it be better if the nightlies used
the commit's author date instead of the current timestamp? It would reduce
confusion for which builds correspond to which commits.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/wrye-bash/wrye-bash/issues/415?email_source=notifications&email_token=AAKNIV7TYPOZMVU6NYIUSALQCI74VA5CNFSM4FISMXX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3JCITQ#issuecomment-517088334,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKNIV5GGLDGYFS2CNPRNB3QCI74VANCNFSM4FISMXXQ
.

[GandaG]

You mean the commit with the date?

Yes, this is about that commit 😛 So, a few things about the current the build scripts:

1. The nightly flag is default;
2. The nightly flag uses bass.AppVersion plus the build timestamp;
3. The production flag uses whatever is in bass.AppVersion;
4. The script uses a wee hack that modifies bass.AppVersion to match the version being used.

This means that

1. If bass.AppVersion is not an integer, the nightly flag will fail to build;
2. If building with the nightly flag then bass.AppVersion (and therefore the bugdump) will have the nightly timestamp.

Which essentially means that if you're doing those shenanigans then the build scripts aren't being useful. I just want to know what I can do to MAKE THE BUILD SCRIPTS GREAT AGAIN (sorry). Why is it important that

people that use the python nightly have a plain 307 version in their bugdumps

And why use the commit with the date? For easier tracking? Please eli5, I'm slow

[Utumno]

when I had the utumno-wip and I posted a build on the forums I
modified the bass.AppVersion
as you do - so I could tell who was using my new build - then when I
modified my utumno-wip I removed the commit with the date and went back to
plain 306 or 307as the case might be - few people are using the python
branch anyway

the only thing you need changing is basically not revert the bass.AppVersion
change - or automatically commit that although not really needed if this
proves tough

On Thu, Aug 1, 2019, 20:24 Daniel Nunes [e-mail] wrote:

You mean the commit with the date?

Yes, this is about that commit 😛 So, a few things about the current the
build scripts:

1. The nightly flag is default
   https://github.com/wrye-bash/wrye-bash/blob/f419cb128e6e57fe29d91fbf91388d5e0cce45a4/scripts/build.py#L175
   ;
2. The nightly flag uses bass.AppVersion plus the build timestamp
   https://github.com/wrye-bash/wrye-bash/blob/f419cb128e6e57fe29d91fbf91388d5e0cce45a4/scripts/build.py#L103
   ;
3. The production flag uses whatever is in bass.AppVersion
   https://github.com/wrye-bash/wrye-bash/blob/f419cb128e6e57fe29d91fbf91388d5e0cce45a4/scripts/build.py#L118
   ;
4. The script uses a wee hack that modifies bass.AppVersion to match
   the version being used
   https://github.com/wrye-bash/wrye-bash/blob/f419cb128e6e57fe29d91fbf91388d5e0cce45a4/scripts/build.py#L662
   .

This means that

1. If bass.AppVersion is not an integer, the nightly flag will fail to
   build;
2. If building with the nightly flag then bass.AppVersion (and
   therefore the bugdump) will have the nightly timestamp.

Which essentially means that if you're doing those shenanigans then the
build scripts aren't being useful. I just want to know what I can do to
MAKE THE BUILD SCRIPTS GREAT AGAIN (sorry). Why is it important that

people that use the python nightly have a plain 307 version in their
bugdumps

And why use the commit with the date? For easier tracking? Please eli5,
I'm slow

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/wrye-bash/wrye-bash/issues/415?email_source=notifications&email_token=AAKNIV27FX5SR2RHNQAIEWTQCML6HA5CNFSM4FISMXX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LKAEQ#issuecomment-517382162,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKNIV57NNAQNHTBHXWW3TLQCML6HANCNFSM4FISMXXQ
.

[GandaG]

Right, so iiuc the point is to only deploy the commits with date for easier tracking, yes? Which would be more akin to a weekly/monthly than a nightly?

[Utumno]

Yep! Basically "nightly" is the builds for people on the forums to use - at
some point, I used to post them quite often hence nightly but basically,
they are not regular, although the more the merrier. If the version is in
bass then we can see it in the bugdump. But once we push new stuff on
nightly branch there are still people who use python and to avoid confusion
we should remove that commit - usually, people who use python are close to
devs and we know which version of nightly they are on

Thanks!

On Fri, Aug 2, 2019 at 9:28 PM Daniel Nunes [e-mail]
wrote:

Right, so iiuc the point is to only deploy the commits with date for
easier tracking, yes? Which would be more akin to a weekly/monthly than a
nightly?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/wrye-bash/wrye-bash/issues/415?email_source=notifications&email_token=AAKNIV7DVALS42M3IA4KTOTQCR4FHA5CNFSM4FISMXX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OQLDA#issuecomment-517801356,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKNIVZXT3XSHZVZ3LHCDV3QCR4FHANCNFSM4FISMXXQ
.

[GandaG]

If the version is in bass then we can see it in the bugdump.

This already happens, the build script changes the version number on the file before building. Either way, I've added a --commit flag to build.py, testing now.

[Utumno]

Great @GandaG thanks!

[Infernio]

@GandaG I moved your two build script commits (774e0549807b26a0857683889809826426c6c74c and 7ec5fc16ab7a3bd5096377324072ecb97eb85bec) to nightly, since I needed them for building the new version anyways.

[Infernio]

Some experiences with a pyinstaller nightly:

• The whole 'progress dialog causes WB to lose focus' issue is gone (see https://github.com/wrye-bash/wrye-bash/issues/15#issuecomment-555668148)...
• ... but that's probably because progress dialogs are now no longer native, so they're uglier and we lose all the speed gains from wx3
• In addition, the BP is slower. Way slower. Like, 'takes three times as long for SSE' levels of slower.

What's really weird is that I don't remember the PyInstaller build Ganda originally gave me having these issues, but maybe my memory is just flaky on that