Ungoogled-chromium: Update Windows support

[Wyse-]

ungoogled-chromium version: 68.0.3440.106-1

Configuration and/or platform(s): Windows x64

Problem description

I'm assuming Windows is not supported yet on version 68.0.3440.106-1.

The build instructions seem outdated: trying to adapt the ones from other platforms (get_package.py windows build/src/ungoogled_packaging) leads to the files build.bat and package.bat.in being created in the build/src/ungoogled_packaging directory. However the build.bat script doesn't seem to be updated to version 68 as it expects the Chromium source files to be already downloaded.

Looking at the other platforms' build scripts I assume the Windows one should handle everything from source downloading to packaging as well.

[Eloston]

Looking at the other platforms' build scripts I assume the Windows one should handle everything from source downloading to packaging as well.

Your assumption is correct. The build scripts will need to be reworked to support the new packaging strategy (i.e. handle everything itself).

A little while ago, @squalus suggested that we should rework the build scripts in Python, but it would make some tasks more difficult, such as providing the shell variables from vcvarsall to certain commands (e.g. the GN bootstrap script). I'm not familiar enough with batch scripting to determine whether it is worth the effort to switch to Python entirely, or stay with the hybrid approach we have with batch scripts. What are your thoughts?

[Eloston]

With 7c2d1b8a216f7e9dacc1d6a78d0a6a4f98076f5f, the scripts have been rewritten in Python (but they haven't been tested). Patches haven't been touched and still need to be updated.

[Wyse-]

Sorry for the delay.

I just tested the new scripts and this is what I get when running py ungoogled_packaging\build.py:
https://paste.debian.net/plainh/a61acaac

It also doesn't seem clear which python version is supposed to run which script: they all have #!/usr/bin/env python3 above the license header, however build.py does some version checks against python 2.7.
Specifying the python version as an argument to py (e.g. py -2) in the instructions might be a good idea.

[Eloston]

EDIT: I just saw the log again, and you're using python, not py. python always refers to Python 2 for compatibility with Chromium scripts, and py is a command introduced by Python 3 that has some logic to auto determine whether to use Python 2 or 3.

That's interesting... you were able to use get_package.py, but you couldn't use build.py? Both use Python 3-only syntax, so it's surprising to see one work but not another, especially since the py command reads the shebang line in all of the files to determine which Python interpreter to use (and they're all python3)

Maybe you're using an old Python 3? buildkit uses additional unpacking generalization (PEP 448), which was added in 3.5.

If that doesn't work for some reason, you can also use py -3 to force usage of Python 3.

[Wyse-]

Ah, I see, I thought py was included in both python versions. As to why I ran build.py with python instead of py: running the script with py gave me build.py: error: Unable to find pypiwin32 in Python 2 installation., so I assumed it was supposed to be ran with python 2.7 (the instructions also specify pip install pypiwin32 for python2.7.

Assuming all ungoogled-chromium scripts should be ran with python 3.7, does this mean that pypiwin32 is needed for python 3.7 as well? I tried running pip install pypiwin32 with the python 3.7 pip executable but I still get the same error as above (build.py: error: Unable to find pypiwin32 in Python 2 installation.).

[Eloston]

At some point in time, Chromium's Python 2 scripts used pypiwin32, hence the requirement. None of my scripts require that.

In 366b1eed5ce975ab537bbd3062b77a0bb5e3064f, the scripts now explicitly check for Python 3.

[Wyse-]

Tested the the scripts on the latest commit and build.py still seems to check for pypiwin32 in lines 85-87. After commenting these out I get the following:
https://paste.debian.net/plainh/cc66f445

[Eloston]

Tested the the scripts on the latest commit and build.py still seems to check for pypiwin32 in lines 85-87

Is the check not working? What happens if you launch Python 2 and import pypiwin32?

[Wyse-]

My bad, I assumed the script was looking for pypiwin32 in the python3 installation, I should've taken a look at the code.

Apparently the check is not working even though I did install pypiwin32 through pip (pip install pypiwin32). If I run import pypiwin32 in a Python 2 shell I get ImportError: No module named pypiwin32.
From what I can see it seems like pypiwin32 is not supposed to be imported with its name, but with import win32api.

Maybe line 85 should be changed to
result = subprocess.run((python2_exe, '-c', 'import win32api'))

After this change the script works, until, while applying patches, I get this:
https://paste.debian.net/plainh/155d21b4

[Eloston]

My bad, I assumed the script was looking for pypiwin32 in the python3 installation, I should've taken a look at the code.

I'm guessing the build instructions were confusing to you, because the exact steps to take are all there. Could you tell me what confused you, or submit a PR with fixes?

From what I can see it seems like pypiwin32 is not supposed to be imported with its name, but with import win32api.

Fixed with #488. Thanks for letting me know regardless.

After this change the script works, until, while applying patches, I get this:

That's expected. The patches haven't been updated yet. I might have an opportunity to update them this and/or next week, but I can't guarantee anything.

EDIT: Feel free to take the initiative to update the patches if you'd like. Portable Linux and Debian stretch are higher priorities for me right now.

[Wyse-]

I'm guessing the build instructions were confusing to you, because the exact steps to take are all there. Could you tell me what confused you, or submit a PR with fixes?

Actually the instructions are just fine, the only thing that confused me was that the build.py script was checking for a Python 2 module while being ran with Python 3: I assumed that it was looking for the module in the Python 3 installation, which in turn made me assume that either the check was not needed or that pypiwin32 was needed in Python 3 as well.
That was all caused by the check not passing even though pypiwin32 was installed correctly (and by me not actually checking the code!), but now that it's fixed with https://github.com/Eloston/ungoogled-chromium/pull/488 I doubt that it'll be a problem for anyone else.

That's expected. The patches haven't been updated yet. I might have an opportunity to update them this and/or next week, but I can't guarantee anything.

I see, no worries. I'll proceed to building and submitting the binaries when the patches are ready.

[ghost]

Chrome 69 is out now, figure you may as well just skip to that yeah?

[Eloston]

@92847586 Yeah, development on 68 has ended ever since xsmile's update to 69 was merged.

[Eloston]

c1a95aa5c50924e96ef51ab8c3b937fc4a764a2f has the existing patches refresh with moderate guessing involved. I have not tested anything yet.

[shiromichi]

I tried build at 8815e41

 2018-09-15 00:19:11,129 - INFO: * Applying windows-fix-enum-conflict.patch (119/119)
 2018-09-15 00:19:11,129 - DEBUG: E:\work\github\Eloston\ungoogled-chromium\build\src\third_party\git\usr\bin\patch.exe -p1 --ignore-whitespace -i E:\work\github\Eloston\ungoogled-chromium\build\src\ungoogled_packaging\patches\ungoogled-chromium\windows\windows-fix-enum-conflict.patch -d E:\work\github\Eloston\ungoogled-chromium\build\src --no-backup-if-mismatch --forward
 (Stripping trailing CRs from patch; use --binary to disable.)
 patching file extensions/browser/api/feedback_private/feedback_private_api.cc
 (Stripping trailing CRs from patch; use --binary to disable.)
 patching file extensions/common/api/feedback_private.idl
 2018-09-15 00:19:18,003 - INFO: Path has no substitutions: chrome/browser/plugins/plugins_resource_service.cc
 2018-09-15 00:19:19,229 - INFO: Path has no substitutions: chrome/browser/safe_browsing/client_side_model_loader.cc
 2018-09-15 00:19:22,079 - INFO: Path has no substitutions: chrome/common/page_load_metrics/page_load_metrics_util.cc
 2018-09-15 00:19:24,438 - INFO: Path has no substitutions: components/dom_distiller/content/browser/dom_distiller_viewer_source.cc
 2018-09-15 00:19:24,441 - INFO: Path has no substitutions: components/dom_distiller/core/html/preview.html
 2018-09-15 00:19:24,452 - INFO: Path has no substitutions: components/domain_reliability/google_configs.cc
 2018-09-15 00:19:24,742 - INFO: Path has no substitutions: components/google/core/browser/google_url_tracker.cc
 2018-09-15 00:19:28,218 - INFO: Path has no substitutions: components/variations/net/variations_http_headers.cc
 2018-09-15 00:19:55,330 - INFO: Path has no substitutions: net/dns/dns_transaction.cc
 2018-09-15 00:19:56,141 - INFO: Path has no substitutions: net/tools/transport_security_state_generator/transport_security_state_generator.cc
 2018-09-15 00:19:57,408 - INFO: Path has no substitutions: rlz/lib/lib_values.cc
 2018-09-15 00:20:20,196 - INFO: Path has no substitutions: third_party/catapult/third_party/polymer/components/font-roboto/roboto.html
 2018-09-15 00:20:21,196 - INFO: Path has no substitutions: third_party/catapult/tracing/third_party/gl-matrix/jsdoc-template/static/default.css
 2018-09-15 00:20:27,430 - INFO: Path has no substitutions: third_party/crashpad/crashpad/doc/support/crashpad_doxygen.css
 2018-09-15 00:21:04,376 - INFO: Path has no substitutions: tools/md_browser/base.css
 E:\work\github\Eloston\ungoogled-chromium\build\src>call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" >nul

 E:\work\github\Eloston\ungoogled-chromium\build\src>"C:\Python27\python.EXE" "tools\gn\bootstrap\bootstrap.py" "-oout\Default\gn.exe"
 ninja: Entering directory `E:\work\github\Eloston\ungoogled-chromium\build\src\out\Release\gn_build'
 ninja: error: unknown target 'gn'
 Traceback (most recent call last):
   File "tools\gn\bootstrap\bootstrap.py", line 92, in <module>
     sys.exit(main(sys.argv[1:]))
   File "tools\gn\bootstrap\bootstrap.py", line 87, in main
     ['ninja', '-C', gn_build_dir, 'gn', '-w', 'dupbuild=err'])
   File "C:\Python27\lib\subprocess.py", line 190, in check_call
     raise CalledProcessError(retcode, cmd)
 subprocess.CalledProcessError: Command '['ninja', '-C', 'E:\\work\\github\\Eloston\\ungoogled-chromium\\build\\src\\out\\Release\\gn_build', 'gn', '-w', 'dupbuild=err']' returned non-zero exit status 1

 E:\work\github\Eloston\ungoogled-chromium\build\src>exit
 Traceback (most recent call last):
   File "ungoogled_packaging\build.py", line 187, in <module>
     main()
   File "ungoogled_packaging\build.py", line 175, in main
     shutil.which('python'), 'tools\\gn\\bootstrap\\bootstrap.py', '-o'
   File "ungoogled_packaging\build.py", line 65, in _run_build_process
     **kwargs)
   File "C:\Users\TEST\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 468, in run
     output=stdout, stderr=stderr)
 subprocess.CalledProcessError: Command '('cmd.exe', '/k')' returned non-zero exit status 1.

 E:\work\github\Eloston\ungoogled-chromium\build\src>

['ninja', '-C', gn_build_dir, 'gn', '-w', 'dupbuild=err']

The fourth parameter 'gn' probably needs to be 'gn.exe'

[Eloston]

E:\work\github\Eloston\ungoogled-chromium\build\src>"C:\Python27python.EXE" "tools\gn\bootstrap\bootstrap.py" "-oout\Default\gn.exe"

Look like I forgot a comma in the build script. The last argument is wrong, but it still worked I guess...

I have no clue why the "gn" target doesn't exist. It looks like a bug in bootstrap.py

[shiromichi]

I tried building the GN manually
However, in the Japanese version of the compiler has failed

FAILED: tools/gn/command_gen.obj 
ninja -t msvc -- cl.exe /nologo /showIncludes /FC @tools/gn/command_gen.obj.rsp /c E:\work\github\Eloston\ungoogled-chromium\build\src\tools\gn\tools/gn/command_gen.cc /Fotools/gn/command_gen.obj
e:\work\github\eloston\ungoogled-chromium\build\src\tools\gn\tools\gn\command_gen.cc: error C2220: 警告をエラーとして扱いました。'object' ファイルは生成されません。
e:\work\github\eloston\ungoogled-chromium\build\src\tools\gn\tools\gn\command_gen.cc: warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。データの損失を防ぐために、ファイルを Unicode 形式で保存してください。

error C2220: 警告をエラーとして扱いました。'object' ファイルは生成されません。

error C2220: warning treated as error - no 'object' file generated

warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。データの損失を防ぐために、ファイルを Unicode 形式で保存してください。

warning C4819: The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss

Adding the /utf-8 parameter to cl.exe solves it

--- a/tools/gn/build/build_win.ninja.template
+++ b/tools/gn/build/build_win.ninja.template
@@ -1,12 +1,12 @@
 rule cc
-  command = ninja -t msvc -- $cc /nologo /showIncludes /FC @${out}.rsp /c ${in} /Fo${out}
+  command = ninja -t msvc -- $cc /utf-8 /nologo /showIncludes /FC @${out}.rsp /c ${in} /Fo${out}
   description = CC ${out}
   rspfile = ${out}.rsp
   rspfile_content = ${defines} ${includes} ${cflags} ${cflags_c}
   deps = msvc

 rule cxx
-  command = ninja -t msvc -- $cxx /nologo /showIncludes /FC @${out}.rsp /c ${in} /Fo${out}
+  command = ninja -t msvc -- $cxx /utf-8 /nologo /showIncludes /FC @${out}.rsp /c ${in} /Fo${out}
   description = CXX ${out}
   rspfile = ${out}.rsp
   rspfile_content = ${defines} ${includes} ${cflags} ${cflags_cc}

[Wyse-]

Just started testing, there's a problem with this line, the $ character should be a %, like in the other URLs.

After this change I get this error:
https://paste.debian.net/plainh/442baf75

[Eloston]

@Wyse- Strange, I didn't change anything from when @shiromichi ran the script in that area. I noticed your current working directory is changing between vcvars64.bat and bootstrap.py. I don't know what is happening there...

[shiromichi]

@Wyse- I tried to build d307bb0, but failed to download LLVM(HTTP Error 404: Not Found)
So, I'm downloading "LLVM" manually and building d307bb0

However, on my side, "can not open file" error did not occur...
Please make sure chromium-69.0.3497.92.tar.xz is extracted correctly

E:\work\github\Eloston\ungoogled-chromium\build\src>call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" >nul

E:\work\github\Eloston\ungoogled-chromium\build\src>"C:\Python27\python.EXE" "tools\gn\bootstrap\bootstrap.py" "-o" "out\Default\gn.exe"
ninja: Entering directory `E:\work\github\Eloston\ungoogled-chromium\build\src\out\Release\gn_build'
ninja: error: unknown target 'gn'
Traceback (most recent call last):
  File "tools\gn\bootstrap\bootstrap.py", line 92, in <module>
    sys.exit(main(sys.argv[1:]))
  File "tools\gn\bootstrap\bootstrap.py", line 87, in main
    ['ninja', '-C', gn_build_dir, 'gn', '-w', 'dupbuild=err'])
  File "C:\Python27\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja', '-C', 'E:\\work\\github\\Eloston\\ungoogled-chromium\\build\\src\\out\\Release\\gn_build', 'gn', '-w', 'dupbuild=err']' returned non-zero exit status 1

E:\work\github\Eloston\ungoogled-chromium\build\src>exit
Traceback (most recent call last):
  File "ungoogled_packaging\build.py", line 186, in <module>
    main()
  File "ungoogled_packaging\build.py", line 175, in main
    shutil.which('python'), 'tools\\gn\\bootstrap\\bootstrap.py', '-o', 'out\\Default\\gn.exe')
  File "ungoogled_packaging\build.py", line 65, in _run_build_process
    **kwargs)
  File "C:\Users\TEST\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 468, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('cmd.exe', '/k')' returned non-zero exit status 1.

Full(Expiration 90days):
https://paste.debian.net/plain/1042283

[Wyse-]

@shiromichi

@Wyse- I tried to build d307bb0, but failed to download LLVM(HTTP Error 404: Not Found)

Yes, that's what I meant with

there's a problem with this line, the $ character should be a %, like in the other URLs.

The version number is not getting properly replaced in the URL and the script is calling
https://prereleases.llvm.org/win-snapshots/LLVM-$(version)s-win64.exe
which delivers a 404, instead of
https://prereleases.llvm.org/win-snapshots/LLVM-8.0.0-r339319-win64.exe
which properly downloads LLVM

The corrected line would be
url = https://prereleases.llvm.org/win-snapshots/LLVM-%(version)s-win64.exe

@Eloston

I noticed your current working directory is changing between vcvars64.bat and bootstrap.py

You're correct, I believe the script is switching directory after executing vcvars64.bat, C:\Users\Wyse\source is a Visual Studio related directory. vcvars64.bat behaves the same way when called manually:
https://paste.debian.net/plainh/ccad1c34

The fixes I'm testing on Wyse-/ungoogled-chromium seem to be working for me, however I now get the same issue as @shiromichi:
https://paste.debian.net/plainh/ad3e9a70

[Wyse-]

The latest fix (https://github.com/Eloston/ungoogled-chromium/commit/dc456c6943bb9fcf025705c6b25dfd40a3b45afb) worked for me, however I now get this:
https://paste.debian.net/plainh/a30ba8c5

[shiioo]

It fails on my end too, using the "master" repo to get the latest fixes for Windows:
https://paste.ee/p/0QlCw

[crazy-max]

@Eloston I also tried to build a release on my side by fixing patches and also in a Python script but I'm encountering a strange error now : #598