So, we have a few PR where the Travis CI check seems to be blocked.
I did a quick Google search, and it seems Travis CI is no longer providing free CI for Open Source Software:
What's our situation with Travis CI? Do we have a plan (/ should consider on taking one)?
Many people seem to move to Gihub Action. Should we plan such migration?
Should we consider other CI services?
Do we have a plan?
I don't know any plan. We used the free offering which apparently went away. I noticed this recently-ish and wrote a comment here: https://github.com/awesomeWM/awesome/pull/3253#issuecomment-774415231
Should we consider other CI services?
GitHub Actions seems to be the popular choice these days (yay for more centralisation...). I will not try to port CI over to actions. (You might have noticed that my involvement declined.)
@psychon I can look into porting the CI into Github actions but this couldn't be done w/o your involvement I'm afraid. Providing I manage to "re-create" the CI on a local fork, would it be merged into this repo ? (I'm not looking for a condition-less promise, but given the time my last PR is under review, I'm reluctant doing such work if nobody requires it).
I don't really know how difficult something like this would be, but I can give it a shot.
edit: this doesn't have to be Github actions either I guess. There seem to be more options here
this couldn't be done w/o your involvement I'm afraid.
I took a quick look at the settings for this repo. On the "actions" tab, it says "Allow all actions". Nothing in there looks like it needs to be changed to allow more. So... dunno? Also, I would guess that one can also test actions on a private fork without having to go through this repo.
given the time my last PR is under review
Luckily, it turns out I already approved that one. :-)
I just asked for reviews from everyone to see if that makes something happen.
Everyone: What are our options? What are the pros/cons?
Personally, GH Actions seems to be "what everyone does" and this was also the reason why we used Travis, right?
CC @actionless @blueyed @Elv13 What do you think?
yup, that credit thing also stroke me unexpectedly on one of my other repos few days ago, i also consider moving to gh actions
but first i wanted to give it few days/weeks in case its some mistake on travis side
@actionless #3253 was 24 days ago.
i checked up the links in the first message: indeed it seems we have to start moving into github actions
With a second read over https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
We will be offering an allotment of OSS minutes that will be reviewed and allocated on a case by case basis. Should you want to apply for these credits please open a request with Travis CI support stating that you鈥檇 like to be considered for the OSS allotment. Please include:
- Your account name and VCS provider (like travis-ci.com/github/[your account name] )
- How many credits (build minutes) you鈥檇 like to request (should your run out of credits again you can repeat the process to request more or discuss a renewable amount)
Usage will be tracked under your account information so that you can better understand how many credits/minutes are being used
If I understand correctly, not all hopes on Travis CI are lost. We have to ask them for "renewable credit". I guess this arrangement between Awesome and Travis can only be done by owners, tho...
GitHub Actions seems to be the popular choice these days (yay for more centralisation...).
I feel you... But nowadays, every git host comes with its own integrated CI/CD service. I haven't already tried github action but I hope it can make the CI integration be a more native experience...
@ShayAgros if you feel like it, feel free to work on github action. I'm also eager to try it. Migrating the CI pipeline can be a good exercise.
However, I belive such migration needs approbation from other maintainers. So I think it's better to wait for the input of everyone.
i'm totally up for idea of moving to github actions :+1:
communicating with travis support from time to time -- feels like an annoying and unnecessary step
regarding centralization -- unless new CI config gonna rely on some github-specific features, but instead just use some generic shell scripts or make targets -- that shouldn't be an issue
(This comment doesn't announce any actual progress done in this area, just clears up the intentions and gives some FYIs)
From what I understand in this discussion, it is agreed to give Github Actions a shot (even if Travis turns out to be working again).
I've gone over some docs about Github Actions. We should get up to 2000 minutes per month of run time for these actions for free (see pricing).
Also it is also configured using a yaml file and is ran on Ubuntu machines (we can run in on macOS and Windows as well XD ).
I'll work on migrating the .travis yaml file into a Github action one and would update (by the end of this weekend I believe). Will update this ticket with any findings in any case
Thanks for the update and thanks for working on this.
We should get up to 2000 minutes per month
I didn't actually know that there was a limit, but it makes a lot of sense. I tried to figure out what the limit for Travis is, but I only found "10000 Credits". Does anyone know what this actually means? How/when is one credit used?
I am asking, because: How long would it take before we exceed the limit? Do we have to trim down CI somehow?
How/when is one credit used?
According to the table near the end of https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing, each started build minute deducts 10 Credits from the associated account. So each build seems to cost about 200 - 230 Credits, which means those free 10000 Credits lasted for some 40-ish builds.
Looking at the travis build in pr 3185, it writes that the travis check took 7m and 51 sec. This would theoretically allow for [2000 / (7 + (51/60))] = 254 builds with Github Actions.
This is theoretical, and I'm hoping it would take less with Github Actions since a simple time make after make distclean runs for 10.61 sec on my laptop (this is of course not a real measure since all the dependencies are already installed, and I don't know how strong Github's machine is).
I think this migration to Github Actions would be a good opportunity to see how we can reduce the build time as well (since almost 8 minutes for 4 builds is a lot ...)
Looking at the travis build in pr 3185, it writes that the travis check took 7m and 51 sec.
That is just the maximum time of one of the five builds that ran in parallel. If you open that build in Travis, it'll say "Total time 25 min 45 sec". And I would assume that that is the number the account is billed for.
With the build log, we can actually check how long each of the steps take. For the longest one, the major steps are:
| step | time |
|-|-|
| apt-get | 11.08s |
| xcb-util-xrm | 14.40s |
| luarocks install busted | 12.90s |
| make | 43.78s |
| run checks (no granularity, sadly) | 85.10s |
| check prev commits | 242.32s |
With GitHub Actions, one thing we could do to eliminate the need for installing/building dependencies is to run everything in a Docker container that we prepare with all the stuff we need (or a set with different Lua versions).
Additionally, we could consider running checks for older Lua versions on manual trigger only, so that the on_push builds for PRs aren't as costly.
check prev commits 242.32s
Of course, this depends on how many commits are in the PR, but I propose to drop that part. (Even though it was only added because of me.)
(It runs "everything" for each commit in the PR AFAIR)
It's certainly useful. If it does run everything right now, it might be enough to change it to run a build, and maybe unit tests. I think that should cover most things that could stop a bisect.
Just found this at https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions#about-billing-for-github-actions:
About billing for GitHub Actions
GitHub Actions usage is free for both public repositories and self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account.
So the 2000 minutes doesn't actually seem to apply for this repository.
Progress: https://github.com/sclu1034/awesome/actions/runs/648933065
I would love it if you wrote that you're working on doing this CI as well considering the time I already spent on it....
I also managed to make similar progress with it.
https://github.com/ShayAgros/awesome/actions/runs/649138720
@sclu1034 I decided to work on it not because I have lots of time on my hands, but simply because I spotted a need (after all this ticket is open for half a month) and decided to help. The reason I declared that I'm starting to work on it is to make sure I'm not doing redundant job
I decided to work on it not because I have lots of time on my hands, but simply because I spotted a need (after all this ticket is open for a month) and decided to help. The reason I declared that I'm starting to work on it is to make sure I'm not doing redundant job
That wasn't clear to me until now. While you did announce that you were planning to work on it, I don't read an extra "but only if no one else is doing it" into such a line, unless it is explicitly stated.
And while I might not have announced that I would look into it the moment I started, I did make sure to do so early into my progress.
Additionally, while this might be a personal view, I don't consider something like that redundant. We will likely have different approaches/findings, which could be compared for improvements. And the current situation shows that a working CI is quite important, so having multiple people with the knowledge to change/fix it can be quite helpful.
That wasn't clear to me until now. While you did announce that you were planning to work on it, I don't read an extra "but only if no one else is doing it" into such a line, unless it is explicitly stated.
Well, at least with me, if it's not too much trouble, please let me know that you're working on the same thing I announce I'm gonna work on. I'd guess it is true for other people here, but I'll let them speak for themselves
Additionally, while this might be a personal view, I don't consider something like that redundant. We will likely have different approaches/findings, which could be compared for improvements. And the current situation shows that a working CI is quite important, so having multiple people with the knowledge to change/fix it can be quite helpful.
I've gone over your workflow file. We are taking different approaches. I rely much more on the package manager, and don't build from source packages like the travis file does.
I'm working on expanding it to other lua versions now, in a hope to see if I'm eventually forced to build from source some packages (seems like luarocks would have to be built from source \=)
Well it's late (at least where I live it is). Can't say I managed to do all the things I wanted but here is an update of my progress:
I managed to write a workflow document which triggers the tests, and got awesome to build for lua5.3 and lua5.2 (it might not look like it if you check, but that just because I keep tweaking it and things break). This is my workflow file.
I didn't add any additional tests as for now to the CI (the tests are far from being as thorough as Travis ones and what Lucas has). Since I found out Lucas works on it too I tried to concentrate on how to improve the time it takes to do these tests.
These are issues we're facing if we want to make them faster:
ruby gems and luarocks directory downloading, but this doesn't provide a solution for apt packages (unless we're gonna cache /usr/bin, /usr/local/bin etc.).apt calls is to create a container that has all the packages we need and load it when we do our tests as suggested in this SO post. Going with this approach means that we're gonna have ~80MB image in our repository which every user which does git clone would have to download. This doesn't seem like a very good approachapt-get download to download the apt files, cache them, and install them each test cycle. This sounds like a perfect idea, but apt doesn't download packages recursively, which would force us to download them when we install the "downloaded packages".I guess we're looking more into having tests ASAP, in which case I guess a PR that does (at least basic) tests on pull-requests could be submitted during this week (let's see how far along Lucas got). Did learn a lot about dev-ops problems (brought up a docker for the first time).
For testing I mostly used a tool which is called act. It brings up a docker on your machine and runs the tests in a similar fashion as they are on Github (somehow it is less strict in some tests).
My current status is:
Issues I've run into:
actions/upload-artifact. I don't know if actions/cache works.gears.filesystemtput (in tests/run.sh) sometimes fails, complaining about a missing $TERM and I haven't been able to find a way to debug it yet.Caching APT packages works fine for me (apt-get install --download-only <packages...> includes all missing dependencies). But package installation wasn't as big of an issue for me anyways, with a non-cached run taking about 40 seconds.
Using a docker image might still be able to reduce dependency setup time and make it easier to create parallel jobs. And by hosting the image at Docker Hub or GitHub's Container Registry, that should not affect regular usage of the repository at all.
What I'm still missing in terms of functionality is:
git bisect stuff)Status update: It works, mostly.
Current issues:
make check-themes fails consistently with LGI 0.8.0 (example run).make check-integration often (but not always) fails with either a timeout or X error: error=BadDrawable.The extra workflow to build and publish API docs is still missing.
I took a quick look and also have no idea. Sorry. The GetGeometry error comes from one of the eight calls to xcb_get_geometry_unchecked.
make check-integrations also fails consistently for LGI 0.8.0 with Error: timeout waiting for signal in step 11/138 (@100) (/home/runner/awesome/tests/test-awful-client.lua:221).BadDrawable issue to awesome.c:scan (log output). But since I don't know anything about Xorg or XCB, I have no idea how to proceed.I've managed to narrow down the BadDrawable issue to awesome.c:scan (log output). But since I don't know anything about Xorg or XCB, I have no idea how to proceed.
TL;DR: I also have no idea how to proceed.
During startup, the window manager first asks the X11 server "give me a list of all existing (top-level) windows" (xcb_query_tree). Then, for each existing window it checks if that window should be managed by the WM by querying information about this window.
BadDrawable means that a window ID was queried that does not exist. Thus, I guess that means that the window was destroyed between the time of xcb_query_tree and the later xcb_get_geometry. Perhaps something was still being closed / nothing waited for all the previous clients to exit before starting a new test?
However, the code you linked to basically does two other X11 requests on the window before the GetGeometry request. It is really weird that the two earlier requests do not cause error?!?
The error itself is harmless, I guess. It just means that AwesomeWM will ignore the window. Since we have no tests that should open windows before starting AwesomeWM, that shouldn't really cause any problems.
Only downsides:
The test immediately before the failing test opened some xterms, but the test runner should complain if some of them were still open at the end: https://github.com/awesomeWM/awesome/blob/413d47d5a5b6b9d72b6ea01839601efb092ec583/tests/_runner.lua#L55-L58
My best ideas so far would be a sleep 0.5 in tests/run.sh's start_awesome or the same in more unobvious:
diff --git a/tests/run.sh b/tests/run.sh
index ec4395c6a..ff04ebbec 100755
--- a/tests/run.sh
+++ b/tests/run.sh
@@ -183,6 +183,11 @@ fi
start_awesome() {
cd "$build_dir"
+ # Ugly hack to slow things down a bit / make sure that all X11 connections
+ # from the last test are really closed.
+ DISPLAY="$D" xwininfo -root
+ DISPLAY="$D" xwininfo -root
+
# Kill awesome after $TEST_TIMEOUT seconds (e.g. for errors during test setup).
# SOURCE_DIRECTORY is used by .luacov.
DISPLAY="$D" SOURCE_DIRECTORY="$source_dir" \
Perhaps something was still being closed / nothing waited for all the previous clients to exit before starting a new test?
You seem to be right on that one. A call to xlsclients returns some clients occasionally: https://github.com/sclu1034/awesome/runs/2141278454?check_suite_focus=true#step:25:443
While even just adding debug calls to xlsclients did seem to improve things slightly already, it required a proper wait_until_sucess to get tests passing consistently. And adding a time, it looks like it's quite necessary, with some clients staying alive for a full minute.
The only thing that's left now is LGI 0.8.0. For this one, I've been able to reproduce some things in a local docker container, including testing Ubuntu 16.04 (Travis), 18.04 and 20.04 (GitHub Actions).
On all three versions, I get the above Error: timeout waiting for signal in step 11/138 (@100) (/home/runner/awesome/tests/test-awful-client.lua:221).
Additionally, on 18.04 and 16.04 it adds this error:
_client /usr/bin/lua5.2: (command line):86: /usr/local/share/lua/5.2/lgi/override/GObject-Object.lua:84: bad argument #8 to 'object_new' (string expected, got userdata)
_client stack traceback:
_client [C]: in function 'coro'
_client (command line):86: in function <(command line):75>
_client [C]: in function 'main'
_client (command line):92: in main chunk
_client [C]: in ?
Tests succeed.
I get this once
2021-03-18 20:29:28 E: awesome: Error during a protected call: /usr/local/share/lua/5.2/lgi/override/GObject-Object.lua:84: bad argument #4 to 'object_new' (table expected, got nil)
stack traceback:
[C]: in function 'object_new'
/usr/local/share/lua/5.2/lgi/override/GObject-Object.lua:84: in function '_construct'
/usr/local/share/lua/5.2/lgi/override/GObject-Object.lua:149: in function </usr/local/share/lua/5.2/lgi/override/GObject-Object.lua:148>
(...tail calls...)
lib/beautiful/gtk.lua:100: in function 'get_theme_variables'
/home/runner/awesome/build/themes/gtk/theme.lua:89: in main chunk
[C]: in function 'dofile'
[C]: in function 'xpcall'
lib/gears/protected_call.lua:36: in function <lib/gears/protected_call.lua:35>
(...tail calls...)
lib/beautiful/init.lua:285: in function 'init'
/tmp/tmp.Y2tdvFbgr5:41: in main chunk
2021-03-18 20:29:28 E: awesome: beautiful: error loading theme: got a nil from: /home/runner/awesome/build/themes/gtk/theme.lua
and then a bunch of these
2021-03-18 20:29:28 E: awesome: Failed to load 'inactive': Failed to open file 'inactive': No such file or directory
stack traceback:
lib/gears/surface.lua:101: in function <lib/gears/surface.lua:93>
(...tail calls...)
lib/wibox/widget/imagebox.lua:151: in function 'load_and_apply'
lib/wibox/widget/imagebox.lua:198: in function 'set_image'
lib/awful/titlebar.lua:783: in function 'update'
lib/awful/titlebar.lua:687: in function <lib/awful/titlebar.lua:683>
Dumps this when trying to test the GTK theme:
2021-03-18 20:32:49 E: awesome: signal_fatal:490: signal 11, dumping backtrace
/home/runner/awesome/build/awesome(backtrace_get+0x5e) [0x55e049d73e01]
/home/runner/awesome/build/awesome(+0x16ebe) [0x55e049d5debe]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f807aef5210]
/usr/local/lib/lua/5.2/lgi/corelgilua51.so(lgi_object_2lua+0xbc) [0x7f807998d6ac]
/usr/local/lib/lua/5.2/lgi/corelgilua51.so(+0x1394e) [0x7f807998d94e]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xcec2) [0x7f807b10cec2]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0x18d47) [0x7f807b118d47]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xd280) [0x7f807b10d280]
/lib/x86_64-linux-gnu/liblua5.2.so.0(lua_callk+0x3f) [0x7f807b10925f]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0x1d135) [0x7f807b11d135]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xcec2) [0x7f807b10cec2]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xd246) [0x7f807b10d246]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xc7d4) [0x7f807b10c7d4]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xd4e3) [0x7f807b10d4e3]
/lib/x86_64-linux-gnu/liblua5.2.so.0(lua_pcallk+0x7b) [0x7f807b10932b]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0x1cc04) [0x7f807b11cc04]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xcec2) [0x7f807b10cec2]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0x18d47) [0x7f807b118d47]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xd280) [0x7f807b10d280]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xc7d4) [0x7f807b10c7d4]
/lib/x86_64-linux-gnu/liblua5.2.so.0(+0xd4e3) [0x7f807b10d4e3]
/lib/x86_64-linux-gnu/liblua5.2.so.0(lua_pcallk+0x7b) [0x7f807b10932b]
/home/runner/awesome/build/awesome(+0x20dfd) [0x55e049d67dfd]
/home/runner/awesome/build/awesome(luaA_find_config+0x21) [0x55e049d692f7]
/home/runner/awesome/build/awesome(luaA_parserc+0x11) [0x55e049d694a0]
/home/runner/awesome/build/awesome(main+0xee2) [0x55e049d5f180]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f807aed60b3]
/home/runner/awesome/build/awesome(_start+0x2e) [0x55e049d5da8e]
Yay, LGI fun... :-(
Since I have no immediate idea: Do you happen to know what the "obvious differences" between the current/old travis build and your actions are? Here is a link to a random (recent-ish) Travis build with LGI 0.8.0: https://travis-ci.com/github/awesomeWM/awesome/jobs/476723988
Error: timeout waiting for signal in step 11/138 (@100) (/home/runner/awesome/tests/test-awful-client.lua:221).
Puh. That's one of those tests where the step numbers are useless. :-(
lib/beautiful/gtk.lua:100: in function 'get_theme_variables'
Uhm, that line is simply local label = Gtk.Label(). Constructing a GtkLabel already fails!? Since the error in question is "(table expected, got nil)", perhaps this needs to be local label = Gtk.Label({})? (Or local label = Gtk.Label{} for extra obfuscation :-) ).
Failed to load 'inactive': Failed to open file 'inactive': No such file or directory
That's awful.titlebar not finding a file called inactive, which really does not sound like a file name at all. That seems like quite a broken beautiful theme?!? Uhm... Looking at the code: If none of the beautiful theme lookups find an entry, then it just does something like this. Perhaps we shouldn't do that and instead do:
diff --git a/lib/awful/titlebar.lua b/lib/awful/titlebar.lua
index 734aecb27..d1ac355de 100644
--- a/lib/awful/titlebar.lua
+++ b/lib/awful/titlebar.lua
@@ -730,6 +730,8 @@ function titlebar.widget.button(c, name, selector, action)
or beautiful["titlebar_" .. name .. "_button_" .. prefix .. "_inactive"]
if theme then
img = theme
+ else
+ img = nil
end
end
ret:set_image(img)
Dumps this when trying to test the GTK theme:
Sorry, I pass. Debugging segfaults that only occur in old versions of a library does not sound like fun nor like it can have helpful results. The backtrace does not seem helpful either.
Do you happen to know what the "obvious differences" between the current/old travis build and your actions are?
I tried to be as faithful to the build steps in Travis as possible, so I don't think there are any major differences in how things run.
The virtualized hardware also seems to be fairly similar: Travis vs. GitHub. But it's not like we have much control over that anyways.
So changing Ubuntu versions already was the only obvious thing to try that I could think of.
Before investigating the individual errors further, are there any LGI + Ubuntu version pairs that we could rule out as "won't fix"? As mentioned in a previous post, current distributions are on LGI >=0.9 already and Ubuntu 16.04 will drop out of LTS at the end of April.
Simplest solution: Drop LGI 0.8 support. That also allows getting rid of some code that works around problems (see beginning of awful.spawn).
Certainly the easiest to implement.
I guess, I'll just do that and if anyone has objections, they may speak up here and/or in the upcoming PR.
Looks like the awesome repo and the awesome-www share their Travis credit. I just merged https://github.com/awesomeWM/awesome-www/pull/158 and no Travis build was started. https://www.travis-ci.com/github/awesomeWM/awesome-www talks about negative credit.
Can we disable Travis builds for this repo so that at least awesome-www works? Keeping them enabled here does not provide much value (we'd quickly run out of credit again), but keeping the awesome-www builds working seems quite useful.
CC @bzgec so that you know why your new content won't end up on the web page for now. Sorry.
Yeah, it seems that Credits are billed per account, in this case the awesomeWM organization.
I don't see harm in disabling Travis here. But it might still be worth applying for OSS Credits for both repos to get PR builds going again, until the migration to GitHub Actions is done.
I changed the plan to the paid one and bought some credits. I would like not to pay 85$ US per month forever, so I still propose we make it work with actions. But _for now_, the CI is back, albeit with only 1 parallel job because the cost of the 5 job plan is ludicrous.
Thank you elv 鉂わ笍
I hope we'll have a CI working soon with Github actions thanks to the work of @sclu1034 and @ShayAgros
Most helpful comment
I changed the plan to the paid one and bought some credits. I would like not to pay 85$ US per month forever, so I still propose we make it work with actions. But _for now_, the CI is back, albeit with only 1 parallel job because the cost of the 5 job plan is ludicrous.