Crystal: Continuous testing on Windows

[oprypin]

So we've been getting occasional PRs saying that they make this or that spec pass on Windows. That made me curious whether the status of specs passing/failing is being tracked anywhere.

@straight-shoota said that there hasn't been such a concerted effort, or even a list of specs that are expected to pass under the current status.
Well, not until he implemented this PR :slightly_smiling_face: https://github.com/crystal-lang/crystal/pull/8664

Then, how can we actually run these?
The current instructions https://github.com/crystal-lang/crystal/wiki/Porting-to-Windows require steps on Linux before getting to Windows.

I was inspired by the multi-stage approach of CircleCI, and indeed it allows running 2 sequential jobs on these different systems and pass files among them.

My work in progress: https://github.com/oprypin/crystal/blob/ci/.circleci/config.yml

I'm just following the instructions from the wiki quite exactly. CircleCI even has MSVC and buildtools preinstalled :slightly_smiling_face:

Sadly, CircleCI provides Windows only 250 minutes / week. And I've already hit that limit for myself...
https://circleci.com/gh/oprypin/crystal

[rishavs]

Can we not use GitHub workflow itself for the CI/CD?
https://github.com/features/actions

[straight-shoota]

@rishavs Might be worth a try. In the long run it would probably be best to use a single service for all CI. But for now this could be a good path for exploration.

We could also contact circleci to see how we can make this work. It seems the free credits for OSS projects can't be used for windows machines. But maybe we can figure something out.

@bcardiff WDYT?

[rishavs]

Azure DevOps, is yet another option where the CI/CD is completely free for OSS projects.

When we went for CircleCI, a lot of these options weren't present. Maybe its time to check out the market for alternatives. Should we open a new issue for the discussion instead of stealing the thunder from Orpypin's PR? :D

[straight-shoota]

@rishavs Yes, let's please keep focused on win32 CI here.
There is already the ancient #3721. But I doubt completely swapping out circleci should be a topic right now.

[bcardiff]

If GitHub works for starting a CI on windows, let's start with that option.

[RX14]

I'd prefer to keep the number of CI providers down.

If circleci can possibly work it'd be a much better option. You'd have to ask them for windows build credit i suspect.

[bcardiff]

Been there. done that. :-)

Windows is not an option for crystal-lang organization. We are on an old free pricing plan plus some perks that would go away if we move to windows. They are already providing a lots and lots of fuel for us.

[j8r]

Agree to keep the number of CI low, but CircleCI doesn't support aarch CI, nor Travis - and Crystal will need it.
To sumup, having a CI that supports without too many limitations (or not to costly):

• Linux aarch64
• Windows
• Bonus: FreeBSD!

As previously said (https://github.com/crystal-lang/crystal/issues/7587#issuecomment-476206235):

• Drone CI, which seems to support Windows on their Cloud, but doesn't support FreeBSD
• Cirrus CI, but doesn't support ARM.

The option is to ditch Travis to keep the number of different CIs at 2, which is reasonable.

I personally like Drone CI, which is used by Alpine Linux and is open source.
It is simple, but quite basic, the documentation is sparse. I really don't know all the limits of the free cloud offer.

[oprypin]

In my experience, GitHub Actions has one of the weakest configuration language and low flexibility. So I would strongly doubt that it allows pipelines that require both Linux and Windows as dependent steps.
From what I've seen, CircleCI seems to be unique in offering this ability.

[bcardiff]

Ok, let's make a try with https://cloud.drone.io/ then.

There would be a need for a PR with a .drone.yml file. I had activated cloud.drone.io in the organization.

Note that we won't accept a CI that depends on other than official crystal releases. Otherwise, the release process is not smooth.

Let's focus first on some initial Windows CI.

[oprypin]

https://docs.drone.io/configure/pipeline/multiple/

Multi-machine pipelines do not share state. It is not possible, for example, for two pipelines to access the same underlying file system or generated files.

¯\_(ツ)_/¯

[bcardiff]

And GitHub allows storing artifacts and the dependencies between jobs seem to not care about the platform. https://help.github.com/en/actions/automating-your-workflow-with-github-actions/persisting-workflow-data-using-artifacts#passing-data-between-jobs-in-a-workflow

@oprypin is there another limitation you had in mind?

Otherwise, it seems then that using GitHub is still the best option for me. But I am happy with whatever provider works.

[oprypin]

OK yes, then I underestimated it. Worth checking out.

[oprypin]

OK so turns out GitHub Actions is pretty sweet.

Aaaand it works!
We have a "Hello world!"

https://github.com/oprypin/crystal/runs/382421709#step:11:26

https://github.com/oprypin/crystal/blob/gh-ci/.github/workflows/win.yml
https://github.com/oprypin/crystal/actions?query=branch:gh-ci

[j8r]

State can be shared between steps, by creating volumes (see my projects for examples), but not pipelines, which is fine. Why would different pipelines share a same state?

The total limitation of GitHub Actions is 2000 hours in the free tier (Windows + in the future ARM Linux for GNU and musl).
If it is enough, it can be the best option.

[oprypin]

https://help.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions

It's actually 2000 minutes per month 😬
Or 1000 in the case of Windows.
So 16.7 hours per month.

But it seems like the limits don't apply to public repositories at all!

[j8r]

Ho right, quite low :astonished:
Yes, it's said Free for public repositories.
So :+1: for GitHub Actions.
Even so I prefer Drone CI on personal projects, and i like being Open Source, I think GH Actions will fit more the Crystal Lang organization (for instance user permissions).