Elixir: Support FreeBSD and macOS in Cirrus CI

[eksperimental]

Opening this issue to see if now that the migration to Cirrus CI has been settled, we can start supporting FreeBSD in our CI?
Also since macOS is supported, if we have plans of supporting it as well.

/cc @fertapric

[josevalim]

FreeBSD definitely. I would skip macOS, as it should be covered under Linux (i.e. we had no plans in testing all Linux distributions).

[fertapric]

@eksperimental yes, we can start working on FreeBSD support.

A PR is welcomed :)

[eksperimental]

Sorry, but why would macOS be covered by Linux? They are different OSes, very similar though, I know there are differences in the shell, but other than that, I don't know if there are other cases that we need to cover too. I have been been a macOS user, so that's why I'm asking.
Maybe just test it on the latest stable OTP?

[eksperimental]

@eksperimental yes, we can start working on FreeBSD support.

A PR is welcomed :)

I can work on that one

[josevalim]

I won't deny we will get more coverage by testing on macOS but there is also a point where we get diminishing returns and I think macOS is one of them, especially a couple members of the core team already uses it. Basically I don't think the cost in terms of build time, resources, etc is worth it.

[eksperimental]

yes. I was also thinking that. it get's more coverage than any other OS as far as I can see

[eksperimental]

I'm already working on the FreeBSD task.

this is the list of available images.
https://cirrus-ci.org/guide/FreeBSD/

freebsd-12-0
freebsd-11-3-snap (stable freebsd-11-3 doesn't boot properly at the moment)
freebsd-11-2
freebsd-10-4

Should we go with freebsd-12-0? (released in December 2018) or with the lowest one freebsd-10-4 (released Octuber 2017)

[eksperimental]

how are the builds in https://repo.hex.pm/builds/otp/ubuntu-14.04/ done?
Can we have the same for the desired FreeBSD version?

[josevalim]

Either 12-0 or 10-4 sound good to me. Can we use pkg install erlang to install it?

[eksperimental]

Either 12-0 or 10-4 sound good to me. Can we use pkg install erlang to install it?

I think so but I guess we won't be able to run it against every OTP version. Let me write up a first version and from there we see how we can move from there on

[josevalim]

@eksperimental I don't think we need to run against every OTP version. Same reason as above regarding diminishing returns. If we were to test all Erlang/OTP combinations across all OSes, it would be quite wasteful. I think testing the Erlang/OTP combinations on a given OS and then one or two versions for the other OSes are fine. For FreeBSD in particular, the most important thing we want to check is in regards to the scripts, and those do not depend on the Erlang/OTP version. :)

[wojtekmach]

Worth mentioning that macOS and FreeBSD have pretty similar user land (e.g. they both have BSD sed as opposed to GNU sed that many Linux distributions have) so if the focus is on scripts just checking on FreeBSD should be enough indeed.

[hauleth]

@wojtekmach there is one huge difference between macOS and Linux/FreeBSD: case insensitive filesystem by default and this was cause of the problems in the past #8782

[eksperimental]

I was thinking about this the other day, if we don't want to check in every commit, we can randomly check the commits, only when the branch is master, with a probability of, let's say, 1 in 20.
And when the branch is a release branch and/or has a tag, such as *-rc always test on these platforms.
Additionally, add to these platfors, a 32 bit Linux operating system. something like alpine Linux.

We don't need to detect these edge-case errors straight away, but eventually before a release is out

[josevalim]

Erlang/OTP already has continuous integration setup with all these systems, which is why I don't think we need to test against all of these systems using all OTP versions. Let's pick one, preferably the latest, and we should be good to go. If that leads to some gaps, then we can enhance the matrix later.

[josevalim]

Closing in favor of PR.