Elixir: Add make compiler

[josevalim]

The goal is to make it easy for those who have to compile native dependencies.

[joshnuss]

:+1:

would love to use this for FFI

[whatyouhide]

@josevalim may I plug this https://github.com/whatyouhide/make_compiler? :) It's not feature complete and missing windows, but I guess a good start. I can tackle this if you want.

[ericmj]

I think it would be better to support some basic C compilation without having to resort to makefiles and then fall back to a makefile compiler.

[josevalim]

Honestly I don't want to support C compilation because we simply won't
solve the problem as well as makefiles. See rebar.

So I would rather make it easy to plug the makefile and compile with it.

Thoughts?

_José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D_

[fhunleth]

+1 on this feature. I'll certainly use it on Nerves-based projects. Also, I have a preference for Makefiles, since I'm comfortable with what I need to do to make them support cross-compilation.

[whatyouhide]

@josevalim @ericmj I'm strongly in favour of makefiles and not compiling C code. C compilation is crazy according to my experiences, and Makefiles already handle that relatively smoothly.

[arpunk]

IMO we want to play nice with Makefiles in order to avoid breaks under multiple build systems, specially embedded ones with mixed toolchains.

[lexmag]

Agree on make compiler > C compiler.

[ericmj]

@arpunk That means that every embedded system someone would like to use has to be supported in the Makefiles of every dependency used.

I'm not advocating for doing C compilation in Mix but it seems like many here are missing the pain points of using Makefiles. Just yesterday someone asked on IRC on how to cross compile their NIF dependency and there was no solution for it because the dependency didn't support it in its Makefile, the only solution was to compile on the target machine instead.

[arpunk]

@ericmj I see your point. I've only worked with buildroot and the OpenWRT build suite for building custom firmwares, so I'm pretty much opinionated towards Makefiles.

[josevalim]

We have added this feature as a separate project so we can get some initial feedback before v1.3 is out: http://github.com/elixir-lang/elixir_make

Please give it a try and let us know if it works as desired!

[jpmec]

That looks nice jose, thanks for adding the env option. One question, could a "pre-build" callback be added that expects a config and returns a new config?

[whatyouhide]

@jpmec what would the use case be?

[jpmec]

@whatyouhide, the use case I had in mind is that I want to call a System.cmd before make is called to query the environment variables needed for nmake. I found that I can call the vcvarsall.bat, then get the environment variables, and then set them before nmake is called.

See the example I made a while back here:

https://gist.github.com/jpmec/a405dd202d083066a842

see on line 9 where the Exvc.vc_env() is called? I would like to perform this call and return a new config with the make_env variable set to the return value of Exvc.vc_env()

[josevalim]

@jpmec we could make it so make_env can also be a function. In such cases, it would be invoked before being used. If that works for you, please send a PR to elixir_make or open up an issue there. :D

However, you should also consider moving this logic to makefile itself (if possible).

[fhunleth]

FYI: I gave it a try with nerves_uart on Linux and Windows, and it works great.

[josevalim]

@fhunleth awesome, thank you for the feedback!

[josevalim]

@jpmec @riverrun @joshnuss any feedback on the elixir_make project so far? :)

[jpmec]

Hi @josevalim,

I tried it out on comeonin here:

https://github.com/jpmec/comeonin/blob/feature/make_compiler/mix.exs

It seemed to get stuck in an endless loop? It kept printing

        mix compile

Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

I'll look at it some more and see if I can't figure out what is going on.

[jpmec]

Something that would be helpful would be to have an option to print out what elixir_make is doing, specifically what is the final command that will be executed?

[riverrun]

I'm working on a test project here. I haven't got it working yet, but it does build.
@jpmec you need to edit the Makefile so that the all option points to the prive\bcrypt_nif.dll line, or you might be able to define your own make command in the config.

[josevalim]

Hello everyone,

This is another ping to see how the elixir_make integration is going. The plan is to release Elixir v1.3-rc this week and we won't include elixir_make in it so far. If we get enough usage and positive feedback before final, we can definitely include it, otherwise we will hold until v1.4, so please let us know.

@jpmec we will make it support the --verbose flag.

Thank you!

[riverrun]

Comeonin develop branch is now using elixir_make.
This has just been tested on Linux, so feedback on using it on other platforms would be very welcome.

[jpmec]

Hi all, sorry for the delay @josevalim. Here is what I've done on Windows with the develop branch of comeonin

git clone https://github.com/elixircnx/comeonin.git elixircnx_comeonin -b develop
cd .\elixircnx_comeonin
mix deps.get
mix compile

I (as expected) get an error about nmake not being found.

Here are my thoughts: I would very much like to be able to configure mix compile from the command line. However, it appears that "the way" is to require the user to setup the environment external to mix and then invoke mix from within this environment? I think this is okay, but not ideal. For instance what if I already know the path to nmake, or I have a mingw version of make installed? I would like to be able to pass in my own settings without having to change what the author intended, or what elixir_make uses as its defaults. There is a very good chance that this is possible, but I am just ignorant of how to do it?

I have tried adding a config/config.exs file to the cloned project that looks like this:

use Mix.Config

config :elixir_make,
   make_executable: "make"

However I still get an error about nmake not found when I run mix compile? Is this right?
I do have make on my path, that will call mingw version of make

[josevalim]

@jpmec if we want to support such, I think we need to do so by using environment variables. Using Mix project or the config files is not going to work when compiling a dependency. Maybe MIX_MAKE_EXECUTABLE? Any other flags you would need?

[jpmec]

@josevalim hmm, environment variables are an okay work around for smaller stuff. I think this goes back to the ultimate problem of the choices the library author made to decide how to compile may not be the choices I need. Perhaps for now, just settling on nmake for windows is the best? I'm okay using Visual Studio, but the awareness needs to be that with this compiler, then Visual Studio becomes a pretty hard dependency. This may or may not be something that others are okay with.

[jpmec]

@josevalim and about the environment variables, I think the MIX_MAKE_EXECUTABLE would be enough for my needs for now.

[josevalim]

@jpmec doesn't the current compiler automatically picks up make if you are inside something like MSyS/MinGW?

[fhunleth]

Regarding environment variables, there's a convention to use $MAKE to point to the make executable and to provide arguments. On Windows, I've only seen this used with Cygwin and Mingw projects, so I'm not sure it applies to Visual Studio. The use cases that I can think of would be to set $MAKE to gmake or gnumake if your default make is the BSD variety. The other one is if your project is being built as part of a larger build system (like Buildroot if you're cross-compiling). Buildroot will set $MAKE to the path to /usr/bin/make -j4 or similar to do parallel builds. Also, they might add debug options to the make invocation so that you can get more printouts. There's also the $MAKEFLAGS variable for these extra options, but I've seen projects use either that or $MAKE, so if using $MAKE, you'll have to be aware that it could contain options. I can't say that I feel very strong about this, but if an environment variable name is chosen to override the make executable, then using $MAKE would fit in nicely with other tools.

Also, regarding MinGW (actually Mingw-w64), I use it for nerves_uart on Windows. Here's what I did when I integrated with elixir_make: https://github.com/nerves-project/nerves_uart/blob/elixir_make/mix.exs. While I like using MinGW, it's not ideal with Elixir since the pre-built version of Erlang is built with Visual Studio. I had thought that since I was using pure C that I wouldn't have ABI issues with the erl_interface library when using gcc, but I did. nerves_uart builds a port executable, so it wasn't hard to work around this ABI issue. I would guess that if I was building a NIF that I would have given up and used Visual Studio. I'm not sure how many people will use MinGW on Windows because of this - perhaps the only redeeming values of it are trivial install with Chocolatey and a friendlier interface if you're more of a Unix person. Also, fwiw, make with the default MinGW install is called mingw32-make.

[josevalim]

Thanks @fhunleth. We could use the MAKE environment variable but we would need to change how we currently invoke the MAKE command since we make a distinction between executable and arguments. I am not sure if that is worth the trade-off.

[fhunleth]

I think the separation in elixir_make between executable and arguments is the way to go, and don't want that changed. The point that I meant to make was that if there's an environment variable, I like it being called MAKE. It just needs to be sanitized first since there's a common use (abuse?) of it to pass command line arguments.

Here's another data point: If a Makefile calls make recursively, the best practice is to call $(MAKE) in the recipe rather than referring to it as just make. That call will use the $MAKE environment variable. For example, take this Makefile:

all:
    @echo "Sub-make will use: $(MAKE)"

Here are some sample invocations:

$ unset MAKE
$ make
Sub-make will use: make
$ export MAKE=/my/other/make
$ make
Sub-make will use: /my/other/make

Based on this, I think that I'd want elixir_make to check if the MAKE environment variable is set and use it. If it doesn't do that and the MAKE environment variable is set, then the top level Makefile will be invoked differently than the sub-Makefiles. That might be confusing.

[riverrun]

I was also having problems with setting config values with elixir_make (I was trying to change the default makefile). Adding values to a config.exs file didn't seem to have any effect. When I checked the output of Mix.Project.config, none of the config values were present - there was just a value for the config_path.
Has anyone managed to get this to work?

[josevalim]

@riverrun the configuration happens in your def project in your mix.exs (similar to the other compilers):

def project do
  [make_executable: "special_make"]
end

[josevalim]

Hi everyone, we have currently decided to not include elixir_make in Elixir v1.3. We will wait for it to get stable and release it as part of Elixir v1.4. The benefit is that you can use it today in your projects as a dependency as it works with older Elixir versions. This way we can get more mileage before we effectively include it in Elixir v1.4. If someone disagrees with this decision, please let us know and we can further discuss it. :)

[riverrun]

Sounds good. I just have a question about how you will handle the transition from using elixir_make (as a separate library) to having make as part of the standard library. Is that going to cause many headaches?

[josevalim]

It should not. The reason we named the project :elixir_make with name of
ElixirMake is so when we move it to Elixir we can rename it to :make and
Make. So you will only need to change the compiler name and everything
should work with no conflicts.

On Thursday, June 9, 2016, David Whitlock [email protected] wrote:

Sounds good. I just have a question about how you will handle the
transition from using elixir_make (as a separate library) to having make as
part of the standard library. Is that going to cause many headaches?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elixir-lang/elixir/issues/4082#issuecomment-224814544,
or mute the thread
https://github.com/notifications/unsubscribe/AAAlbpATKHtRCU7mEa5rm7659Wvg5Xh1ks5qJ7higaJpZM4G3Kjy
.

_José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D_

[josevalim]

It has been a while since folks are using elixir_make and we have not heard of any bugs. It seems to be time to merge it back into Elixir. What do you folks think?

[tony612]

👍 I used it recently, it works well

[whatyouhide]

For reference, as of now elixir_make has ~3k downloads on Hex, so we know a few people should have used it :).

[fhunleth]

We're giving it quite a bit of use on Nerves-based projects. I also use it on Windows, and it has been working fine there as well.

[whatyouhide]

Hello beautiful community,

I had a talk with @josevalim and we decided that for now elixir_make won't be a part of Elixir's standard library. There are a few reasons: first of all, it works well as a package so you can use it as any other dependency. Also, it has been out for a while now and only has ~5k downloads on Hex, so it's not being used like crazy. For this reason, we think that the cost of bringing it into core (maintenance, less agility) outweigh the cost of having it as a package (the only downside of which is, well, not having it into the standard library).

We are not excluding that elixir_make will make (get it?) it into Elixir some day though, time will tell us.

Please, if you don't agree with this decision and/or see reasons to bring it in core that we did not consider, don't hesitate to say so (this issue is a good place to discuss for now). Ping us if you have any questions as well, we'll be happy to answer :)

Thanks all!