Elixir: Introduce `Mix.target`

[josevalim]

Nerves applications have different targets (raspberry pi, beaglebone black, etc). Each of those targets may have different dependencies. Since there is no built-in support for targets in Mix, the choice of dependencies is done conditionally:

def project do
  [
    ...,
    deps: deps(@nerves_target)
  ]
end

defp deps(:rpi3) do
  [...] ++ deps()
end

defp deps(:bbb) do
  [...] ++ deps()
end

This approach comes with some downsides. First of all, there is no command to fetch all dependencies across all targets. So developers may need to run mix deps.get multiple times.

Second of all, having distinct blocks means the dependency resolution happens in parts, which may lead to conflicting lock files. Imagine the :rpi3 section has a dependency with a dependency on poison ~> 3.0 and the :bbb section has a dependency with a dependency on poison ~> 4.0. Every time you run mix deps.get, you will either get an error OR the lock file will change. Users may attempt to solve this with distinct lock files per target but then it means you lose a consistent view of the system.

Those issues are not unfamiliar to us, since in earlier Elixir versions, dependencies were handled in the same way, by having a call to deps(Mix.env). Back then, we solved this issue by introducing the :only option to dependencies. Now we propose to introduce the :target option.

Solution

Instead of using conditional blocks, we will allow the :target option:

{:some_dep, "~> 1.0", target: :rpi3}

In order to make it fully functional, we will need the following changes:

• [x] Introduce Mix.target/0 and Mix.target/1. The default target will be :host and it can be set with the MIX_TARGET environment variable;
• [x] Allow the :target option when specifying dependencies. It will work the same way as :only, except we will filter on the value of Mix.target rather than Mix.env;
• [x] Allow the --target option in mix deps.get and mix deps.update if someone is interested in fetching or updating dependencies for a given target;
• [x] Change MixProject.build_path/1 to make the build path be #{target}-#{env} when the target is not :host;

Implementation wise those changes should be straight-forward as target will behave the same way as :only does today.

/cc @mobileoverlord @fhunleth @ConnorRigby

[ConnorRigby]

One thing that isn't that big of a problem is using atoms for the target. We used to concat the target to choose the nerves_system_#{target} dep. I don't think we do that anymore. Atoms work with this also, it just looks funny: :"nerves_system#{target}"

Anyway this seems like a good idea to me

[ConnorRigby]

Another concern I have, that I don't actually think will be an issue is sub dependencies.

I and many other Nerves users use path: dependencies to separate concerns. Ideally this Mix.target() would be global for all deps. It sounds like that is what is proposed, I just want to be sure.

This is similar to something @pragdave brought up on the mailing list I believe about Mix.env() and path: dependencies.

[josevalim]

@ConnorRigby can you please expand your points? How are you using path dependencies? How is Mix.target() related to @pragdave's discussion? I believe I was part of those discussions but I have terrible memory. Plus there are other people here which are likely not familiar with those discussions, so the most precise and the more context you can give, the better. Thank you.

EDIT: I recall what @pragdave brought up on the mailing list and I believe it is not related to this issue. Dave was about the :env flag which is the environment a dependency was compiled and it is not related to how dependencies are retrieved/fetched.

[ConnorRigby]

Sorry i will explain a bit better. (FWIW @GregMefford made an Embedded Elixir post about this)

basically you have the root of a project with one or more mix projects inside of it. For example:

mkdir super_cool_project && cd super_cool_project
mix nerves.new scp_firmware
mix phx.new scp_ui
mix new scp_common

where scp_common could be a set of utilities (probably not an otp application)
both scp_ui and scp_firmware would rely on scp_common, and scp_firmware would rely on scp_ui. This seems like a tangled mess at first, but it actually works out really nicely for separating concerns. (scp_ui doesn't know anything about the platform in which it's running on)

The issue i want to avoid is having to set target: :rpi3 for each of the path: deps. (or really any deps for that matter) I suppose it is not a huge deal if well documented.

[josevalim]

@ConnorRigby assuming that scp_firmware depends on three targets and describes those dependencies with target: :xyz, then if scp_ui depends on scp_firmware, the target definitions from scp_firmware will be automatically seen from scp_ui. You won't need to tag those.

[GregMefford]

if scp_ui depends on scp_firmware, the target definitions from scp_firmware will be automatically seen from scp_ui. You won't need to tag those.

One correction here is that scp_ui would _not_ depend on scp_firmware. The idea is that you could have someone working on the Phoenix-based scp_ui application on their laptop, totally oblivious to Nerves, but the scp_firmware would pull in that application as a dependency, wrapping all the Nerves deployment magic around it.

---

To make this more concrete, let's look at how Nerves actually does this with our very-simple blinky example.

In the mix.exs, we have some deps that are included whether you're running on the host or the target, defined here:

  defp deps do
    [{:nerves, "~> 1.0", runtime: false}] ++ deps(@target)
  end

Then you have some that are only included when running on the host (i.e. not an embedded device):

  defp deps("host"), do: []

In this case, there aren't any of those, but it could be some dependency that is aware of the host/target distinction and has some kind of mock for when you're running in host mode.

Finally, we have target-specific deps that are used for non-host targets:

  defp deps(target) do
    [
      {:shoehorn, "~> 0.2"},
      {:nerves_runtime, "~> 0.4"},
      {:nerves_leds, "~> 0.8.0"},
      {:nerves_init_gadget, "~> 0.4.0"}
    ] ++ system(target)
  end

These would need to apply to any non-host target; not to a specific target. I'm not sure how you'd specify that in an obvious way, because it would need to be something like except_target: :host or something, which seems confusing to me. For example, nerves_init_gadget does all kinds of stuff to set up networking and firmware updates on your device. It doesn't have any useful (or safe) functions to perform on your host.

Also note that at the end, we append _actually target-specific_ deps based on the current target. In practice, I've never seen anything in one of these that's different based on the target, except for these Nerves System dependencies. Maybe @ConnorRigby has, because he's seen/done lots of weird things with Nerves 😁. These would make total sense to include in a flat deps list with an option like only: :rpi0.

  defp system("rpi"), do: [{:nerves_system_rpi, "~> 1.0", runtime: false}]
  defp system("rpi0"), do: [{:nerves_system_rpi0, "~> 1.0", runtime: false}]
  defp system("rpi2"), do: [{:nerves_system_rpi2, "~> 1.0", runtime: false}]
  defp system("rpi3"), do: [{:nerves_system_rpi3, "~> 1.0", runtime: false}]
  defp system("bbb"), do: [{:nerves_system_bbb, "~> 1.0", runtime: false}]
  defp system("ev3"), do: [{:nerves_system_ev3, "~> 1.0", runtime: false}]
  defp system("qemu_arm"), do: [{:nerves_system_qemu_arm, "~> 1.0", runtime: false}]
  defp system("x86_64"), do: [{:nerves_system_x86_64, "~> 1.0", runtime: false}]
  defp system(target), do: Mix.raise("Unknown MIX_TARGET: #{target}")

[josevalim]

One correction here is that scp_ui would not depend on scp_firmware. The idea is that you could have someone working on the Phoenix-based scp_ui application on their laptop, totally oblivious to Nerves, but the scp_firmware would pull in that application as a dependency, wrapping all the Nerves deployment magic around it.

That sounds good as well. The main focus on the previous example was to show that the target is inherited.

To make this more concrete, let's look at how Nerves actually does this with our very-simple blinky example.

This example would then be written to this:

  defp deps do
    [
      {:shoehorn, "~> 0.2"},
      {:nerves_runtime, "~> 0.4"},
      {:nerves_leds, "~> 0.8.0"},
      {:nerves_init_gadget, "~> 0.4.0"}

      {:nerves_system_rpi, "~> 1.0", target: :rpi, runtime: false},
      {:nerves_system_rpi0, "~> 1.0", target: :rpi0, runtime: false},
      {:nerves_system_rpi2, "~> 1.0", target: :rpi2, runtime: false},
      {:nerves_system_rpi3, "~> 1.0", target: :rpi3, runtime: false},
      {:nerves_system_bbb, "~> 1.0", target: :bbb, runtime: false},
      {:nerves_system_ev3, "~> 1.0", target: :ev3, runtime: false},
      {:nerves_system_qemu_arm, "~> 1.0", target: :qemu_arm, runtime: false},
      {:nerves_system_x86_64, "~> 1.0", target: :x86_64, runtime: false}
    ]
  end

Is there any reason why you chose the target to be a string rather than an atom?

[GregMefford]

This example would then be written to this:

 defp deps(target) do
 # ...
 end

Yes, but this still has the target being passed in as an indirection point for non-host deps. Isn't that the thing this proposal is trying to eliminate?

Is there any reason why you chose the target to be a string rather than an atom?

Because it ultimately comes from an OS environment variable. This might be solve-able though in the proposed solution where it's more of a "real thing." I'm not sure about that, though, because at some point, the developer needs to specify what target they're using, which could be something custom they've developed with a different name than any we support officially.

[josevalim]

Yes, but this still has the target being passed in as an indirection point for non-host deps. Isn't that the thing this proposal is trying to eliminate?

That was a mistake. Target there is not meant to be necessary. :) I fixed it.

which could be something custom they've developed with a different name than any we support officially.

I see, thanks. The conversion to atom should still be fine though.

[josevalim]

The other thing to point out is that, in the blinky example, mix deps.get would download all nerves_system_#{target}. Is that ok?

[fertapric]

Should the deps in the blinky project be like the example below to not download any dependency for the :host target?

defp deps do
  [
    {:nerves, "~> 1.0", runtime: false},

    {:shoehorn, "~> 0.2", target: [:rpi, :rpi0, :rpi2, :rpi3, :bbb, :ev3, :qemu_arm, :x86_64]},
    {:nerves_runtime, "~> 0.4", target: [:rpi, :rpi0, :rpi2, :rpi3, :bbb, :ev3, :qemu_arm, :x86_64]},
    {:nerves_leds, "~> 0.8.0", target: [:rpi, :rpi0, :rpi2, :rpi3, :bbb, :ev3, :qemu_arm, :x86_64]},
    {:nerves_init_gadget, "~> 0.4.0", target: [:rpi, :rpi0, :rpi2, :rpi3, :bbb, :ev3, :qemu_arm, :x86_64]},

    {:nerves_system_rpi, "~> 1.0", target: :rpi, runtime: false},
    {:nerves_system_rpi0, "~> 1.0", target: :rpi0, runtime: false},
    {:nerves_system_rpi2, "~> 1.0", target: :rpi2, runtime: false},
    {:nerves_system_rpi3, "~> 1.0", target: :rpi3, runtime: false},
    {:nerves_system_bbb, "~> 1.0", target: :bbb, runtime: false},
    {:nerves_system_ev3, "~> 1.0", target: :ev3, runtime: false},
    {:nerves_system_qemu_arm, "~> 1.0", target: :qemu_arm, runtime: false},
    {:nerves_system_x86_64, "~> 1.0", target: :x86_64, runtime: false}
  ]
end

I'm not really familiar with Nerves (never used it), but looking at the Farmbot codebase it seems there are dependencies that should only be loaded if the target is :host, and dependencies that are common to all the targets expect for :host. Taking this into consideration, I would write the deps function of the Farmbot project like:

defp deps do
  [
    {:nerves, "~> 1.1", runtime: false},
    {:elixir_make, "~> 0.4.2", runtime: false},
    {:gen_stage, "~> 0.14.0"},
    {:phoenix_html, "~> 2.11"},
    {:poison, "~> 3.1.0"},
    {:jason, "~> 1.1"},
    {:httpoison, "~> 1.2"},
    {:jsx, "~> 2.8"},
    {:timex, "~> 3.3"},
    {:fs, "~> 3.4"},
    {:nerves_uart, "~> 1.2"},
    {:nerves_leds, "~> 0.8"},
    {:cowboy, "~> 2.0"},
    {:plug, "~> 1.6"},
    {:ranch_proxy_protocol, "~> 2.0", override: true},
    {:cors_plug, "~> 1.5"},
    {:rsa, "~> 0.0.1"},
    {:joken, "~> 1.5"},
    {:ecto, "~> 2.2.2"},
    {:sqlite_ecto2, "~> 2.2.1"},
    {:uuid, "~> 1.1"},
    {:socket, "~> 0.3.13"},
    {:amqp, "~> 1.0"},
    {:recon, "~> 2.3.2"},
    {:ring_logger, "~> 0.4.1"},
    {:bbmustache, "~> 1.5"},
    {:apex, "~> 1.2"},
    {:net_logger, "~> 0.1"},

    # Host dependencies
    {:ex_doc, "~> 0.18", target: :host, only: :dev},
    {:excoveralls, "~> 0.9", target: :host, only: :test},
    {:dialyxir, "~> 1.0.0-rc.3", target: :host, only: :dev, runtime: false},
    {:credo, "~> 0.9.3", target: :host, only: [:dev, :test], runtime: false},
    {:inch_ex, "~> 0.5", target: :host, only: :dev},
    {:mock, "~> 0.3.1", target: :host, only: :test},
    {:faker, "~> 0.10", target: :host, only: :test},

    # Target dependencies

    ## Common target dependencies
    {:shoehorn, "~> 0.3", target: :rpi3, except: :test},
    {:nerves_runtime, "~> 0.6.1", target: :rpi3},
    {:nerves_firmware, "~> 0.4", target: :rpi3},
    {:nerves_init_gadget, "~> 0.4.0", target: :rpi3, only: :dev},
    {:nerves_network, "~> 0.3", target: :rpi3},
    {:nerves_wpa_supplicant, github: "nerves-project/nerves_wpa_supplicant", target: :rpi3, override: true},
    {:dhcp_server, "~> 0.4.0", target: :rpi3},
    {:elixir_ale, "~> 1.0", target: :rpi3},
    {:mdns, "~> 1.0", target: :rpi3},

    ## rpi3 target dependencies
    {:nerves_system_farmbot_rpi3, "1.2.1-farmbot.2", target: :rpi3, runtime: false}
  ]
end

If Farmbot would add another target, the last part would be:

# Target dependencies

## Common target dependencies
{:shoehorn, "~> 0.3", target: [:rpi3, :rpi2], except: :test},
{:nerves_runtime, "~> 0.6.1", target: [:rpi3, :rpi2]},
{:nerves_firmware, "~> 0.4", target: [:rpi3, :rpi2]},
{:nerves_init_gadget, "~> 0.4.0", target: [:rpi3, :rpi2], only: :dev},
{:nerves_network, "~> 0.3", target: [:rpi3, :rpi2]},
{:nerves_wpa_supplicant, github: "nerves-project/nerves_wpa_supplicant", target: [:rpi3, :rpi2], override: true},
{:dhcp_server, "~> 0.4.0", target: [:rpi3, :rpi2]},
{:elixir_ale, "~> 1.0", target: [:rpi3, :rpi2]},
{:mdns, "~> 1.0", target: [:rpi3, :rpi2]},

## rpi2 target dependencies
{:nerves_system_farmbot_rpi3, "1.2.1-farmbot.2", target: :rpi2, runtime: false}

## rpi3 target dependencies
{:nerves_system_farmbot_rpi3, "1.2.1-farmbot.2", target: :rpi3, runtime: false}

Would that be ok?

Minor thingy: mix deps.tree should also support the --target flag.

[ConnorRigby]

Wow i didn't realize how many dependencies Farmbot uses lol.

That looks good to me tho @fertapric

[mobileoverlord]

The other thing to point out is that, in the blinky example, mix deps.get would download all nerves_system_#{target}. Is that ok?

When we reorder the compilation during loadpaths we obtain a list of dependencies from Mix.Project.deps_paths. If we could filter this list by target it would be fine.

[GregMefford]

Yeah, we definitely don’t want to download all the system dependencies (and the toolchain sub-dependencies) because it could add up to GBs, probably. That may not be true though because the dependency package itself is small, it’s Nerves magic that downloads the artifact, as Justin mentioned.

I suppose it would be reasonable to put a @supported_targets [:rpi, :rpi3, :bbb] thing at the top, so you could easily pass it into the target option on the any-target-but-host deps.

[lackac]

Looking at those dependency lists from @fertapric examples I think this will soon be generated with something like:

@supported_targets [:rpi, :rpi3, :bbb]

defp deps do
  [
    {:nerves, "~> 1.1", runtime: false},
    # ... other deps that are target independent

    # Host dependencies
    {:ex_doc, "~> 0.18", target: :host, only: :dev},
    # ... other host deps

    # Common target dependencies
    {:shoehorn, "~> 0.3", target: @supported_targets, except: :test},
    # ... other common target deps
  ] ++
    # target system dependencies
    Enum.map(@supported_targets, &{:"nerves_system_farmbot_#{&1}, "1.2.1-farmbot.2", target: &1, runtime: false})
end

I wonder if it would make sense to support something like target: {:except, :host} for common target dependencies. I don't think it helps too much since the @supported_targets approach suggested by @GregMefford is more explicit and useful for generating the system deps as well.

[josevalim]

Yeah, we definitely don’t want to download all the system dependencies (and the toolchain sub-dependencies) because it could add up to GBs, probably. That may not be true though because the dependency package itself is small, it’s Nerves magic that downloads the artifact, as Justin mentioned.

Right. I think there are two important things to clarify here:

1. mix deps.get / mix deps.update works across all targets but not the other commands, such as mix deps.compile. This means you will download everything upfront but you will only compile what is necessary.

2. It is important to separate the blinky example from actual projects. Maybe blinky should come with all targets commented out and the developer should choose one of them. The question here should be: for actual projects, is it ok if you download the dependencies across all targets?

[ConnorRigby]

I'd say most projects only actually use one target plus host. Farmbot supports bbb, rpi3, and rpi0, but i don't put a ton of time supporting other than rpi3. I'd like the ability to have multiple targets, but i wouldn't like to download all deps across targets - it seems like a waste of my time as a developer to wait for system and toolchain assets, along with those _many_ deps listed up there ~4 times. (especially since only a small subset of them actually need to know about there being a different target).

[ConnorRigby]

_but_ i don't want to influence a feature being only one project. I'd like to hear what others think about this.

[GregMefford]

Maybe blinky should come with all targets commented out and the developer should choose one of them.

The Nerves project could change this (we would likely make _some_ change anyway if this proposal is approved), but currently, our mix nerves.new generator creates the file just like the blinky example shows, so the developer can easily delete the systems they don't want to support, but still have them all there to choose from at the beginning. I agree that it's unusual to support many targets; it will normally be one to three targets in addition to the host target.

I think it would be fine to download all of the hex packages for all the systems, if that's what it takes. It'll be a very small download to have them there. My concern is that Nerves somehow needs to know to _not_ download all the actual artifacts unless the project really depends on them. We recently moved that artifact download from mix compile time to mix deps.get time because we thought that made more sense than having a surprise when you try to do an offline compile. This might just be a discussion for the Nerves team to figure out independent of this proposal, though.

[fhunleth]

The commercial Nerves projects that I'm aware of are almost all one target plus host.

The Nerves example projects are unusual in the large number of targets they support. However, they're also the first projects that people try when coming to Nerves, so having a good first time experience is important. Downloading over 1 GB of artifacts to build the blinky example feels wrong. My understanding from @mobileoverlord's comment is that this won't happen, so I think we're ok.

[fhunleth]

Just to add, I don't think that manually commenting in deps will work for blinky. All of the examples get built for all targets as part of our CI process, so we need to programmatically enable each target.

[josevalim]

Regarding downloading dependencies, it seems that this is done as a separate step by the nerves archive. So you can probably control what happens here and what not. This means that the packages will be downloaded, but not the system stuff.

This proposal has two main gains:

• Fetching all dependencies at once
• Resolving dependencies together and in a single lock file

It seems you are not interested on the former, for understsandable reasons. Is the second enough reason to warrant this feature though?

[OvermindDL1]

Maybe an option can be specified to only download the dependency if mix deps.get is run with an environment that it is assigned to exist in.

[mobileoverlord]

Maybe an option can be specified to only download the dependency if mix deps.get is run with an environment that it is assigned to exist in.

Downloading the artifacts for only the dependencies that are loaded into the environment will solve the issue and prevent excessive amounts of disk space from being consumed all at once. If the user switches targets, they will have to be instructed that they may need to call mix nerves.deps.get to fetch just the artifacts for the new target.

This proposal has two main gains:

Fetching all dependencies at once
Resolving dependencies together and in a single lock file
It seems you are not interested on the former, for understsandable reasons. Is the second enough reason to warrant this feature though?

I believe that this is a great proposal with solvable problems. I can't think of a time where I had a project with multiple targets where I did not want them to lock their shared dependency versions together. One of the only possibilities I have thought of is if two different targets systems both used the same toolchains, but different versions. Which I could imagine is an edge case.

[josevalim]

Then maybe, opposite to environments, we always only download the target for the currently specified target?

To be more precise, if MIX_TARGET is host, we only download host. If MIX_TARGET is rpi3, we only download rpi3, etc.

[ConnorRigby]

RE: sharing a lockfile

I think this warrants the feature enough for me anyway. I have a script that manually does this for all supported targets and it is tedious.

[mobileoverlord]

@josevalim Its okay if mix fetches all deps source for all targets, but it should selectively compile / download artifacts based on the set target.

I believe that is the same for only with the exception that elixir doesn't have the notion of a download artifact phase.

[josevalim]

PR sent here: #8443 By default we are also fetching deps across all targets.

@ConnorRigby @fhunleth @mobileoverlord, can you please take it for a spin and give us any feedback? Thank you! Let's continue the discussion on the PR.

[RumataEstor]

Great feature, thanks for it.

I see it only allows single atom to be a target. Could it be extended to allow a list so it can be used to configure dependencies to enable some features? One example is Honeybadger package to enable Plug extensions. Another example is phoenix_ecto could be merged to phoenix to simplify maintenance.

[josevalim]

@RumataEstor that's not the use case for target though. ANd if people have to explicitly set some flags to get some HoneyBadger behaviour, it is most likely they will forget it (or even worse, forget to do it on prod and then have different behaviours). For instance, phoenix_ecto works out of the box as is, I wouldn't want to change it into something that requires conditional compilation.