Oceananigans.jl: HDF5 and plotting utilities are heavy dependencies

[glwagner]

Today @qingli411 and I attempted to install Oceananigans on his laptop, but ran into a problem when the HDF5 compilation (through julia) stalled for something like 15 minutes. We were only able to proceed by completely removing the dependency of Oceananigans on HDF5 and NetCDF (after which we got the code to run!)

I think we should consider moving heavy dependencies like HDF5 and NetCDF into separate packages (say, OceananigansOutput.jl) to make a simple barebones installation of Oceananigans quick, easy, and painless.

[ali-ramadhan]

Thanks for bringing this up, didn't realize it could be that bad. I agree it would be nice to have a barebones version of the package. Maybe we can approach this the same way we wanted to approach having "dynamic dependencies" for running examples that depend on Plots.jl (#195). That way, NetCDF.jl and HDF.jl only get installed and built if you decide to read/write NetCDF. I suppose this is the point of the Requires.jl package.

Out of curiousity, is @qingli411's laptop running Windows? Build times on Linux and Mac (e.g. on Travis CI and GitLab CI) tends to be quite reasonable, so I'm wondering if it's a Windows thing.

PS: I'm pretty sure this is also the reason why Appveyor is super slow #89.

[qingli411]

It's a Mac. @glwagner, I just tried to install it on my home laptop (also a Mac) by simply add Oceananigans and the build time of HDF5 is much shorter (~1 min). Not sure what was happening on the other laptop this afternoon...

[glwagner]

Thanks @qingli411.

One obvious way to introduce the concept of dynamic dependencies is to break functionality out into multiple small packages (OceananigansPlotting, OceananigansOutput, etc). @vchuravy argued that this is a good philosophy for packages; perhaps even more so for a complex project like this one: we may want to keep the core as simple as possible.

This would also make the tests run faster, and might make development easier...

[ali-ramadhan]

I'm strongly opposed to having multiple small packages.

Maybe sub-modules could be a solution. Loading Oceananigans.Plotting can "dynamically" install Plots.jl, and loading stuff from Oceananigans.Output will dynamically install packages depending on the output writer loaded. We just have to keep things modular and neat/tidy.

If the tests are taking a while, then we could split things up into quick unit tests and more comprehensive integration tests, or we need to pay for dedicate CI resources.

[glwagner]

I'm strongly opposed to having multiple small packages.

Why is that, given the benefits?

[ali-ramadhan]

I'm strongly opposed to having multiple small packages.

Why is that, given the benefits?

Mainly I think we can get all the same benefits by maintaining a single unified and tidy repository with sub-modules. I don't see Oceananigans as a complex project yet, we're barely at 2,500 lines of code (minus turbulence closure operators which aren't integrated yet).

I can see reasons for having a separate repository for examples and tutorials if there are enough of them, but I don't see e.g. the output writers as being logically separate from the main code. As a stand alone package, OceananigansOutput.jl doesn't do anything useful. Something like TurbulenceClosures.jl could though.

Keeping things in tidy submodules within a single repository increases code visibility, which I think is important. When I see a package or framework spread across multiply small packages I find it hard to get an idea about what it can do.

Another nice thing is we currently have "unified versioning" so when releases occur we know everything works together. We don't have to worry about running with Oceananigans v0.x with OceananigansPlotting v0.y and OceananigansOutput#master to fix some issue.

From a development perspective, issues and pull requests are now spread across multiple repositories and we can end up with pull requests that depend on other pull requests in other repositories, e.g. when adding in halo regions the main code got refactored but so did the output writers and plotting code would change as well. We can also end up with multiple test suites that need to pass, plus possibly multiple CI pipelines to manage. So right now it's easy to make atomic changes that touch upon multiple pieces of code (and we'll probably be in this stage where we need to refactor frequently for a while).

I also don't have to git clone a bunch of repositories just to work on a simple example.

But this is getting off-topic.

[christophernhill]

https://github.com/MikeInnes/Requires.jl ?

On Fri, Jun 14, 2019 at 10:47 AM Ali Ramadhan [e-mail]
wrote:

I'm strongly opposed to having multiple small packages.

Why is that, given the benefits?

Mainly I think we can get all the same benefits by maintaining a single
unified and tidy repository with sub-modules. I don't see Oceananigans as a
complex project yet, we're barely at 2,500 lines of code (minus turbulence
closure operators which aren't integrated yet).

I can see reasons for having a separate repository for examples and
tutorials if there are enough of them, but I don't see e.g. the output
writers as being logically separate from the main code. As a stand alone
package, OceananigansOutput.jl doesn't do anything useful. Something like
TurbulenceClosures.jl could though.

Keeping things in tidy submodules within a single repository increases
code visibility, which I think is important. When I see a package or
framework spread across multiply small packages I find it hard to get an
idea about what it can do.

Another nice thing is we currently have "unified versioning" so when
releases occur we know everything works together. We don't have to worry
about running with Oceananigans v0.x with OceananigansPlotting v0.y and
OceananigansOutput#master to fix some issue.

From a development perspective, issues and pull requests are now spread
across multiple repositories and we can end up with pull requests that
depend on other pull requests in other repositories, e.g. when adding in
halo regions the main code got refactored but so did the output writers and
plotting code would change as well. We can also end up with multiple test
suites that need to pass, plus possibly multiple CI pipelines to manage. So
right now it's easy to make atomic changes that touch upon multiple pieces
of code (and we'll probably be in this stage where we need to refactor
frequently for a while).

I also don't have to git clone a bunch of repositories just to work on a
simple example.

But this is getting off-topic.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/climate-machine/Oceananigans.jl/issues/284?email_source=notifications&email_token=AA27DYA55SSJKHXQBFUSJDLP2OVODA5CNFSM4HXEYRJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXXAHFI#issuecomment-502137749,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA27DYEHNERHN4MHAGJQZH3P2OVODANCNFSM4HXEYRJQ
.

[glwagner]

Thanks @ali-ramadhan, that's helpful.

It does seem that Requires.jl can solve this issue and satisfies @ali-ramadhan's desire for verticality as mentioned in his comment. We could perhaps isolate the output functionality into a submodule (eg, Oceananigans.Output) that is loaded only when HDF5 (for example) is present. Ditto for Oceananigans.Plotting.

I've changed the name of the issue to be more descriptive of the general issue we are discussing (heavy dependencies).

[glwagner]

This is resolved, eh?

[ali-ramadhan]

I think so. HDF5 is no longer a dependency and we can include plotting in the examples using the import pattern you use in deepening_mixed_layer.jl.