Vm: Build with TerraForm and release with Travis

[enoch85]

UPDATE:
Someone told me about TerraForm at a conference which seems to be the better choice.

It should be possible to automatically release builds to Github when we want to release a new version. It shouldn't build on every commit, rather when we decide to create a new release.

I have read documentation around Travis / Vagrant / Docker, and it should be possible to create a VM inside Travis and push it to Github.

Things to think about

• In the VMs install script the user are given options that should be answered automatically. We don't want to create a new user if not needed, and we don't want to change install mirrors, we don't want to change keyboard layout either (default is US).
• A user with sudo permissions (ncadmin) is needed to run the install script properly. Running the script as a separate root user will work, but is not optimal.
• Contacts, Calendar and Webmin are shipped by default in the release version, here we need to answer "OK" in the install script as they are prompted in a whiptail (while loop in the end of the install script).
• The VM must meet these requirements: https://github.com/nextcloud/vm/blob/master/README.md#build-your-own-vm-or-install-on-a-vps

We are just about to finish the rewrite, but this could be something to add in the next project we do. Or if someone is up for the task, please make a PR and push it to a new branch so that we can discuss and improve.

Thanks!

[Ark74]

So can we start testing the new release?

[enoch85]

@Ark74 The new release is in master already. So feel free to test. :)

[enoch85]

Maybe @pierreozoux can help out here?

[pierreozoux]

Hello @enoch85 nice to meet you :)
I didn't know about this project, so, it is nice to hear! (And funny, that we are kind of concurrent but collaborating! I love the free software style!

I don't exactly know what this project is about, but I can see right away, that it duplicates effort, or maybe it is not the right way to say.

There are already tons of "packages" for Nextcloud:

• yunohost
• docker
• deb
• this vm

I'm all for diversity. And this proves resiliency of a community.

But as you put docker in the title of this issue, I'm asking now :) Do you want to reuse another packaging method? If yes, I'd advise docker, obviously :)
If not, then I'd advise deb.
What does your bash gives that the deb doesn't? Could you PR the deb for it to provide it?

Then, to release a VM CI/CD style, I'd advise to use packer, this is the tool to go. For the configuration management, I'd advise ansible. You can go bash obviously, but, it is not really perfect.

You could also offer VMs, with http://libre.sh installed inside, and reuse our Nextcloud package then we would maintain this together! This would give you another list of possibilities of applications.

Sorry, for my broad answer, that is maybe asking more questions than answer any, but I need a bit more background, a bit more from the problem space before telling you what is the best solution IMHO.

[enoch85]

@pierreozoux Thank you for your reply, and:

#!/bin/sh
if [ $("$pierreozoux" grep 'hello' https://github.com/nextcloud/vm/issues/193) == "hello" ]; then
echo "hello to you as well :)"
fi

I think you missed the point tbh. :) This is not a "package". The Nextcloud/ownCloud VM has been around since 2014 and I scripted it in 2015. The scripts in this repo creates a production environment with all the fancy tweaks needed for a (IMHO) perfect Nextcloud server running. You should try it yourself to see what I'm talking about. It takes around 10 minutes to run on a decent server. You can use these scripts on a bare metal server, VPS, or to build your own VM and customize it as you'd like depending on if you want to have SSL, Collabora, Nextant (and some other apps), Webmin etc etc..

The point of this issue is that I want to be able to do automatic releases on Github every time there is a new version. So, if that is done with Vagrant, Docker idk yet, but I thought you could shed some light over it maybe?

So in other words, when a new release is tagged - automatically start building it with Travis and release the RAR or ZIP together with the release here on Github. Does that make more sense?

[pierreozoux]

Ok, sorry, I just read quickly, but now I read a bit more. It is still not totally clear.

What do you call a release?
If I read instructions here, it looks like, I just nee to boot an ubuntu VM anywhere, and run this script.
So, from these instructions, it looks like, the artifact is just this script.

But then, I check the release page, and I see a big rar file. At this point, I guess, that this contains some sort of VM image (but still not sure what kind, vmware, hyperV or vbox?).
And then I check https://www.techandme.se/nextcloud-vm/ and I see options with OVA, VMDK, and 500GB and 1TB and hyperV.

I'm an IT, and I have to admit, I'm really confused. (I tell it, in an attempt to help, not to criticize, I know myself, that my projects are far from being perfect, and I do appreciate when people explain me that it is a total mess :) )

Can you clarify why you need OVA/VMDK?
Can you clarify why 5000GB and 1TB? (Isn't it just a fs resize at the end of the day?)

Then, the question, I believe, is how to:

• automatically build these VMs
• upload the artifact(s)
• create a GitHub release with links to those artifacts.

Looks like travis + packer should do the job :)

https://github.com/travis-ci/travis-packer-build
https://docs.travis-ci.com/user/deployment/releases/

And for the script, you could have something like:

curl https://vm.next.cloud/ | sudo bash`

This would be nice :) you would just need to upload the latest install script to this location during your build.

hope it helps!

[enoch85]

I'm an IT, and I have to admit, I'm really confused. (I tell it, in an attempt to help, not to criticize, I know myself, that my projects are far from being perfect, and I do appreciate when people explain me that it is a total mess :) )
Can you clarify why you need OVA/VMDK?

Haha :)

The reason for the different files are that there are different versions that you can choose. If you want an OVA you can download that, or if you are running Hyper-V you can download the Hyper-V image, or maybe you want a 1 TB thick (all the VMs are thick) image instead, well then you can download that. So, in other words, there are options so that the user can get whatever he/she wants. All to make it as convenient as possible.

All the files are packed in RAR and contains pre-built images. Those are the releases.

So options the user have is to:

• Built the VM by themselves with the scripts
• Download a pre-configured VM that presents the user with the final setup script when mounted:
  a) Hyper-V
  b) 500 GB
  c) 1 TB
  d) 20 GB (this is the free one that you can see on Github and Tech and Me as the first choice).

As I said, you should use the scripts yourself to get a better picture of what I'm talking about here. I think that would make more sense. :)

When I release I build everything from a "base VM" that I pre-created which is basically a VM with Ubuntu installed with specific settings. This could be done with Vagrant instead, I know.

I got inspired when you mentioned Packer and found this: https://github.com/geerlingguy/packer-ubuntu-1604 which seems kind of what I'm looking for, without the Github part.

The aim of this is to:

• Remove the need of "base VMs" and build the VMs directly with a template which sets the user, language, size, what to install (OpenSSH) and so on.
• Run the script on the newly created VM
• When the script is run then poweroff the VM and package it
• Release the packaged version on Github and also save one compressed version for release on Tech and Me.

So basically, run something like: release_500GB_version.sh and then just wait for it to finish.

I hope that explains it.

[pierreozoux]

Yes, you should be good to go with packer and travis.
And deploy the result with:
https://docs.travis-ci.com/user/deployment/releases/

The only issue I see is how to provide a VirtualBox, VMware or HyperV inside travis to let packer build the artifact.

[enoch85]

The only issue I see is how to provide a VirtualBox, VMware or HyperV inside travis to let packer build the artifact.

Seems to me that Packer supports VMware, which I use. Will investigate this weekend. Thank you for your input!

[enoch85]

Ok, spent some time on this and we have a working Packer now: https://github.com/techandme/nextcloud-packer

Alot to do before it feels stable, but at least a start. cc @pierreozoux cc @nextcloud/vm

[enoch85]

TODO:

• Run the initiial install as ncadmin, currently runs as root which makes it fail in some aspects.

[enoch85]

I talked to some guys at Foss North and they recommended Terraform. Will definitely look into it.

Is that anything that you are familiar with @pierreozoux?

[enoch85]

Let's close this in favour of the other issue opened right now