Dataverse: Docker for production

[omaralsoudanii]

Hello ,
Is there official production ready docker container for data verse ? and if available where can i find the docs?
Thank you

[pdurbin]

@omaralsoudanii thanks for opening this issue! It's been nice talking to you over at https://github.com/nds-org/ndslabs-dataverse/issues/8 but this main issue tracker for Dataverse is a better place to discuss the state of Docker support. I can try to explain.

The short answer is that unfortunately, there is no production-ready Docker image for Dataverse as of this writing. I'm still learning Docker and its ecosystem so I'd like to know from you and others who are interested in Docker support how you expect it to work. Do you expect all of Dataverse to be running in a single container? Do you expect some of the components such as PostgreSQL and Solr to be running in different containers? Do you expect to use docker-compose? Do you expect to run Docker images in an orchestration platform such as Kubernetes?

If you look at the source tree, this is what you will find today:

• conf/docker: Docker images intended to be run in OpenShift, which is a distribution of Kubernetes. See #4040 and http://guides.dataverse.org/en/4.8.6/developers/containers.html#future-production-use-on-minishift-openshift-kubernetes
• conf/docker-aio: A single "all in one" Docker image that some of use to run the API test suite. See http://guides.dataverse.org/en/4.8.6/developers/containers.html#all-in-one-docker-images-for-testing

I hope this helps. If you and others interested in Docker support can answer the questions above or other questions I don't even know to ask, it would be very helpful! Thanks!

[omaralsoudanii]

@pdurbin Thanks for this ,
It would be nice to have 3 containers :
1- Data verse repo container
2- Postgres container
3- Solr container

And a docker-composer.yml file that will build 3 images / 3 containers for those automatically ,
Also it would be good to include an entry point to manage update scripts to date verse and Postgres sql changes , See https://docs.docker.com/engine/reference/builder/#entrypoint .

Because the problem we're having now is in order to update and maintain data verse we need to adjust DockerFile manually (change data verse instance version) and do the update process manually (run each Postgress schema changes manually ) , So some automated script that takes cares of those would be nice .

Another suggestion would be that the data verse team adjusts the DockerFile configuration each time an update to data verse is released so that we can simply pull docker changes from data verse repo and do :

docker-compose down

docker-compose up --build

For our current version (4.7) though we're trying to update to the latest version (4.8.6) in order to use the license and restrict files native API but i need a fresh data verse Postgres database , Is there a script or sql file for that?

Thank you.

[pameyer]

@omaralsoudanii Any thoughts on an additional container for apache? My understanding is that it's preferable to not expose glassfish/dataverse directly to the outside world, so it might make sense to deal with it at this level of abstraction (but I'm not currently planning on using dockerized dataverse in production, so input from folks that are would be interesting).

[vsoch]

hey everyone! I think I can definitely help with this, and I'd like to suggest starting with a simple docker-compose setup, and then using kompose to transition to kubernetes when the time is right! I'll start poking around this weekend (I'm not familiar with the code base) and then we can loop back next week (when the RedHat intern joins up?) How would you like to have our mentoring / discussion? Github? Slack? Other?

[pdurbin]

@vsoch thanks for the offer to help with this effort! I haven't even met the intern yet but @danmcp mentioned internship goals the other day at http://irclog.iq.harvard.edu/dataverse/2018-05-22#i_67516 . I believe @djbrooke said this individual might be starting as soon as next week and I said I'd be happy to mentor him or her on all things related to Dataverse. I'm happy to help you and others dive in as well, of course!

Perhaps a timeline of events (from my perspective anyway) would help ground us:

• 2016-03: @craig-willis from @nds-org dockerizes Dataverse and gets it running on Kubernetes for NDS Labs Workbench , which is described from the Dataverse perspective at http://guides.dataverse.org/en/4.8.6/installation/prep.html#nds-labs-workbench-for-testing-only . The code can be found at https://github.com/nds-org/ndslabs-dataverse
• 2017-06 @portante from Red Hat attends the third annual Dataverse Community Meeting and explains that Red Hat might be interested in running Dataverse if it runs on OpenShift, a distribution of Kubernetes. #4040 is opened to track this.
• 2017-09 @pdurbin uses the Docker/Kubernetes work by @craig-willis as a starting point to create pull request #4168 with help from @danmcp to get Dataverse runnig on OpenShift but not for production use. A list of shortcomings appears at http://guides.dataverse.org/en/4.8.6/developers/containers.html#known-issues-with-dataverse-images-on-docker-hub
• 2017-12 @pameyer adds an "all in one" Docker config in pull request #4370 that can be easily spun up for the purpose of integration testing.
• 2018-02 @joelmarkanderson says, "We are starting to actively develop on the Terraform -> Kubernetes path, and I've begun reconfiguring Docker containers to reflect current dataverse versions": https://github.com/IQSS/dataverse-aws/issues/11#issuecomment-368989741
• 2018-03 @aculich expresses interest in running Dataverse on Docker at http://irclog.iq.harvard.edu/dataverse/2018-03-02#i_64000 and follows up with #4714.
• 2018-03 @4tikhonov provides a link to the docker-compose.yml file he's been working on: https://github.com/Dans-labs/dataverse/blob/4.8.5-compose/conf/docker-eu/docker-compose.yml
• 2018-04 @danmcp @pdurbin and @DirectXMan12 wrap up a semester of mentoring BU students Patrick Dillon, Michael Clifford, Ashwin Pillai, and Ryan Morano to start the work of making it possible to scale up the number of Glassfish (#4626) and PostgreSQL (#4599) servers in use within OpenShift. A summary and link to the final class video can be found at https://github.com/IQSS/dataverse/issues/4040#issuecomment-386061972
• 2018-05 @omaralsoudanii creates this issue, indicating interest in running Dataverse in production on Docker. @xibriz sounds interested, based on a comment at https://github.com/IQSS/dataverse/issues/4040#issuecomment-387355618 .

I'm probably forgetting people and events but it's a start. In a future comment I'd like to write more about how I can imagine the work for this issue being broken up into smaller chunks.

[pameyer]

@vsoch The idea of starting with docker-compose sounds like a reasonable place to start. There's an example of docker-compose in this repository (although intended for a different purpose) in conf/docker-dcm.

One thing that may be a factor for production-izing Docker/Dataverse (and that @pdurbin and I have talked about) is how to handle the intersection of the docker entry point and the dataverse installer. It's very likely that the approach I've been taking as been sub-optimal for production usage (and so wouldn't make a good prototype for production).

[4tikhonov]

Probably our docker-compose from DataverseEU project can be interesting to try as well: https://github.com/Dans-labs/dataverse-docker/blob/master/docker-compose.yml

[vsoch]

@pameyer agreed I would not have the command coincide with the entire install, so the container can be restarted without prompting it done again!

@4tikhonov the work you've done for DataverseEU looks great! It's 99% there! Is it just a matter of combining the two repos into a consolidated thing with a good set of docs? With your blessings I can give a first shot at this.

[4tikhonov]

Thanks! We're running it with Kubernetes on Google Cloud already http://dataverse-dev.cessda.eu
Some documentation for Docker module is available here: https://github.com/Dans-labs/dataverse-docker

[vsoch]

So what needs to be done then?

[4tikhonov]

We're still working on the proxy implementation for multilingual support to get different languages on selected path (/fr for French, for example). First we tried with apache but Google Cloud already has nginx as integrated service running inside of Kubernetes, it seems to be better solution.

[pameyer]

@4tikhonov How did you end up handling the entrypoint / installer issue (run installer in entrypoint, something else)?

[4tikhonov]

@pameyer, I remember that you advised me to do health check before running installer second time:
size=$(curl -sI http://localhost:8080/api/info/version | grep Content-Length|awk '{print $2}')

It's in entrypoint indeed, if size > 0 then Dataverse is already installed and further installation steps can be skipped.

[pdurbin]

@4tikhonov I know it's quittin' time for you you but if you're up for a wild Friday night in Dataverse IRC we're chatting now: http://irclog.iq.harvard.edu/dataverse/2018-05-25#i_67861 😄

[vsoch]

What time zone are you guys in? We can arrange a larger group chat maybe
after the holiday weekend after I've gotten a chance to look at the code?

On Fri, May 25, 2018 at 11:37 AM, Philip Durbin [e-mail]
wrote:

@4tikhonov https://github.com/4tikhonov I know it's quittin' time for
you you but if you're up for a wild Friday night in Dataverse IRC we're
chatting now: http://irclog.iq.harvard.edu/dataverse/2018-05-25#i_67861 😄

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/IQSS/dataverse/issues/4665#issuecomment-392145252,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAxs8sWvP5KxLIdDtibHizbjhLKXS3XVks5t2E93gaJpZM4T8zpS
.

--
Vanessa Villamia Sochat
Stanford University

[vsoch]

hey everyone I just tested https://github.com/Dans-labs/dataverse-docker and it is a solid start, could someone again tell me why this isn't what you want?

[pdurbin]

@vsoch hi! Thanks for taking a look. Personally, I've been more focused on the OpenShift use case and documented what we've been up to at http://guides.dataverse.org/en/4.8.6/developers/containers.html

I'm sorry to say that I haven't looked closely at the @Dans-labs work by @4tikhonov but I believe @pameyer has so he might be in a better position than I am to comment.

Ideally we'll have a solution that works on:

• docker-compose
• Kubernetes
• OpenShift
• something that isn't even on my radar 😄

[vsoch]

I think it would be worth having someone look at his good solution before asking someone else to start over :) My comments would be small tweaks to the containers (e.g., adding env DEBIAN_FRONTEND noninteractive and maybe putting the deps inside the containers to begin with, but I don't think it's a good use of time to start from scratch when this project is going that (I think) is in the right direction.

[pdurbin]

@vsoch that makes total sense. I did take a quick look just now.

@omaralsoudanii since you opened this issue, can you please try https://github.com/Dans-labs/dataverse-docker and provide feedback on if it's going to work for you?

@4tikhonov are they any specific bugs or issues with that repo before anyone should use it in production?

@aculich do you want to give that repo a try as well?

[vsoch]

@4tikhonov I think you've done a great start and I can offer to help, but only if you need/want it.

[4tikhonov]

@vsoch, thanks, we're very indeed interested to get feedback. Our goal is to get it finally integrated with master branch at some point.

[pdurbin]

@4tikhonov is there anything holding you back from making a pull request?

[omaralsoudanii]

@pdurbin @4tikhonov
this seems to be working great so far on my local machine ,

One question i have is what's the steps required if say a new dataverse version is up in the repo , Will this docker image auto update the version when using docker-compose build ( without deleting the existing data) or any further steps required ?

I will provide more feedback after testing it with the native API .

Thank you

[pdurbin]

@4tikhonov any thoughts on the question by @omaralsoudanii above?

There's a section of the Installation Guide I wrote called "Choose Your Own Installation Adventure" which is inspired by a series of books I enjoyed as a child. Under "Advanced Installation" I mention community-supported adventures such as https://github.com/IQSS/dataverse-ansible by @donsizemore and it would make sense to me to have the Docker/Kubernetes installation adventure by @4tikhonov fall into the community-supported category as well. That is, @4tikhonov and others in the community could publish Docker images as new releases of Dataverse come out. I'd be happy to give @4tikhonov and others "push" access to a new repo under the @IQSS GitHub organization if that makes sense. Please just let me know!

[4tikhonov]

@omaralsoudanii, it should work for other Dataverse versions as well as latest version should be downloaded to Docker container from GitHub.

@pdurbin, I think it's great idea and I would like to participate in this Docker/Kubernetes installation adventure.

[pdurbin]

@4tikhonov great! What would you like the repo under @IQSS to be called?

• dataverse-docker?
• dataverse-kubernetes?
• dataverse-containers?

I'll set up a team and give you push access. And whoever else you want.

[4tikhonov]

@pdurbin dataverse-docker is fine, thanks, Phil!

[pdurbin]

@4tikhonov sure! I just created https://github.com/IQSS/dataverse-docker and added you to both of these teams:

• https://github.com/orgs/IQSS/teams/dataverse-docker-admin
• https://github.com/orgs/IQSS/teams/dataverse-docker-write

The repo is empty right now but you should be able to push code there and add more collaborators. Please let me know if you have any trouble. Thanks!

[4tikhonov]

@pdurbin, great, I've placed all stuff there and updated instruction on switching languages in the running Docker container. Tested with versions 4.8.5 and 4.8.6, seems to work.

We'll add Kubernetes specifications later this week.

[pdurbin]

@4tikhonov great! I assigned this issue to you so that when you're ready you can make a pull request against doc/sphinx-guides/source/installation/prep.rst to mention the new dataverse-docker repo as a community-supported effort. Thanks!

[omaralsoudanii]

@4tikhonov FYI on fresh ubuntu 16.04 installation , The current script on

dataverse-docker/dataversedock/step1.sh

line 16 :

unzip glassfish-4.1.zip

requires to install unzip by :

sudo apt-get install unzip

else the initial.bash will fail silently (doesn't add glassfish dependency and docker compose will fail on step 7 )

[pdurbin]

@omaralsoudanii good catch. I would suggest opening an issue at https://github.com/IQSS/dataverse-docker/issues and mentioning which commit you are on.

[4tikhonov]

@omaralsoudanii @pdurbin Yes, great idea! Please open issue there and we'll fix it.

[omaralsoudanii]

@4tikhonov @pdurbin done .

Please note there is a problem with publishing i made an issue there as well.
https://github.com/IQSS/dataverse-docker/issues/2

[4tikhonov]

@omaralsoudanii Yes, the default Docker image is coming without credentials for DOI service. Or do you mean that you want to test it locally with some test DOI prefix?

[omaralsoudanii]

@4tikhonov trying to test it with the default DOI prefix ( doi:10.5072) that comes with the dv installation , Currently tried it both locally and online

[pdurbin]

@omaralsoudanii any news? It looks like @4tikhonov fixed both of the issues:

• https://github.com/IQSS/dataverse-docker/issues/1
• https://github.com/IQSS/dataverse-docker/issues/2

[pdurbin]

@omaralsoudanii did you get a chance to test this?

[omaralsoudanii]

@pdurbin @4tikhonov
we published the docker repo on a production environment for our app
(http://data.mel.cgiar.org) with a handle server for publishing , So far using the native API it looks great after testing , Thank you.

[pdurbin]

I'm just copying a comment from @xibriz from https://github.com/IQSS/dataverse-docker/issues/3#issuecomment-415321082 who said, "I have just made a successful deployment on our Kubernetes cluster using GitLab CI, so thanks for the great work :)"

@omaralsoudanii thanks for testing as well!

I'm wondering what the "definition of done" is for this issue. Currently, it's sitting in the new "Community Dev" column on our kanban board:

That's fine. It can continue to sit there. I'm just wondering what it means to be "done" and for this issue to be closed. Thoughts on this are very welcome!

[xibriz]

Just to clarify, I had to make a few changes to be able to use GitLab CI as a deployment method.

But there are some things I personally think needs to be done. Like using Secrets that all the containers in one deployment can use instead of passwords in files. I personally also need to take a closer look at Persistent Volumes and how to best upgrade the database on new releases.

[pdurbin]

@xibriz make sense. Thanks. Over in pull request #4827 for #4763 a similar effort was undertaken to get secrets out of the OpenShift config. Please keep us posted on your experiments with peristent volumes and and upgrades. Speaking of upgrades, you (and all who are reading this) are welcome to leave a comment on #4980.

[pdurbin]

@poikilotherm over at http://irclog.iq.harvard.edu/dataverse/2018-10-18#i_76326 we decided to add you as an assignee for this issue so you can coordinate with @4tikhonov . Thanks!

[4tikhonov]

Hi @poikilotherm and @pdurbin, we already have complete infrastructure in Kubernetes:
https://github.com/IQSS/dataverse-docker/tree/master/kubernetes

It's actually running on Google Cloud for DataverseEU. You can get all Docker images from Docker Hub:
https://hub.docker.com/r/vtycloud/dataverse/
https://hub.docker.com/r/vtycloud/postgres/
https://hub.docker.com/r/vtycloud/solr7

[pdurbin]

@4tikhonov that's awesome. Over on the "Advanced Installation" at http://guides.dataverse.org/en/4.9.4/installation/advanced.html we say, "Advanced installations are not officially supported but here we are at least documenting some tips and tricks that you might find helpful." Are you interested in making a pull request to add your Kubernetes and Docker information there? The file to edit is doc/sphinx-guides/source/installation/advanced.rst.

[4tikhonov]

hi @pdurbin, that's great! The only problem for me is migration from 4.9.2 to 4.9.4 on Cloud as it didn't went smoothly. If people will migrate from previous versions without applying patches we'll get into troubles with distribution of 4.9.4.

[pdurbin]

Yep, over at #5204 I asked you for some output from asadmin deploy.

If you need any help making a pull request for the Advanced Installation page, please let us know. You might find http://guides.dataverse.org/en/4.9.4/developers/documentation.html helpful but it's a bit out of date.

Thanks!

[pdurbin]

I just wanted to note that @craig-willis (the first to dockerize Dataverse as noted in the timeline above) said over at https://github.com/whole-tale/whole-tale/issues/49 that he's reading through this issue and we should help him learn about latest activity with regard to Dataverse and Docker.

[craig-willis]

Thank you, @pdurbin. I'm always amazed by your ability to keep things connected! Forgive me for the lengthy comment, and hopefully this is the right place to put it.

The use case for whole-tale/whole-tale#49 is that I want to quickly spin up an instance of Dataverse to test API integration and external tools support. As you noted in https://github.com/whole-tale/whole-tale/issues/49#issuecomment-438299326, there are other ways to work with development instances of Dataverse: https://demo.dataverse.org, https://dev1.dataverse.org and I could also use docker-aio, but I'm also motivated to freshen up Dataverse support in Labs Workbench so that others can easily do this too. I am interested in contributing in this space and want to make sure I'm putting my effort in the right place.

There seem to be two different but related needs here:

1. A set of well-maintained images ideally with automatic builds associated with each release that can be used to spin up dev and kick-the-tires instances on OpenShift, Labs Workbench, Kubernetes, Swarm, Docker compose etc. These don't need to be production configurations (i.e., scalable/sharded, etc), but should work out of the box with reasonable default settings, persistent storage, and at minimum be restartable.
2. A set of production images and configurations (scalable, reliable, etc) for use on common orchestration platforms (e.g., Kubernetes, Swarm, etc) that addresses the known limitations and other issues proposed in https://github.com/IQSS/dataverse/issues/5292. (Perhaps even installable via a Helm chart?)

Here is my current understanding of Docker support in the Dataverse community:

• My very stale work on Dataverse support in the NDS Labs Workbench was intended for kicking the tires, certainly not production use, and needs to be updated/replaced, which I am working to do. Per https://github.com/nds-org/ndslabs-dataverse/issues/8#issuecomment-422361354, ideally this would use official or community-contributed images and be updated with each release.
• IQSS/dataverse/conf/docker contains the build process for your work to support kicking the tires on OpenShift with noted limitations. The images available at https://hub.docker.com/r/iqss are for this purpose only, not production use. I assume the Docker build process is kept up-to-date with each release, although images are not auto-built and do not appear to be connected with releases. One could use the build process and release tags in Github to build different release images.
• IQSS/dataverse/conf/docker-aio contains the build process for an all-in-one image for development and testing. There appears to be no official build of this image under the iqss organization in Dockerhub. One could use the build process and release tags to build different release images.
• IQSS/dataverse-docker (formerly Dans-labs/dataverse-docker?) contains build process for community-maintained set of images, primarily used in production on GCE by DataverseEU. This is still under active development and per https://github.com/IQSS/dataverse-docker/issues/8 the images are available from https://hub.docker.com/r/vtycloud, with manually pushed images for 4.9.2.x and 4.9.4. You suggested https://github.com/nds-org/ndslabs-dataverse/issues/8#issuecomment-419245749 that I could use these images in Workbench.
• @poikilotherm's recent work on https://github.com/IQSS/dataverse/issues/5292, which if I'm understanding is intended to further refine the build and install process to be more micro-service friendly.

Based on your comment https://github.com/whole-tale/whole-tale/issues/49#issuecomment-439386207 I gather that in core Dataverse development, Docker is primarily used for testing -- this is not part of your production infrastructure. The DataverseEU group is deploying on GCE via Kubernetes and therefore leading the more production-oriented work today.

I am unclear on the future of IQSS/dataverse-docker and IQSS/dataverse/conf/docker, since they are both quite similar. One is maintained by community members intended for real/production deployments, the other is maintained as part of the core project repository, but only intended for tire kicking and testing.

My tentative conclusion is that I should focus on IQSS/dataverse-docker and contribute any changes there, with the hope that these two eventually converge.

Did I miss anything?

[pdurbin]

@craig-willis you have it 95% right. Excellent summary. Thank you!

Unfortunately, you are assuming a level of support and maintenance for OpenShift-based images in "conf/docker" that does not exist. Some refactoring of our installation process to better support OpenShift occurred in pull request #4805 but parts of our traditional installation process broke so we followed up with pull request #5058 to make sure traditional installation works. This is to say that I have no idea if the OpenShift works on the develop branch right now. It's not regularly tested. I'm not sure when the OpenShift effort will resume.

I do think that you should focus on https://github.com/IQSS/dataverse-docker because that's where the momentum is. That said, I'm not sure if anyone is using it production or not. If they're in production, I don't believe they're on our map at https://dataverse.org

[craig-willis]

@pdurbin -- Well, 5% wrong isn't too bad, but at least it was an optimistic assumption!

Some refactoring of our installation process to better support OpenShift occurred in pull request #4805 but parts of our traditional installation process broke so we followed up with pull request #5058 to make sure traditional installation works.

Good to know. In my early work, I terribly hacked the installation process, which was functional for a proof-of-concept but entirely unmaintainable. Even if Docker isn't supported as part of the core project, it seems that the community would need at a minimum a Docker-compatible installation process. I expect you'd be receptive to PRs as long as they don't break the main installer?

[pdurbin]

@craig-willis yep, pull requests against the installer are always welcome. Our official statement on it is at http://guides.dataverse.org/en/4.9.4/installation/installation-main.html#running-the-dataverse-installer and says:

"The script is to a large degree a derivative of the old installer from DVN 3.x. It is written in Perl. If someone in the community is eager to rewrite it, perhaps in a different language, please get in touch. :)"

A couple other factoids about the installer is that the Perl script is not used by https://github.com/IQSS/dataverse-ansible and lately @poikilotherm has been advocating for using the "MicroProfile Config API" in #5293 but this would require switching from Glassfish to Payara #4172. I don't know. it's complicated. Most of the installer hackers hang out in #dataverse on freenode if you ever want to pick some brains.

[poikilotherm]

Hi @craig-willis,

@poikilotherm's recent work on #5292, which if I'm understanding is intended to further refine the build and install process to be more micro-service friendly.

Actually #5292 is not just about building and installation, but also a lot about testing. Personally, I try to make small steps towards the ability to introduce things like Arquillian, etc. in the build chain on the hand and on the other hand let people run a simple mvn docker:run to spin up a test instance in their current branch. Here "testing" means not just executing JUnit or similar tests to me, but also things like UI building or demos. (And of course I want to avoid breaking existing ways of doing stuff, because that makes people upset.)

What I pretty much dislike about the current approaches for Docker/OpenShift/... images: the big and clunky installer script is used in them. IMHO that is not what a Docker setup should look like; for me it feels more like packaging a lightweight VM to call docker run instead of the install script. Again, this is just my understanding how modern webservices should work and it might differ from others. And I am very aware of changing this will need a lot more work than writing some Dockerfiles, which is why I came up with #5292.

I can't make suggestions where an investement of time and ressources will get the most in return for your own work and visions. #5292 needs people who share my vision for the long run and this needs to be shared, explained and discussed with people like @pdurbin, @pameyer, @matthew-a-dunlap, @scolapasta and anyone else who would like to join. (After all I am not part of @IQSS or Harvard and they need to get stuff done. :wink:)

[poikilotherm]

Hey all,
I encourage anyone to check out my simple and general purpose Kubernetes stuff at https://github.com/iqss/dataverse-kubernetes.

This is still WIP, as I want more stuff inside, but the groundwork is done. Once things get moving in #5292 and #4172 the images will change. Any comments appreciated, feel free to open an issue in the project with feedback, wishes, bugs, ...

[vsoch]

@poikilotherm I have a practical question - do you spin up a cluster (and then have to pay for it) on a cloud provider to test? I'm trying to come up with reasonable ways to (cost effectively) develop with k8s but it seems this is the only way to go... minikube it's really the same thing.

[pdurbin]

@vsoch OpenShift is a flavor of k8s that offers a free tier. It turns out Dataverse is too fat memory wise to squeeze into it but it might be worth a look. Back when I was working on #4040 the free tier allowed for 1 GB of memory. I'm talking about "OpenShift Online", the service hosted by Red Hat: https://www.openshift.com/products . My dream was that people interested in installing Dataverse would be able to just click a couple buttons in OpenShift Online and have a free deployment to kick the tires on. Oh well.

If there is other free Kubernetes hosting out there (perhaps for open source projects?) I'd be interested as well.

Since Dataverse developers have access to AWS, I've been thinking it would be nice to try @poikilotherm 's new dataverse-kubernetes repo on Amazon Elastic Container Service for Kubernetes (EKS).

[vsoch]

Yeah this has been my experience too :/ I'm not sure it's the greatest thing that software development is being driven toward "special" infrastructure that is hard to come by.

[pdurbin]

@vsoch I was reminded by @pameyer (thanks) to mention that NDS Labs Workbench ( https://www.workbench.nationaldataservice.org ) is a way to spin up projects like Dataverse, DSpace, iRods, OwnCloud, etc. (full list at https://nationaldataservice.atlassian.net/wiki/spaces/NDSC/pages/4358234/NDS+Labs+Services ) for free if you are eligible ("Data management tool developers" and others according to https://nationaldataservice.atlassian.net/wiki/spaces/NDSC/pages/18448399/Frequently+Asked+Questions ). @craig-willis developed this and you can read more about it from the Dataverse perspective in #4152 and at http://guides.dataverse.org/en/4.11/installation/prep.html#nds-labs-workbench-for-testing-only

@poikilotherm I'm a little blocked trying out dataverse-kubernetes in the cloud because I need to learn how to use EKS. In this issue I just opened, I'm asking if some tips can be be added to the README: https://github.com/IQSS/dataverse-kubernetes/issues/12

@4tikhonov does https://github.com/IQSS/dataverse-docker run on EKS? I'd be happy to open an issue to add some tips to the README there as well, if you want.

[vsoch]

Holy cow @pdurbin you just won the award for the highest density of references and links I've ever seen in a single comment, anywhere! :confetti_ball: Thank you! This looks like an awesome resource and I'll check it out!

[pdurbin]

@vsoch unfortuntely at https://github.com/IQSS/dataverse/issues/4152#issuecomment-479037624 we learned that the instance of NDS Labs Workbench we've been talking about won't be supported any more. The code remains open source if someone else wants to run it.

Meanwhile, some excellent progress was made by @poikilotherm yesterday toward deploying his community-supported dataverse-kubernetes effort on AWS. Even though in https://github.com/IQSS/dataverse-kubernetes/issues/12 I was asking for EKS support (also discussed at http://irclog.iq.harvard.edu/dataverse/2019-04-24#i_91578 ), all I think I really wanted was a way to deploy his solution on AWS somehow. In pull request https://github.com/IQSS/dataverse-kubernetes/pull/45 he has created what looks like fantastic documentation on how to run dataverse-kubernetes on AWS using kops. I haven't tried it myself but from what I hear at http://irclog.iq.harvard.edu/dataverse/2019-04-25#i_91713 it sounds awesome. All you need is an AWS account, which we also document at http://guides.dataverse.org/en/4.13/developers/deployment.html . I haven't tried it myself but I'd like to encourage anyone reading this to try out the readme in that pull request and report back here any feedback. From my perspective, this unblocks this issue, potentially at least. I don't believe anyone is using dataverse-kubernetes in production yet, but @poikilotherm plans to.

Additionally, @4tikhonov recently tweeted at https://twitter.com/4tykhonov/status/1116229640232873984 some slides regarding his dev process which makes use of https://github.com/IQSS/dataverse-docker and a continous integration pipeline. You can see some more discussion about this (and screenshots) at https://github.com/IQSS/dataverse/issues/5725#issuecomment-482083170 and https://github.com/IQSS/dataverse/pull/5751#issuecomment-482316168 . I don't believe anyone is using dataverse-docker in production yet. I think it's mostly used for (fantastic) demos. I'd be interested in knowing how easy it is to deploy dataverse-docker to AWS since that's the cloud resource that's available to me.

@omaralsoudanii what is your status, please? Have you been experimenting with running Dataverse in Docker?

@xibriz I recently learned that @poikilotherm is also using GitLab CI. You two might want to coordinate.

[xibriz]

@pdurbin Since you have an incredible overview of everything, I have to let you know that I will leave my job at UiT in about a month.

As far as I know, no one will be taking over the Kubernetes-trials at UiT :/

[pdurbin]

@xibriz thanks for the update! Good luck on your future adventures! I'd love to see you in Cambridge again some day! 😄

[vsoch]

@pdurbin thanks for letting me know - it's an incredible (and costly) effort to provide this kind of resource and I understand not being able to support it forever. It's really fantastic to see all the great work on Dataverse! Heads up @poikilotherm the link at the top of your repo here is 404.

I still have yet to figure out how to easily develop with an actual cluster (without the immense burden of costs) but I'm learning a lot of GoLang and working on supported tools to hopefully learn a bit regardless. Definitely ping me if I can be of any help, other than offering words of encouragement! :)

[poikilotherm]

@vsoch seems like the link I used was accidentially a link you can only follow when accessing logged in... Thx for pointing that out, will change it.

[vsoch]

I'm not worthy!!!

Just kidding :)

[pdurbin]

@omaralsoudanii are you still interested in this? There are some community efforts to run Dataverse on Docker in production but it is not supported by IQSS.

[pdurbin]

The first installation of Dataverse to advertise itself as running on Kubernetes in production was just added to the map at https://dataverse.org/installations

@poikilotherm is my hero! 🎉 He recently gave a talk at http://talks.bertuch.name/dataverse-k8s-20200124/ about running Dataverse on Kubernetes:

Awesome.

[portante]

The first installation of Dataverse to advertise itself as running on Kubernetes in production was just added to the map at https://dataverse.org/installations

Great news!

[4tikhonov]

Great, congratulations, @poikilotherm! I'm also convinced that Kubernetes is the only way to go, all services should follow the same direction to increase the maturity, not only Dataverse.

[pdurbin]

@portante thanks for the shout out at https://twitter.com/pportante/status/1233367941346885633 !

When should we circle back to #4040 ? 😄

[pdurbin]

@4tikhonov I absolutely agree!

At @pidapalooza 2020 someone told me, "For anything new, it has to run on Kubernetes."

Hmm, come to think of it... what's the definition of done for this issue? 😄

[poikilotherm]

@pdurbin asked me to link to IQSS/dataverse-kubernetes#129 here because of the first bullet in #4040 being about running things on OpenShift. @portante is this still relevant for you?
Phil is really eager to chat with you on IRC and I'd be happy to share thoughts and ideas.

[4tikhonov]

@4tikhonov I absolutely agree!

At @pidapalooza 2020 someone told me, "For anything new, it has to run on Kubernetes."

Wait a little bit and I'm pretty sure we can bring really big fish to the Dataverse community.
It's also providing previewers and other services running on Kubernetes, as a part of this POC:
https://twitter.com/4tykhonov/status/1232276978021126144