Openjdk-infrastructure: Set up a server and associated playbook for testci / trss

[sophia-guo]

We'd like to set up a live version of Testci (a.k.a. Test Results Summary Service / TRSS - whose source is located in: https://github.com/AdoptOpenJDK/openjdk-test-tools) on Adopt public server. So monitoring and triaging the AdoptOpenJDK test jobs would be much easier.

@karianna mentioned there is a machine available

[llxia]

The machine will be used as a web server for our Test Results Summary Service. We will run node.js and mongoDB on it. Below are the detail about the machine that we would like to have:

os: Ubuntu 16.04 LTS
cup: 4 cores (may need to increase in the future)
ram: 8G (may need to increase in the future)
disk space: 250G (may need to increase in the future. This depends on how much history data we want to keep)

[karianna]

@sxa555 LMK if you need a new Hetzner for this (CI sits on a Hetzner - not sure if we want to be consistent there or not)

[llxia]

Any idea when the machine will be ready? It would be great if we could get Test Results Summary Service set up and running before Shelley's conference talk next week. Thanks.

fyi @smlambert

[sxa]

Hetzner makes sense to me if we're going to have a new system for this to be consistent with the others.

[sxa]

@llxia We're looking at it today :-)

[karianna]

@sxa555 A Hetzner has been purchased and the base box is ready to be set up. DM me for login details (which we'll obviously replace with key box access)

[llxia]

Thanks @sxa555 for setting up the machine!

Assuming we are using same domain as adoptopenjdk.net, could you route subdomain for me? Thanks
testci.adoptopenjdk.net

[smlambert]

thanks! http://eclipsedayindia.mybluemix.net/Program#talk11 - want to shine a light on both our on-going work and upcoming plans :)

[karianna]

@llxia is testci the correct prefix here? This is a reporting host as opposed to one that performs actual CI?

[sxa]

Hmmm agree with @karianna - I would have thought trss.adoptopenjdk.net would make more sense.

[llxia]

There are more features that we would like to add to it. We just feel trss.adoptopenjdk.net may limit what we will offer.

[karianna]

Sounds intriguing! What sort of extra features? Sorry to be a pain about this but my current thinking is for there only to be two sources of CI truth in the farm. The main CI jenkins controller, and the JCK Ci Jenkins controller

[llxia]

No need to apologize. :) We are open for suggestions.
Shelley and I talked about this yesterday, we are ok with both testci and trss. We just feel testci is a little better as there are lots of ides for new features. Such as, integrate with PerfNext (a way to configure, tune and launch performance benchmarks to Jenkins servers), connect to Watson analytics for auto test triage, Test Generation Service FVaaS, Change-based test selection service, Core Analytics, Bug prediction, etc.

We are not there yet, so I am ok with trss for now. :)

[karianna]

Ah I see, I'm slightly in favour of trss as I'm worried folks will go to testci and wonder why they don't see tests running there if that makes sense?

Those additions sound awesome!! :-D

[llxia]

sure, trss it is. Thanks.

[llxia]

Below are the things that we need to do in order to set up TRSS.

• install nodejs, mongodb, forever-service and nginx
• set up nginx to listen on port 80/443
• run mongodb as service
• proxy to the node service with nginx
• manage mongodb, forever-service and nginx

Thanks @sxa555. I installed nodejs from tarball on the machine.
While we do not need sudo rights to run, could you please help us to set up and manage the rest? Thanks.

[smlambert]

Where are we at with this issue?

[karianna]

Where are we at with this issue?

Haven't made any progress but I'm very keen to get this up. I'm personally a blocker on things right now, I'll ping #infrastructure to see if anyone is free.

[smlambert]

If no one is free, but what is left are tasks myself, @llxia or @sophia-guo can take on, please let us know.

[karianna]

If no one is free, but what is left are tasks myself, @llxia or @sophia-guo can take on, please let us know.

I think @sxa555 was starting to take a look...

[sxa]

If no one is free, but what is left are tasks myself, @llxia or @sophia-guo can take on, please let us know.

The ideal thing would be if we had an ansible playbook to do all of the above so if anyone's free to put one together that would be good ;-) Also this issue mentions Ubuntu 16.04 - would it be a problem if the machine was upgraded to be the newer Ubuntu 18.04?

[smlambert]

Ubuntu 18.04 would be fine.

I can look at creating a playbook for setting up what is needed for this machine. 👍

[sxa]

That would be awesome if you have the cycles. We've got quite a few systems we don't currently have playbooks for but getting into the routine of generating them for everything new is kinda essential now one way or another.

[smlambert]

I think I will ask for a quick mtg upon your return @sxa555 - I have some playbook 'attempts' in my fork, but would like a quick guide on design intent for playbooks (is it best to have separate roles in separate files, etc).

[smlambert]

Update on this:

@gdams - thanks for setting this up today, I know you could not have used my 'not working' ansible playbook, if you have one that you used, please include here

George has additionally given access to @llxia - who will set up the TRSS server
and setup the DNS for trss.adoptopenjdk.net.

Once Lan gets things running as expected, we will update this issue.

[gdams]

This is what was run on the machine so far. Happy to PR it in. Was waiting for confirmation that I wasn't missing deps though.

---
###########################################
# AdoptOpenJDK Ansible TRSS Playbook for: #
# ------ Ubuntu 16 (tested on x64) ------ #
###########################################

- hosts: infrastructure-godaddy-ubuntu1604-x64-1
  user: root
  become: yes

  tasks:
    - block:
        - name: Load AdoptOpenJDKs variable file
          include_vars: variables/adoptopenjdk_variables.yml

        - name: Set hostname to trss.adoptopenjdk.net
          hostname:
            name: trss.adoptopenjdk.net
          tags: hostname

        - name: OS update -- apt-get upgrade
          apt: upgrade=safe update_cache=yes
          tags: patch_update

        - name: Add Node.js v8.x
          raw: "curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -"
          tags: dependencies

        - name: Import MongoDB key
          raw: "apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927"
          tags: dependencies

        - name: Create list file for MongoDB
          apt_repository:
            repo: deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse
            state: present
            filename: mongodb-org-3.2.list

        - name: Install API prerequisistes
          apt:
            name: "{{ item }}"
            state: present
            update_cache: yes
          with_items:
            - git
            - nodejs
            - mongodb-org
          tags:
            - dependencies
            - skip_ansible_lint

        - name: Start mongoDB
          systemd:
            name: mongod
            state: started

        - name: Create Jenkins user
          action: user name={{ Jenkins_Username }} state=present shell=/bin/bash
          ignore_errors: yes
          tags: jenkins_user

        - name: Install NPM packages globally
          npm:
            name: "{{ item }}"
            global: yes
            state: present
          with_items:
            - forever-service
            - yarn
          tags: forever

        - name: Git Clone openjdk-test-tools.git
          git:
            repo: 'https://github.com/AdoptOpenJDK/openjdk-test-tools.git'
            dest: "/home/{{ Jenkins_Username }}/openjdk-test-tools"
            update: yes
            force: yes
          tags:
            # TODO: Git checkouts must contain explicit version
            - skip_ansible_lint


        - name: Change owner permissions
          file:
            path: /home/{{ Jenkins_Username }}/openjdk-test-tools
            owner: "{{ Jenkins_Username }}"
            group: "{{ Jenkins_Username }}"

        # - name: Start api app
        #   become_user: "{{ Jenkins_Username }}"
        #   become: True
        #   raw: "export PRODUCTION=true && /usr/local/bin/forever start /home/{{ Jenkins_Username }}/openjdk-api/server.js"

        - name: Add cron job to check for updates
          cron: name="Check for Updates every Sunday at 5am"
            weekday="6"
            minute="0"
            hour="5"
            user=root
            job="/usr/bin/apt-get update && /usr/bin/apt-get -y upgrade"
            state=present

[sxa]

Please PR it as WIP until such confirmation has been obtained - cheers :-)

[llxia]

Should we add nginx in the playbook?

[karianna]

Should we add nginx in the playbook?

If it's required then yes :-)

[llxia]

yes, nginx is required.

[llxia]

Just for the record, due to https://github.com/AdoptOpenJDK/openjdk-test-tools/issues/68, I upgraded to MongoDB to latest (v4.0.6) on the server machine.

[sxa]

@gdams @smlambert How close is the playbook to being finalised?

[smlambert]

While I will defer to @llxia for any final comments, the base playbook that George shared above, with the additional of nginx and requirement of mongodb to be at v4.0.6 or above, seems like the complete story.

[smlambert]

We have had the server up and running for quite a while now. Am assuming this is staying open to ensure config is captured and that a playbook is created ?

[sxa]

Yep ;-) @gdams did you ever PR it as mentioned earlier?

[gdams]

playbook added here: https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1168