Odo: Run e2e and integration test on windows and macOS host

[prietyc123]

[kind/Feature]

Which functionality do you think we should add?

We can borrow macOS host on travis and windows from appveyor then only we can start our tests on these hosts.

Why is this needed?

Not yet implemented in CI.

• [ ] Integration tests
  
  
  
  • [ ] Mac OS
  
  
  • [ ] Windows
  
  
• [ ] e2e tests
  
  
  
  • [ ] Mac OS
  
  
  • [ ] Windows
  
  

[amitkrout]

@mohammedzee1000 Can we please add those inputs that we discussed yesterday and before. @prietyc123 FYI...

[mohammedzee1000]

We can use travis.rb which is the official cli for travis ci or we can directly use the API.

The main drawbacks of travis.rb is we cannot create job manually, just restarting existing ones and also while we can set envs, we cannot do it per build

So i propose:

• configure travis job which looks for env and if not present fails
• prow job starts and brings up cluster
• prow job passes on cluster url to travis job with env and restarts it

NOTE as said we will have to consider possible dirty reads with env and figure out how to work around that

[mohammedzee1000]

https://github.com/travis-ci/travis.rb
The problematic bit where dirty reads can occur is https://github.com/travis-ci/travis.rb#env and we should plan around that.

For example after cluster is brought up, the prow job goes into a loop, where it injects env say CLUSTER_URL=10;example.com (where 10 is PR no) and restarts the job, then waits for build to become a success. If any builds fail then it restarts the same for say 2 times after which it is assumed as failed forever (basically full retrigger required).

Travis job on its part reads CLUSTER_URL="10;example.com" reads pr no and if it does not match, it fails. Otherwise, it reads the cluster URL and stores it in different env and resumes tests

❯ travis env set FOO BAR
[+] setting environment variable $FOO
❯ travis env list
# environment variables for mohammedzee1000/odo
FOO=[secure]

TRAVIS_PULL_REQUEST gets the pull req no or false and TRAVIS_PULL_REQUEST_BRANCH for branch name

Steve said:
Prow provides these env vars, so you could get the PR number by:
pr_num="$( jq .refs.pulls[0].num <<<"${JOB_SPEC}" )" # recommended
pr_num="${PULL_NUMBER}" # may be deprecated
https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md#job-environment-variables

@amitkrout @girishramnani wdyt

[mohammedzee1000]

https://github.com/travis-ci/travis.rb/issues/717

[mohammedzee1000]

Steve said:
Prow provides these env vars, so you could get the PR number by:
pr_num="$( jq .refs.pulls[0].num <<<"${JOB_SPEC}" )" # recommended
pr_num="${PULL_NUMBER}" # may be deprecated
https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md#job-environment-variables

[mohammedzee1000]

[kadel]

Would it be possible to run tests against some of the internal clusters (like ocp311.odo) instead of using OpenShift's Prow?

[mohammedzee1000]

That would depend on if we can get something that can talk to both sides, like a public Jenkins that can talk to internal resources, for instance

Of course, we could also do polling from internal Jenkins but that would make it difficult to report status/block pr based on test status and so on. At best we can have odo bot comment on the issue saying test success or failure. Putting a link not accessible to the outside world esp as a test thing would be problematic for community contributions. We could do that of course, if the team is fine with it.

I am looking at public PSI and I have already mentioned this problem for DV

@kadel

[mohammedzee1000]

It might be possible to use API v3 calls to set env for a specific build. We could potentially avoid the loops but that will mean constructing configuration, which according to travis guys, it will not be easy either

[mohammedzee1000]

I feel short team we should start with travis + prow and then once we have 4x cluster and the Jenkins story figured out (optionally DNS as well) then we can move it all to PSI

[amitkrout]

TBH no major movement i did in the current sprint except playing around the travis cli just to figure out the pattern.

The most important part currently i need to check is to figure out how we are going to get a github token/travis token into hands of prow. That is why i moved the state from ready to needs-information.

ping @mohammedzee1000

[amitkrout]

Me and @mohammedzee1000 had a call regarding the steps over bjn. Steps need to be done are

• [ ] Create a prow job - https://github.com/openshift/release/pull/10942
• [x] Install travis cli in the test container - https://github.com/openshift/odo/pull/3732
• [ ] Use github token to login into odo travis job
• [ ] Pick the travis build number where we have mac and windows jobs are running
• [ ] Trigger the travis job with the 4.* cluster login information

[mohammedzee1000]

For the token problem, we could do something of a hide in plain sight. This should not be used to often so should not need to be changed very often so we should discuss offline on how we do that, if we are. Otherwise we are going to have to wait on prow secrets to inject the token.

We also know that bitwarden is used by openshift release as another way to hold secrets. Regardless, since this prow job won't run the actual tests, it should be ok to use the container job thing too. Need to discuss further with prow team

[amitkrout]

For the token problem, we could do something of a hide in plain sight. This should not be used to often so should not need to be changed very often so we should discuss offline on how we do that, if we are. Otherwise we are going to have to wait on prow secrets to inject the token.

We also know that bitwarden is used by openshift release as another way to hold secrets. Regardless, since this prow job won't run the actual tests, it should be ok to use the container job thing too. Need to discuss further with prow team

@mohammedzee1000 Can you please summarize the tread discussion you had with prow team, so that team will come to know our next plan of action.

[mohammedzee1000]

Ok, we are in process of getting the odo-bot GitHub token injected into prow. The current method which is currently the only way is global ie all prow jobs can access it (but the main prow cluster is redhat internal so there should be no leaks).

Once it is in, we should be able to access it using PULL_SECRET_PATH. It will be json which will likely end up in the form of

 "odo-bot": {
      "auth": "blablablabla"
    }

So we should be able to pull it with jq

Once we have the token, we login with travis cli, pull the builds based on PR no, and trigger/retrigger after setting travis env.

Here is a script that I wrote that can already filter based on PR no

❯ cat test.sh
#!/usr/bin/env bash

if [[ -z $PR_NO ]]; then
    echo "Please set PR_NO"
    exit 1
fi

if [[ -z $GITHUB_TOKEN ]]; then
    echo "Please set GITHUB_TOKEN"
    exit 1
fi

REPO=openshift/odo

hist=`mktemp`
hist_nofirst=`mktemp`

# Travis login with token here
echo "Logging in to travis with provided token..."
travis login --github-token $GITHUB_TOKEN --com

# Get travis job history and store in in file after trunicating multiple spaces
# TODO: there is a way to filter history by PR NO in command but is currently broken https://github.com/travis-ci/travis.rb/issues/382
# Once it works we can avoid a bunch of this script
echo "Querying travis history..."
travis history --limit 100 -r $REPO --com | tr -s ' ' > $hist
# Cut away first column as its build no which follows same pattern as pr no so we can match pr no easily
echo "Cutting out first column as its pattern is similar to pr no"
cat $hist | cut -d ' ' -f1 --complement > $hist_nofirst
# get all line nos where pr no is present
echo "Getting all line numbers where the PR no $PR_NO occours"
line_nos=`grep -Fn $PR_NO $hist_nofirst | cut -d : -f1`
echo "Iterating over travis history to find all builds for pr"

for n in $line_nos; do
    build_no=`sed "${n}q;d" $hist | cut -d ' ' -f1`
    build_no="${build_no//#}"
    build_status=`sed "${n}q;d" $hist | cut -d ' ' -f2`
    build_status="${build_status//:}"
    if [[ ! -z $out ]];then
        out="$out "
    fi
    out="$out$build_no:$build_status"
done

echo $out

The output is:

❯ PR_NO=3129 GITHUB_TOKEN=<redacted> sh test.sh
Logging in to travis with provided token...
Successfully logged in as odo-bot!
Querying travis history...
Cutting out first column as its pattern is similar to pr no
Getting all line numbers where the PR no 3129 occours
Iterating over travis history to find all builds for pr
9602:passed 9601:failed 9549:passed

We should likely need to only work with latest build if it succeeds, then test succeeded else we set a travis env travis env set foo bar --public and then, depending on if restarts of jobs cause new env to be read (needs verification), either restart the latest build travis restart 57 or travis restart 57.x or trigger a new one on the PR no (still need to figure out but hopefully wont need to go there).

[mohammedzee1000]

https://github.com/travis-ci/travis.rb/issues/382

[mohammedzee1000]

@amitkrout we should not need to give login information, by the way, just cluster URL should be enough as everything else is fixed in tests

[mohammedzee1000]

Ok it appears that the --pull-request bug has been fixed in travis cli version 1.9.1(https://rubygems.org/gems/travis/versions/1.9.1.travis.1219.9). This is however a prerelease ie gem install travis --pre.

If we are able to go ahead with this then the script can be cut short by a lot i.e

#!/usr/bin/env bash

if [[ -z $PR_NO ]]; then
    echo "Please set PR_NO"
    exit 1
fi

if [[ -z $GITHUB_TOKEN ]]; then
    echo "Please set GITHUB_TOKEN"
    exit 1
fi

REPO=openshift/odo

hist=`mktemp`

# Travis login with token here
echo "Logging in to travis with provided token..."
travis login --github-token $GITHUB_TOKEN --com

echo "Querying travis history..."
travis history -p $PR_NO --limit 4 -r $REPO --com | tr -s ' ' > $hist
echo "Iterating over travis history to find all builds for pr $PR_NO"
while IFS= read -r line
do
        build_no=`echo $line | cut -d ' ' -f1`
        build_no="${build_no//#}"
        build_status=`echo $line | cut -d ' ' -f2`
        build_status="${build_status//:}"
    if [[ ! -z $out ]];then
        out="$out "
    fi
    out="$out$build_no:$build_status"
done < "$hist"

echo $out

Out:

❯ PR_NO=3129 GITHUB_TOKEN=<redacted> sh test1.sh
Logging in to travis with provided token...
Successfully logged in as odo-bot!
Querying travis history...
Iterating over travis history to find all builds for pr 3129
9602:passed 9601:failed 9549:passed 9522:failed

[amitkrout]

Blocked due to https://github.com/openshift/odo/issues/3270

Currently we re running our test as template based test and in the template setup https://github.com/openshift/release/blob/master/ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml#L197-L229 there is no environment variable such as PULL_SECRET_PATH that we are looking for. Also as per @petr-muller, they have no such plan to add such support to it. However as per the https://github.com/openshift/odo/issues/2537#issuecomment-635941209 in multi-stage tests should be able to use the environment variable PULL_SECRET_PATH and others features too.

Issue https://github.com/openshift/odo/issues/3270 has been created to track this change.

[prietyc123]

Blocked due to #3270

As we are now moved to multistage test infra you should be able to get the environment variable such as PULL_SECRET_PATH needed for this issue. Unblocking the issue.

[amitkrout]

https://github.com/openshift/odo/issues/2019 will be fixed as part of this issue.

[amitkrout]

Latest update on the issue - https://github.com/openshift/odo/issues/2540#issuecomment-618401118

[amitkrout]

2019 will be fixed as part of this issue.

https://github.com/openshift/odo/issues/2019#issuecomment-677883378

[amitkrout]

2019 will be fixed as part of this issue.

#2019 (comment)

2019 has been fixed through https://github.com/openshift/release/pull/11166

[amitkrout]

Task list

• [ ] Create a prow job - openshift/release#10942
• [x] Install travis cli in the test container - #3732
• [ ] Use github token to login into odo travis job
• [ ] Pick the travis build number where we have mac and windows jobs are running
• [ ] Trigger the travis job with the 4.* cluster login information

[prietyc123]

As we have already set up our periodic run on PSI. There is a similar plan for PR runs as well, but for this to get implemented we are first looking for our POC on fedora to get succeed as discussed in https://github.com/openshift/odo/issues/4056 . I will be working with @mohammedzee1000 on this.

[prietyc123]

We will wait on POC https://github.com/openshift/odo/issues/4056 outcome because we have similar plan for automation of e2e and integration tests on macOS and windows. So kind of blocked on the POC task

[kadel]

We will wait on POC #4056 outcome because we have similar plan for automation of e2e and integration tests on macOS and windows. So kind of blocked on the POC task

@prietyc123 So plan is to run even Windows and MacOS e2e tests on PSI? I don't see anything about that in @mohammedzee1000 POC.

[prietyc123]

We will wait on POC #4056 outcome because we have similar plan for automation of e2e and integration tests on macOS and windows. So kind of blocked on the POC task

@prietyc123 So plan is to run even Windows and MacOS e2e tests on PSI? I don't see anything about that in @mohammedzee1000 POC.

We have already set up the multi platform machine on psi and there is one poc pr #4063 in WIP . So we will also follow similar flow for other platforms as well. @mohammedzee1000 it would be great if you can share your inputs on merge duration for pr #4063 because I am not sure whether we are in a place to jump on implementing this scenario currently with other platforms like windows and macOS.

[dharmit]

@mohammedzee1000 how long do you think before get #4063 in? Only then we can start working on this, right?