Odo: [FLAKE]make test fails on osx and windows in travis CI

Created on 6 Feb 2020 · 30Comments · Source: openshift/odo

/kind bug
/kind flake

What versions of software are you using?

Operating System:

Output of odo version: master

How did you run odo exactly?

On OpenShift CI

Actual behavior

Failed in getting the details of the changed file TestWatchAndPush on macOS.

Also watch.go is creating some temporary files C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071 which is failed to get detected on windows.

Expected behavior

make test should pass

Any logs, error output, etc?

macOS travis CI

$ make test
go test -race github.com/openshift/odo/cmd/cli-doc github.com/openshift/odo/cmd/odo github.com/openshift/odo/pkg/application github.com/openshift/odo/pkg/application/labels github.com/openshift/odo/pkg/auth github.com/openshift/odo/pkg/catalog github.com/openshift/odo/pkg/component github.com/openshift/odo/pkg/component/labels github.com/openshift/odo/pkg/config 
[...]
ok      github.com/openshift/odo/pkg/catalog    1.280s
E0131 09:18:12.583395    5459 watch.go:183] Failed getting details of the changed file /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/fabric8-analytics-license-analysis259913361/tests868419307/test_read_licenses.py612824246. Ignoring the change
E0131 09:18:12.585345    5459 watch.go:189] Ignoring event for file /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/fabric8-analytics-license-analysis259913361/tests868419307/test_read_licenses.py612824246 as details about the file couldn't be fetched
--- FAIL: TestWatchAndPush (11.10s)
    watch_test.go:690: Running test:  Case 1: Valid watch with list of files to be ignored with a append event
    watch_test.go:690: Running test:  Case 2: Valid watch with list of files to be ignored with a append and a delete event
    watch_test.go:690: Running test:  Case 3: Valid watch with list of files to be ignored with a create and a delete event
    watch_test.go:690: Running test:  Case 4: Valid watch with list of files to be ignored with a folder create event
    --- FAIL: TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event (1.03s)
 [...]

windows travis CI

$ make test
go test  -race github.com/openshift/odo/cmd/cli-doc github.com/openshift/odo/cmd/odo github.com/openshift/odo/pkg/application github.com/openshift/odo/pkg/application/labels github.com/openshift/odo/pkg/auth github.com/openshift/odo/pkg/catalog github.com/openshift/odo/pkg/component github.com/openshift/odo/pkg/component/labels github.com/openshift/odo/pkg/config github.com/openshift/odo/pkg/debug github.com/openshift/odo/pkg/devfile github.com/openshift/odo/pkg/devfile/adapters
[...]
E0303 10:03:25.148080    5044 watch.go:183] Failed getting details of the changed file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071. Ignoring the change
windows failed err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071: The system cannot find the path specified.
E0303 10:03:25.152962    5044 watch.go:190] Ignoring event for file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071 as details about the file couldn't be fetched
windows ignore stat err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071: The system cannot find the path specified.
E0303 10:03:25.153939    5044 watch.go:183] Failed getting details of the changed file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904. Ignoring the change
windows failed err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
E0303 10:03:25.153939    5044 watch.go:190] Ignoring event for file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904 as details about the file couldn't be fetched
windows ignore stat err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
E0303 10:03:25.153939    5044 watch.go:183] Failed getting details of the changed file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904. Ignoring the change
windows failed err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
E0303 10:03:25.153939    5044 watch.go:190] Ignoring event for file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904 as details about the file couldn't be fetched
windows ignore stat err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
received [C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\__init__.py011991377 C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\src445007821\read_licenses.py313912176] which is not same as expected list src/read_licenses.py,__init__.pyFAIL    github.com/openshift/odo/pkg/component  2.452s
ok      github.com/openshift/odo/pkg/component/labels   1.047s
ok      github.com/openshift/odo/pkg/config 1.147s
--- FAIL: Test_getDebugInfo (0.01s)
    --- FAIL: Test_getDebugInfo/case_3:_debug_port_not_listening (0.00s)
        info_test.go:315: getDebugInfo() got = {{OdoDebugInfo v1} 3844 testing-1 app nodejs-ex 5858 9001}, want {{ } 0    0 0}
        info_test.go:318: getDebugInfo() got1 = true, want false
    --- FAIL: Test_getDebugInfo/case_4:_the_process_is_not_running (0.00s)
        info_test.go:302: error while starting fake port listerner, cause: listen tcp 127.0.0.1:9001: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
FAIL
FAIL    github.com/openshift/odo/pkg/debug  0.377s
[...]

Issue ref : https://travis-ci.com/openshift/odo/jobs/281881842#L133 and https://travis-ci.com/openshift/odo/jobs/293340097#L110

aretesting kinbug kinflake needs more information prioritHigh

Source

prietyc123

All 30 comments

@mik-dass Based on your comment https://github.com/openshift/odo/issues/2571#issuecomment-582806408 on the duplicate issue #2571, do we really need this issue ? cc_ @prietyc123

amitkrout on 6 Feb 2020

@mik-dass Based on your comment #2571 (comment) on the duplicate issue #2571, do we really need this issue ? cc_ @prietyc123

Yes this is a different issue than that. This was introduced in https://github.com/openshift/odo/pull/2549#issuecomment-580589581. This was the reason we bumped the go version on the travis CI.

mik-dass on 6 Feb 2020

/area testing

kadel on 13 Feb 2020

/priority high

kadel on 20 Feb 2020

Skipped watch test on windows and macOS in https://github.com/openshift/odo/pull/2562/files#r389347456. @girishramnani

prietyc123 on 6 Mar 2020

https://github.com/openshift/odo/pull/2692 only solved the flake for windows.

mik-dass on 16 Mar 2020

👍1

@mik-dass it seems the flake on mac (travis CI) appears more frequently, even i can now reproduce it locally on my mac setup.

occurrence - https://travis-ci.com/github/openshift/odo/jobs/307573205#L210

amitkrout on 29 Mar 2020

@mik-dass it seems the flake on mac (travis CI) appears more frequently, even i can now reproduce it locally on my mac setup.

I don't have a mac system with me nor a VM. So there is nothing I can do from my side.
Unless someone can get me one :D The unit tests are working fine on Linux and Windows systems.

mik-dass on 30 Mar 2020

also I sometimes get this error on CI too

analytics-license-analysis971352371/tests699243485 FileType:1 ModificationType:create}]
        watch_test.go:1340: error in WatchAndPush Rel: can't make watch_test.go relative to /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/fabric8-analytics-license-analysis971352371
            failed to propagate delete of file watch_test.go as its relative to /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/fabric8-analytics-license-analysis971352371 couldn't be found
            github.com/openshift/odo/pkg/watch.WatchAndPush.func1
                /Users/travis/gopath/src/github.com/openshift/odo/pkg/watch/watch.go:246
            runtime.goexit
                /Users/travis/.gimme/versions/go1.13.1.darwin.amd64/src/runtime/asm_amd64.s:1357
    watch_test.go:1254: Running test:  Case 2: Valid watch with list of files to be ignored with a append and a delete event

mik-dass on 1 Apr 2020

@mik-dass it seems the flake on mac (travis CI) appears more frequently, even i can now reproduce it locally on my mac setup.

I don't have a mac system with me nor a VM. So there is nothing I can do from my side.
Unless someone can get me one :D The unit tests are working fine on Linux and Windows systems.

No worries @mik-dass. I use mac for my day to day test activities. Now onward i am taking the whole ownership of this issue and yes i need your help over bjn, so assigning the issue to you along with me.

amitkrout on 2 Apr 2020

another occurrence https://travis-ci.com/github/openshift/odo/jobs/316815602

I see it quite often (multiple times per day) :-(

kadel on 8 Apr 2020

Any progress on this?

This is a high priority, flakes are slowing us down a lot!

kadel on 9 Apr 2020

Any progress on this?

This is a high priority, flakes are slowing us down a lot!

Since this test passes on Linux and windows, let's keep the issue open and skip it for mac OS for now. WDYT?

mik-dass on 13 Apr 2020

Since this test passes on Linux and windows, let's keep the issue open and skip it for mac OS for now. WDYT?

Let's try to fix it. I don't think that skipping tests is a good idea.

kadel on 14 Apr 2020

occurrence: https://travis-ci.com/github/openshift/odo/jobs/317396190

kadel on 14 Apr 2020

https://travis-ci.com/github/openshift/odo/jobs/317396190

kadel on 15 Apr 2020

On linux

[mrinaldas@localhost odo]$ go test pkg/watch/watch.go pkg/watch/watch_test.go -p 5 -count=1 -v 4
=== RUN   TestWatchAndPush
=== RUN   TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis858884704
=== RUN   TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis082560444
=== RUN   TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis409193560
E0415 20:47:38.141679   15000 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis409193560/tests463270730/test_read_licenses.py613216233. Ignoring the change
E0415 20:47:38.142007   15000 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis409193560/tests463270730/test_read_licenses.py613216233 as details about the file couldn't be fetched
=== RUN   TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis901342260
E0415 20:47:41.145053   15000 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis901342260/tests235530630/test_read_licenses.py381033589. Ignoring the change
E0415 20:47:41.145104   15000 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis901342260/tests235530630/test_read_licenses.py381033589 as details about the file couldn't be fetched
=== RUN   TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis054912111
E0415 20:47:44.152919   15000 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis054912111/tests653345401/test_read_licenses.py735065371. Ignoring the change
E0415 20:47:44.152983   15000 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis054912111/tests653345401/test_read_licenses.py735065371 as details about the file couldn't be fetched
--- PASS: TestWatchAndPush (14.03s)
    --- PASS: TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event (2.01s)
    --- PASS: TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event (3.01s)
    --- PASS: TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event (3.00s)
    --- PASS: TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event (3.01s)
    --- PASS: TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event (3.01s)
PASS
ok      command-line-arguments  14.044s



[mrinaldas@localhost odo]$ go test pkg/watch/watch.go pkg/watch/watch_test.go -p 1 -count=1 -v 4
=== RUN   TestWatchAndPush
=== RUN   TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis893651643
=== RUN   TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis602517927
=== RUN   TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis760146131
E0415 20:48:16.371745   15088 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis760146131/tests268043901/test_read_licenses.py632036960. Ignoring the change
E0415 20:48:16.372775   15088 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis760146131/tests268043901/test_read_licenses.py632036960 as details about the file couldn't be fetched
=== RUN   TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis354528831
=== RUN   TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis675753038
--- PASS: TestWatchAndPush (14.02s)
    --- PASS: TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event (2.00s)
    --- PASS: TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event (3.00s)
    --- PASS: TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event (3.00s)
    --- PASS: TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event (3.01s)
    --- PASS: TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event (3.01s)
PASS
ok      command-line-arguments  14.040s

Increasing the parallel nodes on the watch test to 5 seems to result in 3 Failed getting details of the changed file cases instead of 1 case

mik-dass on 15 Apr 2020

@mik-dass We are hitting the same issue on windows https://travis-ci.com/github/openshift/odo/jobs/337120311#L233

prietyc123 on 20 May 2020

@mik-dass We are hitting the same issue on windows https://travis-ci.com/github/openshift/odo/jobs/337120311#L233

 watch_test.go:823: error in WatchAndPush error watching filesystem for changes: Windows system assumed buffer larger than it is, events have likely been missed.

It was thrown by the fsnotify library https://github.com/fsnotify/fsnotify/blob/master/windows.go#L506 due to some problem with the windows OS.

mik-dass on 20 May 2020

It was thrown by the fsnotify library https://github.com/fsnotify/fsnotify/blob/master/windows.go#L506 due to some problem with the windows OS.

Ok, then we should add patch in the upstream or let’s create an issue there to keep a track of that. Or is there any possibility to fix it from our end ?

prietyc123 on 20 May 2020

Hitting the same issue again - https://travis-ci.com/github/openshift/odo/jobs/337411640#L248

pr reference - https://github.com/openshift/odo/pull/3218

amitkrout on 20 May 2020

Ok, then we should add patch in the upstream or let’s create an issue there to keep a track of that. Or is there any possibility to fix it from our end ?

The error is already handled upstream by throwing an error from their side. To fix this, we probably will need fix from windows side. The only other solution I can think of is ignoring this error in code. But that can lead to other problems.

mik-dass on 20 May 2020

@mik-dass Skip is not a good idea as you mentioned. This issue has been hit twice, and i am sure more will come in recent future. AFAIK travis does not do image update, then how come suddenly our test fails. Or am i missing anything ?

amitkrout on 20 May 2020

one more hit - https://travis-ci.com/github/openshift/odo/jobs/337411640#L248

amitkrout on 20 May 2020

@mik-dass Skip is not a good idea as you mentioned. This issue has been hit twice, and i am sure more will come in recent future.

Yeah more will come and I am sure about that too.

AFAIK travis does not do image update, then how come suddenly our test fails?

Are you sure about that?

mik-dass on 20 May 2020

@mik-dass Skip is not a good idea as you mentioned. This issue has been hit twice, and i am sure more will come in recent future.

Yeah more will come and I am sure about that too.

AFAIK travis does not do image update, then how come suddenly our test fails?

Are you sure about that?

yes, just for your reference i am attaching a screen shoot. Logs collected from the travis CI

23rd Mar ------------------------------------------- Today
Screen Shot 2020-05-20 at 4 54 57 PM

amitkrout on 20 May 2020

yes, just for your reference i am attaching a screen shoot. Logs collected from the travis CI

Ok let me check locally on a windows machine

mik-dass on 20 May 2020

👍1

Ok let me check locally on a windows machine

I have tested the latest master watch test code 100+ times on a windows machine but couldn't reproduce the issue a single time. Also I think we should close this issue and create a new one as, log wise, most probably it has nothing to do with the issue reported.

mik-dass on 20 May 2020

Ok let me check locally on a windows machine

I have tested the latest master watch test code 100+ times on a windows machine but couldn't reproduce the issue a single time. Also I think we should close this issue and create a new one as, log wise, most probably it has nothing to do with the issue reported.

Ok, make sense. @prietyc123 Please go ahead and open an issue with the fresh logs.

amitkrout on 21 May 2020

Ok, make sense. @prietyc123 Please go ahead and open an issue with the fresh logs.

Yes, I have raised a new issue

prietyc123 on 21 May 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Detect that component has not been build yet

kadel · 8Comments

Telemetery should be disabled for CI tests

jbpratt · 5Comments

odo create fails to download files in assigned --context for odo create --starter

anandrkskd · 5Comments

odo push all the target directory content and not the binary file

cmoulliard · 8Comments

component is left in broken state after redeployment of the pod

kadel · 7Comments