Azure-devops-docs: ContainerStructureTest@0 may not be configured properly

[tashrifbillah]

I have a pipeline and test configFile

Firstly, it gives the following warning:

##[warning]Can\'t find loc string for key: FoundDockerConfigStoredInTempPath
##[warning]Can\'t find loc string for key: OldDockerConfigContent
##[warning]Can\'t find loc string for key: AddingNewAuthToExistingConfig

Then, yields the following error:

[command]/home/vsts/work/_tasks/ContainerStructureTest_39bc2c9b-55b7-4835-89cd-6cc699ef7220/0.1.2/container-structure-test test --image ***/entry_point:latest --config /home/vsts/work/1/structure-test.yaml --json
##[error]Error in executing the command: Error: spawnSync /home/vsts/work/_tasks/ContainerStructureTest_39bc2c9b-55b7-4835-89cd-6cc699ef7220/0.1.2/container-structure-test ETXTBSY
##[error]Error: Error in executing the command: Error: spawnSync /home/vsts/work/_tasks/ContainerStructureTest_39bc2c9b-55b7-4835-89cd-6cc699ef7220/0.1.2/container-structure-test ETXTBSY

I do confirm that my configFile is copied into the virtual machine:

/home/vsts/work/1/structure-test.yaml

Since your documentation is not complete yet, I followed the syntax from https://github.com/GoogleContainerTools/container-structure-test

I have a feeling that this task is not configured properly.

For details, here is my DevOps project and here is the GitHub repo.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: ecd40694-540e-98c1-32ef-87af48fde27d
• Version Independent ID: d1b89b4b-e36e-9cd2-2010-f12b3e2506d6
• Content: Container Structure Test Task - Azure Pipelines
• Content Source: docs/pipelines/tasks/test/container-structure-test-task.md
• Product: devops
• Technology: devops-cicd
• GitHub Login: @Additi
• Microsoft Alias: admahesh

[tashrifbillah]

By the way, following command passes the test on my local machine:

container-structure-test-linux-amd64 test --image entry --config structure-test.yaml

where container-structure-test-linux-amd64 executable is downloaded from here

[WilliamAntonRohm]

@tashrifbillah -- Tashrif, thank you for your feedback. You may find answers to your questions here:

• Azure DevOps on Stack Overflow
• Azure DevOps Support Bot

@Additi -- Aditi, please look into adding this information to the documentation.

[tashrifbillah]

@WilliamAntonRohm -- William, thank you for your idea.

Answers relevant to my question doesn't exist in either of the sources you pointed out. Moreover, Azure DevOps Support Bot can only point you to a few pages of the documentation, apparently due to improper and incomplete training.

Aditi, please look into adding this information to the documentation.

Curious, did you refer to Stack Overflow and Support Bot here only?

[navin22]

@tashrifbillah Sorry for the delay in response.
I was able to reproduce this in the hosted agents. I am working on this will update you with my findings on this thread.

[tashrifbillah]

Hi @navin22 , thanks. Because of the issue, I ended up downloading Google container structure test on the host and then executing the tests. I hope you already noticed, but here is my environment file and relevant code snippet:

- script: |
        curl -LOs https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64
        chmod +x container-structure-test-linux-amd64
        ./container-structure-test-linux-amd64 test --image $(imageName):runTest --config $(Agent.BuildDirectory)/$(testConfigFile)
    displayName: 'Test the docker image'
    failOnStderr: true

[Additi]

@tashrifbillah This issue is fixed now. Please try again and let us know if it resolves your issue.

[tashrifbillah]

Thanks for patiently working on it for two months. I have tested your fix. I would like to point out that the following warning didn't go away:

##[warning]Can\'t find loc string for key: FoundDockerConfigStoredInTempPath
##[warning]Can\'t find loc string for key: OldDockerConfigContent
##[warning]Can\'t find loc string for key: AddingNewAuthToExistingConfig

Test runs fine but test output is shown in one single line compared to nicely colored multi line output from manually downloaded GoogleContainerStructureTest tool:

{"Pass":1,"Fail":0,"Total":1,"Results":[{"Name":"Command Test: Test equality of two integers","Pass":true,"Stdout":"============================= test session starts ==============================\nplatform linux -- Python 3.7.4, pytest-5.3.1, py-1.8.0, pluggy-0.13.1\nrootdir: /home/pnlbwh\ncollected 2 items\n\nequality-test.py .. [100%]\n\n============================== 2 passed in 0.13s ===============================\n"}]}
Total Tests: 1, Pass: 1, Fail: 0

A new warning appeared:

##[warning]Failed to publish test run data: System.ArgumentException: Unknown test runner

Given that, I don't think the issue is completely fixed.

For reproduction, use the following:
MS-fix branch on GitHub
Pipeline on devops

[navin22]

@tashrifbillah - Thanks for the feedback. loc warning, I couldn't repro it locally.
Will see how to reproduce that.
Regarding the new warning, what's the Agent Version you are using?

[tashrifbillah]

I appreciate your cooperation. Can you take a look at Initialize Job log to see agent version? The devops project is public so you should be able to see the logs. The log I hyperlinked says:
Current agent version: '2.163.1'

[navin22]

Can you re queue your pipeline?, I switched off some flags at my end.
Wanted to check on the new warning introduced ##[warning]Failed to publish test run data: System.ArgumentException: Unknown test runner

This will help you to see the test results in the Tests tab.

[tashrifbillah]

1. I do confirm that with your modification the ##[warning]Failed to publish test run data: System.ArgumentException: Unknown test runner went away. Here are the new logs.

2. You are right, now I need to figure out a way from your documentation about publishing test results and artificats on the Tests tab. I was following your screenshots here so far and that didn't help entirely.

3. However, the other loc string warnings still exist.

4. Regarding my other finding about STDOUT not being parsed by the agent correctly, i.e. \n is being printed rather than changing to a new line, I hope you will take a look at that as well.

[navin22]

For #3 we will fix it in the next task update, and #1 we will fix it properly in the next agent version. Will see If I can back port the fixes to the previous versions.

Regarding #2, was the test tab helpful in seeing the test details ? I didn't get your question "publishing test results and artifacts on the Tests tab".?

4 - Will fix this, but in general. Since we show the nice report in the Tests about the tests that are passed and failed with error and other details. You still depend on the logs? wanted to know your use case better?

[tashrifbillah]

My apologies, I didn't realize with your fix the results already got exported to the Tests tab. Great!

You are right, I may no longer need to look at the logs. I need to do a few tests and get back to you.

[tashrifbillah]

Okay, test results appeared in the Tests tab:

But the traceback/details of my pytest -v are not in the Tests tab, so I would still need to look at the logs:

{"Pass":0,"Fail":1,"Total":1,"Results":[{"Name":"Command Test: Test equality of two integers","Pass":false,"Stdout":"============================= test session starts ==============================\nplatform linux -- Python 3.7.4, pytest-5.3.1, py-1.8.0, pluggy-0.13.1 -- /home/pnlbwh/miniconda3/bin/python\ncachedir: .pytest_cache\nrootdir: /home/pnlbwh\ncollecting ... collected 2 items\n\nequality-test.py::TestEnigma::test_enigma_AD FAILED                      [ 50%]\nequality-test.py::TestEnigma::test_enigma_FA PASSED                      [100%]\n\n=================================== FAILURES ===================================\n__________________________ TestEnigma.test_enigma_AD ___________________________\n\nself = \u003cequality-test.TestEnigma testMethod=test_enigma_AD\u003e\n\n    def test_enigma_AD(self):\n\u003e       np.testing.assert_equal(0,5)\nE       AssertionError: \nE       Items are not equal:\nE        ACTUAL: 0\nE        DESIRED: 5\n\nequality-test.py:14: AssertionError\n========================= 1 failed, 1 passed in 0.15s ==========================\n","Errors":["Test 'Test equality of two integers' exited with incorrect error code. Expected: 0, Actual: 1"]}]}

However, following this and this instruction, I may be able to publish pytest -v results in a better way. Thanks for everything so far.

[manishtulsiani]

Hi,

The container structure test task is not able to publish the results to the test tab. The test works fine and I can see the results in the logs but publish to test tab fails.

Build Log:

2020-01-23T00:15:51.8154572Z [command]/opt/hostedtoolcache/container-structure-test/1.0.0/x64/container-structure-test test --image */demo-python-app:29 --config /home/vsts/work/1/s/src/docker-structure-test.yml --json
2020-01-23T00:15:53.4516477Z {"Pass":1,"Fail":0,"Total":1,"Results":[{"Name":"File Existence Test: demo.py","Pass":true}]}
2020-01-23T00:15:53.4518622Z Total Tests: 1, Pass: 1, Fail: 0
2020-01-23T00:15:54.1411901Z Result Attachments will be stored in LogStore
2020-01-23T00:15:54.1414682Z Run Attachments will be stored in LogStore
2020-01-23T00:15:54.1648044Z ##[warning]Failed to publish test run data: System.ArgumentException: Unknown test runner
at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.TestDataPublisher.ParseTestResultsFile(TestRunContext runContext, List1 testResultFiles) at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.TestDataPublisher.PublishAsync(TestRunContext runContext, List1 testResultFiles, PublishOptions publishOptions, CancellationToken cancellationToken)

Current agent version: '2.164.3'

Can you advise if there is still an issue with the ContainerStructureTest task ?

[navin22]

@manishtulsiani can you please share the entire logs with debug statements. It should have been fixed in the latest version of the agent version. Since you are saying you are already using 2.164.3. I would like to go through the logs.

[tashrifbillah]

Hi @navin22 , I do confirm that issues I reported earlier went away. However, to corroborate @manishtulsiani 's observation, I don't see the Tests tab (beside Summary tab) either:

[navin22]

@tashrifbillah thanks for confirming, regarding the tests not appearing you are also seeing the similar error "System.ArgumentException: Unknown test runner" ?

[tashrifbillah]

I don't see any such message as System.ArgumentException: Unknown test runner.

As before, feel free to look at my build logs:
https://dev.azure.com/tbillah/ci-cd-with-docker/_build

[manishtulsiani]

Hi Navin, you can check the container structure test logs in debug at:

https://dev.azure.com/manishtulsiani/azure-learning/_build/results?buildId=40&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=67bf4d1b-8b4b-543e-e9d0-24834b0ff75e

[manishtulsiani]

Hi @navin22, do you have any update on this ?

The current agent version is '2.164.7' but the task still fails to publish the results.

[debug]Object of TestLogStore created.

[warning]Failed to publish test run data: System.ArgumentException: Unknown test runner

at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.TestDataPublisher.ParseTestResultsFile(TestRunContext runContext, List1 testResultFiles) at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.TestDataPublisher.PublishAsync(TestRunContext runContext, List1 testResultFiles, PublishOptions publishOptions, CancellationToken cancellationToken)

[debug]Processed: ##vso[results.publish type=ContainerStructure;mergeResults=false;runTitle=Check file exists container structure test;resultFiles=/home/vsts/work/_temp/21b1b490-4935-11ea-80bb-4f34d8bb3e1b.json;testRunSystem=VSTS-PTR;publishRunAttachments=true;]

[debug]Finished publishing the test results to TCM

Logs at : https://dev.azure.com/manishtulsiani/azure-learning/_build/results?buildId=57&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=67bf4d1b-8b4b-543e-e9d0-24834b0ff75e

[arun-022]

[warning]Can\'t find loc string for key: FoundDockerConfigStoredInTempPath

[warning]Can\'t find loc string for key: OldDockerConfigContent

[warning]Can\'t find loc string for key: AddingNewAuthToExistingConfig

I still get these warnings .. is this issue fixed ? .My tests run fine and the results also get published to the tests tab

[cxwtool]

This issue hasn't been updated in more than 180 days, so we've closed it. If you feel the issue is still relevant and needs fixed, please reopen it and we'll take another look. We appreciate your feedback and apologize for any inconvenience.