Shields: Double AppVeyor xUnit tests reported by shields.io

[salaros]

https://ci.appveyor.com/project/salaros/configparser/build/tests

[chris48s]

I don't know much about AppVeyor but if I call https://ci.appveyor.com/api/projects/salaros/configparser/branch/master it reports passedTestsCount: 32 - that's the number we're using on the badge.

[salaros]

@chris48s so probably TestsCount is the correct tag to use, because it matches the number of actual xUnit tests I've created

I will later check other repos and see if the number of the passed tests for AppVeyor is always duplicated

[chris48s]

In case anyone else looks at this thread, this API returns XML by default but you can grab JSON by passing a Accept: application/json header e.g: curl -H "Accept: application/json" "https://ci.appveyor.com/api/projects/salaros/configparser/branch/master". I find this easier to read.

Here is the relevant extract:

    "jobs": [
      {
        "status": "success", 
        "updated": "2018-08-13T07:55:33.7537077+00:00", 
        "compilationWarningsCount": 0, 
        "name": "", 
        "compilationMessagesCount": 0, 
        "created": "2018-08-13T07:53:55.0528139+00:00", 
        "artifactsCount": 1, 
        "passedTestsCount": 32,
        "failedTestsCount": 0,
        "jobId": "8pal7on7owibrand", 
        "started": "2018-08-13T07:54:21.594665+00:00", 
        "finished": "2018-08-13T07:55:33.7537077+00:00", 
        "allowFailure": false, 
        "testsCount": 16,
        "osType": "Windows", 
        "compilationErrorsCount": 0, 
        "messagesCount": 0
      }
    ],

I don't think we should be reporting testsCount as the number of _passed_ tests here. As I read it, it is intended to be a _total_ but there is obviously something unusual happening here. Looking through some other projects using this badge, I have managed to find another strange example:

https://ci.appveyor.com/api/projects/UsefulToolboxNET/internal

    "jobs": [
      {
        "status": "success", 
        "updated": "2018-07-01T12:00:48.032447+00:00", 
        "compilationWarningsCount": 0, 
        "name": "", 
        "compilationMessagesCount": 0, 
        "created": "2018-07-01T11:59:49.6955183+00:00", 
        "artifactsCount": 1, 
        "passedTestsCount": 1206,
        "failedTestsCount": 0,
        "jobId": "lu4blfbhun6ewl3v", 
        "started": "2018-07-01T11:59:56.3211321+00:00", 
        "finished": "2018-07-01T12:00:48.032447+00:00", 
        "allowFailure": false, 
        "testsCount": 201,
        "osType": "Windows", 
        "compilationErrorsCount": 0, 
        "messagesCount": 0
      }
    ], 

I wonder if this is something that can happen if a build gets restarted multiple times, or if the same test suite is run across multiple platforms or something?

[RedSparr0w]

It would be interesting to see what is returned when only some test fail, I wonder if those get duplicated also?

[calebcartwright]

FWIW, I experienced this same thing before with AppVeyor, but the root cause was that I had duplicated test names (copy/paste goof). I found that the value of the testsCount count field in the AppVeyor API doesn't count duplicates (nor does the AppVeyor UI) but the passedTestsCount includes all. Looking at the raw contents of the xunit xml file I was generating helped me find the duplicates.

[salaros]

@calebcartwright hm.. I will check raw XML output of xUnit.net tests and will let you know if it's a 'not a bug'

[paulmelnikow]

It sounds like there's nothing for us to do here. Feel free to reopen if this is still an issue!