Runtime: Collect dumps as part of CI failure

[tannergooding]

There are a number of cases when CI jobs fail and the machine creates a coredump. However, it does not appear that we currently surface said coredumps at any point during the CI process.

Given that these CI failures are sometimes sporadic, may depend on machine state, or may not repro locally at all; it would be ideal if we could collect these coredumps and make them available as part of the CI failure logs.

Some instances of this include:

• https://github.com/dotnet/coreclr/issues/21105
• https://github.com/dotnet/coreclr/issues/19299
• https://github.com/dotnet/coreclr/issues/19190
• https://github.com/dotnet/coreclr/pull/21097
• (and likely others)

[tannergooding]

It looks like we already have some hookup in bringup_runtests.sh which pulls down dumpling.py and uploads to a service.

It would also likely not take much work to update the groovy script to clear the dump folder before each run and to collect any dumps on job failure as part of the build artifacts (this could go to Jenkins storage or blob storage, as long as it is accessible).

[tannergooding]

CC. @jkotas, @CarolEidt, @BruceForstall, @jashook

[tannergooding]

This primarily appears to impact Arm/Arm64 and OSX.

Both can be difficult to get properly configured hardware and the latter can be nearly impossible to configure the OS (most of our machines are 10.12, and OSX does not let you easily downgrade after you've upgraded your MacBook).

I have gotten access to the hardware (including the exact machine a failure just occurred on) in the past, but these often do not repro consistently.

[tannergooding]

If the dumps are not too large, and we can identify which dumps match to which test (maybe by cleaning the folder after each test completes), we could include it as part of the console log (as we do with the out.txt and err.txt files).

[tannergooding]

I have a prototype for this here: https://github.com/dotnet/coreclr/pull/21149 (basically just copied what bringup_runtests.sh was doing, modulo uploading to the dumpling service).

[BruceForstall]

@tannergooding Thanks for doing work to improve this! It looks like dotnet/coreclr#21149 creates the dumps and copies them to be saved, but doesn't actually upload them anywhere? I was never familiar with what dumpling.py did -- is it still alive and being used? Or should we zip/tar the dumps and have netci.groovy upload them to Jenkins for the job (if they aren't already being uploaded)?

[tannergooding]

It looks like dotnet/coreclr#21149 creates the dumps and copies them to be saved, but doesn't actually upload them anywhere

Correct.

I was never familiar with what dumpling.py did -- is it still alive and being used

I'm not quite sure, looking around at some past issues/commits, it looks like the machines might not be properly configured to use dumpling.py (and if they are/were, I can't find any info on how to get the dumps back down).

Or should we zip/tar the dumps and have netci.groovy upload them to Jenkins for the job

I think this would be the easiest solution, but I believe we had some limitations with this last time we did it (cc. @mmitche, @dotnet/dnceng ). Uploading to an azure blob-store might be better/easier

[mmitche]

Do not upload them to the Jenkins service unless you can guarantee a maximum number of dumps (e.g. 1). The last time this happened there were several jobs that suddenly were uploading 30GB of dumps per failure.

From: Tanner Gooding [e-mail]
Sent: Monday, November 26, 2018 5:15 PM
To: dotnet/coreclr [e-mail]
Cc: Matt Mitchell [e-mail]; Mention [e-mail]
Subject: Re: [dotnet/coreclr] Collect dumps as part of CI failure (#21134)
Importance: High

It looks like dotnet/coreclr#21149https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcoreclr%2Fpull%2F21149&data=02%7C01%7Cmmitche%40microsoft.com%7Ceefa210304ee4fb2b4f408d65405ca3b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636788781291012366&sdata=6JpYkO0oLyxPYT8VJk8nsAeA6J64XHxwgdX%2BL%2Fo1MAs%3D&reserved=0 creates the dumps and copies them to be saved, but doesn't actually upload them anywhere

Correct.

I was never familiar with what dumpling.py did -- is it still alive and being used

I'm not quite sure, looking around at some past issues/commits, it looks like the machines might not be properly configured to use dumpling.py (and if they are/were, I can't find any info on how to get the dumps back down).

Or should we zip/tar the dumps and have netci.groovy upload them to Jenkins for the job

I think this would be the easiest solution, but I believe we had some limitations with this last time we did it (cc. @mmitchehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmmitche&data=02%7C01%7Cmmitche%40microsoft.com%7Ceefa210304ee4fb2b4f408d65405ca3b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636788781291012366&sdata=RmXs8qw4ZnZQtVT2KgWpXGTcnnCpBH7DQ%2B0H%2BuOmovc%3D&reserved=0, @dotnet/dncenghttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Forgs%2Fdotnet%2Fteams%2Fdnceng&data=02%7C01%7Cmmitche%40microsoft.com%7Ceefa210304ee4fb2b4f408d65405ca3b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636788781291022375&sdata=8p40Q2e0%2FBRL53ijdUb5Xkt9zn2UJxWa%2B4rUC6WI4Ls%3D&reserved=0 ). Uploading to an azure blob-store might be better/easier

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcoreclr%2Fissues%2F21134%23issuecomment-441860484&data=02%7C01%7Cmmitche%40microsoft.com%7Ceefa210304ee4fb2b4f408d65405ca3b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636788781291022375&sdata=bqUFRupXn%2BQII%2BPw3xee%2FzaZzgR1scIelh5LvaMIO4o%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIUisi6KqVgFcdgA8s2o5Q2iZ64eQD6tks5uzJIjgaJpZM4YsXeY&data=02%7C01%7Cmmitche%40microsoft.com%7Ceefa210304ee4fb2b4f408d65405ca3b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636788781291032379&sdata=6XO71rnrvGNOk5otzgNpDHe81hgzkKndwP%2BJy5HFA4M%3D&reserved=0.

[tannergooding]

unless you can guarantee a maximum number of dumps (e.g. 1)

Right, no guarantee for the maximum, so we need to likely upload to an azure blob-storage (or figure out how dumpling.py works).

[tannergooding]

@BruceForstall, @mmitche. Do we have some existing Azure Blob Storage we could configure the dumps to be uploaded to?

[BruceForstall]

I'm not familiar with any. Maybe @jashook @RussKeldorph could help?

[MattGal]

I'd ask @DrewScoggins or @Anipik as I am 90% sure Dumpling is just using Azure storage anyways.

[RussKeldorph]

@adityamandaleeka Were you involved with dumpling in the past? Any breadcrumbs we could follow to resurrect it?

[tannergooding]

Actually enabling it (dumpling) looks pretty straightforward (it should just be copying/updating the relevant bits from bringup_runtest.sh into runtest.py).

What I am not sure of is:

• Are the machines all configured appropriately (I found https://github.com/dotnet/coreclr/issues/15847, indicating they might not be)
• How to access the dumps after they have been uploaded (I found the relevant URL from the dumpling.py file, but the website only shows the "default" page for a new ASP.NET website)

[danmosemsft]

@ahsonkhan most recently looked at dumpling. It needs some attention and hasn't got much for a while.

[Anipik]

How to access the dumps after they have been uploaded

The link to download the dump is printed as the console output. There is an another link where we can viewdownload all the dumps. cc @DrewScoggins

[ahsonkhan]

I believe this is the site to access the uploaded dumps, but it seems to be down atm (returns 500, timeout): https://dumpling-internal.int-dot.net/

most recently looked at dumpling. It needs some attention and hasn't got much for a while.

Yep, I need to set aside some time and revisit this. Will follow up with @DrewScoggins offline.

[danmosemsft]

Historically it was possible to curl the dumps using the URL in the log, so it was not necessary to go to the site for that purpose.

[tannergooding]

I naively ported the existing dumpling code from bringup_runtest.sh into runtest.py: https://github.com/dotnet/coreclr/pull/21275

It should be fairly easy to create a test that always creates a dump to validate that it works correctly (the test can then be disabled or removed before merging).

We should also be able to use it to investigate any other issues with the service.

[RussKeldorph]

@markwilkie FYI on dumps/dumpling. We could really use centralized support for dumps.