Wpt: Enable Azure Pipelines

[foolip]

Once https://github.com/web-platform-tests/wpt/pull/13769 has landed, the next step will be to enable it in the Azure DevOps control panel: https://dev.azure.com/web-platform-tests/wpt

Once enabled, it has the potential to block PRs if anything breaks, and unlike with Travis it will not be possible for anyone with write access to this repo to restart builds. This means we need to monitor it carefully, document how to turn it off, and make sure enough people have access to turn it off.

@web-platform-tests/admins, if you don't have one, can you create a Microsoft account and send me the emails you used? Then I can add you to the Azure DevOps project.

For monitoring, I'll write a script that looks at all open PRs updated from now on, and checks for failing or ever-pending Azure Pipelines checks.

[foolip]

I've enabled it now, and as part of that there was a first run, which did almost nothing:
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=1&view=logs
https://api.github.com/repos/web-platform-tests/wpt/commits/1fb4c5a78d1af72228e9371845aa1bbadbb7b3aa/statuses (events can be seen there)

[foolip]

It's run on a number of PRs now, some including the infrastructure/ tests. If it starts misbehaving, here's how to disable it:

1. Go to https://dev.azure.com/web-platform-tests/wpt
2. Click Pipelines, then "Azure Pipelines" (that's the name of the pipeline that I chose)
3. Click Edit and then Triggers
4. Under "Pull request validation", select "Override YAML Pull Request trigger from here" and then "Disable pull request validation"

If someone could verify that the steps can be followed before it's urgent and I'm not around, that'd be great.

[foolip]

Node script to list Azure Pipelines status of all PRs opened since it was enabled

'use strict';

const octokit = require('@octokit/rest')();

const GH_TOKEN = process.env.GH_TOKEN;

if (!GH_TOKEN) {
  console.error('Please set the GH_TOKEN environment variable.');
  console.error('https://github.com/settings/tokens');
  process.exit(1);
}

octokit.authenticate({
  type: 'token',
  token: GH_TOKEN
});

// merge time of https://github.com/web-platform-tests/wpt/pull/13769
const SINCE = '2018-10-31T11:06:44Z';

async function paginate(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const { data } = response;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    data.push(...response.data);
  }
  return data;
}

async function paginateSearch(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const items = response.data.items;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    items.push(...response.data.items);
  }
  return items;
}

(async () => {
  const prs = await paginateSearch(octokit.search.issues, {
    q: `repo:web-platform-tests/wpt is:pr is:open updated:>${SINCE}`,
  });

  for (const pr of prs) {
    const commits = await paginate(octokit.pullRequests.getCommits, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      number: pr.number,
    });

    // only look at the final commit
    const ref = commits[commits.length - 1].sha;

    const statuses = await paginate(octokit.repos.getStatuses, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      ref,
    });

    // note: statuses are in reverse chronological order
    const lastAzureStatus = statuses.find(status => status.context == 'Azure Pipelines');
    if (!lastAzureStatus) {
        console.log(`#${pr.number}: no status`);
        continue;
    }

    console.log(`#${pr.number}: ${lastAzureStatus.state}`);
  }
})();

It currently only lists success and "no status", for PRs that have had comments but no new commits. Nothing has failed or gotten stuck yet.

https://dev.azure.com/web-platform-tests/wpt/_build?definitionId=1 lists the builds in the Azure Pipelines UI, but I didn't want to just trust that :)

[kaylangan]

Hey @foolip! I'm a Program Manager for Azure Pipelines. It looks like you've been able to get up and running in other PRs. If you have any questions or suggestions, please don't hesitate to reach out!

[foolip]

Thanks @kaylangan! I have an email thread with @thejohnjansen about some issues I've seen, but I'll paste it here verbatim for all to see:

I just got pytest junit output working on my fork, here are results:
https://dev.azure.com/foolip/wpt/_build/results?buildId=140&view=ms.vss-test-web.test-result-details

That view is really great, so I think I'll try to get that merged, and perhaps use it for Linux testing too instead of Travis. However, I don't quite understand how to use `continueOnError: true`. It seems like the following steps run even without that, and what it actually means is that the overall status will be successful even if the step fails? (I tried using it to make sure test results were published even on test failure.)

Another minor question: the name of the test file doesn't show up in the test hierarchy, just the name I picked ("tools/wpt/ pytest") and the method names. Can this be changed?

More questions:

1) On my fork, the Checks API integration works well: https://github.com/foolip/wpt/pull/11/checks

However, on the main repo, it looks like the old statuses API is used, as there's nothing (useful) in the Checks tab: https://github.com/web-platform-tests/wpt/pull/13844/checks

The way I enabled Azure Pipelines wasn't exactly the same in both cases, but I don't know why there's this difference. I'd love to have the Checks tab populated on the main repo too.

2) What is the expected upgrade time for macOS agent? We're now using macOS-10.13, and that's fine as long Apple keeps releasing Safari Technology Preview for 10.13, but if major OS upgrades ever lag behind a lot, it might become a problem. Cirrus CI 

3) When I push changes and observe what happens, Travis and Taskcluster usually register their runs as queued/started before Azure Pipelines by several seconds. It seems like something might be unnecessarily slow here? On a repo using only Azure Pipelines, this increases the window in which the PR can be merged without having run through any checks.

That's all for now. Thanks for this great product! :D

[foolip]

Good news! With the help of @DavidStaheli I was able to change the authentication method of the Azure Pipelines project so that the Checks API integration is used instead. That means we get the "Checks" tab nicely populated: https://github.com/web-platform-tests/wpt/pull/13856/checks

(And it'll get even nicer: https://github.com/foolip/wpt/pull/11/checks?check_run_id=28196826)

[sideshowbarker]

(And it'll get even nicer: https://github.com/foolip/wpt/pull/11/checks?check_run_id=28196826)

Very cool 👍

[foolip]

So, given that output, I'm wondering if I should try moving these tests from Travis to Azure Pipelines for Linux. @jgraham our plan was to move stuff to Taskcluster, but maybe we can leverage Azure Pipelines for the bits it seems especially good for?

[foolip]

Oh no, first sign of infra trouble!

In https://github.com/web-platform-tests/wpt/pull/13875/checks?check_run_id=28469086 there was this failure:

System.InvalidOperationException: Git fetch failed with exit code: 128
   at Agent.Plugins.Repository.GitSourceProvider.GetSourceAsync(AgentTaskPluginExecutionContext executionContext, RepositoryResource repository, CancellationToken cancellationToken)
   at Agent.Plugins.Repository.CheckoutTask.RunAsync(AgentTaskPluginExecutionContext executionContext, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)

I presume it will go away if I retry. This was my own PR so no big problem.

@kaylangan @thejohnjansen could you look into why this might be happening? Is there an automatic retry mechanism we can use? Taskcluster retries each task (=job in Azure Pipelines ) 5 times, I think mostly to deal with the possibility of the AWS spot instances used being killed.

[Hexcles]

@foolip also happened in #13877 . I think it's perhaps caused by GitHub outage: https://status.github.com/messages

[foolip]

Oh, that's two. If it was just the GitHub outage, wouldn't Travis and Taskcluster also fail?

If it keeps happening, then everyone feel free to disable per the steps above.

[foolip]

After switching from statuses to checks (https://github.com/web-platform-tests/wpt/issues/13818#issuecomment-435187608) the script in https://github.com/web-platform-tests/wpt/issues/13818#issuecomment-434863817 no longer works.

Updated script that checks (ha) for checks instead

'use strict';

const octokit = require('@octokit/rest')();

const GH_TOKEN = process.env.GH_TOKEN;

if (!GH_TOKEN) {
  console.error('Please set the GH_TOKEN environment variable.');
  console.error('https://github.com/settings/tokens');
  process.exit(1);
}

octokit.authenticate({
  type: 'token',
  token: GH_TOKEN
});

// merge time of https://github.com/web-platform-tests/wpt/pull/13769
const SINCE = '2018-10-31T11:06:44Z';

async function paginate(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const { data } = response;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    data.push(...response.data);
  }
  return data;
}

async function paginateSearch(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const items = response.data.items;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    items.push(...response.data.items);
  }
  return items;
}

(async () => {
  const prs = await paginateSearch(octokit.search.issues, {
    q: `repo:web-platform-tests/wpt is:pr is:open updated:>${SINCE}`,
  });

  for (const pr of prs) {
    const commits = await paginate(octokit.pullRequests.getCommits, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      number: pr.number,
    });

    // only look at the final commit
    const ref = commits[commits.length - 1].sha;

    const checks = await paginate(octokit.checks.listForRef, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      ref,
    });

    const azureRun = checks.check_runs.find(run => run.name == 'Azure Pipelines');
    if (!azureRun) {
        console.log(`#${pr.number}: no check`);
        continue;
    }

    if (azureRun.status !== 'completed') {
      console.log(`#${pr.number}: ${azureRun.status}`);
    } else {
      console.log(`#${pr.number}: ${azureRun.conclusion}`);
    }
  }
})();

Right now it lists only #13875 as failing, which seems right.

[foolip]

OK, another problem. On some PRs Azure Pipelines wasn't triggered at all. https://github.com/web-platform-tests/wpt/pull/13884 and https://github.com/web-platform-tests/wpt/pull/13886 are examples.

[foolip]

Given the lack of a check run on https://github.com/web-platform-tests/wpt/pull/13900, I now suspect builds aren't triggered for PRs from forks by default, which is also the default for Taskcluster.

[foolip]

I've fixed the "PRs from forks" problem and documented how in https://github.com/web-platform-tests/wpt/pull/13952.

[foolip]

Running the script in https://github.com/web-platform-tests/wpt/issues/13818#issuecomment-435542445 again now, the only failure listed is #13640, where the failures seems legitimate and the infrastructure/ tests are failing also on Travis.

A lot of PRs don't have a check because they were created before Azure Pipelines was added and just had comments to bump their updated date, and some (like https://github.com/web-platform-tests/wpt/pull/13926) are before builds for PRs from forks were enabled.

[foolip]

Updated script that only includes problematic "no check" cases

'use strict';

const octokit = require('@octokit/rest')();

const GH_TOKEN = process.env.GH_TOKEN;

if (!GH_TOKEN) {
  console.error('Please set the GH_TOKEN environment variable.');
  console.error('https://github.com/settings/tokens');
  process.exit(1);
}

octokit.authenticate({
  type: 'token',
  token: GH_TOKEN
});

// Time of https://github.com/web-platform-tests/wpt/issues/13818#issuecomment-436330922
const SINCE = '2018-11-06T17:07:56Z';

async function paginate(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const { data } = response;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    data.push(...response.data);
  }
  return data;
}

async function paginateSearch(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const items = response.data.items;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    items.push(...response.data.items);
  }
  return items;
}

(async () => {
  const prs = await paginateSearch(octokit.search.issues, {
    q: `repo:web-platform-tests/wpt is:pr is:open updated:>${SINCE}`,
  });

  for (const pr of prs) {
    const commits = await paginate(octokit.pullRequests.getCommits, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      number: pr.number,
    });

    // only look at the final commit
    const ref = commits[commits.length - 1].sha;

    const checks = await paginate(octokit.checks.listForRef, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      ref,
    });

    const azureRun = checks.check_runs.find(run => run.name == 'Azure Pipelines');
    if (!azureRun) {
        // If created before the cutoff time and there are no checks, that's
        // probably because the update was just a commment and no CI has run.
        if (Date.parse(pr.created_at) < Date.parse(SINCE)) {
            continue;
        }
        // Otherwise there might be something wrong.
        console.log(`#${pr.number}: no check`);
        continue;
    }

    if (azureRun.status !== 'completed') {
      const msAgo = Date.now() - Date.parse(azureRun.started_at);
      const minAgo = Math.floor(msAgo / 60000);
      console.log(`#${pr.number}: ${azureRun.status} (started ${minAgo} min ago)`);
    } else {
      console.log(`#${pr.number}: ${azureRun.conclusion}`);
    }
  }
})();

Overall, things seem to be working just fine.

[foolip]

In https://github.com/web-platform-tests/wpt/runs/29626821 the /infrastructure/assumptions/ahem.html failed on macOS Chrome Dev, but not Firefox Nightly or Safari Technology Preview. That suggests there might still be a problem with font installation on macOS :/

[foolip]

I've run the script in https://github.com/web-platform-tests/wpt/issues/13818#issuecomment-436566009 again, no new failures.

[foolip]

From https://github.com/web-platform-tests/wpt/pull/14096 I learned that to be blocking, we also need to make the Azure Pipelines check required in the GitHub settings. I won't do that right away, but should be done before this issue is closed.

[foolip]

I've run the script again. https://github.com/web-platform-tests/wpt/pull/14074 is the single new failure, but appears to be correct, the failures are in the modified code and unittests are also failing on Travis. Overall, things seems to be working pretty well.

[foolip]

https://github.com/web-platform-tests/wpt/issues/14210 is an issue about monitoring Taskcluster on an ongoing basis. Possibly we can share some monitoring code/process across our CI systems.

[foolip]

Checking again for failures:

• #14206 is a legit failure, Travis has also failed in the same way
• #14156 is my own PR, legit failure

That's all. Still working well :)

[jgraham]

Should this issue be closed?

[foolip]

The monitoring part isn't done. I could rename https://github.com/web-platform-tests/wpt/issues/14210 to cover it too, but then that might also end up in a half-done state :)

[jgraham]

I think this should be closed and a bespoke issue opened for whatever monitioring you are planning.

[foolip]

Filed https://github.com/web-platform-tests/wpt/issues/14763.