Tools: Make generic Github Actions for marketplace

[MaxUlysse]

As we're already adding GitHub actions from some of our tests, I think it'll be good to add more GitHub actions for some steps:

• [ ] Install Nextflow (version specific)
• [ ] Install nf-core tools (version specific) and run nf-core lint (version/release specific)
• [ ] Install and run markdownlint (maybe there's already one existing)
• [ ] Install and run YAML lint (maybe there's already one existing)

[MaxUlysse]

• Markdownlint: https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=%22markdownlint%22+OR+%22markdown+lint%22
• YAMLlint: https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=%22YAMLlint%22+OR+%22YAML+Lint%22

[ewels]

You mean make new repos to put these actions on the GH marketplace?

[MaxUlysse]

Exactly

[ewels]

I wondered that at first. But then it seemed like quite a lot of work and I wasn’t sure what the benefit would be for us?

[MaxUlysse]

I looked into that as well, when we started to use Github Actions, and I had issues understanding the whole thing, but now, there are more repos with Actions and they are more detailed.

From what I see here:
https://github.com/gaurav-nelson/github-action-markdown-link-check
It's indeed a little work, but could be worth it.
I think the Install Nextlfow actions, could be done on the nextflow repo and the Install nf-core tools, run nf-core lint could be done in a repo in nf-core.

Also, regarding nf-core tools linting, I'm thinking that it's a tool that could be useful outside the nf-core community, if we could provide a config file to say which tests could be allowed to fail or not.
But that's another issue.

[ewels]

Yup, specifically this other issue: https://github.com/nf-core/tools/issues/389 😅

[MaxUlysse]

I knew we talked about that before.
So definitively, I think this actions could be good for the community

[mashehu]

If you need motivation to resolve this issue: https://githubhackathon.com 🙂

[ewels]

ooh, swag bags!

[maxibor]

@MaxUlysse I'm having a try at it

[maxibor]

So, here is what I have: github.com/maxibor/nf-core-gh-action

• [X] Install Nextflow (version specific)
• [X] Install nf-core tools (version specific) and run nf-core lint (version/release specific)
• [X] Install and run markdownlint (maybe there's already one existing)
• [X] Install and run YAML lint (maybe there's already one existing)

Calling the action looks like this:

name: nf-core action CI

on: [push, pull_request]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: nf-core action CI
      uses: maxibor/nf-core-gh-action@master
      with:
        nextflow_version: '20.04.1'
        nfcore_version: '1.9'

Demo-ed here on a virigin nf-core template pipeline

[MaxUlysse]

@nf-core/core Shall we create an nf-core/actions repo?

[ewels]

This is super cool @maxibor! Moving my thoughts here from Slack:

• Could the defaults be null, that is - get the latest version?
• It'd be nice to be able to use--release for the nf-core lint command
  
  
  
  • It looks like we've lost this in the current actions workflow, that needs fixing..
  
  
• Also the latest version needs a few ENV variables, I don't know how that would work?
  https://github.com/nf-core/tools/blob/28d8d0d943309f9a436a4517899046f967ebf4e3/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/.github/workflows/linting.yml#L56-L59

[ewels]

Also, I kind of like having separate jobs for the different types of linting currently:

It makes it very simple to see from the PR interface what exactly is failing:

I guess that it's not possible to do this with a single action though. So my vote would be to strip out the markdown and yaml linting into their own actions if possible..

[maxibor]

All your comments have been addressed @ewels

• By default, the latest version is installed
• Therelease mode is supported
• The ENV variables can be passed
• The jobs are separated

The linting workflow looks like this now:

name: nf-core linting
# This workflow is triggered on pushes and PRs to the repository.
# It runs the `nf-core lint` and markdown lint tests to ensure that the code meets the nf-core guidelines
on:
  push:
  pull_request:
  release:
    types: [published]


jobs:
  Markdown:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
        with:
          node-version: '10'
      - name: Install markdownlint
        run: npm install -g markdownlint-cli
      - name: Run Markdownlint
        run: markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml
  YAML:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: actions/setup-node@v1
        with:
          node-version: '10'
      - name: Install yaml-lint
        run: npm install -g yaml-lint
      - name: Run yaml-lint
        run: yamllint $(find ${GITHUB_WORKSPACE} -type f -name "*.yml")
  nf-core:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: nf-core CI
      uses: maxibor/nf-core-gh-action@dev
      with:
          nextflow_version: '20.04.1'
          nfcore_version: '1.9'
          mode: ''
          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}

[ewels]

Love it! And I guess mode is for the release flag?

Can we have multiple actions in a single repo or is it one action per repo? Either way, yes I think that we should probably make a new repo for this / these..

Phil

[ewels]

Also, I guess that you can’t get at thegithub.event payload inside the action? As we could have a bunch of the standard logic wrapped in inside the action if so..

[maxibor]

Yes, mode is for the release flag.
One repo = 1 action
GitHub actions only accept ENV variable or arguments, so I don't think the github.event would work directly within the action.

[ewels]

Nice! Does it set up Python somewhere? I can't see that in the code (the uses: actions/setup-python@v1 equivalent).

Can we get it to check out the code too? Would be nice if we could get down to a single step 🤓

Comparison for my own curiosity...

_Current actions code_

  nf-core:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install Nextflow
        run: |
          wget -qO- get.nextflow.io | bash
          sudo mv nextflow /usr/local/bin/
      - uses: actions/setup-python@v1
        with:
          python-version: '3.6'
          architecture: 'x64'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install nf-core
      - name: Run nf-core lint
        env:
          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
        run: nf-core lint ${GITHUB_WORKSPACE}

_New actions code_

  nf-core:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: nf-core CI
      uses: maxibor/nf-core-gh-action@dev
      with:
          mode: ${{ 'release' if (github.event_name == 'release') }}
          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}

Action would be a little easier to use if we change mode to release and make it boolean. Then we can just have release: ${{ github.event_name == 'release' }}. I'm not sure if my example code above would actually work.

Phil

[maxibor]

The repo is now here: https://github.com/nf-core/gh-actions-lint

[ewels]

Although not yet created, the repo name gh-actions-nextflow-run has also been added to the website ignore config so is ready to go when someone is keen.