Upload-artifact: Feature request: wildcards

[StoneCypher]

One of the really useful things about uploading artifacts is caching logs, coverage reports, et cetera. Those often have timestamps or UUIDs as names.

You can hack around it with a compression to a known name step, but, uploading and downloading artifacts by wildcard (or even just download-all) would be a huge benefit

[eine]

Possible duplicate of #3.

[StoneCypher]

Thanks, no, my request is just for wildcards. The thing you asked for is nice, but way more work

[StoneCypher]

Please stop trying to take over my issue. :(

[yacaovsnc]

@StoneCypher thanks for the feedback!

I understand the uploading part, could you please help clarify on the download all request a bit more?

Say we have this file structure:

.
├── aa.txt
├── ab.txt
├── ba.txt
└── folder1
    └── ac.txt

and we use wildcard in path to upload **/a*.txt with a name of my-artifact, that artifact should contain aa.txt, ab.txt, and folder1/ac.txt.

When you say download artifact by wildcard, do you mean download from the UI on github.com; or using the download-artifact action?

If using the action, we need to give the action the artifact name, in this case my-artifact, and it will download everything in this artifact. Will that work?

EDITED: fixed up the matching pattern, thanks @umarcor!

[umarcor]

@yacaovsnc, IMHO a*.txt should not include folder/ac.txt. In order to include aa.txt, ab.txt, and folder1/ac.txt, the param should be **/a*.txt or ['a*.txt', 'folder1/a*.txt']. Please, see https://github.com/actions/upload-artifact/issues/3#issuecomment-524442814.

[zegnus]

In an Android multi-project setup, static analysis and test results are generated under the individual projects, but with the same folder structure. Example:

.
├── module1
    └── build
        └── reports
            └── checkstyle
                └── main.xml
                └── main.html
            └── pmd
                └── main.html
                └── main.xml
            └── findbugs
                └── debug.xml
                └── debug.html
            lint-results.html
            lint-results.xml
├── module2
    └── build
        └── reports
            └── checkstyle
                └── main.xml
                └── main.html
            └── pmd
                └── main.html
                └── main.xml
            └── findbugs
                └── debug.xml
                └── debug.html
            lint-results.html
            lint-results.xml
...

It would be great if we can use a wildcard for the path in the form of:

• **/build/reports/checkstyle/*.html
• **/build/reports/pmd/*.html

Or if possible:

• **/build/reports/**/*.html

Or even better, to archive all *.html files under build/reports folder and subfolders

[StoneCypher]

So my big problem is coverage files.

I want to use artifacts as a way to join coverage from separate runs. This is reasonable because my coverage tool names the files with a timestamp and a uuid

Unfortunately, the only way I can indicate things is by filename, so first I need to zip up things to create a predictable file name, then undo that at the other end. Slows down the build fairly substantially (~20s on top of ~90s)

Globs would solve this

[yacaovsnc]

@yacaovsnc, IMHO a*.txt should not include folder/ac.txt. In order to include aa.txt, ab.txt, and folder1/ac.txt, the param should be **/a*.txt or ['a*.txt', 'folder1/a*.txt']. Please, see #3 (comment).

Thanks for the correction! Yes, **/a*.txt or a list is the right syntax.

[yacaovsnc]

@StoneCypher cool, I definitely understand the desire to upload by wildcards. We are designing the next version of this action and this is what we are considering:

1. Upload by wildcards. The above examples shows the behavior.
2. Download by wildcards are supported at the artifact name level. For example, I can have those two artifacts:
   i. my-artifact1 which includes aa.txt, ab.txt, and ac.txt.
   ii.my-artifact2 which includes ba.txt, bb.txt, and bc.txt.
   We support download my-artifact*, but not a*.txt, i.e. we will not look into each artifact to find out matching files.

Does that work?

[StoneCypher]

I believe that that would work in a pinch, since i could indicate the contents of an entire directory, but the node standard (and surprisingly easy) thing to do would be to use a globber, nominally node-glob

Either way. I really appreciate the help

[StoneCypher]

node-glob would also save you from ever having to deal with a whole bunch of long term bullshit like traversal attacks

[koppor]

For a single file, following is a solution (based on https://github.com/actions/upload-artifact/issues/7#issuecomment-553144747)

      - id: getfilename
        run: echo "::set-output name=file::$(ls build/*fat.jar)"
      - uses: actions/upload-artifact@v1
        with:
          path: ${{ steps.getfilename.outputs.file }}

[solodon4]

I believe upload artifacts should accept an array of path entries much like path triggers do. Positive wildcards then define the set of files to include for upload while negative patterns (those starting from !) exclude files from upload. Patterns should be evaluated in order given by the user to determine the final set of files.

[umarcor]

I believe upload artifacts should accept an array of path entries

See https://github.com/actions/upload-artifact/issues/3#issuecomment-524442814

[konradpabjan]

Wildcard support is now available with the v2-preview

see: https://github.com/actions/upload-artifact/issues/62

[StoneCypher]

thank you @konradpabjan

[konradpabjan]

Going to keep this issue open for a little longer @StoneCypher until v2 is out of preview and this feature is officially out (should be 1-2 more weeks depending on how smooth v2-preview is coming along)

[xenoterracide]

tried this

        uses: actions/upload-artifact@v2-preview
        with:
          workdir: packages
          name: dist
          path: |
            */dist

expecting it to work like this (but obviously with a zip instead)

tar -cJf dist.tar.xz */dist

I see no artifact on my build

[xenoterracide]

oh, it's because upload doesn't support workdir, nevermind

[amccool]

this preview is working for me (and rather necesary). Is there is a scheduled release date?

[konradpabjan]

Plan is to release v2 next week on the 28th. Small mention here: https://github.com/actions/upload-artifact/issues/62#issuecomment-618317648

[konradpabjan]

v2 upload artifact has been released! https://github.blog/changelog/2020-04-28-github-actions-v2-artifact-actions/

You can now use the feature by using actions/upload-artifact@v2

For those that tested wildcards during the v2-preview thank you ❤