Static-web-apps: Github Actions step fails to detect packages from private NPM registry

[mahmoudajawad]

Hello.

I'm giving SWA some advanced scenario where I've published a private package on Github Packages Registry in a repo I own, then I created an app repo that uses this package in it. Locally this is working very well with a .npmrc file that has a PAT issued with the correct scopes. However, SWA Actions step fails to detect the package is from another registry. I'm not sure why because .npmrc should be enough to handle this, unless at time of files process by SWA Actions something is happening and it's causing .npmrc file not to be taken into consideration.

Error message in Actions log:

npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@ORG%2fPACKAGE - Not found
npm ERR! 404 
npm ERR! 404  '@ORG/[email protected]' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of '__oryx_prod_node_modules'
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
npm ERR!     /github/home/.npm/_logs/2020-06-02T16_20_35_102Z-debug.log

[davemurphysf]

I tried to get around this by creating a .npmrc file on the fly in the build command, but this does not work either

app_build_command: 'echo "registry=https://npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}" > .npmrc && npm install'

[mahmoudajawad]

Nice thought @davemurphysf which asserts for some reason npm command in SWA step is ignoring .npmrc file completely.

[miwebst]

So oryx has an environment variable that we can try, can you try adding this below the 'with' block for the build and deploy step? It might need quotes, haven't actually tried this out yet
env:
NPM_REGISTRY_URL: https://npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}

[davemurphysf]

No luck for me. I tried using quotes and no quotes on the value of the environment variable NPM_REGISTRY_URL. I also tried using a generated PAT instead of the GITHUB_TOKEN. I also tried to do the build first and disable Oryx from building via DISABLE_NODEJS_BUILD: true but then it complained that it could not determine the project type.

Here is my action yaml:

name: Azure Static Web Apps CI/CD

on:
  push:
    branches:
    - master
  pull_request:
    types: [opened, synchronize, reopened, closed]
    branches:
    - master

jobs:
  build_and_deploy_job:
    if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.action != 'closed')
    runs-on: self-hosted
    name: Build and Deploy Job
    steps:
    - uses: actions/checkout@v2

    - name: Build And Deploy
      id: builddeploy
      uses: Azure/[email protected]
      with:
        azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_SALMON_CLIFF_0735E8210 }}
        repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
        action: 'upload'
        ###### Repository/Build Configurations - These values can be configured to match you app requirements. ######
        # For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig 
        app_location: '/' # App source code path
        app_artifact_location: 'build' # Built app content directory - optional
        ###### End of Repository/Build Configurations ######
      env:
        NPM_REGISTRY_URL: https://npm.pkg.github.com/:_authToken=${{ secrets.GPR_PAT }}

  close_pull_request_job:
    if: github.event_name == 'pull_request' && github.event.action == 'closed'
    runs-on: ubuntu-latest
    name: Close Pull Request Job
    steps:
    - name: Close Pull Request
      id: closepullrequest
      uses: Azure/[email protected]
      with:
        azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_SALMON_CLIFF_0735E8210 }}
        action: 'close'

Raw Log Output

2020-06-12T23:20:43.1564859Z ##[group]Run Azure/[email protected]
2020-06-12T23:20:43.1564984Z with:
2020-06-12T23:20:43.1565430Z   azure_static_web_apps_api_token: ***
2020-06-12T23:20:43.1565812Z   repo_token: ***
2020-06-12T23:20:43.1565901Z   action: upload
2020-06-12T23:20:43.1565979Z   app_location: /
2020-06-12T23:20:43.1566059Z   app_artifact_location: build
2020-06-12T23:20:43.1566140Z env:
2020-06-12T23:20:43.1566303Z   NPM_REGISTRY_URL: https://npm.pkg.github.com/:_authToken=***
2020-06-12T23:20:43.1566411Z ##[endgroup]
2020-06-12T23:20:43.1586683Z ##[command]/usr/bin/docker run --name dbc4ec7617c60f44f1da6fa505890ffdec0_73a2b9 --label 549dbc --workdir /github/workspace --rm -e NPM_REGISTRY_URL -e INPUT_AZURE_STATIC_WEB_APPS_API_TOKEN -e INPUT_REPO_TOKEN -e INPUT_ACTION -e INPUT_APP_LOCATION -e INPUT_APP_ARTIFACT_LOCATION -e INPUT_API_BUILD_COMMAND -e INPUT_API_LOCATION -e INPUT_APP_BUILD_COMMAND -e INPUT_ROUTES_LOCATION -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/dave/actions-runner/_work/_temp/_github_home":"/github/home" -v "/home/dave/actions-runner/_work/_temp/_github_workflow":"/github/workflow" -v "/home/dave/actions-runner/_work/ft_web/ft_web":"/github/workspace" 549dbc:4ec7617c60f44f1da6fa505890ffdec0
2020-06-12T23:20:43.9544390Z DeploymentId: 6398ef9d-4634-494f-a939-654b0ead882b
2020-06-12T23:20:43.9544801Z 
2020-06-12T23:20:44.0206926Z App Directory Location: '/' was found.
2020-06-12T23:20:44.0207353Z No Api directory specified. Azure Functions will not be created.
2020-06-12T23:20:45.1439330Z Starting to build app with Oryx
2020-06-12T23:20:45.1442019Z Azure Static Web Apps utilizes Oryx to build both static applications and Azure Functions. You can find more details on Oryx here: https://github.com/microsoft/Oryx
2020-06-12T23:20:45.1442646Z ---Oryx build logs---
2020-06-12T23:20:45.1443131Z 
2020-06-12T23:20:45.1443442Z 
2020-06-12T23:20:45.5892281Z Build orchestrated by Microsoft Oryx, https://github.com/Microsoft/Oryx
2020-06-12T23:20:45.5892917Z You can report issues at https://github.com/Microsoft/Oryx/issues
2020-06-12T23:20:45.5893115Z 
2020-06-12T23:20:45.5900460Z Oryx Version      : 0.2.20200521.2, Commit: 409adf05fdb5f3d6e9b99c6c60597c58eb409524, ReleaseTagName: 20200521.2
2020-06-12T23:20:45.5900903Z Build Operation ID: |PorEe0AEbGA=.6574302c_
2020-06-12T23:20:45.5901090Z 
2020-06-12T23:20:46.5933021Z Using intermediate directory '/bin/staticsites/ss-oryx/app-int'.
2020-06-12T23:20:46.5955355Z 
2020-06-12T23:20:46.5957876Z Intermediate directory doesn't exist, creating it...'
2020-06-12T23:20:46.5958960Z 
2020-06-12T23:20:46.5959285Z Copying files to the intermediate directory...
2020-06-12T23:20:46.6897143Z Done in 0 sec(s).
2020-06-12T23:20:46.6897853Z 
2020-06-12T23:20:46.6899636Z Source directory     : /bin/staticsites/ss-oryx/app-int
2020-06-12T23:20:46.6900509Z Destination directory: /bin/staticsites/ss-oryx/app
2020-06-12T23:20:46.6902003Z 
2020-06-12T23:20:46.6902842Z 
2020-06-12T23:20:46.6903361Z Downloading and extracting nodejs version 12.16.3 to /tmp/oryx/platforms/nodejs/12.16.3...
2020-06-12T23:20:47.3093285Z Downloaded in 1 sec(s).
2020-06-12T23:20:47.3094301Z Verifying checksum...
2020-06-12T23:20:47.3721997Z Extracting contents...
2020-06-12T23:20:47.8482713Z Done in 1 sec(s).
2020-06-12T23:20:47.8483396Z 
2020-06-12T23:20:47.8596382Z 
2020-06-12T23:20:47.8597112Z Using Node version:
2020-06-12T23:20:47.8614659Z v12.16.3
2020-06-12T23:20:47.8621492Z 
2020-06-12T23:20:47.8622883Z Using Npm version:
2020-06-12T23:20:47.9755707Z 6.14.4
2020-06-12T23:20:47.9816372Z 
2020-06-12T23:20:47.9816980Z Installing production dependencies in '/bin/staticsites/ss-oryx/app-int/__oryx_prod_node_modules'...
2020-06-12T23:20:47.9817476Z 
2020-06-12T23:20:47.9818099Z Running 'npm install --unsafe-perm --production'...
2020-06-12T23:20:47.9818985Z 
2020-06-12T23:20:53.8088934Z npm WARN tar ENOENT: no such file or directory, open '/bin/staticsites/ss-oryx/app-int/__oryx_prod_node_modules/node_modules/.staging/minipass-pipeline-bca0161e/LICENSE'
<lots more no such file errors>

[miwebst]

@kichalla any ideas here?

[luchsamapparat]

I have a theory. It seems that sometime during the build, Orxy copies the package.json to another directory to only install the production dependencies:

https://github.com/microsoft/Oryx/blob/master/src/BuildScriptGenerator/Node/NodeBashBuildSnippet.sh.tpl#L45

I stumbled upon this because I have a postinstall script that runs a JS file in my project which is missing in the GitHub Actions build.

Installing production dependencies in '/bin/staticsites/ss-oryx/app-int/__oryx_prod_node_modules'...

Running 'npm install --unsafe-perm --production'...


> @ngxp/[email protected] postinstall /bin/staticsites/ss-oryx/app-int/__oryx_prod_node_modules
> node ./decorate-angular-cli.js && ngcc --properties es2015 browser module main --first-only --create-ivy-entry-points

internal/modules/cjs/loader.js:960
  throw err;
  ^

Error: Cannot find module '/bin/staticsites/ss-oryx/app-int/__oryx_prod_node_modules/decorate-angular-cli.js'

However, I have no idea what to do about that...

[mahmoudajawad]

@miwebst @kichalla, updates on whether this is resolvable or not?

[kichalla]

@mahmoudajawad Do you have a repository with the repro project that I can try with?

[kichalla]

@davemurphysf If possible could you share your repro project?

[mahmoudajawad]

Unfortunately @kichalla , it's private code.

[kichalla]

@mahmoudajawad from your original post, so you are still seeing the package being tried from the NPM registry and not the GitHub registry. I just now tried a scenario with the following content in .npmrc file and it worked.

Example content in the repo's .npmrc file

registry=https://registry.npmjs.org/
@kichalla:registry=https://npm.pkg.github.com/

In your GitHub actions workflow, before the static web apps actions get executed you could do the following:
echo "registry=https://npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}" >> .npmrc

[kichalla]

Ignore my earlier example, the following content in .npmrc file worked fine for me:
Here the ${GITHUB_TOKEN} is expanded by npm when ‘npm install’ is called. You just need to make sure this variable's value is available as an environment variable.

registry=https://registry.npmjs.org/
@kichalla:registry=https://npm.pkg.github.com/
//npm.pkg.github.com/:_authToken=${GITHUB_TOKEN}

[mahmoudajawad]

@kichalla, this doesn't seem to work when the package is in another repo.

[kichalla]

@kichalla, this doesn't seem to work when the package is in another repo.

Ok, just curious...irrespective of static web apps, how does it work when you do it locally on your development machine?

[mahmoudajawad]

.npmrc file is all what is needed. On local machine or Azure DevOps, it works internally because release pipelines have access to all repos the user created the release pipeline have access to. I have never run into such a scenario on Github Action to confirm, but I could spare sometime and create a separate Actions pipeline that attempts to build project artifacts and prepare them for download using simple build commands and see the results. Shall I do this?

[mahmoudajawad]

@kichalla has suggested a resolution for this issue which has been spotted and diagnosed by @getfatday, and it has worked with me. The resolution was to move the private-hosted package from dependencies to devDependencies in package.json and it got the workflow to successfully build and deploy the app.

Ref: https://github.com/microsoft/Oryx/issues/636#issuecomment-657013125

[alikazai]

Hi all.
I am still having this issue after trying out all your tips. The repo i am trying to add is the Fontawesome Pro ones. this is the error i am getting..
npm ERR! code E404 npm ERR! 404 Not Found - GET https://registry.npmjs.org/@fortawesome%2fpro-light-svg-icons - Not found npm ERR! 404 npm ERR! 404 '@fortawesome/[email protected]' is not in the npm registry. npm ERR! 404 You should bug the author to publish it (or use the name yourself!) npm ERR! 404 It was specified as a dependency of '__oryx_prod_node_modules' npm ERR! 404 npm ERR! 404 Note that you can also install from a npm ERR! 404 tarball, folder, http url, or git url.

i have the .npmrc file working locally and it is in the my repo i tried running the config commands again in app_build_command i also tried what @getfatday suggested yet nothing seems to be working.

any thing else i could try

[mahmoudajawad]

@alikazai, try to build your project on clean container or VM so you can confirm the settings of the project are correct, and not some cached credentials and tokens are allowing you to access locally, and not on Github Actions. This is common issue. Alas, you could be hitting on another problem that could be or could be not related to this issue.

[jracc]

I am seeing this as well, using a gitlab's npm registry. It appears that the .npmrc file is being ignored as well as the environment variables sent to the docker execution. To cover all bases, I added the following in the environment settings:

      env:
        NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
        NPM_REGISTRY_URL: https://gitlab.com/api/v4/packages/npm/:_authToken=${{ secrets.NPM_TOKEN }}
        INPUT_REGISTRY-URL: https://gitlab.com/api/v4/packages/npm/:_authToken=${{ secrets.NPM_TOKEN }}

They are environment variables set as a part of the docker run syntax. In addition, what I observed is that while GitHub Action syntax permits

   registry-url: https://gitlab.com/api/v4/packages/npm/:_authToken=${{ secrets.NPM_TOKEN }}

the SWA setup does not. It does not honor the .npmrc file, whether it is at the top of the project, or inserted as previous comments have suggested. And yet, it is clearly processed, because using an invalid environment variable within it, causes the build to fail immediately. So it appears that GitHub Action is performing its substitution.

[alikazai]

Thanks @jracc that seemed to have fixed it.

[jracc]

Hey @alikazai - what fixed what?

[alikazai]

@jracc

 env:
        NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
        NPM_REGISTRY_URL: https://gitlab.com/api/v4/packages/npm/:_authToken=${{ secrets.NPM_TOKEN }}
        INPUT_REGISTRY-URL: https://gitlab.com/api/v4/packages/npm/:_authToken=${{ secrets.NPM_TOKEN }}

you said to use this i did that but tweaked it for fontawesome and it worked the build from the private npm packages worked fine.

[jracc]

@alikazai - thanks. I'm glad it helped you.
@kichalla - Any reason why the build ignores my .npmrc and still assumes the registry is registry.npmjs.org?

[keithotto]

Has anyone made progress in solving this issue? I'm trying to deploy an Angular app that has an extra library in a private github package that fails on the npm install portion of the Build and Deploy step.

I tried adding a step before it that made use of the node-setup action and was able to successfully have it run npm install with the correct repos referenced. But it appears that the SWA step ignores those already installed node_modules and runs the npm install itself again and fails.

[mahmoudajawad]

Here's my Actions workflow in a hope it would help those who are still having issues:

name: Azure Static Web Apps CI/CD

on:
  push:
    branches:
    - master
  pull_request:
    types: [opened, synchronize, reopened, closed]
    branches:
    - master

jobs:
  build_and_deploy_job:
    if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.action != 'closed')
    runs-on: ubuntu-latest
    name: Build and Deploy Job
    steps:
    - uses: actions/checkout@v2
    - name: NPM Token
      run: |
        echo "@ORG:registry=https://npm.pkg.github.com/ORG
        //npm.pkg.github.com/:_authToken=$NPM_TOKEN
        always-auth=true" > .npmrc
      env:
        NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
    - name: Confirm NPM Token
      run: cat .npmrc
    - name: Build And Deploy
      id: builddeploy
      uses: Azure/[email protected]
      with:
        azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_XXXXXXXXXXX }}
        repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
        action: 'upload'
        ###### Repository/Build Configurations - These values can be configured to match you app requirements. ######
        # For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig 
        app_location: '.' # App source code path
        app_artifact_location: './dist/OUTPUT' # Built app content directory - optional
        ###### End of Repository/Build Configurations ######

  close_pull_request_job:
    if: github.event_name == 'pull_request' && github.event.action == 'closed'
    runs-on: ubuntu-latest
    name: Close Pull Request Job
    steps:
    - name: Close Pull Request
      id: closepullrequest
      uses: Azure/[email protected]
      with:
        azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_XXXXXXXXXXX }}
        action: 'close'

Notice, there's a step that prints out .npmrc file. I've added this later when it was suggested .npmrc file is not being copied over by Oryx. This would allow you to make sure you have the correct .npmrc file in place.

[persyl]

I have struggle really long time now and is frustrated that this is not documented anywhere?
Tried all you suggestions above but none of them work for me.
It all works locally as my .npmrc file is then being read as it should!

My private repo is at:
https://pkgs.dev.azure.com/XXXXX/_packaging/MYnpm/npm/registry/

My yml-file looks like this:

name: Azure Static Web Apps CI/CD

on:
  push:
    branches:
      - main
  pull_request:
    types: [opened, synchronize, reopened, closed]
    branches:
      - main

jobs:
  build_and_deploy_job:
    if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.action != 'closed')
    runs-on: ubuntu-latest
    name: Build and Deploy Job
    steps:
      - uses: actions/checkout@v2
        with:
          submodules: true
      - name: NPM Token
        run: |
          echo "@ORG:registry=https://pkgs.dev.azure.com/XXXXX/_packaging/MYnpm/npm/registry/
          //pkgs.dev.azure.com/:_authToken=$NPM_TOKEN
          always-auth=true" > .npmrc
        env:
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
      - name: Confirm NPM Token
        run: cat .npmrc
      - name: Build And Deploy
        id: builddeploy
        uses: Azure/[email protected]
        with:
          app_build_command: "npm run build-production"
          azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_XXXXX }}
          repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
          action: "upload"
          ###### Repository/Build Configurations - These values can be configured to match you app requirements. ######
          # For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig
          app_location: "/" # App source code path
          api_location: "api" # Api source code path - optional
          output_location: "dist" # Built app content directory - optional
          ###### End of Repository/Build Configurations ######
        env:
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
          NPM_REGISTRY_URL: https://pkgs.dev.azure.com/XXXXX/_packaging/MYnpm/npm/registry/:_authToken=${{ secrets.NPM_TOKEN }}
          INPUT_REGISTRY-URL: https://pkgs.dev.azure.com/XXXXX/_packaging/MYnpm/npm/registry/:_authToken=${{ secrets.NPM_TOKEN }}

  close_pull_request_job:
    if: github.event_name == 'pull_request' && github.event.action == 'closed'
    runs-on: ubuntu-latest
    name: Close Pull Request Job
    steps:
      - name: Close Pull Request
        id: closepullrequest
        uses: Azure/[email protected]
        with:
          azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_XXXXX }}
          action: "close"

[rumurillo]

I'm on the same boat as @persyl.

This has been very frustrating -- I've got an Azure Static Web App trying to get it deployed through GitHub Actions that need to authenticate with our private NPM Registry stored in Azure DevOps but I always get the now deeply-hated
npm ERR! code E401 npm ERR! Unable to authenticate, need: *** Basic realm="https://pkgsprodcus1.pkgs.visualstudio.com/", TFS-Federated error.

I've tried every single tip/idea in this and lots of other posts/threads; however, I'm still stuck.

Has anyone found a solution/workaround?