Cli: Error on mdapi:deploy asking for location of "server.key"

[nutchanon-pho]

Summary

I have set up a Jenkins pipeline to automatically deploy to orgs using sfdx cli.
The errors which I have been having recently (they just started to happen since 21st May) is

ERROR running force:mdapi:deploy:  ENOENT: no such file or directory, open '/apps/tempjenkins/workspace/y_Package_Multi_Pipeline_develop@tmp/secretFiles/d446ee9f-22e7-4b4c-8f39-2117c88b29ac/server.key'

The authentication to the target org, which will always be performed as the first step for every pipeline job, was fine according to the log.

2019-06-28T03:01:57.428Z] + sfdx force:auth:jwt:grant --clientid **** --jwtkeyfile **** --username xxxxxxxxxxx --setalias sit --instanceurl https://test.salesforce.com

[2019-06-28T03:01:57.797Z] Successfully authorized xxxxxxxx with org ID 00D0p0000008agxxxx

JWT keyfile is supplied using withCredentials in this context.

The only workaround we could do is to uninstall sfdx and reinstall it and everything will work fine for a period of time before this error started coming up again.

Steps To Reproduce:

1. Try jenkins with the following snippet of code:

withCredentials([file(credentialsId: 'SIT-SECRET-FILE', variable: 'jwt_key_file')]) {
                    sh 'sfdx force:auth:jwt:grant --clientid ${sitConsumerKey} --jwtkeyfile ${jwt_key_file} --username xxxxxxxx --setalias sit --instanceurl https://test.salesforce.com'
                    sh "sfdx essentials:filter-metadatas -i ./src -o ./package -p ./src/package.xml"
                    sh(returnStdout: false, script: '''
                        if [ -d ./package/classes ] 
                        then 
                            sfdx force:mdapi:deploy -d ./package -u sit -l RunSpecifiedTests -w -1 -g -r \$(sh testclasses.sh ./package/classes)
                        else 
                            sfdx force:mdapi:deploy -d ./package -u sit -l NoTestRun -w -1 -g 
                        fi
                    '''.stripIndent())
                }

Remark: I only pasted a snippet of the problematic code here not the full code, because the full code would contain references to libraries and steps which are unique to my Jenkins environment.

1. Run the Jenkins job

Expected result

The mdapi:deploy command should work properly without errors.

Actual result

ERROR running force:mdapi:deploy:  ENOENT: no such file or directory, open '/apps/tempjenkins/workspace/y_Package_Multi_Pipeline_develop@tmp/secretFiles/d446ee9f-22e7-4b4c-8f39-2117c88b29ac/server.key'

The command failed with the error above without even reaching to Salesforce org.

Additional information

_Feel free to attach a screenshot_.

SFDX CLI Version sfdx-cli/7.13.0-27dbcb37d3 linux-x64 node-v10.15.3

[clairebianchi]

@nutchanon-pho Thank you, we are tracking this bug internally as it has been reported a few times. The internal work number is W-6086098.

[tnoonan-salesforce]

@nutchanon-pho Can you add sfdx force:auth:logout --username xxxxxxxx before running the JWT command? That will force it to regenerate a local auth file.

[twaugh2020]

Are there any work arounds for this issue? This also affects the force:org:create command

[clairebianchi]

@twaugh2020 Here is the Know Issue https://success.salesforce.com/issues_view?id=a1p3A000001SGxLQAW
It has the workaround documented

[PseudoDarwinist]

Hi,
Were you able to resolve this issue. I am facing the same issue?Please help!

[cumulus-robert]

Hi I've just encountered this. No changes to my jenkins file throughout the day while there were successful deployments earlier in the day. I did perform some git maintenance (merge ins from master), but shouldn't have affected Jenkins itself. One thing I noticed in output is it's trying to pull in data from a directory unrelated to the job. For example, I'm running a multi-branch Pipe that has base directory X_Partial. Jenkins is trying to reference the key file from another job titled X-Sandbox@tmp. I would expect it to pull in from X_Partial@tmp?? This is a big deal for me as it stops my ability to do any deployments. Hoping I can get to a resolution fast.

[nutchanon-pho]

What I did to resolve this issue is putting the jwt key files inside the repository to be persistently referenced by the Jenkinsfile. This approach is highly not recommended as it is not secured. It is meant to be a workaround.

[cumulus-robert]

Mines odd ...

It's trying to reference the key file from a completely different project. This is prompting me to think there is a cache of sorts on that Jenkins project. Trying to delete that project's folder now.

[nutchanon-pho]

Mines odd ...

It's trying to reference the key file from a completely different project. This is prompting me to think there is a cache of sorts on that Jenkins project. Trying to delete that project's folder now.

I did encountered something like this which is really unexpected and I suspected it might have something to do with Jenkins' way of handling files and caches itself.

Try what I recommended above on moving the JWT files inside the repository not the Jenkins. After that, try uninstalling the SFDX CLI following the steps here https://developer.salesforce.com/docs/atlas.en-us.sfdx_setup.meta/sfdx_setup/sfdx_setup_uninstall.htm

Then, try your deployment again.

[cumulus-robert]

That's the last thing I want to do!! I will need to do something to move forward though. I will need to revisit tomorrow. Thanks for the pointers.

[PseudoDarwinist]

That's the same thing i have started encountering, even deleting the previous project which it is referring does not resolve the issue.

[cumulus-robert]

Adding the key to repository doesn't work when I use Jenkins. I have to log in to the CLI and perform the SFDX commands manually. Jenkins is the problem here. There's nothing I can do to overcome this at this time other than performing a fresh install of Jenkins I believe. It's Jenkins that's trying to reference a file in a place that doesn't exist. This really sucks.

[twaugh2020]

You have to log out of the org you are authenticating to at the end of the Jenkins job and when the job starts you have to log in to authenticate. When you log out and log back in it forces jenkins to generate a new path to the JWT Key File. Here is an example that fixed this issue for me:

stages {

    stage('Deploy to DEV') {
        steps {
            withCredentials([file(credentialsId: 'JWT_KEY_CRED_ID', variable: 'jwt_key_file')]) {    
                bat "sfdx force:auth:jwt:grant --clientid ${CONSUMER_KEY} --username ${PROD_USERNAME} --jwtkeyfile ${jwt_key_file}  --setdefaultdevhubusername"
                bat "sfdx force:auth:jwt:grant --clientid ${DEV_CONSUMER_KEY} --username ${DEV_USERNAME} --jwtkeyfile ${jwt_key_file}  --instanceurl ${SFDC_SANDBOX_HOST} --setalias ${DEV_ALIAS}"
                bat "sfdx force:source:convert -d PortalBilling"
                bat "sfdx force:mdapi:deploy --deploydir PortalBilling -u ${DEV_ALIAS} --testlevel RunSpecifiedTests -r ${UNIT_TESTS}  --wait 30"

            }
        }
    }

    stage('Deploy to ITEST') {
        steps {
            withCredentials([file(credentialsId: 'JWT_KEY_CRED_ID', variable: 'jwt_key_file')]) {    
                echo "${PATH}"
                bat "sfdx force:mdapi:retrieve --retrievetargetdir Billing --targetusername ${DEV_USERNAME} -k ./ReleasePackages/deploymentPackage.xml"
                bat "sfdx force:auth:jwt:grant --clientid ${ITEST_CONSUMER_KEY} --username ${ITEST_USERNAME} --jwtkeyfile ${jwt_key_file}  --instanceurl ${SFDC_SANDBOX_HOST} --setalias ${ITEST_ALIAS}"
                bat "sfdx force:mdapi:deploy --zipfile Billing\\unpackaged.zip --testlevel NoTestRun -u ${ITEST_USERNAME} --wait 3"
            }
        }
    }


post {

    cleanup {
        cleanWs()
    }
    always {
        bat "sfdx force:auth:logout -u ${PROD_USERNAME} -p" 
        bat "sfdx force:auth:logout -u ${DEV_USERNAME} -p" 
        bat "sfdx force:auth:logout -u ${ITEST_USERNAME} -p"
    }
}

[cumulus-robert]

I've tried a couple versions of this, none of which produce the actual logout command AFTER it errors out. Perhaps I need a better understanding of the groovy syntax, but this just isn't working for me. I expanded on the groovy code put out by salesforce.com in the JWT Jenkins CI documentation. I'm not doing much else other than this. Any thoughts on how to force a log out? This script still produces this end result as of a moment ago:

`#!groovy
import groovy.json.JsonSlurperClassic
node {

def BUILD_NUMBER=env.BUILD_NUMBER
def RUN_ARTIFACT_DIR="tests/${BUILD_NUMBER}"
def SFDC_USERNAME
def WORKSPACE = env.WORKSPACE
def HUB_ORG_PROD='[email protected]'
def HUB_ORG='[email protected]'
def SFDC_HOST = env.sandbox
def SFDC_HOST_PROD = env.production
def JWT_KEY_CRED_ID = env.JWT_SERVER_KEY
println {$HUB_ORG}
def CONNECTED_APP_CONSUMER_KEY=env.ga_partial_key
def CONNECTED_APP_CONSUMER_KEY_PROD=env.ga_prod_key
def SFDX_USE_GENERIC_UNIX_KEYCHAIN = true
def SANDBOX_TEST_LEVEL='NoTestRun'
def PROD_TEST_LEVEL='RunLocalTests'        
def PACKAGE_XML = '/media/jenkins/properties/default/package.xml'

stage('checkout source') {
    // when running in multi-branch job, one must issue this command
    checkout scm
}

withCredentials([file(credentialsId: 'JWT_CRED_ID_DH', variable: 'jwt_key_file')]) {

    if(env.BRANCH_NAME == 'master'){
     stage("Upload"){
        // Artifact repository upload steps here
        }
     stage("Deploy"){
        // Deploy steps here
       }
    }

    if(env.BRANCH_NAME == 'partial'){
        stage("Authorize to Salesforce"){
            rc = sh returnStatus: true, script: "sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile server.key --setdefaultdevhubusername --instanceurl ${SFDC_HOST} --setalias ${HUB_ORG}"
            if (rc != 0) { error 'hub org authorization failed' }
        }
        stage("Convert to mdapi"){               
            rc = sh returnStatus: true, script: "mkdir mdapi"
            if (rc != 0) { error 'cannot create mdapi diretory' }
            rc = sh returnStatus: true, script: "sfdx force:source:convert -d mdapi"
            if (rc != 0) { error 'cannot convert source to mdapi' }
        }
        stage("Deploy To Partial Sandbox"){
            // Deploy steps here                
            rc = sh returnStatus: true, script: "sfdx force:mdapi:deploy --wait 120 --deploydir ${WORKSPACE}/mdapi --targetusername ${HUB_ORG} --testlevel ${SANDBOX_TEST_LEVEL}"
            if (rc != 0) {
                error 'Salesforce deploy and test run failed.'
            }
        }
        post {
            always {
                rc = sh returnStatus: true, script: "sfdx force:auth:logout -u ${HUB_ORG_PROD} -p"
                if (rc != 0) {
                        error 'Unable to log out of Production Org'
                    }
                rc = sh returnStatus: true, script: "sfdx force:auth:logout -u ${HUB_ORG} -p"
                if (rc != 0) {
                        error 'Unable to log out of Sandbox Org'
                    }
                //bat "sfdx force:auth:logout -u ${HUB_ORG_PROD} -p" 
                //bat "sfdx force:auth:logout -u ${HUB_ORG} -p"                 
            }
        }
    }
    if(env.BRANCH_NAME == 'prod_validate'){
        stage("Authorize to Salesforce"){
            rc = sh returnStatus: true, script: "sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY_PROD} --username ${HUB_ORG_PROD} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST_PROD} --setalias ${HUB_ORG_PROD}"
            if (rc != 0) { error 'hub org authorization failed' }
        }
        stage("Convert to mdapi"){                
            rc = sh returnStatus: true, script: "sfdx force:source:convert -d mdapi"
            if (rc != 0) { error 'cannot convert source to mdapi' }
        }
        stage("Run Production Validation"){
            // Deploy steps here                
            rc = sh returnStatus: true, script: "sfdx force:mdapi:deploy --wait 120 -c --deploydir ${WORKSPACE}/mdapi --targetusername ${HUB_ORG_PROD} --testlevel ${PROD_TEST_LEVEL}"
            if (rc != 0) {
                error 'Salesforce deploy and test run failed.'
                bat "sfdx force:auth:logout -u ${HUB_ORG} -p"                 
            }
        }
    }
    if(env.BRANCH_NAME == 'prod'){
        stage("Authorize to Salesforce"){
            rc = sh returnStatus: true, script: "sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY_PROD} --username ${HUB_ORG_PROD} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST_PROD} --setalias ${HUB_ORG_PROD}"
            if (rc != 0) { error 'hub org authorization failed' }
        }
        stage("Convert to mdapi"){                
            rc = sh returnStatus: true, script: "sfdx force:source:convert -d mdapi"
            if (rc != 0) { error 'cannot convert source to mdapi' }
        }
        stage("Run Production Deployment"){
            // Deploy steps here                
            rc = sh returnStatus: true, script: "sfdx force:mdapi:deploy --wait 120 --deploydir ${WORKSPACE}/mdapi --targetusername ${HUB_ORG_PROD} --testlevel ${PROD_TEST_LEVEL}"
            if (rc != 0) {
                error 'Salesforce deploy and test run failed.'
            }
        }
    }

}

}`

[PseudoDarwinist]

It's frustrating.I have been into it since long.tried everything.Tried what you had said but still the same error.

!groovy

import groovy.json.JsonSlurperClassic
node {

def BUILD_NUMBER=env.BUILD_NUMBER
def RUN_ARTIFACT_DIR="tests/${BUILD_NUMBER}"
def SFDC_USERNAME

def HUB_ORG=env.HUB_ORG_DH
def SFDC_HOST = env.SFDC_HOST_DH
def JWT_KEY_CRED_ID = env.JWT_CRED_ID_DH
def CONNECTED_APP_CONSUMER_KEY=env.CONNECTED_APP_CONSUMER_KEY_DH

println 'KEY IS' 
println JWT_KEY_CRED_ID
println HUB_ORG
println SFDC_HOST
println CONNECTED_APP_CONSUMER_KEY
def toolbelt = tool 'toolbelt'

stage('checkout source') {
    // when running in multi-branch job, one must issue this command
    checkout scm
}

withCredentials([file(credentialsId: JWT_KEY_CRED_ID, variable: 'jwt_key_file')]) {
    stage('Deploye Code') {
        if (isUnix()) {
            rc = sh returnStatus: true, script: "${toolbelt}/sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
        }else{
             rc = bat returnStatus: true, script: "\"${toolbelt}\"sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile \"${jwt_key_file}\" --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
        }
        if (rc != 0) { error 'hub org authorization failed' }

        println rc

        // need to pull out assigned username
        if (isUnix()) {
            rmsg = sh returnStdout: true, script: "${toolbelt}sfdx force:mdapi:deploy -d force-app/. -u ${HUB_ORG}"
        }else{
           rmsg = bat returnStdout: true, script: "\"${toolbelt}\"sfdx force:mdapi:deploy -d force-app/main/default/classes -u ${HUB_ORG}"
        }

        printf rmsg
        println('Hello from a Job DSL script!')
        println(rmsg)


        post {

            cleanup {
                cleanWs()
            }
            always {
                bat "sfdx force:auth:logout -u ${HUB_ORG} -p" 
            }
        }

    }
}

}

[cumulus-robert]

Yes it is frustrating that one day a script works and the next it doesn't when no changes were made to infrastructure and/or scripts. This is definitely an infrastructure issue specific to SFDX/Jenkins, IMHO. I'm very disappointed/frustrated to be going through this. I don't need to be spending time on this after having just moved my whole Jenkins architecture from ANT to SFDX in October 2019 because of the Eclipse retirement. I thought I was normalized and this was in the past. Apparently not. Scripts written and tested in October were working fine up until yesterday. There has to be a reason for this. It's unacceptable that my primary deployment tool is now a glorified web server with SFDX installed. If SFDX requires a log out, then the sample script provided by salesforce.com should include just that. Jenkins right now is a brick and unusable for my practice.

[shetzel]

The fix for this issue will be included in the patch release this Thursday, 1/9/2020. Apologies for the delay in getting the fix out and for all the CI headaches it's caused!

[PseudoDarwinist]

I just cant believe it..i uninstalled Jenkins completely and installed it afresh with all the plugins and error still refers to the old directory that even my memory does not remember.

:Program Files (x86)JenkinsworkspaceIberia_RM_Salesforce_>"C:Program FilesSalesforce CLIbin"sfdx force:auth:jwt:grant --clientid 3MVG9od6vNol.eMoqx7 --username [email protected] --jwtkeyfile "**" --setdefaultdevhubusername --instanceurl https://test.salesforce.com
Successfully authorized [email protected] with org ID
[Pipeline] echo
0
[Pipeline] isUnix
[Pipeline] bat
ERROR running force:mdapi:deploy: ENOENT: no such file or directory, open 'C:Program Files (x86)[e-mail]'

C:Program Files (x86)JenkinsworkspaceIberia_RM_Salesforce_>"C:Program FilesSalesforce CLIbin"sfdx force:mdapi:deploy -d force-app/main/default/classes -u niit.[email protected]
1108503 bytes written to C:WindowsTEMPclasses.zip using 7480.948ms
Deploying C:WindowsTEMPclasses.zip...

[PseudoDarwinist]

Can somebody please let me know the logic why jenkins is looking for server.key on that folder when i have uploaded it on Jenkins as global variable?

[cumulus-robert]

I believe this is the bug you’re seeing. I didn’t have any issues with my key being in a secret file referenced in script prior to this week. Something changed inside SFDX causing scripts to fail.

From: PseudoDarwinist [e-mail]
Reply-To: forcedotcom/cli [e-mail]
Date: Thursday, January 9, 2020 at 3:02 AM
To: forcedotcom/cli [e-mail]
Cc: cumulus-robert [e-mail], Comment [e-mail]
Subject: Re: [forcedotcom/cli] Error on mdapi:deploy asking for location of "server.key" (#124)

Can somebody please let me know the logic why jenkins is looking for server.key on that folder when i have uploaded it on Jenkins as global variable?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[cumulus-robert]

The ONLY workaround I’ve been able to put in is revert back to my ANT scripts for deployments. ANT still works great and was what I was using for CI prior to Oct-19. The challenge here is ANT uses username/password/token and SFDX uses OAUTH JWT Headless. You’d have to have two sets of credentials in Jenkins for the interim until this gets resolved.

Nothing else, no other feedback provided here allows the SFDX script to work again in my case. It just worked one day and stopped working the next. Unacceptable in my eyes, but at least I’m no longer dead in the water.

From: PseudoDarwinist [e-mail]
Reply-To: forcedotcom/cli [e-mail]
Date: Thursday, January 9, 2020 at 3:02 AM
To: forcedotcom/cli [e-mail]
Cc: cumulus-robert [e-mail], Comment [e-mail]
Subject: Re: [forcedotcom/cli] Error on mdapi:deploy asking for location of "server.key" (#124)

Can somebody please let me know the logic why jenkins is looking for server.key on that folder when i have uploaded it on Jenkins as global variable?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[PseudoDarwinist]

The fix for this issue will be included in the patch release this Thursday, 1/9/2020. Apologies for the delay in getting the fix out and for all the CI headaches it's caused!

Was the patch deployed?It was supposed to be on Thursday!!

[cumulus-robert]

I'm curious too. I just ran 'sfdx update' and says I'm on latest
version.Just ran my pipeline in Jenkins and continue to receive the error.

ERROR running force:mdapi:deploy: ENOENT: no such file or directory, open

On Fri, Jan 10, 2020 at 12:28 AM PseudoDarwinist [e-mail]
wrote:

The fix for this issue will be included in the patch release this
Thursday, 1/9/2020. Apologies for the delay in getting the fix out and for
all the CI headaches it's caused!

Was the patch deployed?It was supposed to be on Thursday!!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/forcedotcom/cli/issues/124?email_source=notifications&email_token=AB6JAJM4K76OJKGDGW4DOXLQ5AWT5A5CNFSM4H4B3KT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEITC2NI#issuecomment-572927285,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB6JAJK7R7GQYIZ2RB55IITQ5AWT5ANCNFSM4H4B3KTQ
.

--
Robert Brown
Senior Consultant

www.cumulusvision.com
Direct 415.463.1202
25 Taylor Street | San Francisco | CA | 94102

[clairebianchi]

@cumulus-robert @PseudoDarwinist, we are currently fixing the build for our installers that is causing an issue with sfdx update it should be fixed today but if you need the newest version of the CLI you can uninstall your CLI and use NPM install which does have the latest version https://developer.salesforce.com/docs/atlas.en-us.sfdx_setup.meta/sfdx_setup/sfdx_setup_install_cli.htm#sfdx_setup_install_cli_npm

[PseudoDarwinist]

Was it fixed?Do i need to uninstall my CLI and install it again because i keep receiving the same error.

[karanjakrpushkar]

Is this resolved yet. I am still facing the same error message . Do we need to uninstall CLI and reinstall ?Also I face this error message only for a deployment to a certain SF instance and not any other instance.

[cumulus-robert]

I got to say this is pretty disappointing. To have built a solution based on salesforce.com telling me I was losing Eclipse and Visual Studio is the way to go AND oh by the way it’s in a completely different format so you have to reorganize stuff AND here’s the tools (SFDX) to do so. I migrated our whole Jenkins instance using these tools. As a consultant operating with 10+ organizations, the only way I’m efficient is using Jenkins. This really broke the way we do things and to this day is continuing to cause me problems. The fact that this still does not work to this day and I’ve had to regress back to my old ANT scripts is frustrating to say the least. I’m faced with reverting back to my ANT scripts on less of a workaround basis until Salesforce.com can confidently say this has been resolved with step by step instructions to perform such resolution.

To add to this, the issue seems to be project specific. I’ve identified one of the orgs I support which is able to run: sfdx force:source:retrieve -p $WORKSPACE/force-app/main/default/ -u $username. This commands works just fine on one project, but get this error on another using the same command. Because of this, I’m unconvinced root cause has been identified at this time. I’m not convinced SFDX should be used inside Jenkins until this is fully resolved. Is this a wider spread issue? Is this happening ONLY on Jenkins? Are other pipeline tools experiencing same issue?

Is Jenkins really the issue here and not SFDX? How many other people here continue to be unable to use SFDX with their Jenkins instance?

From: karanjakrpushkar [e-mail]
Reply-To: forcedotcom/cli [e-mail]
Date: Monday, January 13, 2020 at 4:45 AM
To: forcedotcom/cli [e-mail]
Cc: cumulus-robert [e-mail], Mention [e-mail]
Subject: Re: [forcedotcom/cli] Error on mdapi:deploy asking for location of "server.key" (#124)

Is this resolved yet. I am still facing the same error message . Do we need to uninstall CLI and reinstall ?Also I face this error message only for a deployment to a certain SF instance and not any other instance.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[cumulus-robert]

This continues to be a problem for me. This morning I thought there was progress because jobs that were previously failing started working again. When I started to go remove my workarounds one by one, I found that other jobs were still failing and some weren't. Same behavior as my last comment.

I can run the exact same 'retrieve' command on a job having the exact same jenkins config. I created a new one off of the one that was working just to confirm while only changing repository and credential details, leaving script and build steps untouched.

There is nothing special about this script. It's just doing a source retrieve. The issue I have simply is the source retrieve works on one project and not the other.

ssh -T [email protected]
/opt/bitnami/git/bin/git config --global user.email "[email protected]"
/opt/bitnami/git/bin/git config --global user.name "Jenkins"
sfdx force:auth:jwt:grant --clientid $CONNECTED_APP_CONSUMER_KEY --username $username --jwtkeyfile //$jwt_key_file --setdefaultdevhubusername --instanceurl $sandbox -a $username
cat sfdx-project.json|jq '.sfdcLoginUrl = "https://test.salesforce.com"' > sfdx-project.json1
mv sfdx-project.json1 sfdx-project.json
sfdx force:source:retrieve -p $WORKSPACE/force-app/main/default/ -u $username
/opt/bitnami/git/bin/git status
/opt/bitnami/git/bin/git add .
/opt/bitnami/git/bin/git commit -a -m "Pulled down changes from partial sandbox." || true

Is anyone else continuing to have a problem?

[clairebianchi]

@cumulus-robert, I am so sorry that this is still happening. Is it the exact same error? Are you on CLI version sfdx-cli 7.42.1 and salesforcedx 47.15.0?

[cumulus-robert]

Hi @clairebianchi - Yes, same error. I had updated version last week to 7.41.0. I just updated version again and confirmed it is now 7.42.1. I can confirm too that salesforcedx is version 47.15.

Version summary:

Jenkins console output:

I built these two job right now on version 7.42.1. One works and one doesn't. Same script, different Jenkins project. Different repository. Note on the failure how it is looking for a key file in a completely different location than the current project. This is the same behavior as when I first encountered this problem at the beginning of the month. No change to my infrastructure and had been working fine prior to December/January.

[amphro]

I'm late to this thread but I'm hoping I can provide some additional context and help to get to the bottom of this.

The Salesforce CLI stores all authentication in the <home>/.sfdx folder. When the JWT auth command is run we do NOT override the auth file for existing usernames which is why I think someone suggested to logout first. This means if multiple jobs are using the same devhub and/or the Jenkins workspace changes, the path to the JWT file will be different during the withCredentials but it will NOT update the auth file. When we need to exchange an expired access token for a new one, we will go look in the auth file and use the path to the JWT file to request a token. If the auth file is pointing to a JWT key that is in a different project and doesn't exist because you are no longer withing the withCredential form that job, it will fail with this error.

Can someone try to set the HOME directory to the Jenkins workspace at the beginning of the job?
For example, set the environment variable HOME=env.WORKSPACE for the entire job? Or you could create a groovy function that did something like:

// withHome.groovy

/**
 * Wrap given closure in a <code>withEnv</code> setting common sfdx env vars to the dir given by <code>home</code>
 * 
 * Home default value is env.WORKSPACE and if both are null the withHome does not map env vars.
 * 
 * <br>Env Vars assigned to home
 * 
 * <li>HOME
 * <li>APPDATA
 * <li>USERPROFILE
 * <li>XDG_DATA_HOME
 * </br>
 * @param home
 * @param additions - list of additional env vars to apply to doWithHome
 * @param doWithHome
 * @return
 */
def call(home = null, additions = [], Closure doWithHome) {
    if (doWithHome == null) {
        error("withHome doWithHome closure cannot be null")
    }
    def useThisHome = home ?: env.WORKSPACE
    def envParams = (useThisHome != null) ? [
            "HOME=${useThisHome}",
            "APPDATA=${useThisHome}",
            "USERPROFILE=${useThisHome}",
            "XDG_DATA_HOME=${useThisHome}"
    ] : []
    debug "home: ${home}, env.WORKSPACE: ${env.WORKSPACE}, useThisHome: ${useThisHome}, withHome envParams: ${envParams}"
    envParams.addAll(additions)
    return withEnv(envParams) { doWithHome.call(); }
}

Then use it like you do withCredentials...

withHome() {
  withCredentials([file(credentialsId: ... variable: 'jwt_key_file')]) {
    sh 'sfdx force:auth:jwt:grant --clientid ${sitConsumerKey} --jwtkeyfile ${jwt_key_file} ...'
    ....
  }            
}

If that turns out to be the problem, maybe the CLI can also provide additional options to store auth files in a specific location something similar.

[cumulus-robert]

My issue has been resolved with the feedback of ensuring scripts are setting home environment local to the job vs. global home environment.

Wrapping my 'With credentials' groovy script with:

withEnv(["HOME=${env.WORKSPACE}"]) {
    withCredentials(
    ....
    )
}

For Jenkins jobs outside of pipeline, adding home environment specification (export HOME=$WORKSPACE) to the script fixed my outside Jenkins jobs:

ssh -T [email protected]
/opt/bitnami/git/bin/git config --global user.email "[email protected]"
/opt/bitnami/git/bin/git config --global user.name "Jenkins"
export HOME=$WORKSPACE
sfdx force:auth:jwt:grant --clientid $CONNECTED_APP_CONSUMER_KEY --username $username --jwtkeyfile //$jwt_key_file --setdefaultdevhubusername --instanceurl $sandbox -a $username
#make sure project file points to sandbox
cat sfdx-project.json|jq '.sfdcLoginUrl = "https://test.salesforce.com"' > sfdx-project.json1
mv sfdx-project.json1 sfdx-project.json
#retrieve source code from sandbox/prod
sfdx force:source:retrieve -p $WORKSPACE/force-app/main/default/ -u $username
#commit any changes found to version control
/opt/bitnami/git/bin/git status
/opt/bitnami/git/bin/git add .
/opt/bitnami/git/bin/git commit -a -m "Pulled down changes from full sandbox." || true

Thank you to @clairebianchi and team for taking the time to review and ultimately resolve my issue.

[clairebianchi]

@cumulus-robert thank you for the update. We are updating our Jenkins documentation with this information. I am going to close this issue now.

[PseudoDarwinist]

How would fix the below script. I am still receiving the error.I have tried exporting workspace but i think i am doing it wrong.

!groovy

import groovy.json.JsonSlurperClassic
node {

def BUILD_NUMBER=env.BUILD_NUMBER
def RUN_ARTIFACT_DIR="tests/${BUILD_NUMBER}"
def SFDC_USERNAME

def HUB_ORG=env.HUB_ORG_DH
def SFDC_HOST = env.SFDC_HOST_DH
def JWT_KEY_CRED_ID = env.JWT_CRED_ID_DH
def CONNECTED_APP_CONSUMER_KEY=env.CONNECTED_APP_CONSUMER_KEY_DH

println 'KEY IS' 
println JWT_KEY_CRED_ID
println HUB_ORG
println SFDC_HOST
println CONNECTED_APP_CONSUMER_KEY
def toolbelt = tool 'toolbelt'

stage('checkout source') {
    // when running in multi-branch job, one must issue this command
    checkout scm
}

withCredentials([file(credentialsId: JWT_KEY_CRED_ID, variable: 'jwt_key_file')]) {
    stage('Deploye Code') {
        if (isUnix()) {
            rc = sh returnStatus: true, script: "${toolbelt} force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
        }else{
             rc = bat returnStatus: true, script: "\"${toolbelt}\" force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile \"${jwt_key_file}\" --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
        }
        if (rc != 0) { error 'hub org authorization failed' }

        println rc

        // need to pull out assigned username
        if (isUnix()) {
            rmsg = sh returnStdout: true, script: "${toolbelt} force:mdapi:deploy -d manifest/. -u ${HUB_ORG}"
        }else{
           rmsg = bat returnStdout: true, script: "\"${toolbelt}\" force:mdapi:deploy -d manifest/. -u ${HUB_ORG}"
        }

        printf rmsg
        println('Hello from a Job DSL script!')
        println(rmsg)
    }
}

}

[cumulus-robert]

try this:

`#!groovy
import groovy.json.JsonSlurperClassic
node {

def BUILD_NUMBER=env.BUILD_NUMBER
def RUN_ARTIFACT_DIR="tests/${BUILD_NUMBER}"
def SFDC_USERNAME

def HUB_ORG=env.HUB_ORG_DH
def SFDC_HOST = env.SFDC_HOST_DH
def JWT_KEY_CRED_ID = env.JWT_CRED_ID_DH
def CONNECTED_APP_CONSUMER_KEY=env.CONNECTED_APP_CONSUMER_KEY_DH

println 'KEY IS'
println JWT_KEY_CRED_ID
println HUB_ORG
println SFDC_HOST
println CONNECTED_APP_CONSUMER_KEY
def toolbelt = tool 'toolbelt'

stage('checkout source') {
// when running in multi-branch job, one must issue this command
checkout scm
}

withEnv(["HOME=${env.WORKSPACE}"]) {

withCredentials([file(credentialsId: JWT_KEY_CRED_ID, variable: 'jwt_key_file')]) {
    stage('Deploye Code') {
        if (isUnix()) {
            rc = sh returnStatus: true, script: "${toolbelt} force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
        }else{
             rc = bat returnStatus: true, script: "\"${toolbelt}\" force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile \"${jwt_key_file}\" --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
        }
        if (rc != 0) { error 'hub org authorization failed' }

        println rc

        // need to pull out assigned username
        if (isUnix()) {
            rmsg = sh returnStdout: true, script: "${toolbelt} force:mdapi:deploy -d manifest/. -u ${HUB_ORG}"
        }else{
           rmsg = bat returnStdout: true, script: "\"${toolbelt}\" force:mdapi:deploy -d manifest/. -u ${HUB_ORG}"
        }

        printf rmsg
        println('Hello from a Job DSL script!')
        println(rmsg)
    }
}

}
}`

[PseudoDarwinist]

Uggghh..Still the same error :(
ERROR running force:mdapi:deploy: ENOENT: no such file or directory, open 'C:Program Files (x86)[e-mail]'

C:Program Files (x86)JenkinsworkspaceLocalJenkins_>"C:Program FilesSalesforce CLIbinsfdx" force:mdapi:deploy -d manifest/. -u niit.csingh
318 bytes written to C:WindowsTEMP..zip using 114.021ms
Deploying C:WindowsTEMP..zip...
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

[PseudoDarwinist]

used this script:

!groovy

import groovy.json.JsonSlurperClassic
node {

def BUILD_NUMBER=env.BUILD_NUMBER
def RUN_ARTIFACT_DIR="tests/${BUILD_NUMBER}"
def SFDC_USERNAME

def HUB_ORG=env.HUB_ORG_DH
def SFDC_HOST = env.SFDC_HOST_DH
def JWT_KEY_CRED_ID = env.JWT_CRED_ID_DH
def CONNECTED_APP_CONSUMER_KEY=env.CONNECTED_APP_CONSUMER_KEY_DH

println 'KEY IS'
println JWT_KEY_CRED_ID
println HUB_ORG
println SFDC_HOST
println CONNECTED_APP_CONSUMER_KEY
def toolbelt = tool 'toolbelt'

stage('checkout source') {
// when running in multi-branch job, one must issue this command
checkout scm
}

withEnv(["HOME=${env.WORKSPACE}"]) {

withCredentials([file(credentialsId: JWT_KEY_CRED_ID, variable: 'jwt_key_file')]) {
stage('Deploye Code') {
if (isUnix()) {
rc = sh returnStatus: true, script: "${toolbelt} force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
}else{
rc = bat returnStatus: true, script: ""${toolbelt}" force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile "${jwt_key_file}" --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
}
if (rc != 0) { error 'hub org authorization failed' }

    println rc

    // need to pull out assigned username
    if (isUnix()) {
        rmsg = sh returnStdout: true, script: "${toolbelt} force:mdapi:deploy -d manifest/. -u ${HUB_ORG}"
    }else{
       rmsg = bat returnStdout: true, script: "\"${toolbelt}\" force:mdapi:deploy -d manifest/. -u ${HUB_ORG}"
    }

    printf rmsg
    println('Hello from a Job DSL script!')
    println(rmsg)
}

}
}
}

[cumulus-robert]

hi @clairebianchi -- this issue crept back up for me. I had to rebuild my Jenkins server, but was sure to install sfdx 47.15.0 during rebuild. This was the version that scripts worked on before the rebuild. Now with the same scripts prior to the rebuild, I'm getting this:

• sfdx force:mdapi:deploy --wait 120 -c --deploydir /opt/bitnami/jenkins/jenkins_home/workspace/TAN-pipe_prod_validate/mdapi --targetusername robert.[email protected] --testlevel RunLocalTests
  ERROR running force:mdapi:deploy: We encountered a JSON web token error, which is likely not an issue with Salesforce CLI. Here’s the error: ENOENT: no such file or directory, open '/opt/bitnami/jenkins/jenkins_home/workspace/TAN-pipe_prod_validate@tmp/secretFiles/ef828807-fd4d-41a0-87da-554870598fce/server.key'

robert@jenkins-4-vm:~$ sfdx plugins --core
@oclif/plugin-autocomplete 0.1.5 (core)
@oclif/plugin-commands 1.2.3 (core)
@oclif/plugin-help 2.2.3 (core)
@oclif/plugin-not-found 1.2.3 (core)
@oclif/plugin-plugins 1.7.9 (core)
@oclif/plugin-update 1.3.9 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/sfdx-trust 3.0.7 (core)
analytics 1.7.1 (core)
generator 1.1.2 (core)
salesforcedx 47.15.0 (47.15.0)
├─ salesforcedx-templates 47.16.0
└─ salesforce-alm 47.14.0

The environment variables still exist that were added February. Is there any other details than can be shared to potentially overcome this?

This is a new Jenkins instance on Google Cloud Console Marketplace, Jenkins Certified by Bitnami. The previous instance was the same.

[cumulus-robert]

@clairebianchi - Sorry for false alarm. Was worried I'd be in another month long no Jenkins scenario at first. It seems there was opportunity in order of operations. I think because I have to install SFDX first with current version, then revert to previous version, any Jenkins project created while on current version needed to be deleted. I deleted my projects and recreated. It seems to be working again.

It is interesting however, that the default install of SFDX is producing this same behavior with Jenkins again, even with the environment variables being set for linux.

[cumulus-robert]

Hi @clairebianchi - I want to mention that I continue to have issues with the previous comment. Every day now, I have to reset pipes inside Jenkins because of this failure:

sfdx force:mdapi:deploy --wait 120 -c --deploydir /opt/bitnami/jenkins/jenkins_home/workspace/TAN-pipe_prod_validate/mdapi --targetusername robert.[email protected] --testlevel RunLocalTests
ERROR running force:mdapi:deploy: We encountered a JSON web token error, which is likely not an issue with Salesforce CLI. Here’s the error: ENOENT: no such file or directory, open '/opt/bitnami/jenkins/jenkins_home/workspace/TAN-pipe_prod_validate@tmp/secretFiles/ef828807-fd4d-41a0-87da-554870598fce/server.key'

Because I have a workaround (deleting and recreating project), it's not urgent but do want to figure out as I have have to delete and recreate projects every day now inside Jenkins.

I'm struggling to understanding why this is happening.

Has anyone seen anything like this. It can be recreated daily. Is there any additional pointers you or team have to address this?

[clairebianchi]

@cumulus-robert can you open a new issue and reference this one? That will bubble this up to the top and we can take a look. I have not heard of this happening to anyone else, sorry it is an issue again.

[ThomasCreemers]

@clairebianchi
we are experience the exact same issue. Just freshly installed the latest SFDX Cli on a new t3.medium AWS Linux 2 Server.
Build ran perfect for 20 runs or so. Then suddenly this error is popping up.

/usr/local/bin/sfdx force:mdapi:deploy --targetusername jenkins@deloitte_sf_project.com.test --deploydir mdapi/target/src --testlevel RunLocalTests --json --ignorewarnings
{
"status": 1,
"name": "JwtGrantError",
"message": "We encountered a JSON web token error, which is likely not an issue with Salesforce CLI. Here’s the error: ENOENT: no such file or directory, open '/var/lib/jenkins/workspace/SF_Project@tmp/secretFiles/4f5a2156-a68c-4d6a-97aa-e4f0215ebb8b/server.key'",
"exitCode": 1,
"commandName": "MdapiDeployCommand",
"stack": "JwtGrantError: We encountered a JSON web token error, which is likely not an issue with Salesforce CLI. Here’s the error: ENOENT: no such file or directory, open '/var/lib/jenkins/workspace/SF_Project@tmp/secretFiles/4f5a2156-a68c-4d6a-97aa-e4f0215ebb8b/server.key'n at Function.create (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/@salesforce/core/lib/sfdxError.js:141:16)n at fsReadFile.then.then.catch.error (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/lib/core/force.js:201:32)n at tryCatcher (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/util.js:16:23)n at Promise._settlePromiseFromHandler (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/promise.js:517:31)n at Promise._settlePromise (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/promise.js:574:18)n at Promise._settlePromise0 (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/promise.js:619:10)n at Promise._settlePromises (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/promise.js:695:18)n at _drainQueueStep (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/async.js:138:12)n at _drainQueue (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/async.js:131:9)n at Async._drainQueues (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/async.js:147:5)n at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/bluebird/js/release/async.js:17:14)n at runCallback (timers.js:705:18)n at tryOnImmediate (timers.js:676:5)n at processImmediate (timers.js:658:5)",
"warnings": []
}

[pawan22sfdl]

Can anyone let me know what is final resolution for the issue ??
ERROR running force:mdapi:deploy: ENOENT: no such file or directory

[pawan22sfdl]

@clairebianchi can you please let me know what is the final solution for the issue as its very very urgent for me to solve the below error
ERROR running force:mdapi:deploy: ENOENT: no such file or directory

[cumulus-robert]

@pawan22sfdl @ThomasCreemers This ticket is closed. You may want to either create a new one for yourself and/or add to the one I created the other day (https://github.com/forcedotcom/cli/issues/363). This is actually the second time in 3 months I've experienced this issue with Jenkins/SFDX. It seems to me there are environmental issues with SFDX. I'm getting nowhere with Jenkins / Bitnami. Everything leads to SFDX being the culprit. I'm about to go back to my ANT solution in Jenkins as SFDX is still not ready for continuous integration, IMHO. Or, at very least platform testing for solutions like Jenkins (or other pipeline tools) isn't rigorous enough prior to a new SFDX version release. I had to go back to version SFDX version 47.0.15 as the default install of SFDX WILL NOT work. I immediately get this error. If I go back to version 47.0.15, I at least have a partially working solution. I can go a day without getting the error, but then have to perform maintenance on Jenkins server in order for scripts to work again.

[bobteplitsky]

Doing the following fixed this issue for me:

• wrapping all stages in withEnv
• wrapping all stages in withCredentials
• adding force:auth:logout --noprompt -u username prior to force:auth:grant