I'm using the Deploy Azure App Service task to deploy a nodejs application provided as a zip file.
The process always fails with the error "EMFILE: too many open files, open d:\a\r1\a\temp_web_package_..."
@nlips , Thanks for reporting the issue.
Can you please share your debug logs (by enabling system.debug=true) ?
2017-04-26T08:13:53.9630134Z ##[section]Starting: Deploy Azure App Service
2017-04-26T08:13:53.9740136Z ==============================================================================
2017-04-26T08:13:53.9740136Z Task : Azure App Service Deploy
2017-04-26T08:13:53.9740136Z Description : Update Azure App Service using Web Deploy / Kudu REST APIs
2017-04-26T08:13:53.9740136Z Version : 3.3.0
2017-04-26T08:13:53.9740136Z Author : Microsoft Corporation
2017-04-26T08:13:53.9740136Z Help : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-04-26T08:13:53.9740136Z ==============================================================================
2017-04-26T08:13:55.1739231Z ##[debug]agent.workFolder=d:\a
2017-04-26T08:13:55.1739231Z ##[debug]loading inputs and endpoints
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_1f5cf8f5-bcdd-4930-95cb-397eb569f6ef
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_1f5cf8f5-bcdd-4930-95cb-397eb569f6ef_SERVICEPRINCIPALID
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_1f5cf8f5-bcdd-4930-95cb-397eb569f6ef_SERVICEPRINCIPALKEY
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_1f5cf8f5-bcdd-4930-95cb-397eb569f6ef_TENANTID
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_SCHEME_1f5cf8f5-bcdd-4930-95cb-397eb569f6ef
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2017-04-26T08:13:55.1739231Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2017-04-26T08:13:55.1739231Z ##[debug]loading INPUT_CONNECTEDSERVICENAME
2017-04-26T08:13:55.1739231Z ##[debug]loading INPUT_DEPLOYTOSLOTFLAG
2017-04-26T08:13:55.1739231Z ##[debug]loading INPUT_EXCLUDEFILESFROMAPPDATAFLAG
2017-04-26T08:13:55.1739231Z ##[debug]loading INPUT_GENERATEWEBCONFIG
2017-04-26T08:13:55.1739231Z ##[debug]loading INPUT_PACKAGE
2017-04-26T08:13:55.1739231Z ##[debug]loading INPUT_REMOVEADDITIONALFILESFLAG
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_RENAMEFILESFLAG
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_SCRIPTPATH
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_SERVERFARMID
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_SETPARAMETERSFILE
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_TAKEAPPOFFLINEFLAG
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_USEWEBDEPLOY
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_WEBAPPKIND
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_WEBAPPNAME
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_WEBCONFIGPARAMETERS
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_XMLTRANSFORMATION
2017-04-26T08:13:55.1749236Z ##[debug]loading INPUT_XMLVARIABLESUBSTITUTION
2017-04-26T08:13:55.1749236Z ##[debug]loaded 25
2017-04-26T08:13:55.1749236Z ##[debug]AZURE_HTTP_USER_AGENT=VSTS_fd46871c-8676-480a-acd1-bd9e9a77f181_release_1_7_7_1
2017-04-26T08:13:55.1749236Z ##[debug]AZURE_HTTP_USER_AGENT=VSTS_fd46871c-8676-480a-acd1-bd9e9a77f181_release_1_7_7_1
2017-04-26T08:13:55.1749236Z ##[debug]check path : d:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.3.0\task.json
2017-04-26T08:13:55.1749236Z ##[debug]set resource file to: d:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.3.0\task.json
2017-04-26T08:13:55.1749236Z ##[debug]system.culture=en-US
2017-04-26T08:13:55.1749236Z ##[debug]ConnectedServiceName=1f5cf8f5-bcdd-4930-95cb-397eb569f6ef
2017-04-26T08:13:55.1749236Z ##[debug]WebAppName=[MyWebAppService]
2017-04-26T08:13:55.1749236Z ##[debug]DeployToSlotFlag=false
2017-04-26T08:13:55.1749236Z ##[debug]ResourceGroupName=null
2017-04-26T08:13:55.1749236Z ##[debug]SlotName=null
2017-04-26T08:13:55.1749236Z ##[debug]Package=d:\a\r1\a\**\*.zip
2017-04-26T08:13:55.1749236Z ##[debug]VirtualApplication=null
2017-04-26T08:13:55.1749236Z ##[debug]UseWebDeploy=false
2017-04-26T08:13:55.1749236Z ##[debug]SetParametersFile=d:\a\r1\a
2017-04-26T08:13:55.1749236Z ##[debug]RemoveAdditionalFilesFlag=false
2017-04-26T08:13:55.1749236Z ##[debug]ExcludeFilesFromAppDataFlag=false
2017-04-26T08:13:55.1759233Z ##[debug]TakeAppOfflineFlag=true
2017-04-26T08:13:55.1759233Z ##[debug]RenameFilesFlag=false
2017-04-26T08:13:55.1759233Z ##[debug]AdditionalArguments=null
2017-04-26T08:13:55.1759233Z ##[debug]WebAppUri=null
2017-04-26T08:13:55.1759233Z ##[debug]XmlTransformation=false
2017-04-26T08:13:55.1759233Z ##[debug]JSONFiles=null
2017-04-26T08:13:55.1759233Z ##[debug]XmlVariableSubstitution=false
2017-04-26T08:13:55.1759233Z ##[debug]ScriptType=null
2017-04-26T08:13:55.1759233Z ##[debug]InlineScript=null
2017-04-26T08:13:55.1759233Z ##[debug]ScriptPath=d:\a\r1\a
2017-04-26T08:13:55.1759233Z 1f5cf8f5-bcdd-4930-95cb-397eb569f6ef exists true
2017-04-26T08:13:55.1759233Z ##[debug]1f5cf8f5-bcdd-4930-95cb-397eb569f6ef exists true
2017-04-26T08:13:55.1759233Z ##[debug]GenerateWebConfig=true
2017-04-26T08:13:55.1759233Z ##[debug]WebConfigParameters=-Handler iisnode -NodeStartFile server.js -appType node
2017-04-26T08:13:55.1759233Z ##[debug]WebAppKind=app
2017-04-26T08:13:55.1759233Z ##[debug]DockerNamespace=null
2017-04-26T08:13:55.1759233Z ##[debug]1f5cf8f5-bcdd-4930-95cb-397eb569f6ef auth param serviceprincipalid = ********
2017-04-26T08:13:55.1759233Z ##[debug]1f5cf8f5-bcdd-4930-95cb-397eb569f6ef auth param serviceprincipalkey = ********
2017-04-26T08:13:55.1759233Z ##[debug]1f5cf8f5-bcdd-4930-95cb-397eb569f6ef auth param tenantid = ********
2017-04-26T08:13:55.1759233Z ##[debug]1f5cf8f5-bcdd-4930-95cb-397eb569f6ef data subscriptionid = [MySubscriptionId]
2017-04-26T08:13:55.1759233Z ##[debug]1f5cf8f5-bcdd-4930-95cb-397eb569f6ef data environmentAuthorityUrl = https://login.windows.net/
2017-04-26T08:13:55.1759233Z ##[debug]1f5cf8f5-bcdd-4930-95cb-397eb569f6ef=https://management.azure.com/
2017-04-26T08:13:55.1759233Z ##[debug]Requesting for Auth Token: https://login.windows.net/********/oauth2/token/
2017-04-26T08:13:56.1638978Z ##[debug]Requesting Azure App Service ID: https://management.azure.com/subscriptions/[MySubscriptionId]/resources?$filter=resourceType EQ 'Microsoft.Web/Sites' AND name EQ '[MyWebAppService]'&api-version=2016-07-01
2017-04-26T08:13:56.1638978Z ##[debug]Web App details : /subscriptions/[MySubscriptionId]/resourceGroups/[MyResourceGroup]/providers/Microsoft.Web/sites/[MyWebAppService]
2017-04-26T08:13:56.1638978Z ##[debug]Azure Resource Group Name : [MyResourceGroup]
2017-04-26T08:13:56.1638978Z ##[debug]Requesting for Auth Token: https://login.windows.net/********/oauth2/token/
2017-04-26T08:13:56.1638978Z ##[debug]Requesting Azure Publish Profile: https://management.azure.com/subscriptions/[MySubscriptionId]/resourceGroups/[MyResourceGroup]/providers/Microsoft.Web/sites/[MyWebAppService]/publishxml?api-version=2016-08-01
2017-04-26T08:13:56.1638978Z Got connection details for Azure App Service:'[MyWebAppService]'
2017-04-26T08:13:56.1638978Z ##[debug]Performing the deployment of webapp.
2017-04-26T08:13:56.1638978Z ##[debug]Finding files matching input: d:\a\r1\a\**\*.zip
2017-04-26T08:13:56.1638978Z ##[debug]Matching glob pattern: d:\a\r1\a\**\*.zip
2017-04-26T08:13:56.1638978Z ##[debug]Index of first wildcard: 10
2017-04-26T08:13:56.1638978Z ##[debug]find root dir: d:\a\r1\a\
2017-04-26T08:13:56.1638978Z ##[debug]findPath: 'd:\a\r1\a\'
2017-04-26T08:13:56.1638978Z ##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
2017-04-26T08:13:56.1638978Z ##[debug]findOptions.followSymbolicLinks: 'true'
2017-04-26T08:13:56.1638978Z ##[debug] d:\a\r1\a\ (directory)
2017-04-26T08:13:56.1638978Z ##[debug] d:\a\r1\a\[MyBuildDefinitionName] (directory)
2017-04-26T08:13:56.1638978Z ##[debug] d:\a\r1\a\[MyBuildDefinitionName]\drop (directory)
2017-04-26T08:13:56.1648983Z ##[debug] d:\a\r1\a\[MyBuildDefinitionName]\drop\9.zip (file)
2017-04-26T08:13:56.1648983Z ##[debug]4 results
2017-04-26T08:13:56.1648983Z ##[debug]patternRoot: ''
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.debug: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.nobrace: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.noglobstar: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.dot: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.noext: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.nocase: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.nonull: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.matchBase: 'true'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.nocomment: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.nonegate: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]matchOptions.flipNegate: 'undefined'
2017-04-26T08:13:56.1648983Z ##[debug]pattern: 'd:\a\r1\a\**\*.zip'
2017-04-26T08:13:56.1648983Z ##[debug]expanding braces
2017-04-26T08:13:56.1648983Z ##[debug]pattern: 'd:/a/r1/a/**/*.zip'
2017-04-26T08:13:56.1648983Z ##[debug]applying include pattern against original list
2017-04-26T08:13:56.1648983Z ##[debug]1 matches
2017-04-26T08:13:56.1648983Z ##[debug]1 final results
2017-04-26T08:13:56.1648983Z ##[debug]System.DefaultWorkingDirectory=d:\a\r1\a
2017-04-26T08:13:56.1648983Z ##[debug]extracting d:\a\r1\a\[MyBuildDefinitionName]\drop\9.zip to d:\a\r1\a\temp_web_package_5957852916326374
2017-04-26T08:14:05.3019000Z ##[debug]task result: Failed
2017-04-26T08:14:05.3078983Z ##[error]Error: EMFILE: too many open files, open 'd:\a\r1\a\temp_web_package_5957852916326374\modulaar\node_modules\make-error-cause\dist\index.spec.d.ts'
2017-04-26T08:14:05.3078983Z ##[debug]Processed: ##vso[task.issue type=error;]Error: EMFILE: too many open files, open 'd:\a\r1\a\temp_web_package_5957852916326374\modulaar\node_modules\make-error-cause\dist\index.spec.d.ts'
2017-04-26T08:14:05.3098981Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: EMFILE: too many open files, open 'd:\a\r1\a\temp_web_package_5957852916326374\modulaar\node_modules\make-error-cause\dist\index.spec.d.ts'
@nlips , thanks for the logs.
We recently found an issue with our zip extraction module with large zip files.
For a workaround, Can you please add the web.config file in your package ? In this way, the zip package is not extracted and it proceeds with the deployment.
@vincentdass How does something like this get fixed? Would it be a new agent?
I'm seeing the same exact problem, but I'm not clear about your web.config comment since I already have web.config in the root of my zipfile.
Can you provide further details of the workaround. Thanks
This is my rough directory structure inside my zip (that is being extracted in my case to C:\vstsagent\DEPLOY-RRCRM_work\r1\a\temp_web_package_6270807555411011)
bin
ClientApp
node_modules
wwwroot
web.config
@simeyla , if any options in File Transforms & Variable Substitution Options is enabled, the zip package is extracted
to a temporary location to perform the selected operation ( In case of folder, it is copied
to temporary location).
After performing the above operation, the extracted package is compressed to a zip package. (To maintain the same package structure). Currently, the task uses 'Decompress-zip' node module to perform the extraction and for larger packages, the module seems to fail with EMFILE error.
For @nlips , Generate Web.config option is enabled and to unblock him, we recommended to have the web.config file inside the package and disable the option.
@simeyla and @nlips , sorry for the late response.
For windows, this issue is resolved with 2.117.1 agent version.
Can you please check the hosted agent version and rerun the release ?
@vinca thanks. I actually managed to dramatically reduce the size of my
release to get around this issue. Glad to see it is fixed but I won't be
able to verify it at this time.
On Tue, Jun 13, 2017 at 4:41 AM, Vincent A (vinca) <[email protected]
wrote:
@simeyla https://github.com/simeyla and @nlips
https://github.com/nlips , sorry for the late response.
For windows, this issue is resolved with 2.117.1 agent version.
Can you please check the hosted agent version and rerun the release ?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vsts-tasks/issues/4127#issuecomment-308089855,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADfKVhvo8YZVR4FnsgRRYcP4Rlk0xNBgks5sDnVtgaJpZM4NIHVM
.
Summarizing the fix for reference :
1) Increase the max. limit of open file descriptors for the account where the automation agent is running.
2) Use Extract file
task to decompress the web package zip to temporary location and use the same for web package in Azure App Service Deploy Task
.
@vincentdass I just ran into this issue today when trying to deploy my nodejs application as a AzureWebService. My template application was working fine but once started adding modules it couldn't deploy.
Deploy Azure App Service is generating my web.config file
Your alternate solution to Extract Files first was what I did and it has resolved the issue.
Just wanted to point out that it seems like either the issue came back in 3.* or it wasn't resolved in 2.*.
@skylerjokiel , thanks for reaching out.
Can you please share the following details for debugging?
system.debug
variable to true)I am still getting this issue with a 2.117.1 agent on a windows platform
here are my debug logs
debuglog.txt
zip size: 47.6 MB
@hendecasyllabic , thanks for reporting the issue.
To unblock, can you try any of the following workarounds?
- Create a zip/folder that contains only core app content. (without node_modules)
- Run **npm install** from post deployment script to install app dependencies.
I have currently got it working using the extract file work around. I might try the post deployment script as it might be faster. thanks.
Same issue here. I'm using the 3.* agent version. Got it working using extract file work around.
@hilfor , Can you please share the logs and Package size?
@vincentdass The last logs from the release process, the package size is about 20MB :
2017-12-18T18:12:53.9936453Z ##[section]Starting: Download artifact - ********- ********
2017-12-18T18:12:54.0216948Z ==============================================================================
2017-12-18T18:12:54.0217269Z Task : Download Build Artifacts
2017-12-18T18:12:54.0217525Z Description : Download Build Artifacts
2017-12-18T18:12:54.0217735Z Version : 0.127.5
2017-12-18T18:12:54.0217959Z Author : Microsoft Corporation
2017-12-18T18:12:54.0218321Z Help :
2017-12-18T18:12:54.0218644Z ==============================================================================
2017-12-18T18:12:57.8268381Z Linked artifacts count: 1
2017-12-18T18:12:57.8279446Z Downloading artifacts from: https://********.visualstudio.com//_apis/resources/Containers/2163734?itemPath=drop&isShallow=true&api-version=4.1-preview.4
2017-12-18T18:12:58.3652174Z Downloading drop/33.zip to d:\a\r1\a\********- ********\drop\33.zip
2017-12-18T18:12:59.2463333Z Downloaded 'drop/33.zip' to 'd:\a\r1\a\********- ********\drop\33.zip'
2017-12-18T18:12:59.8391807Z Total Files: 1, Processed: 1, Skipped: 0, Failed: 0, Download time: 2.01secs, Download size: 20.301MB
2017-12-18T18:12:59.9683800Z Successfully downloaded artifacts to d:\a\r1\a/********- ********
2017-12-18T18:12:59.9719698Z ##[section]Finishing: Download artifact - ********- ********
2017-12-18T18:12:59.9742525Z ##[section]Starting: Deploy Azure App Service
2017-12-18T18:12:59.9755839Z ==============================================================================
2017-12-18T18:12:59.9756853Z Task : Azure App Service Deploy
2017-12-18T18:12:59.9757256Z Description : Update Azure WebApp Services On Windows, Web App On Linux with built-in images or docker containers, ASP.NET, .NET Core, PHP, Python or Node based Web applications, Function Apps, Mobile Apps, Api applications, Web Jobs using Web Deploy / Kudu REST APIs
2017-12-18T18:12:59.9757643Z Version : 3.3.32
2017-12-18T18:12:59.9757849Z Author : Microsoft Corporation
2017-12-18T18:12:59.9758109Z Help : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-12-18T18:12:59.9758512Z ==============================================================================
2017-12-18T18:13:04.9522166Z Got connection details for Azure App Service:'********-********'
2017-12-18T18:13:36.4765413Z ##[error]Error: EMFILE: too many open files, open 'd:\a\r1\a\temp_web_package_30652997078345234\node_modules\globule\test\fixtures\expand\README.md'
2017-12-18T18:13:41.6795011Z Successfully updated deployment History at https://********-********.scm.azurewebsites.net/api/deployments/191513620816467
2017-12-18T18:13:43.0956656Z ##[section]Finishing: Deploy Azure App Service
2017-12-18T18:13:43.1054184Z ##[section]Finishing: Release
I can't use @vincentdass 's answer because after extracting it has a long directory, something like: Content/D_C/a/1/s/Project.MVC/obj/release/Package/PackageTmp these directory changes over time, would not make it a stable release.
Please tell me where to "Increase the max. limit of open file descriptors for the account where the automation agent is running." on the "Hosted" agent or "Hosted VS2017" agent.
Or how to ignore the node_modules folder because as far as deployment goes it should only be used as content, why is it "opening" these files anyway?
Post deployment scripts seems no option because not executing the npm install & build during solution build will throw other errors. And my feeling says it shouldn't be in the release because you don't want to "Build" something within a "Release", that is what a "Build" should be for.
What is causing this issue anyway?
A bit of log:
2018-10-30T15:18:04.3630475Z ##[debug]Finding files matching input: D:\a\r1\a\Babywatcher-Acceptance\AcceptanceDrop\Babywatcher.MVC_Package\Babywatcher.MVC.zip
2018-10-30T15:18:04.3646259Z ##[debug]Agent.TempDirectory=D:\a\_temp
2018-10-30T15:18:04.3646769Z ##[debug]Agent.TempDirectory=D:\a\_temp
2018-10-30T15:18:04.3657261Z ##[debug]extracting D:\a\r1\a\Babywatcher-Acceptance\AcceptanceDrop\Babywatcher.MVC_Package\Babywatcher.MVC.zip to D:\a\_temp\temp_web_package_6610637606854615
2018-10-30T15:18:51.7265379Z ##[debug]task result: Failed
2018-10-30T15:18:51.7266856Z ##[error]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_6610637606854615\Content\D_C\a\1\s\Babywatcher.MVC\obj\release\Package\PackageTmp\node_modules\resolve\test\resolver\other_path\root.js'
2018-10-30T15:18:51.7268057Z ##[debug]Processed: ##vso[task.issue type=error;]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_6610637606854615\Content\D_C\a\1\s\Babywatcher.MVC\obj\release\Package\PackageTmp\node_modules\resolve\test\resolver\other_path\root.js'
2018-10-30T15:18:51.7269319Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_6610637606854615\Content\D_C\a\1\s\Babywatcher.MVC\obj\release\Package\PackageTmp\node_modules\resolve\test\resolver\other_path\root.js'
Total zipfile size: 136MB
@Ruud-cb , Is there any file transformation option enabled in the task? (Web config generation , XML transform, XML sub or JSON sub)?
@vincentdass
Same problem for me, worked previously, added some extra code / NPM package to my app and now starting to receive this error, log:
2019-02-23T21:31:11.5335177Z ##[debug]Evaluating condition for step: 'Deploy Azure App Service'
2019-02-23T21:31:11.5335795Z ##[debug]Evaluating: succeeded()
2019-02-23T21:31:11.5335932Z ##[debug]Evaluating succeeded:
2019-02-23T21:31:11.5336128Z ##[debug]=> True
2019-02-23T21:31:11.5336336Z ##[debug]Result: True
2019-02-23T21:31:11.5336507Z ##[section]Starting: Deploy Azure App Service
2019-02-23T21:31:11.5344853Z ==============================================================================
2019-02-23T21:31:11.5344966Z Task : Azure App Service Deploy
2019-02-23T21:31:11.5345068Z Description : Update Azure App Services on Windows, Web App on Linux with built-in images or Docker containers, ASP.NET, .NET Core, PHP, Python or Node.js based Web applications, Function Apps, Mobile Apps, API applications, Web Jobs using Web Deploy / Kudu REST APIs
2019-02-23T21:31:11.5345188Z Version : 3.4.21
2019-02-23T21:31:11.5345281Z Author : Microsoft Corporation
2019-02-23T21:31:11.5345353Z Help : [More information](https://aka.ms/azurermwebdeployreadme)
2019-02-23T21:31:11.5345455Z ==============================================================================
2019-02-23T21:31:11.7721250Z ##[debug]agent.workFolder=D:\a
2019-02-23T21:31:11.7751462Z ##[debug]loading inputs and endpoints
2019-02-23T21:31:11.7757757Z ##[debug]loading ENDPOINT_AUTH_74788a23-51c2-4664-939a-d8b705e3c579
2019-02-23T21:31:11.7769937Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_74788a23-51c2-4664-939a-d8b705e3c579_ROLE
2019-02-23T21:31:11.7773304Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_74788a23-51c2-4664-939a-d8b705e3c579_SERVICEPRINCIPALID
2019-02-23T21:31:11.7776275Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_74788a23-51c2-4664-939a-d8b705e3c579_SERVICEPRINCIPALKEY
2019-02-23T21:31:11.7781631Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_74788a23-51c2-4664-939a-d8b705e3c579_TENANTID
2019-02-23T21:31:11.7783938Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2019-02-23T21:31:11.7786261Z ##[debug]loading ENDPOINT_AUTH_SCHEME_74788a23-51c2-4664-939a-d8b705e3c579
2019-02-23T21:31:11.7788351Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2019-02-23T21:31:11.7790920Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2019-02-23T21:31:11.7796243Z ##[debug]loading INPUT_BUILTINLINUXPACKAGE
2019-02-23T21:31:11.7797929Z ##[debug]loading INPUT_CONNECTEDSERVICENAME
2019-02-23T21:31:11.7800752Z ##[debug]loading INPUT_DEPLOYTOSLOTFLAG
2019-02-23T21:31:11.7802670Z ##[debug]loading INPUT_DOCKERIMAGETAG
2019-02-23T21:31:11.7805227Z ##[debug]loading INPUT_DOCKERREPOSITORYACCESS
2019-02-23T21:31:11.7807035Z ##[debug]loading INPUT_EXCLUDEFILESFROMAPPDATAFLAG
2019-02-23T21:31:11.7808942Z ##[debug]loading INPUT_GENERATEWEBCONFIG
2019-02-23T21:31:11.7811149Z ##[debug]loading INPUT_IMAGESOURCE
2019-02-23T21:31:11.7813269Z ##[debug]loading INPUT_JSONFILES
2019-02-23T21:31:11.7815412Z ##[debug]loading INPUT_PACKAGE
2019-02-23T21:31:11.7817447Z ##[debug]loading INPUT_REMOVEADDITIONALFILESFLAG
2019-02-23T21:31:11.7822298Z ##[debug]loading INPUT_RENAMEFILESFLAG
2019-02-23T21:31:11.7824900Z ##[debug]loading INPUT_RESOURCEGROUPNAME
2019-02-23T21:31:11.7826739Z ##[debug]loading INPUT_SCRIPTPATH
2019-02-23T21:31:11.7828562Z ##[debug]loading INPUT_SETPARAMETERSFILE
2019-02-23T21:31:11.7831263Z ##[debug]loading INPUT_TAKEAPPOFFLINEFLAG
2019-02-23T21:31:11.7833157Z ##[debug]loading INPUT_USEWEBDEPLOY
2019-02-23T21:31:11.7835590Z ##[debug]loading INPUT_WEBAPPKIND
2019-02-23T21:31:11.7837782Z ##[debug]loading INPUT_WEBAPPNAME
2019-02-23T21:31:11.7839864Z ##[debug]loading INPUT_WEBAPPURI
2019-02-23T21:31:11.7842600Z ##[debug]loading INPUT_XMLTRANSFORMATION
2019-02-23T21:31:11.7844598Z ##[debug]loading INPUT_XMLVARIABLESUBSTITUTION
2019-02-23T21:31:11.7853117Z ##[debug]loading SECRET_CONNECTIONSTRING
2019-02-23T21:31:11.7858779Z ##[debug]loaded 32
2019-02-23T21:31:11.8198538Z ##[debug]agent.proxyurl=undefined
2019-02-23T21:31:11.8199089Z ##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
2019-02-23T21:31:11.8199398Z ##[debug]AZURE_HTTP_USER_AGENT=VSTS_f38fcfda-db7a-4b32-b772-bc9a0ec2084b_release_1_2_2_2
2019-02-23T21:31:11.9028369Z ##[debug]Agent.TempDirectory=D:\a\_temp
2019-02-23T21:31:12.1783981Z ##[debug]check path : D:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.4.21\task.json
2019-02-23T21:31:12.1785060Z ##[debug]set resource file to: D:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.4.21\task.json
2019-02-23T21:31:12.1785263Z ##[debug]system.culture=en-US
2019-02-23T21:31:12.1925457Z ##[debug]ConnectedServiceName=74788a23-51c2-4664-939a-d8b705e3c579
2019-02-23T21:31:12.1927698Z ##[debug]WebAppName=AppName
2019-02-23T21:31:12.1929403Z ##[debug]WebAppKind=app
2019-02-23T21:31:12.1931643Z ##[debug]DeployToSlotFlag=false
2019-02-23T21:31:12.1932021Z ##[debug]VirtualApplication=null
2019-02-23T21:31:12.1934321Z ##[debug]Package=D:\a\r1\a\**\Application*.zip
2019-02-23T21:31:12.1935989Z ##[debug]GenerateWebConfig=false
2019-02-23T21:31:12.1936132Z ##[debug]WebConfigParameters=null
2019-02-23T21:31:12.1937372Z ##[debug]XmlTransformation=false
2019-02-23T21:31:12.1939926Z ##[debug]JSONFiles=**\connectionData.json
2019-02-23T21:31:12.1957277Z ##[debug]XmlVariableSubstitution=false
2019-02-23T21:31:12.1957423Z ##[debug]UseWebDeploy=true
2019-02-23T21:31:12.1957543Z ##[debug]TakeAppOfflineFlag=true
2019-02-23T21:31:12.1957659Z ##[debug]RenameFilesFlag=true
2019-02-23T21:31:12.1957776Z ##[debug]AdditionalArguments=null
2019-02-23T21:31:12.1957874Z ##[debug]ScriptType=null
2019-02-23T21:31:12.1957990Z ##[debug]InlineScript=null
2019-02-23T21:31:12.1958101Z ##[debug]ScriptPath=D:\a\r1\a
2019-02-23T21:31:12.1958415Z ##[debug]DockerNamespace=null
2019-02-23T21:31:12.1958740Z ##[debug]AppSettings=null
2019-02-23T21:31:12.1961150Z ##[debug]ImageSource=Registry
2019-02-23T21:31:12.1961301Z ##[debug]StartupCommand=null
2019-02-23T21:31:12.1962670Z ##[debug]WebAppUri=webAppUrl
2019-02-23T21:31:12.1962823Z ##[debug]ConfigurationSettings=null
2019-02-23T21:31:12.1965670Z ##[debug]RemoveAdditionalFilesFlag=false
2019-02-23T21:31:12.1966914Z ##[debug]SetParametersFile=D:\a\r1\a
2019-02-23T21:31:12.1968057Z ##[debug]ExcludeFilesFromAppDataFlag=false
2019-02-23T21:31:12.1968195Z ##[debug]AdditionalArguments=null
2019-02-23T21:31:12.1978162Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data subscriptionid = 684ec67e-69db-4afd-8571-e303027e74d7
2019-02-23T21:31:12.1978326Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data subscriptionname = 684ec67e-69db-4afd-8571-e303027e74d7
2019-02-23T21:31:12.1981185Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 auth param serviceprincipalid = ***
2019-02-23T21:31:12.1981371Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data environmentAuthorityUrl = https://login.windows.net/
2019-02-23T21:31:12.1982866Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 auth param tenantid = ***
2019-02-23T21:31:12.1984172Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579=https://management.azure.com/
2019-02-23T21:31:12.1984321Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data environment = AzureCloud
2019-02-23T21:31:12.1986506Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 auth scheme = ServicePrincipal
2019-02-23T21:31:12.1987008Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data msiclientId = undefined
2019-02-23T21:31:12.1987353Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data activeDirectoryServiceEndpointResourceId = https://management.core.windows.net/
2019-02-23T21:31:12.1987679Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data AzureKeyVaultServiceEndpointResourceId = https://vault.azure.net
2019-02-23T21:31:12.1987954Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data AzureKeyVaultDnsSuffix = vault.azure.net
2019-02-23T21:31:12.1988240Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 auth param authenticationType = null
2019-02-23T21:31:12.1988488Z ##[debug]credentials spn endpoint
2019-02-23T21:31:12.1989939Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 auth param serviceprincipalkey = ***
2019-02-23T21:31:12.1991773Z ##[debug]74788a23-51c2-4664-939a-d8b705e3c579 data EnableAdfsAuthentication = false
2019-02-23T21:31:12.1996074Z ##[debug]{"subscriptionID":"xxxxxxx","subscriptionName":"xxxxxxx","servicePrincipalClientID":"***","environmentAuthorityUrl":"https://login.windows.net/","tenantID":"***","url":"https://management.azure.com/","environment":"AzureCloud","scheme":"ServicePrincipal","activeDirectoryResourceID":"https://management.azure.com/","azureKeyVaultServiceEndpointResourceId":"https://vault.azure.net","azureKeyVaultDnsSuffix":"vault.azure.net","authenticationType":null,"servicePrincipalKey":***,"isADFSEnabled":false,"applicationTokenCredentials":{"clientId":"***","domain":"***","baseUrl":"https://management.azure.com/","authorityUrl":"https://login.windows.net/","activeDirectoryResourceId":"https://management.azure.com/","isAzureStackEnvironment":false,"authType":"spnKey","secret":***,"isADFSEnabled":false}}
2019-02-23T21:31:12.2034084Z Got connection details for Azure App Service:'AppName'
2019-02-23T21:31:12.2079200Z ##[debug][POST]https://login.windows.net/***/oauth2/token/
2019-02-23T21:31:12.4164307Z ##[debug][GET]https://management.azure.com/subscriptions/xxxxxxx/resources?$filter=resourceType EQ 'Microsoft.Web%2FSites' AND name EQ 'AppName'&api-version=2016-07-01
2019-02-23T21:31:12.6037974Z ##[debug]Resource Group: ResourceGroupName
2019-02-23T21:31:12.6063736Z ##[debug][POST]https://management.azure.com/subscriptions/xxxxxxx/resourceGroups/ResourceGroupName/providers/Microsoft.Web/sites/AppName/publishxml?api-version=2016-08-01
2019-02-23T21:31:13.7557993Z ##[debug]Could not parse response: {}
2019-02-23T21:31:13.7558719Z ##[debug]Response: undefined
2019-02-23T21:31:13.7726244Z ##[debug][GET]http://AppName.azurewebsites.net
2019-02-23T21:31:52.3096673Z ##[debug]Could not parse response: {}
2019-02-23T21:31:52.3097545Z ##[debug]Response: undefined
2019-02-23T21:31:52.3097792Z ##[debug]App Service status Code: '200'. Status Message: 'OK'
2019-02-23T21:31:52.3106396Z ##[debug][POST]https://management.azure.com/subscriptions/xxxxxxx/resourceGroups/ResourceGroupName/providers/Microsoft.Web/sites/AppName/config/publishingcredentials/list?api-version=2016-08-01
2019-02-23T21:31:53.3861100Z ##[debug]set AZURE_APP_SERVICE_KUDU_production_PASSWORD=********
2019-02-23T21:31:53.3877531Z ##[debug]Processed: ##vso[task.setvariable variable=AZURE_APP_SERVICE_KUDU_production_PASSWORD;issecret=true;]***
2019-02-23T21:31:53.4180559Z ##[debug]set webAppUrl=http://AppName.azurewebsites.net
2019-02-23T21:31:53.4237676Z ##[debug]Processed: ##vso[task.setvariable variable=webAppUrl;issecret=false;]http://AppName.azurewebsites.net
2019-02-23T21:31:53.4261493Z ##[debug]Finding files matching input: D:\a\r1\a\**\Application*.zip
2019-02-23T21:31:53.4288701Z ##[debug]Matching glob pattern: D:\a\r1\a\**\Application*.zip
2019-02-23T21:31:53.4298465Z ##[debug]Index of first wildcard: 10
2019-02-23T21:31:53.4358374Z ##[debug]find root dir: D:\a\r1\a\
2019-02-23T21:31:53.4358664Z ##[debug]findPath: 'D:\a\r1\a\'
2019-02-23T21:31:53.4358879Z ##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
2019-02-23T21:31:53.4359078Z ##[debug]findOptions.followSymbolicLinks: 'true'
2019-02-23T21:31:53.4359280Z ##[debug] D:\a\r1\a\ (directory)
2019-02-23T21:31:53.4359652Z ##[debug] D:\a\r1\a\Drop (directory)
2019-02-23T21:31:53.4360275Z ##[debug] D:\a\r1\a\Drop\drop (directory)
2019-02-23T21:31:53.4360570Z ##[debug] D:\a\r1\a\Drop\drop\Application2.zip (file)
2019-02-23T21:31:53.4360825Z ##[debug] D:\a\r1\a\Drop\drop\Tests2.zip (file)
2019-02-23T21:31:53.4361070Z ##[debug] D:\a\r1\a\Drop\drop\windows-webapp-template.json (file)
2019-02-23T21:31:53.4361314Z ##[debug]6 results
2019-02-23T21:31:53.4361552Z ##[debug]patternRoot: ''
2019-02-23T21:31:53.4361793Z ##[debug]matchOptions.debug: 'undefined'
2019-02-23T21:31:53.4362120Z ##[debug]matchOptions.nobrace: 'undefined'
2019-02-23T21:31:53.4362377Z ##[debug]matchOptions.noglobstar: 'undefined'
2019-02-23T21:31:53.4362866Z ##[debug]matchOptions.dot: 'undefined'
2019-02-23T21:31:53.4363118Z ##[debug]matchOptions.noext: 'undefined'
2019-02-23T21:31:53.4363506Z ##[debug]matchOptions.nocase: 'true'
2019-02-23T21:31:53.4363869Z ##[debug]matchOptions.nonull: 'undefined'
2019-02-23T21:31:53.4364069Z ##[debug]matchOptions.matchBase: 'true'
2019-02-23T21:31:53.4364431Z ##[debug]matchOptions.nocomment: 'undefined'
2019-02-23T21:31:53.4364618Z ##[debug]matchOptions.nonegate: 'undefined'
2019-02-23T21:31:53.4364819Z ##[debug]matchOptions.flipNegate: 'undefined'
2019-02-23T21:31:53.4365018Z ##[debug]pattern: 'D:\a\r1\a\**\Application*.zip'
2019-02-23T21:31:53.4365250Z ##[debug]expanding braces
2019-02-23T21:31:53.4365464Z ##[debug]pattern: 'D:/a/r1/a/**/Application*.zip'
2019-02-23T21:31:53.4365667Z ##[debug]applying include pattern against original list
2019-02-23T21:31:53.4365861Z ##[debug]1 matches
2019-02-23T21:31:53.4366052Z ##[debug]1 final results
2019-02-23T21:31:53.4366260Z ##[debug]Agent.TempDirectory=D:\a\_temp
2019-02-23T21:31:53.4366461Z ##[debug]Agent.TempDirectory=D:\a\_temp
2019-02-23T21:31:53.4366673Z ##[debug]extracting D:\a\r1\a\Drop\drop\Application2.zip to D:\a\_temp\temp_web_package_15900609935273824
2019-02-23T21:32:43.6817555Z ##[debug]task result: Failed
2019-02-23T21:32:43.6878318Z ##[error]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_15900609935273824\node_modules\recursive-readdir\test\testdir\a\a'
2019-02-23T21:32:43.6889259Z ##[debug]Processed: ##vso[task.issue type=error;]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_15900609935273824\node_modules\recursive-readdir\test\testdir\a\a'
2019-02-23T21:32:43.6890317Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_15900609935273824\node_modules\recursive-readdir\test\testdir\a\a'
My app does include a Web.config
as part of the source but it does do a JSON replacement for configuration. How do I get around this issue?
Happening to me as well, where is the command to "Increase the max. limit of open file descriptors for the account where the automation agent is running."
????
@jameshowe , @CheetahChrome , Can you please share the agent version?
https://docs.oracle.com/cd/E19623-01/820-6168/file-descriptor-requirements.html
2019-03-03T22:54:45.4056323Z Current agent version: '2.148.0'
2019-03-03T22:54:45.4085370Z Prepare release directory.
2019-03-03T22:54:45.4100521Z ReleaseId=8, …
@vincentdass last failed run was using Agent v2.147.1 (Hosted VS2017)
@vincentdass is it possible to do this on a hosted agent? FWIW I updated the Web App Deploy task version from 3.* to 4.* but it didn't help, it's definitely a limitation of the agent itself.
@vincentdass - Have been seeing this issue in Agent Version 2.146.0 (In house VM Agents). Seems to be happening when we check the "Generate Web.config" section in "Azure App Service Deploy task". Did you get a chance to look into this ?
@CheetahChrome , @jameshowe , @nitinmb , thanks for the info. We are able to repro the issue from our side and working on the fix.
In the meanwhile, you can add the web.config in the repo and package the same with the artifact. (Disable the web.config file)
@vincentdass - Do you have any further updates on this ?
This is still an issue in Oct 2019, kind of ridiculous that nothing has been done about this in over 2 years.
Hi @vincent1173 I am still seeing this issue. The workaround for adding an extract task before the Azure App Service task worked for me
2020-04-02T19:20:35.8686983Z ##[section]Starting: Deploy Azure App Service
2020-04-02T19:20:35.8826146Z ==============================================================================
2020-04-02T19:20:35.8826602Z Task : Azure App Service deploy
2020-04-02T19:20:35.8827086Z Description : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
2020-04-02T19:20:35.8827532Z Version : 4.163.4
2020-04-02T19:20:35.8827796Z Author : xxx
2020-04-02T19:20:35.8828152Z Help : https://aka.ms/azureappservicetroubleshooting
2020-04-02T19:20:35.8828550Z ==============================================================================
2020-04-02T19:20:36.6746855Z Got service connection details for Azure App Service:'xxx'
2020-04-02T19:21:32.7294562Z ##[error]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_2896488148652798\node_modules\office-ui-fabric-react\src\components\HoverCard\docs\HoverCardDonts.md'
This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days
We've encountered this issue a number of times over the last week
Also runnimg into this problem when I'm trying to deploy a next.js project.
Zipped folder with node_modules is 100mb.
I opened a support ticket with MS. I've copy/pasted the current assessment of the situation below.
TL;DR: When JSON variable substitution is used in the task it will unzip the artifact package to perform the replacement in the file. This issue emerges (for us intermittently, around 25% of the time) during this unzip process. Supposedly, the "Extract Files" task has a more reliable unzipping mechanism. So, as a workaround use that and pass the output to the deploy task.
===============================
We have got an update from our product group team. Please go through the below analysis and workaround shared by PG and let us know if you have any questions.
Issue:
Whenever a File transformation or Variable substitution option is selected in the Azure App Service Deploy task, it will first unzip the package and then perform the operations on the relevant files.
Analysis:
Currently, decompress-zip node module is used to unzip the package which seems to have some bug related to EMFILE when dealing with a large number of files. PG is currently exploring other options to unzip the package like 7zip, adm-zip. This change will take some time to roll out as they have to revisit the design and test accordingly.
Workaround:
For now, kindly use the Extract Files task to first extract the zip into a folder path and then provide the folder path into the App Service Deploy task. This will prevent the App service task to unzip the package and hence prevent EMFILE errors.
For YAML pipeline:
steps:
- task: ExtractFiles@1
displayName: 'Extract files '
inputs:
archiveFilePatterns: '$(InputPackageZipPath)'
destinationFolder: '$(OutputUnzippedPath)'
- task: AzureRmWebAppDeployment@4
displayName: 'Azure App Service Deploy: ev2arjunWebAppWebApp-wus'
inputs:
azureSubscription: 'Subscription'
WebAppName: 'app-name'
package: '$(OutputUnzippedPath)'
enableXmlTransform: true
MS Support have clearly just copied that response from the comments here rather than doing any actual investigation or trying to fix it.
This issue has been fixed and it should get deployed in ~3 weeks with the next deployment.
Thank you @amit-avit and team! Look forward to the fix.
This issue is fixed now and has been deployed.
@amit-avit
I am still getting this error, please see below
Error: EMFILE: too many open files, open 'D:\a_temp\temp_web_package_9235376407089184node_modules\tar-fs\test\fixtures\d\file1'
Log:
2021-01-24T08:18:19.9115523Z ##[section]Starting: Azure Web App Deploy: APP_SERVICE_NAME
2021-01-24T08:18:19.9336583Z ==============================================================================
2021-01-24T08:18:19.9336939Z Task : Azure Web App
2021-01-24T08:18:19.9337213Z Description : Deploy an Azure Web App for Linux or Windows
2021-01-24T08:18:19.9337482Z Version : 1.168.3
2021-01-24T08:18:19.9338774Z Author : Microsoft Corporation
2021-01-24T08:18:19.9339107Z Help : https://aka.ms/azurewebapptroubleshooting
2021-01-24T08:18:19.9340184Z ==============================================================================
2021-01-24T08:18:21.4138038Z Got service connection details for Azure App Service:'APP_SERVICE_NAME'
2021-01-24T08:18:22.4206941Z $APP_SERVICE_NAME
2021-01-24T08:18:22.4207745Z APP_SERVICE_NAME.scm.azurewebsites.net
2021-01-24T08:19:30.4109454Z ##[error]Error: EMFILE: too many open files, open 'D:\a_temp\temp_web_package_9235376407089184node_modules\tar-fs\test\fixtures\d\file1'
2021-01-24T08:19:39.5648101Z Successfully updated deployment History at https://APP_SERVICE_NAME.scm.azurewebsites.net/api/deployments/11611476370410
2021-01-24T08:19:39.5650514Z App Service Application URL: http://APP_SERVICE_NAME.azurewebsites.net
2021-01-24T08:19:45.6860095Z ##[section]Finishing: Azure Web App Deploy: APP_SERVICE_NAME
Azure Pipeline Hosted Agent
Version: 2.181.0
Most helpful comment
This is still an issue in Oct 2019, kind of ridiculous that nothing has been done about this in over 2 years.