Renovate: Bitbucket server: prConcurrentLimit seems to take declined PRs into account

Created on 20 Aug 2020  路  23Comments  路  Source: renovatebot/renovate

What Renovate type are you using?

self-hosted Bitbucket server (Stash)

Describe the bug

When we noticed a lack of renovate PRs/commits, I went to check the logs to notice it's claiming there are 3 PRs still open; even though there are no renovate PRs open at all (nor remote branches)

Relevant debug logs


[05:46:40] :     [Step 3/4] DEBUG: branchExists(renovate/vue-simple-svg-2.x) (repository=hcc/mdt-app)
[05:46:40] :     [Step 3/4] DEBUG: branchExists(renovate/wait-on-5.x) (repository=hcc/mdt-app)
[05:46:40] :     [Step 3/4] DEBUG: branchExists(renovate/webpack-merge-5.x) (repository=hcc/mdt-app)
[05:46:40] :     [Step 3/4] DEBUG: branchExists(renovate/z-schema-4.x) (repository=hcc/mdt-app)
[05:46:40] :     [Step 3/4] DEBUG: 3 PRs are currently open (repository=hcc/mdt-app)
[05:46:40] :     [Step 3/4] DEBUG: PR concurrent limit remaining: 0 (repository=hcc/mdt-app)
[05:46:40] :     [Step 3/4] DEBUG: Calculated maximum PRs remaining this run (repository=hcc/mdt-app)
[05:46:40] :     [Step 3/4]        "prsRemaining": 0
[05:46:40] :     [Step 3/4] DEBUG: processBranch with 1 upgrades (repository=hcc/mdt-app, dependencies=@uirouter/angularjs, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:40] :     [Step 3/4] DEBUG: Setting baseBranch to master (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:40] :     [Step 3/4] DEBUG: latest commit (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:40] :     [Step 3/4]        "branchName": "master",
[05:46:40] :     [Step 3/4]        "latestCommitDate": "2020-08-19 09:47:24 +0000"
[05:46:41] :     [Step 3/4] DEBUG: branchExists(renovate/uirouter-angularjs-1.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: getBranchPr(renovate/uirouter-angularjs-1.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: findPr(renovate/uirouter-angularjs-1.0.x, "undefined", "open") (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: getPrList() (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Retrieved Pull Requests (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4]        "length": 17
[05:46:41] :     [Step 3/4] DEBUG: DID NOT Found PR from branch #renovate/uirouter-angularjs-1.0.x (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: branchExists=true (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Branch has 1 upgrade(s) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: recreateClosed is false (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: findPr(renovate/uirouter-angularjs-1.0.x, "Update dependency @uirouter/angularjs to v1.0.28", "!open") (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: getPrList() (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: returning cached PR list (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Found PR #1809 (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Found closed PR with current title (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: getPr(1809) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: branchExists(renovate/uirouter-angularjs-1.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: isBranchStale(renovate/uirouter-angularjs-1.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Closed PR already exists. Skipping branch. (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4]        "prTitle": "Update dependency @uirouter/angularjs to v1.0.28"
[05:46:41] :     [Step 3/4] DEBUG: Found 4 comments (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Ensuring comment "Renovate Ignore Notification" in #1809 (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Comment is already update-to-date (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: deleteBranch(renovate/uirouter-angularjs-1.0.x, closePr=false) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4] DEBUG: No remote branch to delete (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4]        "branchName": "renovate/uirouter-angularjs-1.0.x"
[05:46:41] :     [Step 3/4] DEBUG: No local branch to delete (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:46:41] :     [Step 3/4]        "branchName": "renovate/uirouter-angularjs-1.0.x"
[05:46:41] :     [Step 3/4] DEBUG: processBranch with 1 upgrades (repository=hcc/mdt-app, dependencies=@uirouter/core, branch=renovate/uirouter-core-5.0.x)
[05:46:41] :     [Step 3/4] DEBUG: Setting baseBranch to master (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: latest commit (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4]        "branchName": "master",
[05:46:42] :     [Step 3/4]        "latestCommitDate": "2020-08-19 09:47:24 +0000"
[05:46:42] :     [Step 3/4] DEBUG: branchExists(renovate/uirouter-core-5.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: getBranchPr(renovate/uirouter-core-5.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: findPr(renovate/uirouter-core-5.0.x, "undefined", "open") (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: getPrList() (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: returning cached PR list (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: DID NOT Found PR from branch #renovate/uirouter-core-5.0.x (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: branchExists=true (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: Branch has 1 upgrade(s) (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: recreateClosed is false (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: findPr(renovate/uirouter-core-5.0.x, "Update dependency @uirouter/core to v5.0.23", "!open") (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: getPrList() (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: returning cached PR list (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: Found PR #1810 (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: Found closed PR with current title (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: getPr(1810) (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: branchExists(renovate/uirouter-core-5.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: isBranchStale(renovate/uirouter-core-5.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: Closed PR already exists. Skipping branch. (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4]        "prTitle": "Update dependency @uirouter/core to v5.0.23"
[05:46:42] :     [Step 3/4] DEBUG: Found 4 comments (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: Ensuring comment "Renovate Ignore Notification" in #1810 (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: Comment is already update-to-date (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: deleteBranch(renovate/uirouter-core-5.0.x, closePr=false) (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4] DEBUG: No remote branch to delete (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4]        "branchName": "renovate/uirouter-core-5.0.x"
[05:46:42] :     [Step 3/4] DEBUG: No local branch to delete (repository=hcc/mdt-app, branch=renovate/uirouter-core-5.0.x)
[05:46:42] :     [Step 3/4]        "branchName": "renovate/uirouter-core-5.0.x"
[05:46:42] :     [Step 3/4] DEBUG: processBranch with 1 upgrades (repository=hcc/mdt-app, dependencies=bootstrap, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: Setting baseBranch to master (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: latest commit (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4]        "branchName": "master",
[05:46:42] :     [Step 3/4]        "latestCommitDate": "2020-08-19 09:47:24 +0000"
[05:46:42] :     [Step 3/4] DEBUG: branchExists(renovate/bootstrap-4.5.x) (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: getBranchPr(renovate/bootstrap-4.5.x) (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: findPr(renovate/bootstrap-4.5.x, "undefined", "open") (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: getPrList() (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: returning cached PR list (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: DID NOT Found PR from branch #renovate/bootstrap-4.5.x (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: branchExists=false (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: Branch has 1 upgrade(s) (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: recreateClosed is false (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: findPr(renovate/bootstrap-4.5.x, "Update dependency bootstrap to v4.5.2", "!open") (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: getPrList() (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: returning cached PR list (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: DID NOT Found PR from branch #renovate/bootstrap-4.5.x (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: prAlreadyExisted=false (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)
[05:46:42] :     [Step 3/4] DEBUG: Reached PR creation limit or per run commits limit - skipping branch creation (repository=hcc/mdt-app, branch=renovate/bootstrap-4.5.x)

To Reproduce

Let renovate run fine for weeks, decline (at least) prConcurrentLimit amount of PRs

Additional context

  • There were 3 PRs open during this log run; none of them by renovate
  • No _leftover_ renovate branches reside on Stash anymore
  • Seemed to be working fine for weeks; debug logs do seem to imply there's 100s of renovate branches still found _somewhere_
bitbucket-server priority-3-normal bug
Source
picture jlsjonas

All 23 comments

Possible fix: #7064

Seemed to be working fine for weeks; debug logs do seem to imply there's 100s of renovate branches still found somewhere

Not sure, but may be caused by cache or by Renovate internal repo clone. And can be related to #6988.

zharinov picture zharinov on 23 Aug 2020

@zharinov although @jlsjonas reports that there are no remote branches open, it sounds like Renovate thinks there is. @jlsjonas can you paste your bot config? e.g. are you using persistRepoData=true ?

rarkins picture rarkins on 24 Aug 2020

I am using persistRepoData yes

jlsjonas picture jlsjonas on 24 Aug 2020

@viceice are you still using persistRepoData=true in your self-hosted? I suspect it's not detecting remote branch deletion and therefore impacting our ability to count concurrent branches properly. I think we need to either:

  • "sync" branches after fetch to prune ones missing from remote, or
  • detect these declined PRs and delete their branches locally
rarkins picture rarkins on 24 Aug 2020

@jlsjonas I suggest turning off persistRepoData and the problem likely goes away after the first time the current local copies are deleted.

rarkins picture rarkins on 24 Aug 2020

@jlsjonas actually, before you do that, I'd be curious if you can run git branch --v | grep "\[gone\]" within the local repo to see if that command identifies the branches that have been deleted on the server but not locally

Source: https://stackoverflow.com/a/59228595/13699742

rarkins picture rarkins on 24 Aug 2020

Important extra note: not all branches detected locally have a declined PR, some were created earlier and either due to pendingDays or other limiting reason (like CI/max PR, although the latter doesn't actually check PRs afaik).

Checking declined PRs and delete those branches won't be a full fix.

For now I've disabled persistRepoData, will report if that fixed it (as a workaround) tomorrow

Will check the local git repo before the next run, expect an update in ~1hr

jlsjonas picture jlsjonas on 24 Aug 2020

We don't want to delete all local branches, just the ones with no remote. I can't think of a scenario where we would create a local branch, NOT push it to remote, and then still have a reason to keep it. Therefore if that is correct then pruning all local branches without a remote should work.

FWIW the command suggested on StackOverflow seems to specifically identify branches with a removed remote (rather than on that never existed), which I'm not sure is a good or a bad thing just yet.

rarkins picture rarkins on 24 Aug 2020
👍1

@viceice are you still using persistRepoData=true in your self-hosted? I suspect it's not detecting remote branch deletion and therefore impacting our ability to count concurrent branches properly. I think we need to either:

  • "sync" branches after fetch to prune ones missing from remote, or
  • detect these declined PRs and delete their branches locally

I've disabled it today, after reading the issue here. Maybe that flag is also the cause of sometimes have pr's created for branches with stabillity-days=pending. 馃

viceice picture viceice on 24 Aug 2020

@rarkins the situation is even stranger...

$ git branch --v | grep "\[gone\]"
$ git branch --v
* master 05...

It looks like only master is still listed locally within renovate's repo
Yet renovate logs indicate there's a lot of branches (and also acts accordingly, since the PR limit is hit directly) :thinking:

jlsjonas picture jlsjonas on 24 Aug 2020

What version of Renovate are you running?

rarkins picture rarkins on 24 Aug 2020

21.16.0, looks like we could use an update :)

jlsjonas picture jlsjonas on 24 Aug 2020

Yes, I'm not sure if it will fix this problem but I just observed that one of the log messages you showed does not exist anymore.

rarkins picture rarkins on 24 Aug 2020

Will report back tomorrow (our renovate is configured to run outside office hours), I'll keep persistRepoData enabled for tonight, to avoid _lots of changes fixing the problem_

jlsjonas picture jlsjonas on 24 Aug 2020

@rarkins sadly enough, same behavior on the latest version

[05:47:16] :     [Step 3/4] DEBUG: branchExists(renovate/webpack-merge-5.x) (repository=hcc/mdt-app)
[05:47:16] :     [Step 3/4] DEBUG: branchExists(renovate/z-schema-4.x) (repository=hcc/mdt-app)
[05:47:16] :     [Step 3/4] DEBUG: 3 PRs are currently open (repository=hcc/mdt-app)
[05:47:16] :     [Step 3/4] DEBUG: PR concurrent limit remaining: 0 (repository=hcc/mdt-app)
[05:47:16] :     [Step 3/4] DEBUG: Calculated maximum PRs remaining this run (repository=hcc/mdt-app)
[05:47:16] :     [Step 3/4]        "prsRemaining": 0
[05:47:16] :     [Step 3/4] DEBUG: processBranch with 1 upgrades (repository=hcc/mdt-app, dependencies=@uirouter/angularjs, branch=renovate/uirouter-angularjs-1.0.x)
[05:47:16] :     [Step 3/4] DEBUG: Setting baseBranch to master (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:47:17] :     [Step 3/4] DEBUG: latest commit (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[05:47:17] :     [Step 3/4]        "branchName": "master",
[05:47:17] :     [Step 3/4]        "latestCommitDate": "2020-08-24 14:02:09 +0200"

Interestingly though... if I read it correctly it seems that renovate itself is somehow parsing the local master branch (the only existing one) as all the different renovate/ branches.

Will disable persisting as a next debug step

jlsjonas picture jlsjonas on 25 Aug 2020

@jlsjonas FYI I cannot find anywhere in our current code base where a debug string like branchExists(renovate/z-schema-4.x) should be printed, which is why I initially worked out that you were not on the latest version. I still have a doubt..

rarkins picture rarkins on 25 Aug 2020

Pretty sure it was removed on 4th July: https://github.com/renovatebot/renovate/commit/52cddfde8be2868aa813f7f0a0a9b625e42421fd#diff-7faeb060683d4322d42ed78669008f50 (search for:

logger.debug(`branchExists(${branchName})`);
rarkins picture rarkins on 25 Aug 2020

馃 Something indeed made it run on the older version still, this time I'm 100% confident it is upgraded to 22.25.6 (latest at time of writing); I did run it at least once without persistRepoData, which also could've cleared some weird cache.

I just did a run (outside active hours config-wise) which is giving me the same return pr-wise

[11:22:01] :     [Step 4/5] DEBUG: processRepo() (repository=hcc/mdt-app)
[11:22:01] :     [Step 4/5] DEBUG: Processing 214 branches: renovate/ajv-6.x, renovate/amazon-cognito-identity-js-3.x, renovate/amazon-cognito-identity-js-4.x, renovate/angular-1.x, renovate/angular-bootstrap-toggle-0.x, renovate/angular-monorepo, renovate/angular-translate-2.x, renovate/async-3.x, renovate/autoprefixer-9.x, renovate/aws-sdk-2.x, renovate/axios-0.x, renovate/babel-monorepo, renovate/babel-plugin-istanbul-6.x, renovate/bootstrap-4.5.x, renovate/bowser-2.x, renovate/browserslist-4.x, renovate/chromedriver-84.x, renovate/clean-webpack-plugin-3.x, renovate/commander-2.x, renovate/commander-6.x, renovate/concurrently-5.x, renovate/copy-webpack-plugin-4.x, renovate/copy-webpack-plugin-6.x, renovate/cross-env-7.x, renovate/css-loader-4.x, renovate/cucumber-6.x, renovate/debounce-promise-3.x, renovate/debug-3.x, renovate/debug-4.x, renovate/diacritic-regex-1.x, renovate/dotenv-6.x, renovate/dotenv-8.x, renovate/element-resize-detector-1.x, renovate/eslint-6.x, renovate/eslint-7.x, renovate/eslint-config-airbnb-base-14.x, renovate/eslint-plugin-import-2.x, renovate/eslint-plugin-jsdoc-15.x, renovate/eslint-plugin-jsdoc-30.x, renovate/eslint-plugin-mocha-8.x, renovate/express-4.x, renovate/express-handlebars-3.x, renovate/express-handlebars-5.x, renovate/express-mongo-sanitize-2.x, renovate/express-openapi-validate-0.x, renovate/extract-loader-5.x, renovate/favicons-webpack-plugin-4.x, renovate/fhir-4.x, renovate/file-loader-6.x, renovate/file-saver-2.x, renovate/flatpickr-4.x, renovate/geckodriver-1.x, renovate/gherkin-lint-3.x, renovate/gherkin-lint-4.x, renovate/github-markdown-css-4.x, renovate/hashmap-2.x, renovate/helmet-3.x, renovate/helmet-4.x, renovate/highlight.js-10.x, renovate/highlight.js-9.x, renovate/html-loader-1.x, renovate/html-webpack-harddisk-plugin-1.x, renovate/html-webpack-plugin-4.x, renovate/htmllint-0.x, renovate/i18next-19.x, renovate/i18next-browser-languagedetector-4.x, renovate/i18next-browser-languagedetector-6.x, renovate/i18next-express-middleware-2.x, renovate/js-combinatorics-0.x, renovate/js-combinatorics-1.x, renovate/json2csv-5.x, renovate/jsonwebtoken-8.x, renovate/jsreport-chrome-pdf-1.x, renovate/jsreport-core-2.x, renovate/jwks-rsa-1.x, renovate/karma-4.x, renovate/karma-5.x, renovate/karma-chrome-launcher-3.x, renovate/karma-coverage-2.x, renovate/karma-junit-reporter-2.x, renovate/karma-mocha-2.x, renovate/keyboardjs-2.x, renovate/loader-utils-1.x, renovate/loader-utils-2.x, renovate/markdown-it-11.x, renovate/markdown-table-2.x, renovate/messageformat-1.x, renovate/messageformat-2.x, renovate/mini-css-extract-plugin-0.x, renovate/mkdirp-1.x, renovate/mocha-6.x, renovate/mocha-8.x, renovate/mocha-junit-reporter-2.x, renovate/mockserver-client-5.x, renovate/mockserver-node-5.x, renovate/moment-2.x, renovate/moment-duration-format-2.x, renovate/moment-locales-webpack-plugin-1.x, renovate/moment-timezone-0.5.x, renovate/mongodb-3.x, renovate/mongodb-memory-server-6.x, renovate/mongoose-5.9.x, renovate/mongoose-5.x, renovate/node-sass-4.x, renovate/nodemon-2.x, renovate/npm-6.x, renovate/null-loader-4.x, renovate/nyc-15.x, renovate/opentok-2.8.x, renovate/opentok-2.x, renovate/opentok-client-2.x, renovate/parse5-5.1.x, renovate/parse5-6.x, renovate/parseurl-1.3.x, renovate/patch-storybook-monorepo, renovate/patch-typings, renovate/patch-vue-monorepo, renovate/path-to-regexp-3.x, renovate/path-to-regexp-6.x, renovate/postcss-flexbugs-fixes-4.x, renovate/postcss-loader-2.1.x, renovate/postcss-loader-3.x, renovate/preload-webpack-plugin-3.0.x, renovate/protractor-5.4.x, renovate/protractor-7.x, renovate/protractor-cucumber-framework-6.1.x, renovate/protractor-cucumber-framework-6.x, renovate/proxyquire-2.1.x, renovate/puppeteer-5.x, renovate/qs-6.7.x, renovate/qs-6.x, renovate/raw-loader-4.x, renovate/regenerator-runtime-0.13.x, renovate/request-2.88.x, renovate/request-promise-native-1.0.x, renovate/resolve-url-loader-2.3.x, renovate/resolve-url-loader-3.x, renovate/restify-errors-8.0.x, renovate/restify-errors-8.x, renovate/rewire-5.x, renovate/rimraf-2.6.x, renovate/rimraf-2.x, renovate/rimraf-3.x, renovate/roboto-fontface-0.x, renovate/rxjs-5.5.x, renovate/rxjs-6.x, renovate/rxjs-compat-6.x, renovate/rxjs-marbles-2.3.x, renovate/rxjs-marbles-2.x, renovate/rxjs-marbles-6.x, renovate/sass-loader-10.x, renovate/sass-loader-7.x, renovate/sassdoc-2.7.x, renovate/sassdoc-extras-3.x, renovate/schema-utils-2.x, renovate/scripty-1.x, renovate/scripty-2.x, renovate/selenium-webdriver-4.x, renovate/sinon-7.x, renovate/sinon-9.x, renovate/socket.io-2.x, renovate/socket.io-packages, renovate/spark-md5-3.0.x, renovate/spawn-loader-6.x, renovate/spawn-loader-7.x, renovate/strip-bom-4.x, renovate/style-loader-0.x, renovate/style-loader-1.x, renovate/style-resources-loader-1.x, renovate/stylelint-11.x, renovate/stylelint-13.x, renovate/stylelint-config-sass-guidelines-6.x, renovate/stylelint-config-sass-guidelines-7.x, renovate/stylelint-config-standard-20.x, renovate/svgo-1.x, renovate/svgo-loader-2.x, renovate/swagger-jsdoc-3.2.x, renovate/swagger-jsdoc-3.x, renovate/swagger-jsdoc-4.x, renovate/swagger-tools-0.10.x, renovate/terser-webpack-plugin-3.0.x, renovate/terser-webpack-plugin-3.x, renovate/terser-webpack-plugin-4.x, renovate/toposort-1.0.x, renovate/toposort-2.x, renovate/tough-cookie-2.x, renovate/tough-cookie-4.x, renovate/uirouter-angularjs-1.0.x, renovate/uirouter-core-5.0.x, renovate/uirouter-core-6.x, renovate/uirouter-dsr-1.x, renovate/uirouter-visualizer-7.x, renovate/url-loader-2.x, renovate/url-loader-4.x, renovate/uuid-3.3.x, renovate/uuid-3.x, renovate/uuid-8.x, renovate/vue-docgen-api-4.x, renovate/vue-flatpickr-component-8.1.x, renovate/vue-monorepo, renovate/vue-simple-suggest-1.9.x, renovate/vue-simple-suggest-1.x, renovate/vue-simple-svg-1.3.x, renovate/vue-simple-svg-2.x, renovate/wait-on-3.x, renovate/wait-on-5.x, renovate/webpack-4.x, renovate/webpack-cli-3.3.x, renovate/webpack-dev-server-3.8.x, renovate/webpack-dev-server-3.x, renovate/webpack-merge-5.x, renovate/z-schema-3.24.x, renovate/z-schema-3.x, renovate/z-schema-4.x (repository=hcc/mdt-app)
[11:22:01] :     [Step 4/5] DEBUG: Enforcing prConcurrentLimit (3) (repository=hcc/mdt-app)
[11:22:08] :     [Step 4/5] DEBUG: 3 PRs are currently open (repository=hcc/mdt-app)
[11:22:08] :     [Step 4/5] DEBUG: PR concurrent limit remaining: 0 (repository=hcc/mdt-app)
[11:22:08] :     [Step 4/5] DEBUG: Calculated maximum PRs remaining this run (repository=hcc/mdt-app)
[11:22:08] :     [Step 4/5]        "prsRemaining": 0
[11:22:08] :     [Step 4/5] DEBUG: processBranch with 1 upgrades (repository=hcc/mdt-app, dependencies=@uirouter/angularjs, branch=renovate/uirouter-angularjs-1.0.x)
[11:22:08] :     [Step 4/5] DEBUG: Setting current branch to master (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[11:22:09] :     [Step 4/5] DEBUG: latest commit (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[11:22:09] :     [Step 4/5]        "branchName": "master",
[11:22:09] :     [Step 4/5]        "latestCommitDate": "2020-08-24T14:02:09+02:00"
[11:22:09] :     [Step 4/5] DEBUG: getBranchPr(renovate/uirouter-angularjs-1.0.x) (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[11:22:09] :     [Step 4/5] DEBUG: findPr(renovate/uirouter-angularjs-1.0.x, "undefined", "open") (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[11:22:09] :     [Step 4/5] DEBUG: getPrList() (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[11:22:09] :     [Step 4/5] DEBUG: Retrieved Pull Requests (repository=hcc/mdt-app, branch=renovate/uirouter-angularjs-1.0.x)
[11:22:09] :     [Step 4/5]        "length": 17

This is with persistRepoData still disabled

jlsjonas picture jlsjonas on 25 Aug 2020

Can you post the full debug log for a repo to a gist?

BTW the logic we use is a little different to what you tested manually last time:

  try {
    await git.raw(['show-branch', 'origin/' + branchName]);
    config.branchExists[branchName] = true;
    return true;
  } catch (err) {
    checkForPlatformFailure(err);
    config.branchExists[branchName] = false;
    return false;
  }

Essentially, it would be running a command like git show-branch origin/renovate-foo-1.x and expecting it to throw if the branch doesn't exist.

rarkins picture rarkins on 25 Aug 2020
👍1

Please find the (redacted) log here

jlsjonas picture jlsjonas on 25 Aug 2020

@jlsjonas a few quick observations:

  • I see git fetch completed (repository=hcc/mdt-app) so it's fetching rather than cloning which I think means data is still persisted?
  • Lots of hits for branch has been modified which means it can't prune. Can you configure [email protected] and see if those branches are pruned?
rarkins picture rarkins on 25 Aug 2020

I assume that it needed a(n/ few) extra run(s) to actively clear out the git repo; which seems to have resolved the blockage.
persist-repo-data got turned on again so if it's a recurring issue I'll be able to report back about it in a few weeks.

I'll close the issue (feel free to reopen) for now until I have new info, assuming the underlying issue got resolved in between.

jlsjonas picture jlsjonas on 26 Aug 2020
👍1

Going to re-open this with an asterix...
We came back from the weekend to 13 open PRs from renovate, even though we (still) have prConcurrentLimit set to 3.
Any advice on debug steps? (given previous context, you're aware we had no renovate branches left, so everything was gone)
we had 3 prs created (as expected) on Thursday, 1 PR was left open before _the extras_ came

jlsjonas picture jlsjonas on 31 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kogai picture kogai  路  3Comments
OmgImAlexis picture OmgImAlexis  路  4Comments
jycouet picture jycouet  路  4Comments
ChristianMurphy picture ChristianMurphy  路  4Comments
ZauberNerd picture ZauberNerd  路  4Comments