Concourse: 3.9.2 - Pipeline Trigger Resources appear to continue to fire even after pipeline deleted

[keithkroeger]

Bug Report

• Concourse version: 3.9.2
• Deployment type (BOSH/Docker/binary): BOSH
• Infrastructure/IaaS: Openstack
• Browser (if applicable): N/A

To test our deployments, we create dozens of pipelines. What we find is that the timers appear to still fire, even after the pipeline is deleted.

We see this in our logs as:

<14>1 2018-03-20T13:48:55.968697Z a719de49-023b-4ae9-aaa6-367b7b8e897b.web.default.qa.bosh atc - - [instance@47450 director="" deployment="qa" group="web" az="z1" id="a719de49-023b-4ae9-aaa6-367b7b8e897b"] {"timestamp":"1521553735.963469744","source":"atc","message":"atc.syncer.test-46:scheduler.tick.scheduling.try-start-next-pending-build.scan.release.released","log_level":0,"data":{"build-id":19162,"build-name":"1","id":[0,17463],"input":"build","resource":"build","session":"42.102.1.3.2.2.8061"}}
<14>1 2018-03-20T13:48:55.968704Z a719de49-023b-4ae9-aaa6-367b7b8e897b.web.default.qa.bosh atc - - [instance@47450 director="" deployment="qa" group="web" az="z1" id="a719de49-023b-4ae9-aaa6-367b7b8e897b"] {"timestamp":"1521553735.963504076","source":"atc","message":"atc.syncer.test-46:scheduler.tick.scheduling.try-start-next-pending-build.scan.lock.did-not-get-lock","log_level":0,"data":{"build-id":19162,"build-name":"1","input":"build","resource":"build","session":"42.102.1.3.2.2.1"}}

Pipeline:

resources:
- name: 5m
  type: time
  source: {interval: 5m}

- name: 10m
  type: time
  source: {interval: 5m}

- name: 15m
  type: time
  source: {interval: 5m}

- name: repo
  type: git
  source:
    branch: feature/ref-app-test
    uri: https://.....
  check_every: 60m

- name: build
  type: git
  source:
    branch: develop
    uri: ......
    private_key: ((vault-key-repo-private-key))
  check_every: 60m

jobs:
- name: mvn-build
  interruptible: true
  plan:
  - aggregate:
    - get: build
    - get: repo 
    - get: 5m
    - get: 10m
    - get: 15m   
  - task: build
    file: build/tasks/maven/task.yml
    params:
      <<: *maven-params
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: ubuntu}
      run:
        path: /bin/bash
        args:
          - -exc
          - |
            rand=$((20 + RANDOM % 90))
            sleep $rand
            echo "Hello world $rand"


maven-params: &maven-params
  MVN_GOALS: clean package
  MVN_SETTINGS: settings.xml
  MVN_POM: pom.xml
  MAVEN_OPTS: -Xmx1024m

[marco-m]

I think you will increase the chances of a response from the Concourse team if you format properly your bug description :-)

[vito]

Those logs don't indicate to me that a build is running. They're trying to acquire a lock for checking a resource, probably because the build was manually triggered, and waiting on the lock because there's already a check running (somewhere). It may flail for a bit trying to do that, but by the time it gets to actually running the build it should see that the build is no longer there and not run anything.

Destroying a pipeline doesn't currently guarantee an immediate cancellation of all in-flight plumbing, but it will guarantee that new builds don't get created. Those logs are probably coming from here:

https://github.com/concourse/concourse/blob/cd76db60a4ab3bca25eb447724ad9ccb92af1272/atc/scheduler/buildstarter.go#L106

...and looking at the code immediately after, just about any of those calls will fail if the pipeline is destroyed, and the scheduling will not continue.

[keithkroeger]

Thank you, @vito