Concourse: Feature Request: Per directory configuration

[philwebb]

What challenge are you facing?

I currently manage quite a few concourse pipelines and it's often hard to keep track of the target names and pipeline names that I'm running. It would be very useful if the target and pipeline options could be skipped and instead picked up from a config file that could be checked in alongside the pipeline.

What would make this better?

I'd like to be able to commit a .flyrc file that defines the URL of the CI and the pipeline name. The file could be checked into version control, allowing any member of the team to quickly run commands against the default CI instance.

For example, we might have a CI failure that necessitates fly hijack. To do this currently I need to run something like:

$ fly -t spring hijack -j spring-boot-2.0.x/build   

I also need to ensure that my target has been logged into the correct team.

$ fly login -t spring -n spring-boot

Mostly when I'm running a command like this, it's from a git checkout of the project. For the example above it would be the 2.0.x branch of https://github.com/spring-projects/spring-boot/

If we could commit a .flyrc file into the repository with something the following information:

default:
  api: https://ci.spring.io
  team: artifactory-resource
  pipeline: spring-boot-2.0.x

Then the fly command could become a lot simpler:

$ fly hijack build   

Are you interested in implementing this yourself?

I'm not that familiar with go, but I'd be happy to try.

[marco-m]

Hello, this looks like a Concourse antipattern: Concourse wants the fly target to be always explicit on purpose, to avoid errors that happen with other tools that allow this implicit behavior (and I strongly agree with the reasoning :-)

Documentation: https://concourse-ci.org/fly.html:

The -t flag is intentionally stateless and must be explicitly added to each command. This reduces the risk of accidentally running a command against the wrong environment when you have multiple targets defined.

See for example discussion in previous tickets:

• https://github.com/concourse/fly/issues/101
• https://github.com/concourse/fly/pull/157

In addition, for your specific proposal, I have two observations:

• it seems you make the assumption that a given directory containing a pipeline configuration file can give life to only one pipeline on the ATC. Actually it is not the case: we use 1 pipeline configuration file and generate N pipelines, 1 per branch, so having an implicit target/pipeline combination would almost always target the wrong pipeline instance.
• For the specific example of fly intercept/hijack you mention, there is a shortcut: if you pass --url, you don't have to pass the target, see https://github.com/concourse/fly/commit/6456bddd2519fe1abf446aff56b1f20280b10a45

[disclaimer: I am just a Concourse user, not a member of the Concourse team]

[philwebb]

Concourse wants the fly target to be always explicit on purpose, to avoid errors that happen with other tools that allow this implicit behavior (and I strongly agree with the reasoning :-)

I agree this is indeed a fine design decision, especially where you might be pushing the same pipeline to multiple targets. However, I do think there are probably many users like myself where a single pipeline has a single target. In these cases, always needing to specify the target actually increases the risk that you might push to the wrong location.

you make the assumption that a given directory containing a pipeline configuration file can give life to only one pipeline

I'm assuming that for many users this is will be the case, but I don't think it will hold true for all users. In the case of multiple pipelines per file, the "default pipeline" feature doesn't make any sense.

...there is a shortcut: if you pass --url, you don't have to pass the target

That's super useful. Thanks for the hint! I'll certainly be using that in the future.