Sts4: concourse vs code extension not recognise YAML anchors, references, extend

[chaoyangnz]

anchor &
reference *
extend <<

[kdvolder]

I think I know what you mean by 'anchor' and 'reference'. But can you please explain a bit more about 'extend'. What is that exactly?

An example of something that uses these (especially the 'extend <<' one since I'm not familiar with it) would be very welcome as well.

[kdvolder]

Actually.. I found this: https://blog.daemonl.com/2016/02/yaml.html
That seems to explain all three. Adding this link for future reference.

If you have an example pipeline file that uses these features it would be still be welcome. It is allways nice to use a 'real' example as a test case instead of making up something totally artificial by myself.

[kdvolder]

Talk is cheap, show me the code.

Quoting your own moto.... Show me the code... means. Show me your pipeline example please :-)

[chaoyangnz]

I anchor a reference:

dcind: &dcind
  type: docker-image
  source:
    repository: kiwiops/stuff-mem-dcind
    tag: latest
...

I got unknow property 'dcind' for type 'Pipeline'.

- task: deploy-ssp-devint
    privileged: true
    config:
      platform: linux
      image_resource: 
        <<: *dcind
      inputs:
      - name: kms
...

Here, I got Unknow property '<<' for type ImageResource.

[kdvolder]

Hmmm... interesting. Technically, I think the error about unknown property 'dcind' is actually not incorrect. Because... your yaml does define a property called 'dcind' and it isn't in the schema. But I suppose it is probably a common way to define something with anchor. You see... this is why a real example matters. I probably wouldn't thought of that without the example.

In other words... thanks for the nice example. If you have more interesting cases of 'stuff' you would like to have working don't hesitate to attach as well.

PS: It may be some time before I get to implementing support for this. It doesn't seem straightforward so its not something I can just pick off when I have some spare time between other chores. But have every intention of fixing this eventually.

[marco-m]

@kdvolder another YAML documentation that I found useful to understand anchors and extend/merge is https://camel.readthedocs.io/en/latest/yamlref.html. I will provide another example of anchors and extend from my own pipelines. BTW I appreciate your kindness!

Another article explicitly on using YAML anchors and merge for Concourse pipelines: https://www.digitalocean.com/community/tutorials/how-to-set-up-continuous-integration-pipelines-with-concourse-ci-on-ubuntu-16-04

[marco-m]

here is a full pipeline:

     1  pool-template: &pool-template
     2    uri: ((pool-git-backing-store-uri))
     3    branch: master
     4    pool: OVERRIDEME
     5    private_key: ((pool-git-backing-store-private-key))
     6  
     7  sleep: &sleep
     8    config:
     9      platform: linux
    10      image_resource:
    11        type: docker-image
    12        source:
    13          repository: alpine
    14          tag: latest
    15      run:
    16        path: sh
    17        args:
    18        - -exc
    19        - sleep 60
    20  
    21  ##########
    22  
    23  resource_types:
    24  
    25  - name: pool
    26    type: docker-image
    27    source:
    28      repository: ((pool-resource-docker-repo))
    29      tag: ((pool-resource-tag))
    30  
    31  ##########
    32  
    33  resources:
    34  
    35  - name: acquire-pool
    36    type: pool
    37    source:
    38      <<: *pool-template
    39      pool: acquire-pool
    40  
    41  - name: claim-pool
    42    type: pool
    43    source:
    44      <<: *pool-template
    45      pool: claim-pool
    46  
    47  ##########
    48  
    49  jobs:
    50  
    51  - name: acquire-1
    52    plan:
    53      - put: acquire-pool
    54        params: {acquire: true}
    55      - task: sleep
    56        <<: *sleep
    57    ensure:
    58      put: acquire-pool
    59      params: {release: acquire-pool}
    60  
    61  - name: claim-1
    62    plan:
    63      - put: claim-pool
    64        params: {claim: slot-1}
    65      - task: sleep
    66        <<: *sleep
    67    ensure:
    68      put: claim-pool
    69      params: {release: claim-pool}
    70  

The extension reports the following (wrong) errors:

[kdvolder]

Perhaps as a first 'modest' goal we could take the given examples and just try and make the 'false positives' go away. In other words, you won't get all the content assist etc as you use anchors and references, but at least we could have the validation be smart enough to recognize you are doing something it doesn't understand and stop complaining about that. That probably isn't too hard especially given these excellent examples to use for crafting some test cases :-)

[marco-m]

The 'modest' goal would be good for me :-)

[marco-m]

Hello @kdvolder, I was wondering if you had any chance/plan/interest to work on this feature ?

[kdvolder]

Interest, yes. Chance... not yet. Plan... nothing concrete. Meaning this ticket has not been added to one of our milestones or short term todo lists. Unfortunately there's a lot of things me and the other folks on the tooling team are working and that are competing for our attention.

Sorry I don't have better news for you. I do have every intention of trying to fix this eventually, because I think its an important feature. But because its not something simple I can just fix in a couple hours between other tasks, it will only happen once it gets put into a milestone or todo list during one of our team formal planning sessions.

[marco-m]

@kdvolder I understand perfectly; thanks for the answer!

[kdvolder]

Good news everyone :-)

This should now be working. Validation now understands ">>", "&" and "*". Please give it a try from snapshot build. You can download a vscode .vsix snapshot from this webpage: http://dist.springsource.com/snapshot/STS4/nightly-distributions.html

If you find any problems with it, please let me know by raising additional bugs (and do attach specific examples as that's really very very helpful :-)

[martinlippert]

Should we also offer content-assist for references?

[marco-m]

@kdvolder thanks a lot!

I tested the nightly build and it works perfectly! ❤️

For people wanting to try a unreleased vscode extension, the full documentation is at https://code.visualstudio.com/docs/editor/extension-gallery#_install-from-a-vsix

In short

• Find latest version from http://dist.springsource.com/snapshot/STS4/nightly-distributions.html
• curl -O http://s3-test.spring.io/sts4/vscode-extensions/snapshots/vscode-concourse-1.4.0-201902061358.vsix
• code --install-extension vscode-concourse-1.4.0-201902061358.vsix
• if you had vscode open, reload the extension.

NOTE: I think I read somewhere that if you install a VSIX then auto update doesn't work anymore. If that is the case, I think it is enough to uninstall the extension and then install it again from the release.

[kdvolder]

Should we also offer content-assist for references?

I thought about it, but it doesn't exactly fit in with the existing content-assist infrastructure. It's probably not extremely hard to implement something though. But I think it should be treated as a separate ticket/work item.

[goflow80]

Hi! I‘m not sure if this is a problem, but just in case, please use „>>“ (reference https://blog.betomorrow.com/yaml-tips-and-tricks-for-concourse-ci-35a3b577a239). Hope this helps.

[kdvolder]

@goflow80 I think you mean "<<". The people who raised this ticket are using anchors, references and "<<". The issue is about extending the concourse editor to support this and not give bogus error messages. The issue has been closed because the editor has been fixed accordingly.