Yq: Can't handle multiple yaml documents within one file

[IdanAdar]

Given the following Kubernetes YAML configuration file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
   name: somename
spec:
   revisionHistoryLimit: 5
   minReadySeconds: 10
   strategy:
      type: RollingUpdate
      rollingUpdate:
         maxUnavailable: 0
         maxSurge: 1
   replicas: 1
   template:
      metadata:
         labels:
            microservice: somename
      spec: 
         containers:
         - name: serviceapi
           image: myimage:0.0.6
           ports:
           - containerPort: 8080
           env:
##### ENVIRONMENT VARIABLES
            - name: somename
              valueFrom:
                configMapKeyRef:
                   name: serviceapi-environment-configuration
                   key: somekey
---
apiVersion: v1
kind: Service
metadata:
  name: somename
  labels: 
     microservice: somename
spec:
   type: NodePort
   ports:
   - port: 8080
   selector:
      microservice: somename
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: some-name
  namespace: default
data:
  someData: data

This command will work:

./yaml w -i myfile.yaml spec.template.spec.containers[0].image myimage:0.0.19

The image value will be replaced, however, the file will then change to this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
   name: somename
spec:
   revisionHistoryLimit: 5
   minReadySeconds: 10
   strategy:
      type: RollingUpdate
      rollingUpdate:
         maxUnavailable: 0
         maxSurge: 1
   replicas: 1
   template:
      metadata:
         labels:
            microservice: somename
      spec: 
         containers:
         - name: serviceapi
           image: myimage:0.0.19
           ports:
           - containerPort: 8080
           env:
##### ENVIRONMENT VARIABLES
            - name: somename
              valueFrom:
                configMapKeyRef:
                   name: serviceapi-environment-configuration
                   key: somekey

As you can see, the Services and ConfigMap sections (denoted with ---) have disappeared.

[mikefarah]

Currently this uses go-yaml to parse yaml files and that does not yet support multiple yaml documents embedded into a separate file (https://github.com/go-yaml/yaml/issues/232) :(

[IdanAdar]

Too bad... hopefully this will be added - soon!

[IdanAdar]

What are the chances of a workaround?

[mikefarah]

Only thing I can think of is:

• preprocessing the document before hand to split it out into seperate docs
• running the update as per usual on the selected doc number (provided via the cli)
• post processing to stitch the documents back together

If I had more time I'd look into it, happy to review and merge a pull request though....

[therealplato]

go-yaml recently merged a fix to their development branch. Really looking forward to this feature for scripting with kube manifests!

[mikefarah]

oh neat :)

[JeanMertz]

was looking for this as well, looks like this is the fix that makes this now possible:

https://github.com/go-yaml/yaml/pull/301

[IdanAdar]

Does this require updating the go version in yq?
@mikefarah

[therealplato]

@mikefarah what needs to be done to resolve this issue? i need it so badly! i'm happy to try and implement it, but not being familiar with either codebase, i could use a hint.

API ideas:
Given a.yml is:

foo: 1
---
bar: 2

When I type yq read a.yml foo
Then I expect output 1
When I type yq read a.yml bar
Then I expect output 2

Given b.yml is:

foo: 1
---
foo: 2

When I type yq read b.yml foo
Then I expect output

1
---
2

I haven't used yq write so I don't have an expectation for the behavior when invoke yq write on a multi-document file.

A totally separate idea: For multi-document use, treat the document number as a top level key that vanishes:

yq read b.yml ---0.foo
1

[mikefarah]

Gday, it would be great to get a PR for this.
I'm thinking the document number as a top level key works well, and we could assume index 0 when it's not provided.

The path idea does have a nice intuitiveness to it too, however ----0 is not super obvious that it's selecting a document imo. I also don't want to force people to select document 0 for single document yaml files, as that is the most common scenario - so as long as we can get away with that then it should work.

yq read b.yml ---d1.foo
2
yq read b.yml foo
1

It could also be an argument ala

yq read --d[ocument] 1 b.yml foo
2

yq read b.yml foo
1

Thoughts? Happy to have further opinions?

I think you'll only need to update yq.go. readData currently takes a filename and populates parsedData, perhaps if that also took a document number and only populated parsedData with that selected document. That should work for read. For write - I'm not sure if you can write to a specific document index with go-yaml - so it might be a bit trickier.

[therealplato]

Updated ideas:

// read second document
yq read document.yaml @1.a.b.c
yq read document.yml @1 a.b.c

I haven't used new, merge, write commands. Any ideas based on your workflows @IdanAdar @JeanMertz ?

[ctaggart]

You can also use the log file example in the yaml 1.2 spec for testing. I need this bad for all kubernetes tools like kubectl and helm.

[mikefarah]

Just started working on this - multi read is working in master, though I haven't built a release yet :)

I will create a release once write is working too.

[ctaggart]

Awesome! I haven't checked it out yet. Was thinking on the design that given the spec example, it currently reads the first document, so it acts like cat log.yaml | yq -d 0 r -. It would be nice to have a --documentcount too. I'd stick with zero-based index like bash/go and most languages. What design did you go with?

---
Time: 2001-11-23 15:01:42 -5
User: ed
Warning:
  This is an error message
  for the log file
---
Time: 2001-11-23 15:02:31 -5
User: ed
Warning:
  A slightly different error
  message.
---
Date: 2001-11-23 15:03:17 -5
User: ed
Fatal:
  Unknown variable "bar"
Stack:
  - file: TopClass.py
    line: 23
    code: |
      x = MoreObject("345\n")
  - file: MoreClass.py
    line: 58
    code: |-
      foo = bar

$ cat log.yaml | yq r -
Time: 2001-11-23 15:01:42 -5
User: ed
Warning: This is an error message for the log file

[mikefarah]

so far, it's like you guessed, I'm adding a '-d' parameter to the commands which allows you to index the documents where 0 is the first and default.

yq w  my_file.yml a.b.c cat // updates the first doc, outputs all documents
yq w -d 1 my_file.yml a.b.c cat // updates the second doc, outputs all documents

Under the hood - this is a significant change, and one of the things that will have to go is JSON output for all commands except read.

Basically (my current plan) is to have 'read' as the only command that outputs a single document (and you can select which one), so a JSON toggles makes perfect sense.

yq r  my_file.yml a.b.c // reads a.b.c from first doc
yq r  -d 1 my_file.yml a.b.c // reads a.b.c from second doc
yq r  my_file.yml  // reads first doc
yq r  -d 1 my_file.yml  // reads second doc

The other commands, like 'write' will need to output all the documents in the yaml (as you are updating a field, but still need the rest of the docs), and so it doesn't make as much sense to support json output. If you need JSON output still for those commands, then you'll need to pipe:

yq w -d 1 my_file.yml a.b.c cat | yq r -d1 -j -

Thoughts?

[mikefarah]

Pre release: https://github.com/mikefarah/yq/releases/tag/2.0-beta

[fkrauthan]

I saw the announcement in the 2.0 beta release notes in regards supporting multiple yaml documents in one file. But the examples always talk about reading information (which require to define the index of the document) what about changing and removing things? And is there going to be a operate on all documents option?

My usecase is for example to remove yaml paths across all yaml files and documents in that file. Therefor I don't know upfront how many yaml documents are in each file.

[mikefarah]

Changing and removing paths for a specific document index is supported. I haven't thought about operating on all documents - though it wouldn't be difficult to add 🤔

Feel free to add this as a separate issue - if there are enough votes/interest I'd be more inclined to work in it. Or even better if there is a pull request ;)

[mikefarah]

You know what - never mind I'll add it in - easier than I thought :)

[mikefarah]

Fixed by https://github.com/mikefarah/yq/releases/tag/2.0.0

[deitch]

I haven't thought about operating on all documents - though it wouldn't be difficult to add
You know what - never mind I'll add it in - easier than I thought :)

I'll bite @mikefarah . How do I read from all documents? Or at least get the number so I can iterate through?

[mikefarah]

Gday - so I've added the ability to operate update commands against all documents

yq w -d'*' examples/multiple_docs.yaml user.surname farah

And you can read from a specific doc

yq r -d1  examples/multiple_docs.yaml user.surname

What I haven't got is the ability to read from multiple

yq r -d'*'  examples/multiple_docs.yaml user.surname

Not sure if that would actually be a useful thing though?

[mikefarah]

Feel free to open another issue (or even better a pull request)

[deitch]

Hi @mikefarah - and a good afternoon. :-)

Simple use case: I am processing a directory with 3 yml files. Eg I need to extract the name and location property from each of them, or perhaps convert them to json for processing by some other system. I can for loop through all three files, so that’s good.

2 of the files are single document, so no problem for yq to extract the properties or convert. The third is a multi document file. I do not know in advance how many docs it contains. How would I process it (extract the props or convert)?

Either of the above would work. If I could -d ‘*’ then it would process all of the docs in the file. If I could extract the count of docs in the file, then I could for loop through each, passing to -d.

Feel free to open another issue

Sure. Happy to do so and copy this comment there as well.

better a pull request

I’ll try; I suspect you’ll get to it more quickly. Currently got open work going on in more repos than I care to think of (calico, linuxkit, kubesync, storas, not to mention closed stuff for clients... :-) ).

[deitch]

Opened.

[francescobianco]

@IdanAdar @deitch @JeanMertz @mikefarah My solution is use Yamlinc a tool to build multiple file into one with special tag $include take a look here https://github.com/javanile/yamlinc