Fluentd-kubernetes-daemonset: Sending logs to Graylog

Created on 27 May 2018 · 21Comments · Source: fluent/fluentd-kubernetes-daemonset

There is no example of Graylog DaemonSet, I tried to create my own by modifying elasticsearch example. But I receive the following error:

[out_graylog] failed to flush the buffer. error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="out_graylog"

I also noticed that protocol is not configured in fluentd.conf, so I tested UDP and TCP with the same result.

fluentd-daemonset-graylog.yaml I created:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.2-debian-graylog
        env:
          - name:  FLUENT_GRAYLOG_HOST
            value: "logs-graylog.default.svc.cluster.local"
          - name:  FLUENT_GRAYLOG_PORT
            value: "12201"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Any idea?
Thanks

Source

cloud-rocket

Most helpful comment

Update, TCP is working properly, I just had to change the attribute shared and switch the GELF input in graylog to TCP.

My RBAC version:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: yourdockerpath/your:image
        env:
          - name:  FLUENT_GRAYLOG_HOST
            value: "tcp.log.com"
          - name:  FLUENT_GRAYLOG_PORT
            value: "12201"
          - name: FLUENT_GRAYLOG_PROTOCOL
            value: "tcp"
        resources:
          limits:
            # ===========
            # Less memory leads to child process problems.
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        securityContext:
          privileged: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

miguelortize on 29 May 2018

👍3 😕1

All 21 comments

I also want to add this daemonset as a default log collector for Graylog helm chart I am developing - https://github.com/imubit/graylog-helm-chart

cloud-rocket on 27 May 2018

Hi!

I noticed that the master nodes are the ones with that error message, if you look at the other pods logs, you will notice that they have successful outputs.

UDP is the default protocol set as default: https://github.com/bodhi-space/fluent-plugin-gelf-hs/blob/master/lib/fluent/plugin/out_gelf.rb

I was able to change it successfully to TCP by adding an attribute:

<match **>
   @type gelf
   @id out_graylog
   log_level info
   include_tag_key true
   host "#{ENV['FLUENT_GRAYLOG_HOST']}"
   port "#{ENV['FLUENT_GRAYLOG_PORT']}"
   protocol "#{ENV['FLUENT_GRAYLOG_PROTOCOL']}"
   buffer_chunk_limit 4096K
   buffer_queue_limit 512
   flush_interval 5s
   max_retry_wait 30
   disable_retry_limit
   num_threads 8
</match>

However, even when fluentd is able to generate the connection (graylog shows 8 connections which is the amount of pods I have set in the daemonset), no inputs show up.

I created a TCP GELF input.

Is TCP supported?

miguelortize on 29 May 2018

Also, increase the resource limit to at least 400! Otherwise it will be getting killed every-time without you noticing.

miguelortize on 29 May 2018

Update, TCP is working properly, I just had to change the attribute shared and switch the GELF input in graylog to TCP.

My RBAC version:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: yourdockerpath/your:image
        env:
          - name:  FLUENT_GRAYLOG_HOST
            value: "tcp.log.com"
          - name:  FLUENT_GRAYLOG_PORT
            value: "12201"
          - name: FLUENT_GRAYLOG_PROTOCOL
            value: "tcp"
        resources:
          limits:
            # ===========
            # Less memory leads to child process problems.
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        securityContext:
          privileged: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

miguelortize on 29 May 2018

👍3 😕1

Encountered the same problem, but on slave nodes too.

Error is followed by:

2018-06-01 11:32:28 +0000 [warn]: #0 [out_graylog] failed to flush the buffer. error_class="Encoding::UndefinedConversionError" error="\"\\xE2\" from ASCII-8BIT to UTF-8" plugin_id="out_graylog"
2018-06-01 11:32:28 +0000 [error]: #0 [out_graylog] error on output thread error_class=NoMethodError error="undefined method `+' for nil:NilClass"
  2018-06-01 11:32:28 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.1/lib/fluent/plugin/buffer.rb:478:in `block in takeback_chunk'
  2018-06-01 11:32:28 +0000 [error]: #0 /usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
  2018-06-01 11:32:28 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.1/lib/fluent/plugin/buffer.rb:473:in `takeback_chunk'

danielefranceschi on 1 Jun 2018

I think there is a problem with encoding, some values from the payload sent to graylog have â which is not being able to be converted to UTF-8 when JSON.parse is trying to do its thing.

Check your graylog, that pod should be sending logs, as far as I see, it only doesnt send a payload when it fails to convert.

What I do not know is if this is being worked on.

miguelortize on 1 Jun 2018

having the same issue:

[out_graylog] failed to flush the buffer. error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="out_graylog"

alanh0vx on 6 Jun 2018

See: https://docs.fluentd.org/v1.0/articles/faq#i-got-enconding-error-inside-plugin.-how-to-fix-it?

repeatedly on 7 Jun 2018

👍1

Great ! Thanks @miguelortize
I created my Daemonset based on your fluent.conf and daemonset yaml file.

Here is my fluentd-kubernetes-daemonset. I forked the official https://github.com/fluent/fluentd-kubernetes-daemonset and made some changes:

Change the Dockerfile to use another mirrors in my country(China);
Add FLUENT_GRAYLOG_PROTOCOL to conf/fluent.conf
Build my own docker image
Create a fluentd-kubernetes-daemonset.yaml file
- I also commented the tolerations cause I don't want pod schedule onto master nodes;
- Set my own image
- Set my own graylog host
- Commented terminationGracePeriodSeconds cause pod won't be created by daemonset if not do this.

ykfq on 28 Aug 2018

To clarify what @repeatedly sent, I managed to get rid of the conversion error by making a custom build and installing an extra plugin:

gem "fluent-plugin-record-modifier"

I then edited the fluent.conf to convert all incoming messages to UTF-8:

<filter **>
      @type record_modifier
      char_encoding utf-8
</filter>

bjaworski3 on 29 Aug 2018

👍1

i am facing a issue for graylog integration, ,this is my k8s pod logs.
kops.int@ansible-dt:~/prometheus/fluentd/git$ kubectl logs -f fluentd-rq46w -n kube-system chown: changing ownership of '/fluentd/etc/..2018_10_04_09_10_33.609619388/fluent.conf': Read-only file system chown: changing ownership of '/fluentd/etc/..2018_10_04_09_10_33.609619388': Read-only file system chown: changing ownership of '/fluentd/etc/fluent.conf': Read-only file system chown: changing ownership of '/fluentd/etc/..data': Read-only file system chown: changing ownership of '/fluentd/etc': Read-only file system 2018-10-04 09:16:09 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf" 2018-10-04 09:16:09 +0000 [info]: starting fluentd-0.12.43 2018-10-04 09:16:09 +0000 [info]: gem 'fluentd' version '0.12.43' 2018-10-04 09:16:09 +0000 [info]: adding match pattern="fluent.**" type="null" 2018-10-04 09:16:09 +0000 [info]: adding match pattern="**" type="gelf" 2018-10-04 09:16:09 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Unknown output plugin 'gelf'. Run 'gem search -rd fluent-plugin' to find plugins" 2018-10-04 09:16:09 +0000 [info]: process finished code=256 2018-10-04 09:16:09 +0000 [warn]: process died within 1 second. exit. kops.int@ansible-dt:~/prometheus/fluentd/git$

my config map for fluentd is as below.

  fluent.conf: |
    <match fluent.**>
    type null
    </match>

    <match **>
     @type gelf
     @id out_graylog
     log_level info
     include_tag_key true
     host "{ENV['FLUENT_GRAYLOG_HOS']}"
     port "{ENV['FLUENT_GRAYLOG_PORT']}"
     protocol  "{ENV['FLUENT_GRAYLOG_PROTOCOL']}"
     buffer_chunk_limit 4096K
     buffer_queue_limit 512
     flush_interval 5s
     max_retry_wait 30
     disable_retry_limit
     num_threads 8
    </match>

    <source>
      type tail
      path /var/log/syslog,/var/log/messages
      pos_file /var/log/fluentd-system-logs.log.pos
      encoding UTF-8
      time_format %b %d %H:%M:%S
      tag system.*
      format syslog
      message_format rfc3164
      read_from_head true
    </source>

ghost on 4 Oct 2018

I'm running into this error as well but only on nodes that are running tiller pods.

I've tried the approaches suggested in FAQ but nothing helps.

Error messages in fluentd logs:

~
2018-11-29 10:47:04 +0000 [debug]: #0 [out_graylog] flush_thread actually running
2018-11-29 10:47:05 +0000 [warn]: #0 [out_graylog] failed to flush the buffer. error_class="Encoding::UndefinedConversionError" error="\"\xC2\" from ASCII-8BIT to UTF-8" plugin_id="out_graylog"
2018-11-29 10:47:05 +0000 [debug]: #0 [out_graylog] taking back chunk for errors. chunk="57bca9ffa02c61587f4aee7e61d725e7"
2018-11-29 10:47:05 +0000 [warn]: #0 [out_graylog] failed to flush the buffer. retry_time=0 next_retry_seconds=2018-11-29 10:47:06 +0000 chunk="57bca9ffa02c61587f4aee7e61d725e7" error_class=Encoding::UndefinedConversionError error="\"\xC2\" from ASCII-8BIT to UTF-8"
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:160:in encode' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:160:into_json'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:160:in notify_with_level!' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:119:innotify!'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-gelf-hs-1.0.7/lib/fluent/plugin/out_gelf.rb:92:in block in write' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/event.rb:323:ineach'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/event.rb:323:in block in each' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/buffer/file_chunk.rb:172:inopen'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/event.rb:322:in each' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-gelf-hs-1.0.7/lib/fluent/plugin/out_gelf.rb:90:inwrite'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/compat/output.rb:131:in write' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/output.rb:1123:intry_flush'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/output.rb:1423:in flush_thread_run' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/output.rb:452:inblock (2 levels) in start'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create' 2018-11-29 10:47:05 +0000 [warn]: #0 [out_graylog] failed to flush the buffer. error_class="Encoding::UndefinedConversionError" error="\"\\xC2\" from ASCII-8BIT to UTF-8" plugin_id="out_graylog" 2018-11-29 10:47:05 +0000 [debug]: #0 [out_graylog] taking back chunk for errors. chunk="57bca9fad9ef52bef0003e382dea77ef" 2018-11-29 10:47:05 +0000 [warn]: #0 [out_graylog] failed to flush the buffer. retry_time=0 next_retry_seconds=2018-11-29 10:47:06 +0000 chunk="57bca9fad9ef52bef0003e382dea77ef" error_class=Encoding::UndefinedConversionError error="\"\\xC2\" from ASCII-8BIT to UTF-8" 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:160:inencode'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:160:in to_json' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:160:innotify_with_level!'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:119:in notify!' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-gelf-hs-1.0.7/lib/fluent/plugin/out_gelf.rb:92:inblock in write'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/event.rb:323:in each' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/event.rb:323:inblock in each'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/buffer/file_chunk.rb:172:in open' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/event.rb:322:ineach'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-gelf-hs-1.0.7/lib/fluent/plugin/out_gelf.rb:90:in write' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/compat/output.rb:131:inwrite'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/output.rb:1123:in try_flush' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/output.rb:1423:inflush_thread_run'
2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin/output.rb:452:in block (2 levels) in start' 2018-11-29 10:47:05 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.3.0/lib/fluent/plugin_helper/thread.rb:78:inblock in thread_create'
2018-11-29 10:47:06 +0000 [warn]: #0 [out_graylog] failed to flush the buffer. error_class="Encoding::UndefinedConversionError" error="\"\xC2\" from ASCII-8BIT to UTF-8" plugin_id="out_graylog"
2018-11-29 10:47:06 +0000 [debug]: #0 [out_graylog] taking back chunk for errors. chunk="57bca9ffa02c61587f4aee7e61d725e7"
~

The buffer content is kept in the following gist
https://gist.github.com/harshal-shah/719c894b37fe8f51e5f1bc49b1c6249a

harshal-shah on 29 Nov 2018

@harshal-shah It looks like encoding conversion problem. I made custom image and added plugin to fix it as suggested by @bjaworski3 above.

My repo is here: https://github.com/dekses/fluentd-kubernetes-daemonset

ysubach on 29 Nov 2018

@miguelortize Can you explain why you needed to add privileged here? Other examples doesn't use this. Just curious.

...
        securityContext:
          privileged: true

shinebayar-g on 4 Apr 2019

Hello all,
I also tried to create my own by modifying elasticsearch example. But I receive the following error:

2019-08-08 09:38:05 +0000 [info]: adding match pattern="**" type="gelf"
2019-08-08 09:38:05 +0000 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown output plugin 'gelf'. Run 'gem search -rd fluent-plugin' to find plugins"
2019-08-08 09:38:05 +0000 [info]: process finished code=256
2019-08-08 09:38:05 +0000 [error]: fluentd main process died unexpectedly. restarting.
2019-08-08 09:38:05 +0000 [info]: starting fluentd-0.12.43

It will keep like that and restart service in a infinity loop.
Do anyone has public images fluentd-graylog that has work well :D.

Tks in advance,

thodquach on 8 Aug 2019

@thodquach use v1.4.2-debian-graylog-1.1

I'll submit my graylog working example to this repo very soon.

shinebayar-g on 8 Aug 2019

@thodquach use v1.4.2-debian-graylog-1.1

I'll submit my graylog working example to this repo very soon.

Tks @shinebayar-g , Hope to see you soon :D

thodquach on 8 Aug 2019

Submitted PR https://github.com/fluent/fluentd-kubernetes-daemonset/pull/342 @thodquach

shinebayar-g on 11 Aug 2019

👍1

Submitted PR #342 @thodquach

Tks so much @shinebayar-g =))) ( BTW, the icon Tyrannosaurus Rex looks funny )

thodquach on 12 Aug 2019

😄1

@miguelortize any chance to get this working with TLS?
There is an issue from the rancher gui when using fluentd to send logs to graylog https://github.com/rancher/rancher/issues/23052

chuegel on 25 Sep 2019

@huegelc Create follow-up issue regarding TLS support: #355. I'll be working on it this week, can pull you in for reviews as well if you like :+1:

invidian on 7 Oct 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Images unable to start with Read-Only ConfigMaps

nrmitchi · 11Comments

fluent-plugin-google-cloud is outdated

ozayss · 4Comments

[error]: config error file="/fluentd/etc/fluent.conf" error="Invalid Kubernetes API v1 endpoint https://172.28.0.1:443/api: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed"

Jacky2 · 6Comments

json log not getting parsed to the output record fields

ericln · 3Comments

Unify multi-arch tags

billinghamj · 4Comments