Loki: Promtail crashes when running in Windows Container

[Gerthum]

Describe the bug
When attempting to run the Promtail Windows Binary in a Windows Container it crashes with the following output:

fatal error: PowerRegisterSuspendResumeNotification failure

runtime stack:
runtime.throw(0x240ebbd, 0x2e)
        /usr/local/go/src/runtime/panic.go:774 +0x79 fp=0x3cdfdb0 sp=0x3cdfd80 pc=0x4331a9
runtime.monitorSuspendResume()
        /usr/local/go/src/runtime/os_windows.go:294 +0x1a9 fp=0x3cdfe50 sp=0x3cdfdb0 pc=0x42f309
runtime.goenvs()
        /usr/local/go/src/runtime/os_windows.go:531 +0x1ba fp=0x3cdfea8 sp=0x3cdfe50 pc=0x42fc8a
runtime.schedinit()
        /usr/local/go/src/runtime/proc.go:554 +0xa9 fp=0x3cdff00 sp=0x3cdfea8 pc=0x435ba9
runtime.rt0_go(0x3cdff30, 0x7ff81f387974, 0x3cdff30, 0x0, 0x7ff81f387974, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/runtime/asm_amd64.s:214 +0x13d fp=0x3cdff08 sp=0x3cdff00 pc=0x45f84d

To Reproduce
Steps to reproduce the behavior:

1. docker run --entrypoint powershell -it mcr.microsoft.com/windows/servercore:ltsc2019 powershell
2. C:\promtail\promtail-windows-amd64.exe --config.file C:\promtail\promtail_config.yaml

Expected behavior
To run as normal

Environment:

• Infrastructure: Laptop
• Deployment tool: Manual

Screenshots, Promtail config, or terminal output

It looks to be related to the following bug in Golang https://github.com/golang/go/issues/35447 which is resolved in 1.14

Any idea when the move to 1.14 will happen?

[cyriltovena]

Should be soon we were wait for a fix on boltdb blocking 1.14.

See https://github.com/cortexproject/cortex/pull/2480

[Gerthum]

Thank you for the feedback I will test once its upgraded.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[sachinmsft]

any updates on this issue ?

[Gerthum]

We've built our own binary with go 1.14 and it's running without issue in windows containers.

[sachinmsft]

Thanks @Gerthum . were you able to run it on Kubernetes. if yes, Can you please share the promtail config ?

[Gerthum]

Yes we are running it successfully in K8s. We use the standard config apart from the following changes:

1. Dockerfile:

FROM mcr.microsoft.com/windows/servercore:ltsc2019 
RUN mkdir C:\Loki
WORKDIR "C:\Loki"
COPY promtail.exe C:\Loki\
ENTRYPOINT ["promtail.exe"]

1. Daemonset

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: loki-promtail-win
  namespace: logging
  labels:
    app: promtail-win
spec:
  selector:
    matchLabels:
      app: promtail-win
      release: loki
  template:
    metadata:
      labels:
        app: promtail-win
        release: loki
      annotations:
        prometheus.io/port: http-metrics
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: loki-promtail-chart
      imagePullSecrets:
      - name: 
      containers:
        - name: promtail-win
          image: "private-registry/promtail-win:1.0.0"
          imagePullPolicy: IfNotPresent
          args:
            - "-config.file=/etc/promtail/promtail.yaml"
            - "-client.url=http://loki.logging.svc.cluster.local:3100/loki/api/v1/push"
            - "-log.level=debug"
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: varlog
              mountPath: C:\var
              readOnly: true
            - name: varlibdockercontainers
              mountPath: C:\ProgramData\Docker\containers
              readOnly: true

          env:
            - name: HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          ports:
            - containerPort: 3101
              name: http-metrics
          securityContext:
            readOnlyRootFilesystem: true
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            {}
      nodeSelector:
        kubernetes.io/os: windows
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
      volumes:
        - name: config
          configMap:
            name: loki-promtail-win
        - name: varlog
          hostPath:
            path: C:\var
        - name: varlibdockercontainers
          hostPath:
            path: C:\ProgramData\Docker\containers
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
    name: loki-promtail-win
    namespace: logging
    labels:
      app: promtail-win
spec:
    privileged: false
    allowPrivilegeEscalation: false
    volumes:
    - 'secret'
    - 'configMap'
    - 'hostPath'
    hostNetwork: false
    hostIPC: false
    hostPID: false
    runAsUser:
        rule: 'RunAsAny'
    seLinux:
        rule: 'RunAsAny'
    supplementalGroups:
        rule: 'RunAsAny'
    fsGroup:
        rule: 'RunAsAny'
    readOnlyRootFilesystem: true
    requiredDropCapabilities:
    - ALL

1. Make sure the Service Account etc. names match the ones you use for the linux daemonset.

2. Configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-promtail-win
  namespace: logging
  labels:
    app: promtail-win
data:
  promtail.yaml: |
    client:
      backoff_config:
        max_period: 5s
        max_retries: 20
        min_period: 100ms
      batchsize: 102400
      batchwait: 1s
      external_labels: {}
      timeout: 10s
    positions:
      filename: C:\var\positions.yaml
    server:
      http_listen_port: 3101
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: kubernetes-pods-name
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_label_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: C:\var\log\pods\*$1\*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
    - job_name: kubernetes-pods-app
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        source_labels:
        - __meta_kubernetes_pod_label_name
      - source_labels:
        - __meta_kubernetes_pod_label_app
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: C:\var\log\pods\*$1\*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
    - job_name: kubernetes-pods-direct-controllers
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        separator: ''
        source_labels:
        - __meta_kubernetes_pod_label_name
        - __meta_kubernetes_pod_label_app
      - action: drop
        regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
        source_labels:
        - __meta_kubernetes_pod_controller_name
      - source_labels:
        - __meta_kubernetes_pod_controller_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: C:\var\log\pods\*$1\*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
    - job_name: kubernetes-pods-indirect-controller
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: .+
        separator: ''
        source_labels:
        - __meta_kubernetes_pod_label_name
        - __meta_kubernetes_pod_label_app
      - action: keep
        regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
        source_labels:
        - __meta_kubernetes_pod_controller_name
      - action: replace
        regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
        source_labels:
        - __meta_kubernetes_pod_controller_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: C:\var\log\pods\*$1\*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
    - job_name: kubernetes-pods-static
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: ''
        source_labels:
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_label_component
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ''
        source_labels:
        - __service__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - replacement: C:\var\log\pods\*$1\*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
        - __meta_kubernetes_pod_container_name
        target_label: __path__

[cyriltovena]

Feel free to send a pr to support docker windows image.

[Gerthum]

I'm keen to assist in getting the Windows Image supported.

Just a question does it mean integrating it into the build pipeline as well as adding it to helm?

If so might just need a bit of guidance to get started.

[cyriltovena]

Let’s start with just the dockerfile and instructions/docs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[cloudcafetech]

@Gerthum

With servercore image (mcr.microsoft.com/windows/servercore:ltsc2019) its heavy (image size close to more than 4GB).

I tried with powershell nanoserver image () which is light weight but facing following error ..

failed to try resolving symlinks in path "\\var\\log\\pods\\logging_lo