Azure-docs: Parsing errors when using pod annotation

[stefanrothnet]

When I try to configure using pod annotation
interval = "10s"
monitor_kubernetes_pods = true
- prometheus.io/scrape:"true"
- prometheus.io/port:"8080"
I am getting parsing errors....
****Start Config Processing*****
Parsing File /etc/config/settings/prometheus-data-collection-settings
config::configmap container-azm-ms-agentconfig for settings mounted, parsing values from /etc/config/settings/prometheus-data-collection-settings
config::error::Exception while parsing toml config file:
parse error on value "prometheus" (IDENTIFIER), using defaults
cat: config_env_var: No such file or directory
/opt/main.sh: line 134: config_env_var: No such file or directory
****Start Prometheus Config Processing*****
config::configmap container-azm-ms-agentconfig for settings mounted, parsing values for prometheus config map
config::error::Exception while parsing toml config file for prometheus config:
parse error on value "prometheus" (IDENTIFIER), using defaults
****End Prometheus Config Processing******
How do I implement it so it works? I only got to work it when I specify an URL to the pod like
urls = ["http://sample-go.default:8080/metrics"]

Please provide a working example...

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 75504e44-7da6-ac6c-c972-500dda560684
• Version Independent ID: f8728d39-8aa6-4588-4127-1d27a471fcb0
• Content: Configure Azure Monitor for containers agent data collection
• Content Source: articles/azure-monitor/insights/container-insights-agent-config.md
• Service: azure-monitor
• GitHub Login: @MGoedtel
• Microsoft Alias: magoedte

[SwathiDhanwada-MSFT]

@stefanrothnet Thanks for your comment. We are actively investigating and will get back to you shortly.

[MGoedtel]

@stefanrothnet - reviewing this issue and will get back to you shortly. #in-progress

[vishiy]

@stefanrothnet - all you need to specify in the config map for annotation based scraping is below (under[prometheus_data_collection_settings.cluster]) -
monitor_kubernetes_pods = true

The below are annotations you need to specify in your pod (not in your configmap) for the agent to know where to scrape metrics from, for your specific pod(s) -

    # - prometheus.io/scrape: true -------Enable scraping for this pod
    # - prometheus.io/scheme: http   --------------If the metrics endpoint is secured then you will need to set this to `https` 
    # - prometheus.io/path: /mymetrics ------------If the metrics path is not /metrics, define it with this annotation.
    # - prometheus.io/port: 8080 ----------------If port is not 9102 use this annotation

[stefanrothnet]

Hi vishi

Thank you for your investigation. The documentation is very confusing, e.g. if I take this section here...
Configure and deploy ConfigMaps

The article says we should modify the config map and in the examples the article sections are uncommented compared to the config map file. In addition it is not clear what to modify in my pod and what in the config map file.

I'll play with an example and get back to you, how it goes...

[vishiy]

@stefanrothnet - thanks for feedback. we will fix the doc soon. @MGoedtel

[MGoedtel]

@stefanrothnet & @vishiy - Already started working on this yesterday after realizing the details/steps related to this aren't clear enough based on Stefan's response. #in-progress

[MGoedtel]

@stefanrothnet - I've just completed a set of updates to this section after sitting down with engineering to validate the corrections. They should help clarify what changes are applied where with regards to Prometheus metric scraping. Other minor changes included during this edit so it should go live tomorrow. #please-close