Envoy: proposal: make configuration of a tracing provider dynamic

[yskopets]

Title: Make configuration of a tracing provider dynamic

Context:

• at the moment, tracing provider is configured statically as part of bootstrap config
• while this approach is very pragmatic, it breaks overall user experience that Envoy can reconfigured on-the-fly
• although it would be possible to achieve a similar user experience through a hot restart, it feels like a workaround and doesn't seem a long-term solution

Proposal:

• Make configuration of a tracing provider dynamic

Example use cases:

• change a tracing provider on-the-fly (e.g., zipkin to opentracing)
• reconfigure an active tracing provider (e.g., address of a collector)

Anticipated scope of changes:

• extend Envoy bootstrap config with a new trace driver - envoy.tracers.dynamic
• implement configuration model for dynamic tracer, e.g. as a new independent xDS API or as a higher-level abstraction on top of RTDS (Runtime Discovery)
• implement a dynamic tracer as a smart reference to the actual implementation

Open questions:

• which configuration model to choose: new xDS, RTDS or something else? Does Envoy already have an example of turning a part of bootstrap config into a dynamic configuration ?
• it's possible that some tracing providers might not support being configured multiple times (e.g., envoy.dynamic.ot and envoy.tracers.opencensus look questionable). Hopefully, it's acceptable to sacrifice them in the beginning and let them improve over time

[mattklein123]

@yskopets can we just remove tracing from bootstrap entirely and just define it as part of the HCM configuration that would come down as part of LDS? This is similar to what @ramaraochavali did around the rate limit service configuration. Can we follow the same pattern?

[yskopets]

@mattklein123 Let me take a look into rate limit example

[yskopets]

@mattklein123 Here is the updated proposal that follows rate limit example:

• Update configuration of model of HttpConnectionManager to support in-place definition
  of a tracing_provider, e.g.

  message HttpConnectionManager {
  ...
  message Tracing {
    ...
  
    // Configuration for an external tracing provider.
    //
    // Within depracation window, this field may be omitted,
    // in which case Envoy will fall back the to tracing provider configuration
    // from the :ref:`Bootstrap <envoy_api_msg_config.bootstrap.v3.Bootstrap>` config.
    //
    // Once deprecation window is over, tracing provider configuration will be excluded
    // from the :ref:`Bootstrap <envoy_api_msg_config.bootstrap.v3.Bootstrap>` config
    // and this field will become mandatory.
    trace.v3.Tracing tracing_provider = 9;
  }
  }
  

  Sample configuration will look the following way:

  filter_chains:
  - filters:
  - name: envoy.http_connection_manager
    typed_config:
      "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v3.HttpConnectionManager
      tracing:
        tracing_provider:
          http:
            name: envoy.zipkin
            typed_config:
              "@type": type.googleapis.com/envoy.config.trace.v2.ZipkinConfig
              collector_cluster: zipkin
              collector_endpoint: "/api/v2/spans"
              collector_endpoint_version: HTTP_JSON
      codec_type: auto
      ...
      http_filters:
      - name: envoy.router
        typed_config: {}  
  

  Notice tracing :: tracing_provider :: http path. Probably, it should be simplified.

• Introduce HttpTracerCache interface

  /**
  * Interface for a cache of HttpTracers.
  *
  * The cache is responsible for de-duplicating tracers with identical configuration.
  *
  * Notice that the cache doesn't deal with threading since HttpTracers are required to be thread-safe
  * and already address this problem internally.
  */
  class HttpTracerCache {
  public:
  virtual ~HttpTracerCache() = default;
  
  /**
   * Get existing HttpTracer or create a new one for a given configuration.
   * @param config supplies the configuration for the tracing provider.
   * @return HttpTracerSharedPtr.
   */
  virtual HttpTracerSharedPtr getOrCreateHttpTracer(
      const envoy::config::trace::v3::Tracing& config) PURE;
  };
  

• Introduce HttpTracerCacheImpl as a singleton
  class HttpTracerCacheImpl : public Singleton::Instance, public HttpTracerCache { ... }
• Refactor server startup process to initialize HttpTracerCache singleton
  and use it to create an HttpTracer out of Bootstrap config
• Exclude HttpTracer from everywhere, namely from
  
  
  
  • Envoy::Http::Context
  
  
  • Envoy::Server::Configuration::Main
  
  
  • Envoy::Server::Configuration::FactoryContext
  
  
• Update HttpConnectionManagerConfig to use HttpTracerCache singleton to get/create
  an HttpTracer and store it in the HttpConnectionManagerConfig
• Update Http::ConnectionManagerImpl::ActiveStream to use HttpTracer from
  HttpConnectionManagerConfig

[mattklein123]

+1 looks great. @ramaraochavali WDYT?

[yskopets]

@mattklein123 I'm actively working on this feature

[yskopets]

@kyessenov @dio Tracing configuration is now fully dynamic, both in v2 and v3 APIs.
The remaining PRs are optimizations and refactorings.

If you were planning to use this feature, you can give it a try already.

[kyessenov]

Thanks for all your work!