Mapbox-gl-js: Style spec: Pull more than one data source into a layer

Created on 3 Mar 2017 · 5Comments · Source: mapbox/mapbox-gl-js

Style layers can currently only specify a single data layer. With data driven styling, we're looking at rendering the same data with fewer style layers. I propose adding support for pulling multiple data sources into one style layer. In a data source, the split between putting features into separate data layers vs. putting them in the same data layer and using tags to filter/separate them is somewhat arbitrary. Adding support for multiple data layers also means that they could stem from different sources, e.g. a regular tileset and a GeoJSON file, or runtime annotations.

In terms of style spec change, we could make the source field an array that allows specifying multiple data source/layer combinations:

{
  "id": "landcover",
  "type": "fill",
  "source": [
    ["streets", "landcover"],
    ["geojson", "glaciers"]
  ],
  ...
}

Alternatively, we could use a different attribute like data and move source, source-layer, and filter to that:

{
  "id": "landcover",
  "type": "fill",
  "data": [
    {
      "source": "composite",
      "source-layer": "landcover",
      "filter": [
        "all",
        ["in", "class", "crop", "grass", "scrub", "wood", "snow"]
      ]
    },
    {
      "source": "glaciers.geojson"
    }
  ],
  ...
}

cross-platform

Source

kkaefer

👍9

Most helpful comment

Is this something that might come in a not too far future?

anderslarssonvbg on 18 Sep 2018

👍8

All 5 comments

Previously:

jfirebaugh on 3 Mar 2017

Combining several sources in one layer might be a big challenge because of the need to synchronize data between workers (e.g. if you have VT + GeoJSON). However, implementing support for pulling from multiple source layers in one source could be much easier. We could e.g. optionally move the source-layer property inside the filter syntax for that:

{
  "source": "composite",
  "filter": [
    "any",
    ["all", ["==", "$layer", "landcover"], ["in", "class", "crop", "grass", "scrub", "wood", "snow"]],
    ["all", ["==", "$layer", "hillshade"], ["in", "foo", "bar"]]
  ]
}

mourner on 3 Mar 2017

👍4

Is this something that might come in a not too far future?

anderslarssonvbg on 18 Sep 2018

👍8

Brought forward from #9317 by @sansumbrella

Motivation

I want to use heatmaps and cluster visualizations to display historic data grouped in different time intervals. I already have my data stored in a series of 1-day groups. At runtime, I want to be able to choose what timeframe to visualize: any single day, or any span of seven or thirty days. Because of my visual design choices, the source data needs to be grouped together for display on a single layer.

Design Alternatives

Store all possible data in a single source and filter the data in an expression. This results in always downloading the most possible data and puts a potentially large computing cost on the client device.

Manage source aggregation earlier in the pipeline and change the layer source to the desired aggregation at runtime. I opened a separate ticket for enabling this kind of aggregation when creating tilesets.

Ideally, source aggregation would be supported at multiple points in the application lifecycle to allow for different use cases. Currently application developers must develop their own aggregation scheme and provide a single source when they interface with Mapbox tools.

Design

In the layer style specification, add a sources key that accepts an array of sources. If an array of sources is provided, a singular source and source-layer should not be provided.

Mock-Up

Library developers will support a new variant of the Layer type that contains multiple sources.

interface Source {
  id: string; // Name of a source description to be used in this layer
  layer?: string; // Layer to use from a vector tile source
}

interface LayerBase {
  // …
  // existing style spec for layers omitting source, source-layer, and sources
}

interface LayerWithSingleSource extends LayerBase {
  source: string;
  "source-layer"?: string;
}

interface LayerWithMultipleSources extends LayerBase {
  sources: Source[];
}

type Layer = LayerWithSingleSource | LayerWithMultipleSources;

Application developers can add an array of sources instead of a single source+source-layer combo.

{
  type: "heatmap"
  sources: [ { id: "vehicles-2020-02-16", layer: "vehicles" }, { id: "vehicles-2020-02-17", layer: "vehicles" }, { id: "vehicles-2020-02-18", layer: "vehicles" } ]
}

Concepts

New concept: you can have multiple sources for a single visual layer.

The sources should have compatible feature types for the style layer using them (e.g. all lines or polygons for a line layer).

Implementation

In a simplest implementation, an AggregateSource acts as a facade to a collection of underlying sources. When data is requested from the source, it passes the request to all sources in the collection, aggregates their responses into a single response object, and provides that aggregate response to the code requesting data. Styles and other consumers of sources should not need to know anything about the source or how it works.

Pseudocode ignoring asynchrony and other library details:

class AggregateSource {
  sources: Source[];

  loadTile(tile: Tile): Data[] {
    return this.sources.reduce((aggregate, source) => {
      return aggregate.concat(source.loadTile(tile));
    }, []);
  }
}