Vega: Down-sampling large datasets

[kevinmickey]

When I visualize larger datasets, I am experiencing some sluggishness (using svg or canvas). Is there a way to down-sample points, so that you only plot one point per pixel, for instance? I'd like for the graph to be interactive, like the overview+detail example, so I don't think I can just down-sample before plotting.

Here are some examples. I think this one just downsamples uniformly:
Dygraphs plugin: http://kaliatech.github.io/dygraphs-dynamiczooming-example/example1.html

These examples try a smarter algorithm:
Flot plugin: http://flot.base.is/
Pure JS: https://github.com/joshcarr/largest-triangle-three-buckets.js

[kevinmickey]

My first attempt in this direction was to use the expression datum._id % 5000 == 0 in the test for a data transform filter. (Note that this would not be adaptive to zooming, like I eventually want.) This seems to be causing problems, though...

[SaleRise]

Any update on this one. This looks similar to other issue I posted...

[jheer]

The basic way to accomplish this in Vega is to use an aggregate transform in conjunction with a bin transform (or similarly functioning formula transform) to discretize the domain. You will have to determine the appropriate bin granularity yourself (e.g., using Vega expressions). Then the aggregate can group by these bins and generate the appropriate down-sampled value(s) (average, median, min, max, ...).

To make that all a bit more concrete, here is an example. The pixelRes signal indicates the desired pixel granularity of downsampling (samples up to every 1 pixel, up to every 2 pixels, etc), which helps parameterize the bin transform. The data is then downsampled via groupby aggregation, with the ability to view different aggregate operations.

{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "width": 180,
  "height": 390,
  "padding": 10,

  "signals": [
    {
      "name": "pixelRes", "value": 1,
      "bind": {"input": "range", "min": 1, "max": 10, "step": 1}
    },
    {
      "name": "aggrOp", "value": "mean",
      "bind": {"input": "select", "options": ["mean", "median", "min", "max"]}
    }
  ],

  "data": [
    {
      "name": "sp500",
      "url": "data/sp500.csv",
      "format": {
        "type": "csv",
        "parse": {"price": "number", "date": "date"}
      },
      "transform": [
        {
          "type": "extent",
          "field": "date",
          "signal": "dateExtent"
        }
      ]
    },
    {
      "name": "downsampled",
      "source": "sp500",
      "transform": [
        {
          "type": "bin",
          "field": "date",
          "extent": {"signal": "dateExtent"},
          "step": {"signal": "pixelRes * span(dateExtent) / width"},
          "nice": false,
          "as": ["datebin0", "datebin1"]
        },
        {
          "type": "aggregate",
          "groupby": ["datebin0"],
          "ops": [{"signal": "aggrOp"}],
          "fields": ["price"],
          "as": ["price"]
        }
      ]
    }
  ],

  "scales": [
    {
      "name": "xDetail",
      "type": "time",
      "range": "width",
      "domain": {"signal": "dateExtent"},
      "nice": false
    },
    {
      "name": "yDetail",
      "type": "linear",
      "range": "height",
      "domain": {"data": "sp500", "field": "price"},
      "nice": true, "zero": false
    }
  ],

  "axes": [
    {"orient": "bottom", "scale": "xDetail", "tickCount": 3},
    {"orient": "left", "scale": "yDetail"}
  ],

  "marks": [
    {
      "type": "line",
      "from": {"data": "downsampled"},
      "sort": {"field": "datum.datebin0"},
      "encode": {
        "enter": {
          "stroke": {"value": "steelblue"},
          "strokeMiterLimit": {"value": 1}
        },
        "update": {
          "x": {"scale": "xDetail", "field": "datebin0"},
          "y": {"scale": "yDetail", "field": "price"}
        }
      }
    },
    {
      "type": "symbol",
      "from": {"data": "downsampled"},
      "encode": {
        "enter": {
          "size": {"value": 16}
        },
        "update": {
          "x": {"scale": "xDetail", "field": "datebin0"},
          "y": {"scale": "yDetail", "field": "price"},
          "fill": {"value": "steelblue"}
        },
        "hover": {
          "fill": {"value": "firebrick"}
        }
      }
    }
  ]
}

[SaleRise]

@jheer Thanks. It works great for single points. Struggling to have it worked for grouped stack like multiple line charts. Try playing with it and change config. However seems to not distinguish between different line charts. for ref, https://vega.github.io/editor/#/examples/vega/line-chart

[alp82]

Thanks @jheer for that example. Is there a way to show the whole range of data instead of calculating a single aggregated value per pixel? For example: zoomed out i would have five values for one pixel: [2,3,3,4,8]. Total is 20, average is four. Now could we just show an area chart that has minimum of 2 and maximum of 8 for that one pixel? (similar to this highcharts example: https://www.highcharts.com/demo/arearange)