Mapbox-gl-js: Add data, datastart and dataend events

[scothis]

There are events for when individual tiles load, but it's difficult to know when all the tiles for a source have finished loaded. Essentially, I'd like an API to know if tiles requests are in flight, or if the all the data I would expect to view for a given view is already loaded. Firing an event when the state transitions would also be helpful.

Knowing when a source is fully loaded is useful for smoothly transitioning between different overlay sources without a flash as tiles load.

As a quick sketch, I'd imaging something like:

map.on('source.loading', function (e) {
  e.status; // one of 'fetching', 'idle', 'errored'
});

[jfirebaugh]

https://www.mapbox.com/mapbox-gl-js/api/#Map.load ?

[anandthakker]

@jfirebaugh I think Map.load doesn't account for view changes requiring new tiles to be downloaded?

@scothis A related issue I encountered today (copying from a #sneak-peek slack conversation):

I’m trying to use featuresIn on load, and i think i’m running into a race. I am waiting until i see source.load events for the sources that i need features from, but even though source.load has fired, sometimes some of the individual ​_tiles_​ aren’t loaded. I just tried a workaround where, if !source.loaded(), wait for tile.load and check again, but the problem is that if there’s a missing tile, source.loaded() never becomes true. I’m thinking about hacking this in by setting an errored flag here: https://github.com/mapbox/mapbox-gl-js/blob/master/js/source/vector_tile_source.js#L77
and then having tile_pyramid.loaded check for loaded || errored instead of just loaded.

Just tried that change here: https://github.com/mapbox/mapbox-gl-js/compare/master...anandthakker:fix-source-loaded. Seems to resolve my particular issue; lemme know if you want a pr, though maybe you'd rather solve this more systematically.

[anandthakker]

^ Hmm, okay so my hack doesn't really resolve the problem after all: even after correcting pyramid.loaded() and then waiting for it, I end up with missing data in my featuresIn request. My guess is that loaded() comes back true when all the tiles that have been requested have come back, but there more yet that need to be requested for the current view.

I guess this is my :+1: for something like what @scothis proposed.

[scothis]

@jfirebaugh looking for something like the load event that stays up to date, and doesn't only fire once.

[jfirebaugh]

Got it, thanks. I think maybe we should make load fire whenever the view quiesces. This would be a breaking change and we'd need to switch most examples to map.once('load', ...).

[derrickpelletier]

Just wanted to +1 that I'm looking for something similar for when a geojson-source data url has loaded. Which I suppose would inherently trigger tile-loaded events. Just a slightly different case than a regular tile source.

[lucaswoj]

I'm in favor of building this and calling the event something other than load.

Some strawman proposals:

• quiesce
• coalesce
• stabilize

[tmcw]

complete?

[DannyDelott]

What about updating the function signatures of map#addSource to take accept an optional callback as the last argument?

 map.addSource(id, source, function(){ 
   // source loaded, do stuff...
});

[mourner]

Leaflet uses load for this, although I agree that it's a bit ambiguous. complete sounds fine.

[tristen]

+1. Related to https://github.com/mapbox/mapbox-gl-js/issues/1715#issuecomment-162101219 I would expect source.load to fire twice in this scenario and sometimes there's a long lag between the geojson layer finally added to the map:

[timjcook]

I have successfully been able to detect when my features have loaded by watching the render event that the map fires when a single style is rendered and each time checking map.loaded(). I can then know it is ok to call featuresIn() and I will get my features.

map.on("render", function() {
  if(map.loaded()) {
    map.featuresIn(...);
  }
});

Ideally, an event at the exact time all layers on the map are ready is better, rather than checking all the render events but this is getting me by. Looking forward to the update.

[DannyDelott]

+1 @timjcook

[ryanbaumann]

+1 @timjcook that worked for use case as well, adding a loading bar while source.setData() is executing.

[lucaswoj]

I'm thinking about resolving this issue with

• Mapⓔdata: a resource (tile, sprite, glyph) has been loaded OR changed with GeoJSONSource#setData
• Mapⓔdataend: no future data events will be fired (unless map is interacted with)

This parallels the structure of

• Mapⓔmove: the map's viewport has been moved
• Mapⓔmoveend: no future move events will be fired (unless map is interacted with)

💭 @scothis?

[scothis]

@lucaswoj works for me

[jfirebaugh]

👍

I like the symmetry there.

[bsudekum]

@lucaswoj any movement on these events?

[lucaswoj]

No movement on this PR. In almost every case, render is the best event to use. Closing this PR as stale.

[lucaswoj]

My mistake -- this is the issue not the PR.

[brentlemons]

Is there a target release for this?

[lucaswoj]

Below are explanations of the existing fire-ers and listeners of the events that are going to be refactored:

layer.add

todo

layer.remove

todo

source.add

todo

source.change

• fired by GeoJSON source after a setData operation has completed http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/geojson_source.js#L122
• fired by ImageSource after setCoordinates operation has completed http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/image_source.js#L109
• listened to by SourceCache to trigger a Map#reload and Map#update http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/source_cache.js#L53
• fired by VideoSource after setCoordinates operation has completed http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/video_source.js#L138
• listened to by AttributionControl to trigger an update of the attributon string http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/ui/control/attribution.js#L58
• listened to by Map to trigger a Map#update http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/ui/map.js#L215

source.load

• fired by GeoJSONSource after the initial data has been sent to the Worker http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/geojson_source.js#L91-L92
• fired by ImageSource after the image data has been loaded http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/image_source.js#L56
• fired by RasterTileSource after the TileJSON has been loaded http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/raster_tile_source.js#L20
• listened to by SourceCache to initialize some properties that might not have been available before this event was fired http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/source_cache.js#L33-L34
• fired by VectorTileSource after the TileJSON has loaded http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/vector_tile_source.js#L26
• fired by VideoSource after the video has loaded http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/video_source.js#L76
• listened to by Style to trigger source-layer validation http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/style/style.js#L79
• listened to by AttributionControl to trigger attribution string update http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/ui/control/attribution.js#L57
• listened to by Map to trigger a Map#update operation http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/ui/map.js#L214

source.remove

todo

style.change

• fired by ImageSprite after both the sprite JSON and sprite image have loaded http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/style/image_sprite.js#L23 http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/style/image_sprite.js#L44
• listened to by ImageSprite to proxy a resized version of ImageSprite 😬 http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/style/image_sprite.js#L61
• fired by Style after a Map#update operation has applied changes http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/style/style.js#L303-L304
• listened to by Style to trigger an update of SpriteAtlas after ImageSprite has new data http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/style/style.js#L721
• listened to by Map to trigger a Map#update operation http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/ui/map.js#L200-L201

style.load

• fired by Style to indicate that the style JSON has loaded http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/style/style.js#L70
• listened to by Map to trigger a jump to the style's specified coordinates and a Map#update operation http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/ui/map.js#L193

tile.add

• fired by SourceCache for each additional tile needed after the viewport has changed http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/source_cache.js#L410

tile.load

• fired by SourceCache after a tile has been populated with data http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/source_cache.js#L169
• fired by Tile after a tile has completed a Tile#redoPlacement operation http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/tile.js#L147
• listened to by Map to trigger a Map#update operation http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/ui/map.js#L216

tile.remove

• fired by SourceCache for each tile as it is no longer needed http://github.com/mapbox/mapbox-gl-js/blob/24f2135f058372bd7906e3c09148c92c8ffebef0/js/source/source_cache.js#L410

[lucaswoj]

Strawman proposal:

• dataloading will be fired when any resource starts loading
• data will be be fired when any resource loads or changes

interface DataEvent {
    type: 'dataloading' | 'data';
    target: Tile | Source | Style | SpriteImage;
    dataType: (
        'geoJSON' |
        'tileJSON' |
        'style' |
        'sprite' |
        'image' |
        'video' |
        'tile'
    );
}

• datastart will be fired when Map#loaded() transitions from true to false
• dataend will be fired after a data event while Map#loaded() is true (there will be no corresponding datastart event if the change doesn't require loading like GeoJSONSource#setData)

[jfirebaugh]

👍 Very thorough.

        'sourceJSON' |
        'tileJSON' |

Is there a distinction between these two?

[lucaswoj]

@jfirebaugh No, that was my mistake. Updated the spec above. Thank you 😄

EDIT: I meant to write styleJSON there

[anandthakker]

@lucaswoj nice! Couple questions:

1. Are errors meant to be abstracted away in these semantics? If so, how will that work? I would assume we'd still want to get a dataend if a resource load yields an error.
2. Would it make sense to attach the source id when datatype= image|video|rasterTile|vectorTile?

[lucaswoj]

Are errors meant to be abstracted away in these semantics? If so, how will that work? I would assume we'd still want to get a dataend if a resource load yields an error.

We will continue to fire error events to indicate that an error occurred.

Good catch on dataend. I updated the spec above to indicate that a dataend event may be fired after an error also.

Would it make sense to attach the source id when datatype= image|video|rasterTile|vectorTile?

A source property will be attached as the event propagates through Style https://github.com/mapbox/mapbox-gl-js/blob/2693ad5842724cda96c0e8c396aa6607bca5d0f4/js/style/style.js#L338

[lucaswoj]

Todo

PR 0

• [x] create event propagation system

PR 1

• [x] replace source.change with data
• [x] augment source.load with data
• [x] replace style.change with data
• [x] augment style.load with data
• [x] replace tile.add with dataloading
• [x] replace tile.load with data
• [x] replace tile.removewith data
• [x] replace source.add with data
• [x] replace source.remove with data
• [x] replace layer.add with data
• [x] replace layer.remove with data
• [x] document data and dataloading events

PR 2

• [x] fire a dataloading event for source data
• [x] fire a dataloading event for style data
• [x] fire a dataloading event for tile data

PR 3

• [x] replace source.load with sourceload and document it
• [x] replace style.load with styleload and document it

PR 4

• [x] add sourcedata & sourcedataloading events
• [x] add styledata & styledataloading events
• [x] add tiledata & tiledataloading events
• [x] refactor to make data & dataloading events aggregates of ☝️ and fired by Map

PR 5

• [ ] implement datastart event
• [ ] implement dataend event

[thiagoxvo]

Any news on this: implement dataend event ?

[stackTom]

Is the best method still to use on render as @timjcook mentions?

[lucaswoj]

That depends on your use case @stackTom. If you want to listen for an event that fires when the map changes, Map#render is a good fit.

[stackTom]

I have a dense mbtiles loaded on my map, and on zoom end, the source re-renders such that more points are rendered than at higher zoom levels. I need to execute a function when all of these points have been fully rendered at the lower zoom level.

[chriswhong]

+1, I would like to implement a loading spinner on my map when it is fetching tiles or waiting for geojson ajax calls to return. I would love it if there was a simple event that would fire when all layers are fully rendered.

[LDF913]

is it possible to load different data layers dependent and a time of day month. ex. broadcast listener of system time

[lucaswoj]

@LDF913 Due to the large volume of issues we get and the relatively small size of our team, we are unable to provide support here on GitHub. I recommend asking this question on Stack Overflow.

[jdeboer-geoplan]

A reliable dataend event would be great, using render and map.loaded() occasionally doesn't work, on the final render event fire the map is still not loaded. Can't reliably reproduce though.

[joehand]

+1, I would like to implement a loading spinner on my map when it is fetching tiles or waiting for geojson ajax calls to return. I would love it if there was a simple event that would fire when all layers are fully rendered.

I was able to fairly reliably check if ajax-type data was loaded via:

 map.on('data', function (data) {
    if (data.dataType === 'source' && data.isSourceLoaded) {
      console.log('data loaded', data)
      // stop listening to map.on('data'), if applicable
    }
  })

The map.on('data') event is pretty noisy though.

[jucor]

@joehand: thanks for map.on('data') , it sounded great at first :) Sadly, it is so noisy that it misses many events, and as such my spinning wheel (same use case as @chriswhong ) often stays stuck, which makes for very poor user experience. Can't wait for this feature!

[jdeboer-geoplan]

@jucor what I've done is to just start a timer on sourcedata event that checks map.loaded() every half a second and then removes the loading wheel. Not had a problem with this approach

[jucor]

@jdeboer-geoplan Nice workaround, thanks!

[wshaver]

Following this thread as my app also needs this.

[kylebebak]

It's amazing that this issue is 2 years old. This feature is table stakes.

[sebastianovide]

is this work in progress ?

[ericjames]

Is this related to https://github.com/mapbox/mapbox-gl-js/issues/3964

I'd like to see it as on complete as in, no more pending source requests

[strech345]

i don't understand how for example get the event tile.remove. Was it removed, is there a documentation about all events?

i use this.map.on('data', event...
with event.tile.uses i check if it's loaded or unloaded. But there are much events missing of unloaded tiles. I compare to map.style.sourceCaches.xxxxx._tiles. This looks correct but i dont get for all unloaded tiles a event.

[dollysingh3192]

@jfirebaugh I think Map.load doesn't account for view changes requiring new tiles to be downloaded?

@scothis A related issue I encountered today (copying from a #sneak-peek slack conversation):

I’m trying to use featuresIn on load, and i think i’m running into a race. I am waiting until i see source.load events for the sources that i need features from, but even though source.load has fired, sometimes some of the individual _tiles_ aren’t loaded. I just tried a workaround where, if !source.loaded(), wait for tile.load and check again, but the problem is that if there’s a missing tile, source.loaded() never becomes true. I’m thinking about hacking this in by setting an errored flag here: https://github.com/mapbox/mapbox-gl-js/blob/master/js/source/vector_tile_source.js#L77
and then having tile_pyramid.loaded check for loaded || errored instead of just loaded.

Just tried that change here: master...anandthakker:fix-source-loaded. Seems to resolve my particular issue; lemme know if you want a pr, though maybe you'd rather solve this more systematically.

Hi @anandthakker,

I am facing the same issue of "sourcedata" callback is not calling if one of the tiles in viewport fails to load, even the map.loaded() is not true.

I have high dependency on this, I set the state of feature on this basis(querySourceFeatures). I am stuck in this, if there is an alternative to this. Please suggest.

Like if we can abort/unload/remove that particular tile and make e.isSourceLoaded to true in callback

if (e.isSourceLoaded && e.sourceId === "src_chairs") {
//code to execute
}

Also, attaching screenshot

Please give me some suggestions on this.

[hjrobinson]

I appreciate jdeboer-geoplan's suggestion; he or she could have been a little more explicit. I think a recursive timeout is the answer. This seems to work (most of the time) for me. I'm not sure if areTilesLoaded() helps or not. Also, isStyleLoaded could be added.

function checkOutThisLoad(){
    if (map.loaded() == true && map.areTilesLoaded() == true) {
         console.log("load is complete");
    } else {
       console.log("load is incomplete");
       setTimeout(function() {
           checkOutThisLoad();
       },1000);
    }
}

[Babeetlebum]

Edit : I went for the idle event which does exactly what I want

My use case : I want to use queryRenderedFeatures after adding a layer.

It's quite similar to the airport example in the documentation. This issue was avoided by displaying a message Drag the map to populate results but I can't affort to do that. I need the list to be updated when the layer loads

If I understand it correctly, dataend should be the event I need, right ?

Seeing how often the render event is fired I went for a hack by removing the listener after some time after adding the layer

// when a new layer is added
layers$.subscribe((layers) => {
      this.map.on("render", checkRenderedFeatures);
      setTimeout(() => this.map.off("render", checkRenderedFeatures), 2000);
});

This way checkRenderedFeatures is called ~30-50 times before the listener is removed

I don't like it one bit but hopefuly dataend will solve this issue for me. I'm fully aware that my "temporary hack" comment will still be in the codebase in 10 years :D

[hjrobinson]

Yes, idle provided a solution for me too. I don't know how many years the idle event has been available but it should have been mentioned a long time ago for this issue. Like anyone else I guess I could have said something earlier.