Folium: Rendering Map Fails at ~3000 Markers

[sherl0cks]

Please add a code sample or a nbviewer link, copy-pastable if possible

import folium
import json
import os
import numpy

from folium.plugins import MarkerCluster

m = folium.Map(
    location=[0, 0],
    zoom_start=2,
    tiles='http://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}',
    attr='Tiles © Esri — Source: Esri, i-cubed, USDA, USGS, AEX, GeoEye, Getmapping, Aerogrid, IGN, IGP, UPR-EGP, and the GIS User Community'
)

marker_cluster = MarkerCluster().add_to(m)

random_marker_count=3000        
colors = numpy.random.randint(low=0, high=0xFFFFFF, size=(random_marker_count,1))
lats = numpy.random.random_sample(random_marker_count)
longs = numpy.random.random_sample(random_marker_count)

for i in range(0, random_marker_count):
    lat = ((lats[i]*2) - 1) * 90
    long = ((longs[i]*2) - 1) * 180
    coor = [lat,long]
    c = "#%06x" % colors[i][0]
    folium.CircleMarker(
        location=coor,
        fill=True,
        radius=5,
        color=c,
        fill_color=c
    ).add_to(marker_cluster)

m

Problem description

Somewhere around 3000 markers I can no longer render the map. I can see the cell, it just never populates. Not seeing any errors in the logs.

Am I creating these data points incorrectly? Perhaps I should be loading a GeoJSON layer instead? This was intended to be a simple scaling test.

Expected Output

Map renders in a < 5 seconds.

I'd expect issues at 10,000's or 100,000's of markers, not 1000's.

Output of folium.__version__

'0.5.0'

[Conengmo]

I tried to replicate your problem. In a Jupyter notebook I can create a map with 3000 markers. At 5000 markers I get an error: IOPub data rate exceeded.. To solve this you'd need to do something with Jupyter.

In regular Python I can go to 20 000 markers and open the html file. It does take a while to create though (~4 mins), because it works on one core, so I haven't tried higher numbers. But I don't see why that wouldn't work.

I think your issue with displaying is not a problem with folium, but with Jupyter.

By the way, if you want to speed up marker creation, check out the FastMarkerCluster plugin: http://nbviewer.jupyter.org/github/python-visualization/folium/blob/master/examples/MarkerCluster.ipynb

[sherl0cks]

Thanks @Conengmo.

FastMarkerCluster doesn't seem to just affect marker creation, it improves rendering in Jupyter by an order of magnitude in my tests. The render cuts out somewhere around 37,000 now. FWIW - it's a noticeable cliff. 35,000 markers takes a second or two and 37,000 was minutes.

That said, if I just save the html and load that in a new browser tab, I can load 100,000 markers in ~10 seconds. This approach works fine for me, so let's focus here. According to chrome profiling, almost all of that time is spent on scripting and looking at the resulting .html, it's basically a big list of the marker points with an "addToCluster" section at the bottom. Any thoughts on executing that function server side and then saving the result?

All in all, a good start, but I'm really keen to get another order of magnitude improvement, or I need to look different technologies. All ideas welcome.

[Conengmo]

I haven't looked at what FastMarkerCluster does exactly, but I reckon it's already quite efficient.

According to chrome profiling, almost all of that time is spent on scripting and looking at the resulting .html

This means your talking about Leaflet functionality, and how to optimize that. You could take a look at the Leaflet plugin that's actually doing the work in the browser: https://github.com/Leaflet/Leaflet.markercluster. If you find anything there to further optimize, that could be added to folium.

On their page they say:

The Clusterer can handle 10,000 or even 50,000 markers (in chrome). IE9 has some issues with 50,000.

That suggests that your 100 000 markers are comparable to their performance claim.

Any thoughts on executing that function server side and then saving the result?

What do you mean by 'saving the result`? You could let folium create html server side, but Leaflet rendering is done in a browser, so client side. You could look for running Leaflet server-side or on a headless system. I don't know much about this, so I can't say if it's worth going down this rabbit hole.

As a general note, Leaflet does the actual work, folium 'only' creates the code/html for Leaflet. So look at Leaflet to figure out how to get better performance, and than implement those calls in folium.

[sherl0cks]

What do you mean by 'saving the result`?

I think you hit this on the head with server side / headless approach.

At this point, I have the information I need. Thanks for your help. Closing

[anderl80]

What is the preferred way with folium to create a big data marker map with more than ~3000 markers? ClusterMarker is not an option as this visually distorts the information I want to display.