Mapbox-gl-js: OpenGL state management system

[kkaefer]

We should maintain all current state in the painter object so that we can avoid doing redundant calls (e.g. to bind buffers or switch shaders). Some of that is already done in the switchShader code, but we're still happily binding buffers repeatedly and setting other WebGL state that isn't strictly necessary.

[lucaswoj]

My dream interface here is to do away with all GL getters / setters in favor of passing all desired state to the draw* call as an object. This is a clean abstraction that will allow us to run the smallest possible # of WebGL api calls, prevent bugs stemming from unexpected GL state, and make our code easier to understand.

[jfirebaugh]

I've been meaning to have a closer look at the API of glium (Rust OpenGL bindings). It promises:

Glium is stateless. There are no set_something() functions in the entire library, and everything is done by parameter passing. The same set of function calls will always produce the same results, which greatly reduces the number of potential problems.

[lucaswoj]

As part of this refactoring, we should merge Painter and gl_utils

[lucaswoj]

:thought_balloon:

gl2.drawElements({
    shader: getShader('circle'),
    mode: gl2.TRIANGLES,

    // Allow circles to be drawn across boundaries, so that
    // large circles are not clipped to tiles
    stencilTest: false,

    depthMask: false,
    depthSubrange: getDepthSubrange(0),

    count: group.getCount(),
    offset: group.getOffset(),
    elementBuffer: group.getElementBuffer(),
    vertexBuffer: group.getVertexBuffer(),

    attributes: {
        a_pos: group.getVertexBuffer().pos,
    },

    uniforms: {
        u_exmatrix: transform.exMatrix,
        u_posmatrix: translatePosMatrix(
            calculatePosMatrix(coord, source.maxzoom),
            tile,
            layer.paint['circle-translate'],
            layer.paint['circle-translate-anchor']
        ),
        u_color: util.premultiply(layer.paint['circle-color'], layer.paint['circle-opacity']),
        u_blur: Math.max(layer.paint['circle-blur'], 1 / browser.devicePixelRatio / layer.paint['circle-radius'])),
        u_size: layer.paint['circle-radius']
    }
});

[lucaswoj]

It'd be cool to build this as an external library, maybe including our Buffer implementation too.

[lucaswoj]

http://jsperf.com/webgl-api-calls/2

[lucaswoj]

As long as performance is acceptable, https://github.com/mikolalysenko/regl would be a perfect fit for this.

[lucaswoj]

As part of this work, I am interested in exploring other abstractions around VAOs

[lucaswoj]

Noting that regl hit v1.0 today 🎉 http://regl.party/

[lucaswoj]

I've come to think that Regl is a poor fit for our project because Regl objects cannot be transferred between threads.

I'm envisioning creating a similar interface using flat objects called RendererAtoms. RendererAtoms could be created within worker threads instead of Buckets, transferred to the main thread, and drawn without any intermediate representation. This architecture could simplify or eliminate the need for Buckets, Buffers, ArrayGroups, draw_* functions and more.

Note that the RendererAtom proposal below does not have an affordance for uniforms (these are tricky -- they are the only part that needs to change frame-by-frame) and may be missing few other minor use cases.

interface RendererAtom {
    mode: GLEnum;
    vertexArrays: Array<StructArray>;
    elementArray: StructArray;
    vertexProgram: string;
    fragmentProgram: string;

    stencil: {
        enable: boolean;
        func: GLEnum;
        ref: number;
        valueMask: GLEnum;
        fail: GLEnum;
        depthFail: GLEnum;
        pass: GLEnum
    };

    depth: {
        enable: boolean;
        func: GLEnum;
        rangeNear: number;
        rangeFar: number;
    };

    texture: {
        enable: boolean;
        width: number;
        height: number;
        format: GLEnum;
        type: GLEnum;
        data: ImageData;
        wrapS: GLEnum;
        wrapT: GLEnum;
        minFilter: GLEnum;
        magFilter: GLEnum;
        mipmap: GLEnum;
    };

    blend: {
        enable: boolean;
        sourceFactor: GLEnum;
        destFactor: GLEnum;
    };
}

[lucaswoj]

Some cool features of RendererAtoms:

• Processing in worker threads would create an _array_ of RendererAtoms per tile. Each RendererAtom would directly correspond to one render call.
• The arrays of RendererAtoms for each tile in a layer could be concatenated in order to create instructions for drawing the entire layer.
• The arrays of RendererAtoms for each layer in a map could be concatenated in order to create instructions for drawing the entire map.
• Each RendererAtom could have its own VertexArrayObject.
• Having multiple RendererAtoms refer to the same StructArray would have no performance impact and allow us to make implicit many complex relationships.
• Inspecting RendererAtoms would be a boon to WebGL debugging
• The ability to cache and re-create RendererAtoms separately (rather than having to work with Buckets) will allow data-driven styling and custom source types to be more efficient in updating the map
• The RendererAtom interface will allow us to capture more layer-type-specific logic in one place
• The RendererAtom interface will eliminate the need to create intermediate data representations (like when we attach random fields to Bucket) because we can directly configure WebGL calls while preprocessing in the worker
• The RendererAtom interface will allow us to track WebGL state and implement new performance optimizations

[kkaefer]

@lucaswoj Ideally, we'd have some abstraction in there though, in particular for creating VAOs. Overall, I'm envisioning a similar system for GL native where we'd remove any state and encode everything into DrawCall objects.

[lucaswoj]

@kkaefer What do you see as the difference between a RendererAtom object and a DrawCall object? Going by on name, I'd guess there's a similar intent behind RendererAtom and DrawCall.

Ideally, we'd have some abstraction in there though, in particular for creating VAOs.

There should be a 1:1 relationship between VAOs and RendererAtom / DrawCall objects. This will make it possible to create the VAOs automatically and transparently within the renderer.

[jfirebaugh]

My experience implementing this in mapbox-gl-native:

• A unified RendererAtom / DrawCall object -- Drawable is what I called it in native -- works well as a means to unify the stateful and procedural GL API into a single function call: context.draw(drawable).
• The value and feasibility of making Drawable _persistent_ between frames was unclear. The components of Drawable come from a variety of sources, many of which differ in terms of their lifetime, ownership semantics, and logical location within the overall program state. Assembling them into a Drawable at render time works well. I didn't see an easy way to make the result persistent without reintroducing complexity.

[ibgreen]

@kkaefer Based on our discussion, we (the luma.gl/deck.gl team) would like to propose the following stand-alone webgl state management system: trackContextState as a possible base for enabling on a true WebGL plugin API in mapbox-gl-js.

In its current incarnation, this state management system basically offers three functions, trackContextState(gl), pushContextState(gl) and popContextState(gl).

Comments:

• While this code is currently part of the luma.gl repository, it has no dependencies on luma and is in the completely independent base webgl-utils folder.
• We can make this available as a separate repository + npm module (or you could copy the code and use this as a base as you see fit).
• Currently this only covers context parameters, but it can be extend it based on your requirements, for instance to also cover buffer bindings etc. You'll need to set the requirements here.
• This module currently also caches state access for speed. If you already have similar solutions we can make sure we disable that part for you.
• As discussed we might be able to collaborate on/support you on a PR to help you integrate this, should you like it.

Let me know if the code is reviewable in its current state or if we need to further preparations.
The following files are of interest:

• https://github.com/uber/luma.gl/blob/master/src/webgl-utils/README.md
• https://github.com/uber/luma.gl/blob/master/src/webgl-utils/set-parameters.js
• https://github.com/uber/luma.gl/blob/master/src/webgl-utils/track-context-state.js

[mourner]

There don't seem to be any clear next actions here after we started tracking most of the GL state. Should we close this @kkaefer? We could outline any potential remaining work in a new ticket.

[ibgreen]

There don't seem to be any clear next actions here after we started tracking most of the GL state. Should we close thi

@mourner Just an update from our side, we have now published our WebGL state tracker as a separate module.

Since it sounds like you have rolled your own system, maybe we should sync to make sure our two state tracking systems are compatible?

Ours is based on WebGL context interception and is thus very generic, is that how you also went about things?

CC: @pessimistress @tsherif

[chloekraw]

cc/ @ansis @asheemmamoowala

[ansis]

Our implementation can mostly be found in https://github.com/mapbox/mapbox-gl-js/blob/master/src/gl/value.js and https://github.com/mapbox/mapbox-gl-js/blob/master/src/gl/context.js. I'm not that familiar with the design choices behind our implementation.

maybe we should sync to make sure our two state tracking systems are compatible?

While this would be nice, do you think the benefits here would be significant? My guess would be that cost of assuming the state is undefined when you switch between libraries once or twice a frame should be pretty insignificant.

[ibgreen]

@ansis Thanks.

do you think the benefits of [syncing to make sure our systems are compatible] would be significant?

If there aren't any problems there should not be a need to spend much time on this. That said it doesn't hurt if we understand each other's systems.

Some observations looking at your code:

• Looks like you are building a nice typesafe wrapper class on top of the WebGLContext.
• Our implementation is more raw, it replaces functions on the webgl context itself.

So in that sense, our implementations do not clash. Thoughts:

• We can track all changes to the context, including the ones your Context class make, as your methods ultimately hit the gl... methods
• However, as far as I can tell, you may not see changes that we make to the context.

• Our solution has push state and pop state (save and restore), I couldn't see if you have functions that completely restores a context or just per-setting management.

• @tsherif who is tech leading luma.gl now.

[mourner]

Closing as an issue without clear next actions — let's open new issues for any specific items we should address.