Cesium: scene.pickPosition returns incorrect position

[hpinkos]

Reported on the forum: https://groups.google.com/forum/?hl=en#!topic/cesium-dev/0E-aKBTLESk

The user created this sandcastle example to demonstrate the problem: http://hosting.virtualcitysystems.de/demos/temp/pickProblem/Apps/Sandcastle/?src=3D%20Tiles.html&label=undefined

The demo is using the latest 3d-tiles branch

pickPosition works fine if you run the example inside sandcastle, but if you click 'Open in New Window' sometimes the returned position sits in front of the building you clicked instead of where the click intersects with it. Maybe it's related to the canvas ratio? See the forum post for more details.

I couldn't reproduce this with any of our sample models, but I asked the user if he could share a tile for us to test with.

[lucasvw]

Here is the tileset which is used in the sandcastle demo: http://hosting.virtualcitysystems.de/demos/temp/b3dm

Here is a video which shows the behavior: https://youtu.be/9twwwMHbjKU
And here is another that shows that it has something to do with the size of the screen: https://youtu.be/IbDNwqKVOik

And zooming : https://youtu.be/nJpHGOdTssY

[lucasvw]

3D tileset zipped: http://hosting.virtualcitysystems.de/demos/temp/b3dm/3dTiles.zip

[hpinkos]

Thanks @lucasvw!

[lilleyse]

I'll take a look at this soon.

[lilleyse]

Just to update this issue, it looks like this is related to inconsistencies with depth picking with multifrustum rendering, so it's a bug in the core engine. I still need to investigate more.

[lucasvw]

Is this going to be part of the bug-bash next week? +1 for yes ;)

[mramato]

@lucasvw I marked us to at least look at it as part of the bash, but depending on how involved the fix is, I can't promise it will be resolved. We'll try our best!

[lucasvw]

@mramato @lilleyse do you perhaps have an idea when you will be able to fix this ?

[lilleyse]

Sorry about the delay, I will spend some more time looking into this.

[lucasvw]

That would be great! Thanks a lot @lilleyse

[lucasvw]

@lilleyse I barely dare to ask... but eh.. any updates on this ?

;)

[lilleyse]

Sorry but not yet. I did find a slight bug with the calculated pick position (https://github.com/AnalyticalGraphicsInc/cesium/pull/4615) but it didn't solve the problem unfortunately.

If I remember correctly the core issue here is that Cesium uses multi-frustum rendering and when the pick position is calculated it looks to see if a depth exists starting from the first frustum. The problem is that each frustum may have a different depth at that pixel and it doesn't know which frustum to use. That's just a theory though, but it could be tough to solve. I promise I'll get to this eventually.

[lucasvw]

Thanks for the update and the explanation!

[kring]

In looking at this while debugging #4855, I saw a case where I had only a single frustum, and yet the position returned by pickPosition was wildly wrong. For example, values in pickGlobe in ScreenSpaceCameraController.js:

That's without anything in the scene except for the globe (and sun and moon and such I suppose, but not explicitly-added primitives).

So I don't think it's necessarily multi-frustum related.

[emackey]

Our "pick position" Sandcastle demo limits your picking to just a 3D model of the milk truck. Check out what happens if you open it up to allow picking the entire globe. It works at close range to the truck, but the more you zoom out, the worse it gets.

var viewer = new Cesium.Viewer('cesiumContainer', {
    selectionIndicator : false,
    infoBox : false
});

var scene = viewer.scene;
var handler;

var modelEntity = viewer.entities.add({
    name : 'milktruck',
    position : Cesium.Cartesian3.fromDegrees(-123.0744619, 44.0503706),
    model : {
        uri : '../../SampleData/models/CesiumMilkTruck/CesiumMilkTruck-kmc.gltf'
    }
});
viewer.zoomTo(modelEntity);

var labelEntity = viewer.entities.add({
    label : {
        show : false,
        showBackground : true,
        font : '14px monospace',
        horizontalOrigin : Cesium.HorizontalOrigin.LEFT,
        verticalOrigin : Cesium.VerticalOrigin.TOP,
        pixelOffset : new Cesium.Cartesian2(15, 0)
    }
});

var sceneModeWarningPosted = false;

// Mouse over the globe to see the cartographic position
handler = new Cesium.ScreenSpaceEventHandler(scene.canvas);
handler.setInputAction(function(movement) {
    var foundPosition = false;

    var scene = viewer.scene;
    var pickedObject = scene.pick(movement.endPosition);
    if (scene.pickPositionSupported) {
        if (scene.mode === Cesium.SceneMode.SCENE3D) {
            var cartesian = viewer.scene.pickPosition(movement.endPosition);

            if (Cesium.defined(cartesian)) {
                var cartographic = Cesium.Cartographic.fromCartesian(cartesian);
                var longitudeString = Cesium.Math.toDegrees(cartographic.longitude).toFixed(2);
                var latitudeString = Cesium.Math.toDegrees(cartographic.latitude).toFixed(2);
                var heightString = cartographic.height.toFixed(2);

                labelEntity.position = cartesian;
                labelEntity.label.show = true;
                labelEntity.label.text =
                    'Lon: ' + ('   ' + longitudeString).slice(-7) + '\u00B0' +
                    '\nLat: ' + ('   ' + latitudeString).slice(-7) + '\u00B0' +
                    '\nAlt: ' + ('   ' + heightString).slice(-7) + 'm';

                var camera = scene.camera;
                labelEntity.label.eyeOffset = new Cesium.Cartesian3(0.0, 0.0, camera.frustum.near * 1.5 - Cesium.Cartesian3.distance(cartesian, camera.position));

                foundPosition = true;
            }
        } else if (!sceneModeWarningPosted) {
            sceneModeWarningPosted = true;
            console.log("pickPosition is currently only supported in 3D mode.");
        }
    }

    if (!foundPosition) {
        labelEntity.label.show = false;
    }
}, Cesium.ScreenSpaceEventType.MOUSE_MOVE);

[duvifn]

@emackey

1. I'm not sure that the 'jittering' in your animation is due to the pickPosition results . On my machine, If I set eyeOffset.z to a fixed value this doesn't happen.

1. I think it's worth adding scene.globe.depthTestAgainstTerrain = true; to the code in your comment, because this is not the default in the sandcastle.
   pickPosition returns different values on terrain if depthTestAgainstTerrain == false , usually below the ellipsoid surface (maybe of the DepthPlane).

2. The Alt string in this example code is truncated so it shows an wrong height.
   for example: -143976.41485355987 becomes 3976.41.

[denverpierce]

What GPUs/Browsers does everyone run into this on? I get this pretty consistently on a discrete AMD GPU, which makes me think that @kring idea (caught up with the zoom issues which were exacerbated by this issue) that it's depth buffer related with AMD cards may have something to do with it.

[jbo023]

@lilleyse
After some hours of debugging i found a possible solution for this.

The problem is that scene.pickPosition sometimes returns a negative height value if used in combination with scene.pick.

I noticed that the number of frustums in the pickPosition rendering pass differs from the number of frustums in the normal rendering update, when pickPosition delivers a corrupt result.

After some digging I figured that the scene.pick function changes the framestate cullingVolume which is then also used from the pickPosition function.

So when I use the pickPosition function without calling any scene.pick() in the same frame, pickPosition will use the cullingVolume from the last rendering pass and delivers the correct position result.

var viewer = new Cesium.CesiumWidget('cesiumContainer', {
        scene3DOnly : true,
        shadows : true
    });

    //var viewer = cesium.getViewer();

    var scene = viewer.scene;

    viewer.camera.up = new Cesium.Cartesian3(0.602242291265668, 0.47745810596428245, 0.6397952638618689);
    viewer.camera.direction = new Cesium.Cartesian3(-0.09199309329813982, 0.8376012488044867, -0.5384806577646075);
    viewer.camera.position = new Cesium.Cartesian3(3785056.8752774675, 901611.4457953185, 5037174.477028298);


    var tileset = scene.primitives.add(new Cesium.Cesium3DTileset({
        url : 'https://d35ei6ur3bjvr1.cloudfront.net/berlin/3c2a32d3-633c-462d-b79c-7215dcfbc44f',
    }));

    var dataSourceCollection = new Cesium.DataSourceCollection();
    var dataSourceDisplay = new Cesium.DataSourceDisplay({
          scene : scene,
          dataSourceCollection : dataSourceCollection
        });

        var clock = viewer.clock;

        var eventHelper = new Cesium.EventHelper();
        clock.onTick.addEventListener(function(clock){
          var time = clock.currentTime;
          dataSourceDisplay.update(time);
        }.bind(this));

    // Mouse over the globe to see the cartographic position
    var handler = new Cesium.ScreenSpaceEventHandler(viewer.scene.canvas);
    handler.setInputAction(function(click) {
        var foundPosition = false;

        var scene = viewer.scene;
        var pickedOject = scene.pick(click.position);        
        if (true) {
            var position = viewer.scene.pickPosition(click.position);            
            var carto = Cesium.Cartographic.fromCartesian(position);
            console.log('pos' + carto);


            if (Cesium.defined(position)) {
                dataSourceDisplay.defaultDataSource.entities.add({
                    position : position,
                    point : {
                        pixelSize : 10,
                        color : Cesium.Color.YELLOW
                    }
                });
            }
        }
    }, Cesium.ScreenSpaceEventType.LEFT_CLICK);

With this Sandcastle code i could reproduce the corrupt height value if i click on the building in the center of the screen. If you just remove the line var pickedOject = scene.pick(click.position); pickPosition works correct.

[lilleyse]

This helps a lot @jbo023.

So why does it matter that the pick pass renders lesser frustums if it is still writing out depth correctly? Is the pick depth-copy broken? Just questions I'm thinking about right now...

[thw0rted]

Hey all, it's been a year and I'm pretty sure that (at least the way I'm using it?) scene.pickPosition is still giving me hilariously-wrong results. I'm passing the endPosition from a MoveEvent and getting back a point that, when converted to Cartographic, has a height in the negative millions. Is there another way I'm supposed to get lat / lon point under the cursor on the rendered globe along with a relatively accurate height-above-ellipsoid? Nothing is working...

[hpinkos]

@thw0rted this is for using pickPosition to pick the globe? I would recommend using either scene.camera.pickEllipsoid if you don't have terrain or scene.globe.pick if you do have terrain. pickPosition works great for getting the position on a 3D tileset or a glTF model, but we haven't had a chance to fix the globe picking issue.

[thw0rted]

I don't think I'm getting sane altitudes from any method. For comparison, I made a quick SandCastle testbed. Zoom pretty far out (at least so you can see the whole globe) then just kind of noodle around -- when I have North America in the center of the globe, I can mouse over the Amazon, near the edge of the visible globe, and get altitude values like -18000 m (!) for Globe.pick. Most values are negative and the magnitude of the value tends to increase the further your view is zoomed out -- once you zoom in far enough, you start to see some positive values show up even outside of major mountain ranges.

The interesting bit is that Globe.pick gives massive negative altitudes even when the Ellipsoid terrain provider is used. In theory, the values for Globe.pick and Camera.pickEllipsoid should line up every time, right? I mean, if the globe is using the ellipsoid to model the mesh, and Globe.pick is based on a mesh/pick-ray intersection. I guess what we're seeing is some "fudge factor" where the actual mesh is contained by the ellipsoid but doesn't actually reach it, since zooming in reduces this discrepancy and eventually removes it -- zoom in to maybe a state or city-sized region and the Globe.pick altitude shows up as (negative) 0m. Using the Ion terrain provider changes the numbers somewhat, but the trend (more negative when zoomed out further) remains.

All that to say: am I making a mistake somewhere, or is it just not possible to get a realistic terrain-height-under-cursor right now?

[hpinkos]

@thw0rted that is expected behavior. Globe.pick is retrieving the height of whatever terrain LOD is currently visible on the globe. When you're zoomed out far, the terrain tiles are very low resolution, hence the negative altitude values.

TerriaJS (a library that uses both Cesium and leaflet) has a mouse over coordinate tool that does a fast approximation of height using globe.pickTriangle, then gets the actual terrain height using sampleTerrainMostDetailed. It displays the approximate height until the result from sampleTerrainMostDetailed is returned. I would recommend doing something similar. See https://github.com/TerriaJS/terriajs/blob/master/lib/ReactViewModels/MouseCoords.js

[thw0rted]

I was looking at something similar on my own, actually, but it turns out the terrain provider I'm using at the moment (an older in-house Google Earth server) doesn't support availability and that's required for sampleTerrainMostDetailed. I could use sampleTerrain but I didn't know a good way to guesstimate the appropriate level to pass, so I put it on the back burner. I just want to know how tall stuff is :-/

ETA: I had a bit of a poke around, it looks like globe.pickTriangle no longer exists -- would that have been any more acurate than globe.pick, in terms of getting a sane height when zoomed out significantly?

[hpinkos]

See https://github.com/AnalyticalGraphicsInc/cesium/issues/6990 for a great code example. pickPosition works decently well on the ellipsoid when globe.depthTestAgainstTerrain = true but doesn't work at all otherwise.

[drmoose]

For others who find this bug because pickPosition is wildly inaccurate and seems to change based on the camera position, on my system (Linux / Firefox / nvidia) the entire problem turned out to be CanvasBlocker intentionally tampering with the WebGL APIs that Cesium uses to query the depth buffer to prevent that from being used for fingerprinting.

[hpinkos]

Also reported by @hieeyh in #7506

[bn-dignitas]

@jbo023 Thanks for that analysis. It has been a couple years since your post, but I ran into something similar with Cesium 1.58, calling drillPick and then pickPosition right after with incorrect results. Luckily I was able to just swap the order of operations to work around it.

[1nstinct]

I faced the same issue - pickPosition returns wrong position. It can be seen while moving camera around. Tried to use globe.depthTestAgainstTerrain = true - it works but all entities that were on the surface dive into it now. The only solution that I can see at this moment is adding height: 10 to all the surface entities to make them lying above the surface.