Webodm: Incorrect scale on GCP file.

[mwfoshee]

How did you install WebODM? (Docker, natively, ...)?
Downloaded the installer and installed. Docker was installed and runs for WebODM to work

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)
Chrome 70 on Windows 10

What is the problem?
Output from WebODM is not scaled correctly ( I think, When I import WebODM output into QGIS as layers, all WebODM output layers match each other, yet when I overlay the other layers (Google, Bing, Open Map and my Ground Control Point Files) the WebODM layers are to small

What should be the expected behavior?
WebODM layers should match layers added from outside sources.

Link to Both Datasets

These are the options I'm using in WebODM - Options: fast-orthophoto: true, force-ccd: 23.4

The only 2 projects I've gotten to process with GCP's have produced Orthophotos in the wrong scale. (see Image from QGIS)

Also, The GCP file generated using the GCP interface in WebODM is dropping precision. Ive had this issue using 2 different coordinate systems, the following GCP files are both in UTM 15N

My GCP File:

+proj=utm +zone=15 +datum=WGS84 +units=m +no_defs
3 491507.911389015 3809328.08667974 134.2
PLStrip 491484.361944006 3809356.76260682 135.78
PumpStation 491536.66764989 3809390.57422016 134.98
1 491524.382773379 3809345.4109657 133.85
4 491440.381027897 3809333.2925389 135.97

GCP file From the GCP interface in WebODM:

+proj=utm +zone=15 +datum=WGS84 +units=m +no_defs
491507.91 3809328.09 134.2 4144.64 1304.98 DSC00834.JPG 3
491507.91 3809328.09 134.2 1328.18 2636.61 DSC00822.JPG 3
491507.91 3809328.09 134.2 1100.54 604.75 DSC00824.JPG 3
491507.91 3809328.09 134.2 2467.37 2289.96 DSC00831.JPG 3
491507.91 3809328.09 134.2 4294.50 2094.75 DSC00833.JPG 3
491440.38 3809333.29 135.97 1656.81 1001.08 DSC00798.JPG 4
491440.38 3809333.29 135.97 1978.99 2118.70 DSC00797.JPG 4
491440.38 3809333.29 135.97 1786.00 3057.25 DSC00818.JPG 4
491484.36 3809356.76 135.78 1516.61 2420.56 DSC00801.JPG PLStrip
491484.36 3809356.76 135.78 1329.88 1375.55 DSC00802.JPG PLStrip
491484.36 3809356.76 135.78 2019.00 1576.46 DSC00815.JPG PLStrip
491536.67 3809390.57 134.98 1545.61 1729.21 DSC00807.JPG PumpStation
491536.67 3809390.57 134.98 1882.75 2330.25 DSC00809.JPG PumpStation
491536.67 3809390.57 134.98 1739.73 1257.14 DSC00810.JPG PumpStation
491524.38 3809345.41 133.85 1771.29 1540.42 DSC00825.JPG 1
491524.38 3809345.41 133.85 1549.62 673.37 DSC00826.JPG 1
491524.38 3809345.41 133.85 1773.30 2721.77 DSC00829.JPG 1
491524.38 3809345.41 133.85 1753.26 1395.95 DSC00830.JPG 1
491524.38 3809345.41 133.85 1773.50 250.75 DSC00831.JPG 1

[pierotofy]

Hi @mwfoshee,

It looks like there is some information missing from your issue that will be needed in order to diagnose and fix the problem at hand. Please take a look at the Issue Template, which will tell you exactly what your issue has to contain in order to be processable.

Also, double check that this is the right place. If you are just asking for information, reporting feedback or proposing a few feature, the right place to ask is the Community Forum, not here.

I'm marking this one now as needing some more information. Please understand that if you do not provide that information within the next week (until 2018-11-05 19:10) I'll close this issue so it doesn't clutter the bug tracker.

Cheers!
~ Your friendly GitIssueBot

PS: I'm just an automated script, not a human being.

[smathermather]

We have two issues here, so think. The incorrect scale is a function of two different reconstructions due to an incomplete pair of flights (not enough overlap in the middle). This is an interesting issue. This can be demonstrated by processing either half of the flight independently, which does appear at the correct scale.

The second issue is that of the GCPs (which may itself be two or three issues — one with the coordinate truncation and one with the application of GCPs to reference a model).

[mwfoshee]

I agree two issues, I'm working on sorting them out now. Hope this helps.

Issue 1 incorrect scale:

Only one of my posts had the missing strip of images, the image above was generated by a complete set of images with consistent overlap.

I haven't been able to process projects in any other CRS than WGS 84 / UTM zone 15N, Attempts in other CRS's (EPGS's 6414, 3434, 102652, 3857, 4326) all failed to process, ending in either a log that said finished but no assets were created or in a code 1 error. My only successful attempts to process projects since I installed WebODM have been when I used WGS 84 / UTM zone 15N or +proj=utm +zone=15 +datum=WGS84 +units=m +no_defs (EPSG:32614 )

Both projects had the same scale issue.

I did get both halves of the one project to successfully process at the correct scale when I did each half separate but not with GCP's. I tried unsuccessfully, but I don't think I tried using UTM 15 N. I'm working on that now.

I might add also that both of the projects were completed previously in Drone Mapper, Global Mapper and online through Maps Made Easy successfully. (The project with the missing strip of images in the middle had to be processed as two halves in global mapper)

Issue 2 coordinate truncation

It occurred to me that the only time more than 2 decimal places is required is when using LAT/LON in a CRS such as WGS84 (EPSG:4326). two decimal places in US Survey feet is hundredths on a foot and 2 decimal places in a Metric CRS is centimeter accuracy, both of which are considered accurate by surveyors standards.

So the truncation issue only appears relevant when using a CRS whose units are decimal degrees. I'm ok with using UTM coordinates all the time.

Thank you guys for your work and patience with me.

I'm reprocessing one image set (of the flight with the missing strip down the middle) now using UTM, I'll post my results.

[mwfoshee]

this project exhibits the same problem.

Link to my DATA in Google Drive

Output without GCP's

Output with GCP's:

Image processed with GCP's Zoomed in to see targets. The red dots are the GCP's plotted in QGIS (UTM15N) I would expect the black and white targets in the WebODM orthophoto to align with the red GCP's

[mwfoshee]

My workflow hen using the GCP interface
1) add the images
2) add the GCP file
3) Select an image
At this point an orange target appears on the image anf the point map on the left, I tried moving those to the desired location but I couldn't get that process to work. so
4) Delete the orange target on the Image and the point map
5) Click the plus sign
6) Click on the center of my target
7) Click on the correct GCP from the point map (both turn green)
8) move to the next image

I checked DSC00960.JPG from the Data set in The Gimp, the x and y coordinates of the center of the target seem to match the x and y from the GCP file generated from the GCP interface and I checked the utm x and y against the the x and y of the GCP's both seem to match.

[mwfoshee]

Just to rule out errors incurred while importing into QGIS, this from the "View Map Button on the Project Task screen inside WebODM

[pierotofy]

The precision issue should be fixed, the scale problem still needs to be fixed.

[denistestemale]

Hello.

• How did you install WebODM? (Docker, natively, ...)? Docker on Ubuntu 18.04 + git.
• What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/) Firefox 63 on Ubuntu Linux.
• What is the problem? I'm posting here because I thought this is related to @mwfoshee problem, but let me know if I shall create another issue. See details here. When using a GCP file (WGS84UTM31N), the orthophoto Geotiff is shifted by 10 meters compared to dsm/dtm Tiff and the cloud point (PLY) ((both seem to be correctly corrected by the use of GCP). Compared to the OP, the scale is OK, so it is not technically the same problem, but might be related.
• What should be the expected behavior? WebODM assets should align with each other (in the same dataset).

Here is a screenshot of orthophoto and dsm (colored) which are not aligned:

I'll post a link to the dataset if needed.

Cheers
denis

[pierotofy]

Hey @denistestemale :hand:, thanks for the report!

A copy of the dataset would be greatly helpful (along with the GCP file).

[denistestemale]

Hello.
You're welcome. A link to the dataset with the gcp file included:
https://drive.google.com/folderview?id=1B1c5BsG19Hx7RYwzUgZ26nZWb702yLbz

Cheers
Denis

Le 12 novembre 2018 15:14:39 Piero Toffanin [e-mail] a
écrit :

Hey @denistestemale ✋, thanks for the report!

A copy of the dataset would be greatly helpful (along with the GCP file).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[pierotofy]

edit: the analysis below is wrong, see below

It seems like some GCPs are not being projected to the right location... I'm not sure yet of what makes certain datasets different than others (in another dataset I just tested this does not happen), but you can see that one particular GCP that was used for georeferencing was projected in the wrong spot by several meters.

GCP location where the (wrong) projection happens (approximately) in red:

3D view (correct GCP position highlighted in red), marker showing (wrong) projected location:

This is puzzling but I'm sure we'll get to the bottom of it.

[denistestemale]

Thanks Piero for looking into that, much appreciated.
Do you mean that the recalculated GCP position (taking into account the offset, as indicated in the georeferencing_log.txt file) is wrong?
I've made some tests in the last days: I tried all the pre-existing set of parameters, no custom, on the same dataset. I also add here some observations that might (or might not) be related to the problem. Sorry if they are completely off topic.

Summary:

• I tried fast orthophoto, default, high quality, and high resolution.
• the cloud points and orthophotos never match (based on point picking in cloudcompare vs position reading in QGIS), but I've never found a 10 meters offset like previously shown, it's more in the 1m range, and mainly/only north-south offset (coincidence?)
• I cannot check if dsm and orthophoto match since dsm lack features, but they seem to agree.
• I never find one single GCP spot on the corresponding field target. Never.

Details:

• parameters fast orthophoto: I don't know if the GCP file is used (no log), the agreement (GCP vs orthophoto) seems to be globally better (than with no GCP file), but still off. No dsm or cloud point to compare to.
• parameters default: the agreement (GCP vs orthophoto) seems to be globally better, but still off. No big offset between orthophoto and cloud points, but still an offset (about 0.5m, mostly north-south).
• parameters high quality: the orthophoto is completely under-resolved, resolution is found to be 1.6666 (60cm/pixel)! Can't check cloud points vs orthophoto then.
• parameters high resolution: the agreement (GCP vs orthophoto) seems to be globally better, but still off. No big offset between orthophoto and cloud points, but still an offset (about 1m, mostly north-south).
• Resolution: the best resolution I can use is 28.450167151, is it a feature? I.e. probably set to what the pictures offer and no oversampling allowed? In my previous version of WebODM (native ubuntu 16.04 install, now I'm running docker on ubuntu 18.04), I remember I could oversample the dataset (that's a bit useless, and lost time and disk space, but that's not the point here). But here, if I use the high resolution option, or the resolution=40 of my initial run (the one that triggered this discussion), I'm back to 28.450167151. Looks like a calculated value based on image metadata (filed of view, altitude, sensor pixel counts)? That's 3.5cm/pixel, the DJI app I used for the drone survey says 4.1cm/pixel, might be the same thing calculated differently?
• Accuracy: As I wrote, I never find one single GCP spot on the corresponding field target. Never. I didn't expect them to match perfectly (like a georeferencing of the global orthophoto in QGIS would do), but the offset is always at least 50cm. I have the same question as @mwfoshee : is it normal? I don't know if in ODM the transformation parameters derived from the analysis of GCP are "flexible" enough (I mean: like a 3rd order polynomial transformation is more "flexible" than a 1st or 2nd order; excuse my lack of technical vocabulary), mostly local (like Thin Plate Spline) or global (like polynomial).

Cheers
denis

[pierotofy]

Thanks for the additional info.

In regard to the oversampling, you can pass --ignore-gsd to oversample a dataset (like in previous versions). Also note the units of orthophoto-resolution and dem-resolution, we've changed those to standard cm / pixel (they used to be pixel / meter and meter / pixel.

[denistestemale]

Thanks for the tip for oversampling. I had seen the change of units indeed.
I've done another test: I repeated the same calculation (the one with my
custom parameters and the 10m offset of the orthophoto). Now there is no
big offset! Trust me I checked I didn't do any mistake with the first
dataset (I redownloaded from webodm dashboard and the offset is definitely
present). So now it is basically like the HR calculation : cloud point and
orthophoto do not match (about 1m offset) and no GCP are spot on the targets.
This is puzzling. Tell me if you need me to try something else.

Cheers.
Denis

Le 14 novembre 2018 15:20:13 Piero Toffanin [e-mail] a
écrit :

Thanks for the additional info.
In regard to the oversampling, you can pass --ignore-gsd to oversample a
dataset (like in previous versions). Also note the units of
orthophoto-resolution and dem-resolution, we've changed those to standard
cm / pixel (they used to be pixel / meter and meter / pixel.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[pierotofy]

My previous hypothesis about GCPs not being properly projected turned out to be wrong. I just mistakenly used the wrong point when making the analysis. Problem is elsewhere.

[pierotofy]

@denistestemale the problem might be due to incorrect points in your gcp_file.txt. For example, this entry:

731612.625 4946166.347 2544.578 4506 944 DJI_0194.JPG

Seems way off.

This results in a less optimal choice of points by odm_georef.

[pierotofy]

Also:

731777.103 4946133.922 2547.348 4425 1405 DJI_0199.JPG

[pierotofy]

Removing those two points I was able to get better accuracy, but odm_georef still reports a mean error of 0.743 meters. I wonder if this is due to the inability of the program to reconcile the difference between the GCP measurements and the errors in the point cloud reconstruction. @smathermather wonder if this is another case that could be improved with a camera calibration database.

[smathermather]

Could be. @denistestemale: were these images fed through a separate camera calibration before processing?

[pierotofy]

mean bias of 0.6m in XY & 0.9m in Z.

http://www.isprs.org/tc2-symposium2018/images/ISPRS-Invited-Fraser.pdf

[denistestemale]

I'm currently checking my GCP file since I've obviously screwed up some points.
The JPG files are directly from the DJI Phantom 4 (Pro or Advanced, don't remember now), no camera calibration whatsoever.
Thanks guys.
d

[denistestemale]

OK, I indeed removed 2 points: I'm really sorry about that. I updated the GCP file in the Google drive folder. I also added another GCP, but you can forget about this one.
I have access to Darktable which can offer geometry distorsion for this camera/objective (from the lensfun database I reckon), but I don't know if this correction is what you want (the straight lines in the original pictures become curved if they are not centered, this doesn't look good to me). But if you want I can try.
I'm doing another run with the correct GCP file.

denis

[denistestemale]

I did again the WebODM analysis of the dataset: parameters "high resolution", with the updated list of GCPs (outliers removed).

• First the analysis of the cloud point (where I picked the position of the targets and compared to the corresponding GCP coordinates) show the same kind of agreement as you said. Just to get an idea I calculated the X/Y/Z difference between the GCP and the corresponding points in the ply file, and calculated the average of their absolute values (should compare to the RMS values you gave above). I found 0.7m xy and 0.3m z. The results are in the Google drive folder (.ods spreadsheet). If I understand the documents you linked, that is the best we can expect given the image deformation and the flat nature of the terrain. I have another dataset with much more height variations and mountain terrain, I'll give a try.
• Orthophoto and cloud point are still offset (1 to 2m approximately). In the spreadsheet you'll see that the targets in the cloud point are never more than 1m away from their expected values. But in the Geotiff, the targets and GCP are always >1m away, sometimes 2m. I show 2 screenshots:
  
  

• Finally, dsm.tiff and orthophoto do not seem to match either (hard to check if the dsm is aligned with the cloud point though), with the same ~2m offset. See screenshot where the edge of the terrain plateau (in the orthophoto) is below (to the south) the corresponding change of altitude in the dsm.
  

OK I need to try another dataset.
Denis

[kikislater]

@pierotofy Is there a way to take in consideration these calibration parameters in ODM ?

  <camera name="FC6310_8.8_5472x3648">
      <imageWidth>5472</imageWidth>
      <imageHeight>3648</imageHeight>
      <pixelSize>2.34527</pixelSize>
      <principalPointXmm>6.41666</principalPointXmm>
      <principalPointYmm>4.27777</principalPointYmm>
      <lensType>perspective</lensType>
      <focalLengthmm>8.60423</focalLengthmm>
      <distortion>5</distortion>
      <radialK1>0.00298599</radialK1>
      <radialK2>-0.00769116</radialK2>
      <radialK3>0.0079115</radialK3>
      <tangentialT1>-0.000129713</tangentialT1>
      <tangentialT2>0.000221193</tangentialT2>
      <cameraModelSource>internalDB</cameraModelSource>
      <bandConfig>
          <band name="Red" centralWaveLength="660" width="0" weight="0.2126"/>
          <band name="Green" centralWaveLength="550" width="0" weight="0.7152"/>
          <band name="Blue" centralWaveLength="470" width="0" weight="0.0722"/>
      </bandConfig>
  </camera>

[denistestemale]

I played with CloudCompare tonight (and learnt how to project rasters from the cloud point), and I can confirm that the dsm from WebODM is perfectly aligned with the .ply cloud from WebODM (that was the last unknown in my "investigation"), but not with the orthophoto.
2 screenshots below where you can see the Orthophoto from WebODM along with a raster projected from the .ply cloud point along the Z axis + the measuring tool to show the 2m offset.

Thank you. I'll be quiet now, I swear! :-)
Cheers
d

[smathermather]

@denistestemale -- a useful thing to try would be to calibrate your images and then feed through to see if we can get the overall error below the 1m that Piero was measuring. That DJI has a fair amount of barrel distortion that ODM doesn't account for (something we should fix long term).

You can fix these using darktable in a single image (apply lens distortion corrections to said image) and then pasting that to the rest of the image set a la: https://photo.stackexchange.com/questions/39055/how-to-batch-edit-a-collection-of-raw-files-in-darktable and then exporting the group of them before using in ODM.

[denistestemale]

Thanks @smathermather . As I said what worries me (but I might be wrong) is that once the geometry corrections are applied straight lines are not straight anymore:

• before:
  

• after:
  

Also, there are several options for the geometry distorsion: see the list in the "correction des objectifs" tool menu on the right of the screen (second screenshot). The darktable manual says:

In addition to the correction of lens flaws, this module can change the projection type of your image. Set this combobox to the aimed projection type, like “rectilinear”, “fish-eye”, “panoramic”, “equirectangular”, “orthographic”, “stereographic”, “equisolid angle”, “thoby fish-eye”.

What do you reckon @smathermather ?

[kikislater]

It's normal, Darktable is for RAW correction in general not jpeg. You have to correct with opencv and parameters I give above

[denistestemale]

Thanks @kikislater . Do you mean those opencv parameters are determined/optimized for the JPG files? I'll investigate how to install opencv and will give it a shot.
d

[smathermather]

@denistestemale -- those curving lines should worry you. You can do what @kikislater recommends above pretty easily I think with these scripts: https://github.com/OpenDroneMap/CameraCalibration

You won't need to do the checkerboard step, as that is replaced by Sylvain's parameters above.

[smathermather]

So I guess this get's at a fundamental question that maybe @kikislater can help answer -- the lensfun parameters are always for RAW and they don't have alternatives for JPG, or does it depend on the camera and what's available in lensfun?

I ask, as we need to be adding a calibration step from known parameters to ODM, especially for cameras with barrel distortion, and I was hoping we could just drop in lensfun... .

[kikislater]

No, lensfun is mostly designed for raw processing but there is some jpeg correction in library. Proprietary like camera raw detect both.
I tried to send some pictures for correction but never been approved due to jpeg format ...

Above corrections I given are minus and difficult to catch with eyes ! Doesn't think it could improve computation but may be worth a try (sharpen seems to be homogneous which is good for photogrammetry) ... Or need another correction different from opencv.
Opensfm take in consideration radial and tangential parameters. line (142 => https://github.com/mapillary/OpenSfM/blob/master/opensfm/commands/undistort.py )
An option could be available in ODM to set radial (k1, k2, k3) and tangential (p1, p2) corrections if it has effectively proven to enhance reconstruction !

Here is my python code if you would like to try : (I will not process dataset, I'm in Africa and downloading dataset will kill my internet connection ^_^ )

# -*- coding: utf-8 -*-

import cv2
import numpy as np
from matplotlib import pyplot as plt

# Read an example image and acquire its size
img = cv2.imread('DJI_0175.JPG')
h,  w = img.shape[:2]
#Sensor properties
ps = 2.34527 # pixel size
ppx = 6.41666 
ppy = 4.27777
f = 8.60423 #focal
fx = (f / ppx) * w * 0.5 #(focal_mm / sensor_width_mm) * image_width_in_pixels
                   # http://answers.opencv.org/question/17076/conversion-focal-distance-from-mm-to-pixels/
fy =(f / ppy) * h * 0.5
cx = w/2
cy = h/2                             
# Corrections
# Polynomial
k1 = 0.00298599 #radial
k2 = -0.00769116
k3 = 0.0079115
p1 = -0.000129713 #tangential
p2 = 0.000221193

# Define camera matrix 
camera_matrix = np.array([[fx, 0., cx], [0., fy, cy], [0., 0., 1.]])

#Define distorsion coefficeints. Order = k1, k2, p1, p2, k3
dist = np.array([k1, k2, p1, p2, k3])
# Generate new camera matrix from parameters
newcameramatrix, roi = cv2.getOptimalNewCameraMatrix(camera_matrix, dist, (w, h), 1, (w, h))

# Generate look-up tables for remapping the camera image
mapx, mapy = cv2.initUndistortRectifyMap(camera_matrix, dist, None, newcameramatrix, (w, h), 5)

# Remap the original image to a new image
newimg = cv2.remap(img, mapx, mapy, cv2.INTER_LINEAR)
# crop and write image
x,y,w,h = roi
dst = newimg[y:y+h, x:x+w]
cv2.imwrite('DJI_0175-calib.jpg',dst)

# Display old and new image
fig, (oldimg_ax, newimg_ax) = plt.subplots(1, 2)
oldimg_ax.imshow(img)
oldimg_ax.set_title('Original image')
newimg_ax.imshow(newimg)
newimg_ax.set_title('Unwarped image')
plt.show()

[denistestemale]

I'm willing to give a try, but I'm reaching the limits of my Python capabilities when I have to deal with python and opencv error messages, such as (undistort.py is the name I give to your python code @kikislater ):

Traceback (most recent call last):
File "undistort.py", line 43, in
dst = dst[y:y+h, x:x+w]
NameError: name 'dst' is not defined

[kikislater]

updated, sorry !

[denistestemale]

OK, update. Based on your python code @kikislater I could figure out what are supposed to be the matrix and distortion txt files needed with the undistort.py code that @smathermather linked above (https://github.com/OpenDroneMap/CameraCalibration). So I could use this one without error messages. On a side note, your matrix is slightly different from @dakotabenjamin example: in his example dataset, fx is not equal to fy, i.e. the 2 first elements of the matrix diagonal. They're close (3601 vs 3595) but different.
I could unwrap the whole dataset, and I agree with you, the differences are very small: 2 pixels in both direction at the maximum. I don't believe it's gonna make a change.

[denistestemale]

I've just seen your correction @kikislater ,thank you. I could run you python code without error, and get the same kind of correction as with the other code. They are not exactly the same, but we are talking 1 pixel deviation.

[kikislater]

Yes my code is wrong ! It miss *0.5 (corrected)

fx = (f / ppx) * w * 0.5
fy = (f / ppy) * h * 0.5
fx
3664.735328036704
fy
3668.760947877048

focal length in pixel calculate in proprietary software and where fx = fy (f in picture) :

[denistestemale]

Well @kikislater , such a small difference is not gonna make a difference, true?

[kikislater]

Focal length pixel or all these small corrections?
Don't know if it could enhance at this level ... Need to try ! Piero already told that problem is too perfect flat fly with no camera angle but it's a good case to try!

[denistestemale]

I was refering to the 1-2 pixel (maximum) shift of the unwrapped images. 1 pixel is more or less the error I make when I pick the target position in the images when building the gcp_file.txt.
I can live with the 0.6m error RMS that Piero mentioned, if it is intrisic of the method (and terrain configuration). What worries me more here is the shift between the orthophoto and the other ODM assets. I'll try on another dataset to check if I see the same problem.
d

[kikislater]

But @denistestemale , could you describe what is inside gcp_file.txt ? It's a ppk flight not a ground collecting GCP ?

[denistestemale]

My GCP coordinates are ground collected (RTKLIB PPK ~2cm accuracy from a U-blox M8T GPS).

[kikislater]

There is a problem in this dataset, it's the same behaviour with well known commercial software. Bowl effect may be or not : most of the points shift to the north and you seems to have enough GCP to correct it ... could be GCP IMO

Sorry but your pictures are really ugly and doesn't help ... Why do you set your p4p to :

Sharpness : Hard

It's really a bad idea and not good for my eyes as well ^_^. Contrast images works well like 5mp camera on Delair tech DT18 or Canon S100 but sharpen I don't think so !

[denistestemale]

Could you elaborate about:

There is a problem in this dataset, it's the same behaviour with well known commercial software. Bowl effect may be or not : most of the points shift to the north and you seems to have enough GCP to correct it ... could be GCP IMO

In particular:

could be GCP IMO

Thanks.

[kikislater]

So you are using rtklib. Just about curiosity : which modules are you using (reach or other) ?
And if you are using rtklib, you were converting wgs84 to utm31. Please share original wgs84 coordinates.

[mwfoshee]

I'm not sure if this is applicable to the problem you guys are working on,
but when considering the scale problem I'm having, and RTKlib, I've been
outputting ellipsiodal and then converting to geodetic when creating .gpx.

Is this correct? Could this be leading to my scale issue?

Thanks

On Tue, Nov 20, 2018, 11:10 PM Sylvain POULAIN <[email protected]
wrote:

So you are using rtklib. Just about curiosity : which modules are you
using (reach or other) ?
And if you are using rtklib, you were converting wgs84 to utm31. Please
share original wgs84 coordinates.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OpenDroneMap/WebODM/issues/549#issuecomment-440533088,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aa9d7bomzKQQmdfActnuweUFexILgSJhks5uxOBNgaJpZM4YMfqE
.

[kikislater]

Yes you need to convert to geodetic. French local grid is better : ign69 - raf09. And for planar, could I know why are you not using rgf93 (epsg:2154) or local CC ?
But please share original without geodetic correction. Did you get only Q1 results ?

[denistestemale]

(Edited my comment about the antenna positions and datum)

Thanks Sylvain. Different elements of answer:

• The module I used is a u-blox M8T (same as in the REACH RS+ if I remember correcty). I PPKed my raw data with 2 baselines (to compare): one GPS only (no GLONASS) from a Bad-Elf surveyor installed onsite as a base, and one recording from an RGP antenna (STV2 at Gap, GPS, GLO, GALILEO). The results are the same (within 1 centimeter).
• The baseline antenna (STV2) positions are expressed xyz in the RGF93 datum (header of the RINEX files). So my RTKLIB results are in this datum (see my next message), under the form of geographical coordinates. But the persons who want to use this dataset prefer WGS84/UTM31 (for historical reasons): I offered to use ETRS89/UTM31 (EPSG:25831) because it is compatible with WGS84, and if I understood correctly ETRS89 is prefered on the european plate. They accepted. The conversion was done with the software Circé from IGN.
• I only got Q1 results (the site is located at 2500m altitude, very clear skies, nothing blocking the horizon) and several GCPs wre measured several times over the course of our 3 days field work: they positions are reproductible (5cm offset, but most likely due to the error I make when repositioning my antenna).
• To get the geodetic altitude I substracted the geodetic correction from the ellipsoidal elevation in QGIS (vector attribute operation).
• To validate the GPS measurements, I have very precise distances and altitude from the site (not visible in the area that we are dealing with in this thread). To make a long story short: it is an astronomic laboratory and they know some antenna positions very well (down to 5mm precision) and the altitude as well. The GPS measurements I did agree with those values (within 5cm in xy and 20cm in geodetic altitude).

So I'm quite confident that the GCP are correct. The only thing I'm not 100% sure is this ETRS89-WGS84 relationship, and the impact on the gcp_file of ODM. I tried hard to document myself on that, but have no clear answer. Your input might be valuable here, in case I make a mistake:

• Circé does convert from RGF93-Lambert 93 to ETRS89-UTM31N. Which I did.
• When in QGIS, I converted the resulting vector layer to WGS84-UTM31N (EPSG:32631), but cannot see any shift (coordinates are the same). I'm talking about doing "save as" to the vector layer with EPSG:25831 coordinates, and choosing EPSG:32631 as the destination SCR. The coordinates end up the same in the resulting csv file. And I cannot not find a way to convert from ETRS89-UTM31N to WGS84-UTM31N, although I understand it is needed. This reference http://www.killetsoft.de/t_1009_e.htm talks about a 0.5m shift.
• So I assumed it was an "epoch" thing or something, and that they can be considered the same, and that's why in my gcp_file.txt, the coordinates are ETRS89/UTM31N but the datum in the header is WGS84. I feel this is wrong (ellispoides are not exactly the same), but I thought that the WGS84 coordinates in the metadata of the JPG files are used to place the images in relation to each other, and that my GCP are used to georeference the resulting cloud. But I could be wrong here and that would explain the RMS 0.6m georeference error?

What I'm gonna do is convert my GCP to WGS84 (decimal degrees) with http://geofree.fr/gf/coordinateConv.asp , change the gcp_file.txt accordingly (header and values) and run ODM again.

Cheers
denis

[denistestemale]

Small addendum to my last message.
rtklibexplorer developer (and other sources over the internet) assured me that the output of rtklib is in the same datum as the antenna position (header of the RINEX files). But you seem to imply (above) that the output is WGS84. Any opinion about that?

[denistestemale]

I downloaded the trial version of http://www.killetsoft.de/p_trda_e.htm to convert my RTKLIB results (that I assumed to be in Geographic RGF93) to WGS84/UTM31N: the values are different from the ETRS89/UTM31N values that I used before (contrarily to both QGIS and http://geofree.fr/gf/coordinateConv.asp which would return the same numbers, I don't understand this difference). I did the ODM thingie with an updated gcp_file.txt. What I find is similar:

• georeferencing RMS error 0.73m (same as before)
• orthophoto shifted ~2m from the GCP (this time the orthophoto is north of the GCP, not south...)
• cloud (.ply) and dsm are aligned with each other, and their alignement with the GCP is in agreement with the 0.73m number, i.e. the targets are within ~[0.2-0.9m] of the GCP, with no trend (I mean they are not shifted on the same side, it looks like a statistically dispersed misalignement (if what I say makes sense)).
• cloud/dsm not aligned with orthophoto.

I think I'm gonna leave things as they are. I'll consider this 0.7m RMS misalignement an intrinsic number of the method and configuration of the terrain, and will compensate by QGIS georeferencing.

Cheers
denis

[smathermather]

QGIS as well as OpenDroneMap use PROJ which has datum transformation issues that are currently being addressed (the original creator of PROJ considered datum issues a separate issue from projections). These issues are being fixed, but in the meantime, datum translations are not a great idea within FOSS software.

That said, once you are in a datum and staying in that datum (QGIS or ODM, or whatever), you are good to go.

[smathermather]

So, we've come a long way from the original issue, and this is great conversation. I propose we close this issue and take up further conversation on http://community.opendronemap.org.

[pierotofy]

Agreed, we should just open a new issue that addresses the scale problem (which I think is still an issue), then move the conversation over.

[denistestemale]

OK, thank you for your time. I'm still puzzled by the offset between the orthophoto and the dsm, but I can live with it: from now on I don't use dsm from odm and generate a new one from the cloud point.
Regards.

[pierotofy]

Good point, @denistestemale could you open a new issue to track the DSM--orthophoto alignment issue separately?

[denistestemale]

Yes Piero, I just did:
https://github.com/OpenDroneMap/WebODM/issues/560

Denis

[denistestemale]

I don't know if this is still the place for the scale problem and the low precision (if not I apologize). I'd like to share some data in relation with these topics, that I obtained with 2 new datasets (in the same area):

• with these new 2 datasets, the agreement between the PC (and dsm) and the GCP is good (less than 20cm), much better than in the previous case (above). So I don't know what's the problem of the low precision in the first dataset.
• despite this good agreement, the "Mean georeference error" reported in the odm_georeferencing_log.txt file is weird (I've seen values of 45, 2 and 88 in datasets where the PC is spot on).
• with one of this new dataset I have the offset orthophoto (that's the other issue opened), and with the second one I just run in the same problem as @mwfoshee : I used a GCP file, the orthophoto is badly scaled (and shifted).

So could the offset and scale problem be the same thing? Since in both cases the PC is good. Let me know if I should stop publishing here and talk on the other thread.

Regards.

ps: what I'd like to emphasize as well is that despite this offset thing which can be compensated in QGIS the results are still fantastic! . Thanks.

[pierotofy]

This should now be fixed. If it isn't, please re-open the issue.

[denistestemale]

Hello.
I still observe the scale problem. I'm investigating with a different GCP file where I don't use GCP that are on the borders of the images, only the central ones, jsut to check. If I still see it, I'll make the dataset available.

Cheers