Mapper: (Slightly) misplaced templates

[jmacura]

Steps to reproduce

1. open .tiff with World file as a template
2. open Geotiff as a template
3. see how much they align

Actual behaviour

They are mutually shifted by some 16 m. It is kind of hard to say which one is correct, but I feel like the Geotiff is misplaced.
Below is a screenshot from Mapper. Transparent b/w background is shifted (GeoTIFF), while the ortophoto below (.tiff+.tfw) is correctly placed (I assume)

Expected behaviour

All georeferenced templates should fit.
Below is a screenshot of those very rasters in QGIS 3.10.

Configuration

Mapper Version: 0.9.1 & last master (20191221.17)
Operating System: Win 10 Pro v1903

Notes

Sadly, I should not be sharing the template files ...
The CRS I use is EPSG:5514. It can be tricky sometimes :-(

[dg0yt]

Thanks for reporting the issue. It will be difficult to find out where this particular issue comes from. The main difference to QGIS is our usage of more recent versions of PROJ and GDAL. That's why we already stumbled over various errors.

We do have test files for world file and GeoTIFF, and they are run as unit tests. During our builds, the unit test for these files does not indicate any issue. You may try yourself with the files from test/data/templates.

If you don't mind sharing the geographic coordinates, than let us know the georeferencing:

• The contents of the world file, and also the width and heigth of the file, in pixels.
• The geoinformation of the TIFF file. On Linux, I would use the command listgeo. This command belongs to the geotiff package. Maybe there is something similar with QGIS. (And listgeo can be used to generate world files.)

For the test files, the output is like this:

OpenOrienteering/mapper/test/data/templates$ cat world-file.pgw

~
0.15
0
0
-0.15
649078
394159
~

mapper/test/data/templates$ listgeo *.tiff

~~~
Geotiff_Information:
Version: 1
Key_Revision: 1.0
Tagged_Information:
ModelTiepointTag (2,3):
0 0 0
649077.925 394159.075 0
ModelPixelScaleTag (1,3):
0.15 0.15 0
End_Of_Tags.
Keyed_Information:
GTModelTypeGeoKey (Short,1): ModelTypeProjected
GTRasterTypeGeoKey (Short,1): RasterPixelIsArea
GTCitationGeoKey (Ascii,21): "ETRS89 / Poland CS92"
GeogCitationGeoKey (Ascii,7): "ETRS89"
GeogAngularUnitsGeoKey (Short,1): Angular_Degree
GeogTOWGS84GeoKey (Double,3): 0 0 0
ProjectedCSTypeGeoKey (Short,1): PCS_ETRS89_Poland_CS92
ProjLinearUnitsGeoKey (Short,1): Linear_Meter
End_Of_Keys.
End_Of_Geotiff.
Warning 1: Cannot find pcs.csv

PCS = 2180 (name unknown)
TOWGS84: 0,0,0
Projection Linear Units: 9001/metre (1.000000m)

Corner Coordinates:
Upper Left ( 649077.925, 394159.075)
Lower Left ( 649077.925, 394135.075)
Upper Right ( 649107.925, 394159.075)
Lower Right ( 649107.925, 394135.075)

[jmacura]

If you don't mind sharing the geographic coordinates, than let us know the georeferencing:

• The contents of the world file, and also the width and heigth of the file, in pixels.

The content of the World file follows:

0.200000
0.000000
0.000000
-0.200000
-874999.900000
-1032000.100000

The resolution of this TIFF is 12500 x 10000 px (width x height).

• The geoinformation of the TIFF file. On Linux, I would use the command listgeo. This command belongs to the geotiff package. Maybe there is something similar with QGIS. (And listgeo can be used to generate world files.)

(sidenote: listgeo is available in OSGeo4W shell installed alongside the QGIS ;-) )
The output of listgeo for the GeoTIFF file follows:

Geotiff_Information:
   Version: 1
   Key_Revision: 1.0
   Tagged_Information:
      ModelTiepointTag (2,3):
         0                 0                 0
         -873813.69        -1032253.4692     0
      ModelPixelScaleTag (1,3):
         0.0999951668923267 0.0999945244500635 0
      End_Of_Tags.
   Keyed_Information:
      GTModelTypeGeoKey (Short,1): ModelTypeProjected
      GTRasterTypeGeoKey (Short,1): RasterPixelIsArea
      GTCitationGeoKey (Ascii,27): "S-JTSK / Krovak East North"
      GeogCitationGeoKey (Ascii,7): "S-JTSK"
      GeogAngularUnitsGeoKey (Short,1): Angular_Degree
      ProjectedCSTypeGeoKey (Short,1): Unknown-5514
      ProjLinearUnitsGeoKey (Short,1): Linear_Meter
      End_Of_Keys.
   End_Of_Geotiff.

PCS = 5514 (S-JTSK / Krovak East North)
Projection = 5510 (Krovak East North (Greenwich))
Projection Method: Unknown-1041
GCS: 4156/S-JTSK
Datum: 6156/System Jednotne Trigonometricke Site Katastralni
Ellipsoid: 7004/Bessel 1841 (6377397.16,6356078.96)
Prime Meridian: 8901/Greenwich (0.000000/  0d 0' 0.00"E)
Projection Linear Units: 9001/metre (1.000000m)

Corner Coordinates:
Upper Left    ( -873813.690,-1032253.469)
Lower Left    ( -873813.690,-1033517.200)
Upper Right   ( -872115.072,-1032253.469)
Lower Right   ( -872115.072,-1033517.200)
Center        ( -872964.381,-1032885.335)

We do have test files for world file and GeoTIFF, and they are run as unit tests. During our builds, the unit test for these files does not indicate any issue. You may try yourself with the files from test/data/templates.

Will do and let you know the result.

[dg0yt]

Maybe I was a little bit too optimistic: Of course your templates are different in size and position. Still there are things you can do:

• Use a known point to verify which template is accurately positioned.
• Use an OSM export to verify locations.
• Export a World file from GeoTiff with listgeo -tfw ..., and then try in Mapper 0.9. The world file takes precedence of the the GeoTiff data.
• With the world file, test in Mapper 0.8.x (old version of PROJ and GDAL, no GeoTIFF support).
• With our example data, test in QGIS.

I tried hard to get the half-pixel offset correct for world file vs. GeoTIFF, and I have no problems with my UTM maps. So I really expect some issue with PROJ vs. GDAL vs. GeoTIFF vs. EPSG 5514, or the level of adjustments we have to make to use modern PROJ correctly (late-binding, no more towgs84...)

[jmacura]

Thank you @dg0yt . I will investigate the bug further in the evening.

[jmacura]

So, there are some new notes on this issue:

• the GeoTIFF template seems to be the one misplaced – TIFF+.tfw aligns well with other datasources
• the GeoTIFF won't load in Mapper 0.8.1 (unrecognized georeferencing)
  
• the TIFF + .tfw loads exactly the same in 0.8.1 and 0.9.master
• I have generated a .tfw for the GeoTIFF with listgeo, but the result is the same as without it in 0.9.master (still shifted)
• the GeoTIFF with .tfw generated via listgeo and opened in 0.8.1 surprisingly aligns with the rest of the templates
  
• I am not sure what to do with the test files from /test/data/templates/ I have opened reo.tiff in QGIS, put OSM behind it and it looks like this:
  

[dg0yt]

the GeoTIFF with .tfw generated via listgeo and opened in 0.8.1 surprisingly aligns with the rest of the templates

Maybe it is not as much surprising as you think. I suppose you see the result of inconsistent definitions of EPSG 5514. (We, or rather PROJ, had issues with that CRS before.)

• The world file includes no CRS definition. Either you supplied a PROJ4 string, or directly used the EPSG:5514 from PROJ.
• The GeoTIFF includes a CRS reference to EPSG:5514. (I don't think it includes a full definition.) Mapper asks GDAL for a PROJ4 string. GDAL may delegate this to libgeotiff.

So if PROJ and GDAL/geotiff use slightly different definitions, the trouble starts. AFAIU there is an ongoing effort to align the databases of libgeotiff, GDAL and PROJ, but it is not trivial, and I cannot report issues without instructions how to reproduce them.

Unfortunately, even if you use a world file with a GeoTIFF in Mapper 0.9, the program needs to know the CRS, and takes it from the GeoTIFF without asking... Maybe we can mitigate your issue by offering more choice here.

[lpechacek]

Just for the record. I've just generated an additional template for my map project in QGIS. Saved with EPSG 5514, converted the GTiff to JPEG+WorldFile with gdal_transform and opened the JPEG in Mapper with the above patches. After selecting georeference from the template file, the template was shifted by 9 meters in the direction of about 30°. Selecting georeference same as the map made the template fit perfectly. I'm not sure how exactly my observations fit this case but I felt it's worth sharing here. In case my comment is off the mark, I apologize for the noise.

I can share the files for further investigation if needed.

[dg0yt]

@lpechacek Thanks for testing - this is how I meant my change to mitigate this issue.

I'm just piping PROJ 6.3.1 through CI, in order to have another PROJ fix in the master snapshot:

Fix wrong use of derivingConversionRef() that caused issues with use of +init=epsg:XXXX by GDAL (affecting R spatial libraries) or in MapServer

I don't know if it is really related, but we do use GeoTIFF via GDAL and PROJ. Check the next master snapshot.

[jmacura]

This issue really got wild in the last days.

I have installed Mapper 0.9.2. When I try to open (any) GeoTIFF template (not including TIFF + .tfw, they are fine) the result is following:

The very same happens for GeoTIFF supplied with additional .tfw created via listgeo.
The very same happens in the last MASTER (v20200214.2).

What is really strange (and probably not related to Mapper, but I am noting it here anyway, 'cause might be affecting more users) is the current behaviour of GeoTIFFs (again, TIFF + .tfw is inaffected) in older versions of Mapper, namely I have tested 0.9.1 and 0.8.1, screenshots are from 0.9.1.

• Map loaded properly:
  
• Opening a GeoTIFF template...
  
• no other dialog appers and voilá.. there is no template:
  
• But it gets really crazy if you move the template under the map...
  
• and zoom the map:
  
• Or worse, pan the map:
  
• But if you disable the GeoTIFF template, you are back in the origin:
  

I really don't know what I can do and what caused this change of behaviour in older Mapper versions (0.8.1 is still the same install as previously, 0.9.1 is reinstalled today). I would blame some Windows update but it's pure guessing...

[dg0yt]

You really need to supply the offending files. Send me a direct e-mail if I am to supply a private storage space.
Note that my available time is quite limited now, but I'm really willing to resolve this as quick as usual.

[lpechacek]

@jmacura The last template issue you are seeing may be unrelated to georeferencing. I saw similar behavior (failure to load template, "smudged" map image) when I loaded a GeoTIFF tat the Qt image loader does not support.

Converting the slope map to a JPEG with any tool that can read the GeoTIFF may help. The world file can remain the same after a pixel-to-pixel conversion but rename to '.jpgw' may be needed. HTH

[jmacura]

@lpechacek Thanks for the tip, Libor. I will try reload TIFF as JPG.
But the weird part is that it's still the same template file you can see (almost) correctly loaded in the original post of this issue.

[lpechacek]

But the weird part is that it's still the same template file you can see (almost) correctly loaded in the original post of this issue.

Ack. That part is absolutely worth investigating. If Kai does not make progress, for whatever reason, I can have a look too.

And now I recognize Kynžvart castle in the first picture! One of my best castle visit experiences ever! :)

[dg0yt]

Let's summarize (please verify):

| | v0.8.4 | v0.9.1 | v0.9.2 | QGIS |
|-----------|----------|----------|----------|---------|
|world file| ok | ok | ok | ok |
|GeoTIFF| n/a | offset | offset, pan issue | ok|
| TIFF lib | Qt | GDAL | GDAL | ? |
| Qt lib | 5.6 | 5.12 | 5.12 | ? |
| GDAL lib | ? | 3.x | 3.x | ? |

And again: I think the issue is limited to particular types of GeoTIFF. The format may carry various data inside. Some variants may need more work.

[jmacura]

@lpechacek @dg0yt I owe you both an apology (and a beer). I was opening different template by mistake. The GeoTIFFs that didn't want to load and thrown errors are not standard images but rather a data layer from terrain analysis. They e.g. have floating-point values in pixels, while the values exceed the standard image range of 0-1... This can be a problem for many programs I believe. QGIS renders them meaningfully so this tricked me (and my file naming convention wasn't very error-resistant in this project :-( )... If you still care about those and want to elaborate with such GeoTIFFs, I can share some excerpts with you.

Actually the state of things is opposite: The "slightly misplaced templates" now opens pretty well in 0.9.2 and master 👏

And yes, Libor, you have recognised Kynžvart pretty well :-) We still hope we will be able to organise an event there in the beginning of May... 😕

[jmacura]

And of course, there is no mysterious regression of older versions -- 0.8.1, 0.9.2, they work just like before. I only haven't previously noticed what happens, if I open invalid template file (because I've never tried it before).

[jmacura]

Just to fill in and refine your table, @dg0yt :
-- | v0.8.4 | v0.9.1 | v0.9.2 | QGIS 3.12.0
-- | ----- | ------ | ------- | -----
world file | ok | ok | ok | ok
GeoTIFF | n/a | offset | ok | ok
GeoTIFF with float | pan issue | pan issue | error message, won't load | ok
TIFF lib | Qt | GDAL | GDAL | ?
Qt lib | 5.6 | 5.12 | 5.12 | 5.11
GDAL lib | ? | 3.x | 3.x | 3.x

[dg0yt]

Dal si bych jedno dvanátcku, prosím... :-)

So there is an offset with some GeoTIFFs in v0.9.2?

We may do something useful with the float type data in the future. I just didn't want to start with the less common variants. What we already handle in GdalImageReader is:

• 8 bit grayscale (256 levels), optionally plus 8 bit alpha
• 8 bit indexed (256 colors palette)
• 24-bit RGB
• 32-bit RGBA

[jmacura]

Once the current pandemic and quarantine is over I hope we can meet at some o-event (as proposed in https://github.com/OpenOrienteering/mapper/issues/1035#issuecomment-489661258) and I will definitely have one with you ;-)

You are right again, I forgot to correct this in the table (now updated), there is no offset of GeoTIFFs in v0.9.2. Or at least no visible offset with the CRS I am using and with the quality of data I have. I can't say for sure it's pixel-wise equal with other templates, but the precision is absolutely satisfying for o-maps (see the last attached image -- the grayscale building fits nicely over the rooftop on the orthophoto).

As I said, the current state is satisfying for most of the cases, handling non-common image files is a long-run goal. Only enhancement that makes sense to me for next OO Mapper point release, is to be a bit more descriptive in the error message: "unable to read image data" could be extended to something like "Unable to read image data. Reason: Only 8 bit grayscale, 8 bit indexed, 24-bit RGB and 32-bit RGBA images can be loaded as templates". What you think?

[lpechacek]

Once the current pandemic and quarantine is over I hope we can meet at some o-event (as proposed in #1035 (comment)) and I will definitely have one with you ;-)

Answered there. I completely forgot about Olles' proposal. My mention of WOC does not exclude an earlier meeting though.

Only enhancement that makes sense to me for next OO Mapper point release, is to be a bit more descriptive in the error message: "unable to read image data" could be extended to something like "Unable to read image data. Reason: Only 8 bit grayscale, 8 bit indexed, 24-bit RGB and 32-bit RGBA images can be loaded as templates". What you think?

+1 I think Mapper should mark the template invalid (red in the list) in this case plus the error message should be more descriptive, as you propose. Please file a separate ticket for this UX issue and close this one. Thanks!

[jmacura]

Shift of standard GeoTIFFs fixed in v0.9.2.

[dg0yt]

For the record, https://github.com/OpenOrienteering/mapper/commit/e824d1d3521b86680062992a0db5fcc4c5f5eaf0 was another related fix: Mapper didn't handle resource file names as expected by PROJ, so it failed to load grid shift files in addition to the projections database.

[lpechacek]

And yes, Libor, you have recognised Kynžvart pretty well :-) We still hope we will be able to organise an event there in the beginning of May...

For the record, the resulting map. Note the OpenOrienteering logo in the lower left corner. :)
http://doma.obricany.cz/domains/doma.obricany.cz/map_images/750.blank.jpg

[jmacura]

For the record, the resulting map. Note the OpenOrienteering logo in the lower left corner. :)

Oh, sure. There was some space left in the end, so I felt the responsibility. Both mappers, and myself as well, have been using OO Mapper exclusively while creating this map 😉 OCAD came into scene only when the map was almost ready and the courses were about to be made.

Pity, I was not on the event, so I could not reach you in person.