Geonode: Uploading Geojson format to geonode 2.10.1, not all items in features gets uploaded

[vijayr1912]

Expected Behavior

Uploading Geojson format to geonode 2.10.1, all items in features should get uploaded to the Geonode layer.

Actual Behavior

Uploading Geojson format to geonode 2.10.1, not all items in features gets uploaded. Have observed that last approximately 20% of the features are not uploaded to Geonode.

For the same layer, the shape file format has all the records in the layer.

Steps to Reproduce the Problem

1. Upload the attached Geojson file to the Geonode using Upload Layers option to the http://master.demo.geonode.org/.
2. After successful upload go to Editing Tools -> Layer -> Edit Data, at bottom of table 60 items will be listed. When there are 72 elements in the Geojson file
   
   
   
   1. Open the geojson file in a text editor confirms that there are 72 features but in geonode layer only 80%, (60 features) are seen.
   
   

Specifications

• GeoNode version: 2:10.1
• Installation method (manual, GeoNode Docker, SPCGeoNode Docker): Geonode Docker and http://master.demo.geonode.org/.

• Platform: Geonode Dockers on Mac
  Archive.zip

• Additional details:
  Zip file contains the Geojson and also its equivalent shape file.

[t-book]

Maybe connected with: https://github.com/GeoNode/geonode/issues/4713

[srggrs]

Me and @vijayr1912 were trying to debug such issue, can point us to the right directions please?

[srggrs]

Would django-osgeo-importer app would help in loading the correct geojson?

[afabiani]

Hello, my guess is that some lines of the GeoJSON contain invalid values.

I would try first to upload it directly by using GeoServer and see what happens.

Loging into GeoServer as admin and try to upload your GeoJSON by using the Importer functionality

[vijayr1912]

Hi

Thanks for the suggestion.

I tried to import the attached geojson file by using Import option of Geoserver. The status is shown as PENDING. whereas for shape file it gets to COMPLETE status.

The same geojson file on opening in online tool http://geojson.io shows the 72 records with its attributes.

Attached

1. Geojson file used for testing as zipped file.
2. Screen shot of Geoserver where the Status is stuck at Pending.

study001_ins00_Sept5_Buffer.geojson.zip

[afabiani]

So, probably an issue on GeoServer importer.
I'll try to do more tests with your attached file as soon as possible. I'm afraid we will need to open a ticker on GeoServer for that.

Would be good if you could open a JIRA issue on GeoServer also, in order to try to speedup the process (hopefully).

[vijayr1912]

Hi

I found a strange observation. I am attaching a zip of geojson files where the number of records are multiples of 30.

I found that

1. If geojson layerhas more than 30 fields, then records that get displayed are closest to multiple of 30.
   For example if layer has 72 records, 60 get uploaded and displayed as part of layer, 12 get discarded.

1. If layer has less than 30 records, even if it 29, it does not get reflected as records in Geonode.

2. If layer has less than 30 records, say 4, and if i copy paste the records to make it to 30 records or more than 30 records, then the layer records are uploaded to Geonode, though there is a duplication in records.

For all layer that is uploaded to Geonode, a corresponding layer store is created in Geoserver.

I uploaded the file using Update Layer option present in Geonode Front End.

If you can confirm you can observe the same with these files or any Geojson file downloaded and play around with 30 records we can confirm the behaviour and debug further.

All my geojson files are dealing with Polygon Geometry with "urn:ogc:def:crs:EPSG::4326" projection. We are not working with any other geometry or projections,

multiples-of-30.zip

Thanks
Vijay

[jnothman]

Yeah, I'd wondered if this bug came from some processing occurring in
batches, and the last (smaller) batch not being processed and saved by the
same code as the previous batches.

[vijayr1912]

Issue Tracker in Geoserver -
https://osgeo-org.atlassian.net/browse/GEOS-9325

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[afabiani]

up

[jnothman]

It would be good to keep an eye on the status of this in geoserver

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jnothman]

I think this is a severe issue, and should not be wontfixed, but should be tested and tracked.

[gannebamm]

If geojson layerhas more than 30 fields, then records that get displayed are closest to multiple of 30.
For example if layer has 72 records, 60 get uploaded and displayed as part of layer, 12 get discarded.

This sounds like a wrong for loop somehow. Maybe they switched rows/columns somehow? Or do not drill properly through the geojson with for each?

[gannebamm]

Would postpone this to get 3.0 faster

This seems to be an harder to tackle issue. In the meantime I would disable GeoJSON uploads.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[t-book]

up

[bruno-ichec]

Having the same issue in 3.0 here, both with geoJSON and KML, only regions up to a multiple of 30 are loaded in the layer. 90 out of 98 on a map, and 990 out of 1018 on another.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jnothman]

I think closing this remains a bad idea. Or disable GeoJSON first, per https://github.com/GeoNode/geonode/issues/4770#issuecomment-606420905

[gannebamm]

@afabiani do your clients use GeoJSON uploads? They are broken and I would strongly advise to disable this format untill we have a clear idea how to fix it. (disclaimer: haven´t tested it with the new GeoServer version though)

[gannebamm]

Sometimes Issues just get resolved. I can not reproduce the error anymore with geonode master. See:

https://master.demo.geonode.org/maps/new?layer=geonode:study001_ins00_Buffer&view=True

https://master.demo.geonode.org/maps/new?layer=geonode:study001_ins00_Buffer_geoJSON&view=True

If others face the issue again, please reopen.