Geonode: GNIP-79: GeoNode REST APIs (v2)

[afabiani]

GNIP-79: GeoNode REST APIs (v2)

Overview

Create a real REST API for GeoNode resources, which is:

• Pluggable: new external dependencies and modules should be easily allowed to add their own API endpoints without modifying the GeoNode code base.
• Modular: each app has their own API module if needed. No more monolithic modules with a ton of classes
• REST compliant: the new API must adhere to the REST paradigm and protocol
• Live together with the standard Tastypie based GeoNode APIs

Proposed By

@afabiani
@giohappy

Assigned to Release

This proposal is for GeoNode 3.2.

State

• [x] Under Discussion
• [x] In Progress
• [x] Completed
• [ ] Rejected
• [ ] Deferred

Motivation

We would like to transition out from old Tastypie based GeoNode APIs, or, at least, provide a more lightweight, REST compliant and secure set of API endpoints.

The proposal is to collect all those endpoints under the /api/v2 URI prefix.

Proposal

Define a common REST router under geonode.api

router = routers.DynamicRouter()

schema_view = get_schema_view(
    openapi.Info(
        title="GeoNode REST API",
        default_version='v2',
        description="Application for serving and sharing geospatial data",
        terms_of_service="https://github.com/GeoNode/geonode/wiki/Community-Bylaws",
        contact=openapi.Contact(email="[email protected]"),
        license=openapi.License(name="GPL License"),
    ),
    public=True,
    permission_classes=(permissions.AllowAny,),
)

Define APIs for each GeoNode module

As an instance for geonode.base.api we can easily plug ResourceBase and Users routes as follows:

from geonode.api.urls import router

from . import views

router.register(r'users', views.UserViewSet)
router.register(r'groups', views.GroupViewSet)
router.register(r'base_resources', views.ResourceBaseViewSet)

Backwards Compatibility

NO Backwards Compatible.

Future evolution

• Get rid of Tastypie
• Allow GeoNode to be shipped in two modules, pure models/APIs and frontend

Feedback

Update this section with relevant feedbacks, if any.

Voting

Project Steering Committee:

• Alessio Fabiani: 👍
• Francesco Bartoli:
• Giovanni Allegri:👍
• Simone Dalmasso:
• Toni Schoenbuchner: 👍
• Florian Hoedt: 👍

Links

• Demo Instance
• Reference Branch: rest_api_v2_proof_of_concept

[t-book]

absolutely +1!

[gannebamm]

I like the idea and current dev implementation. Nonetheless we should discuss how and if this will connect and/or interfere with pygeoapi.

[francbartoli]

I like the general idea to get rid of the old API system. However, I have some concern about how this has been tackled. In particular as far as I can see:

• The data model is driven by the Django models rather than being OpenAPI centric with a document defining, at the least, the minimum viable product for a new GeoNode Core API system in terms of endpoints, resources, parameters, responses, each of them with their own schema to enforce the validation eventually. I understood this has been decided to preserve the retro-compatibility. In other words, the OpenAPI document (formerly swagger) is just the output of the ORM at the moment. It would be great to have (if possible) a baseline for a GeoNode openapi.json definition in the codebase.
• The specification used for OpenAPI is based on OAS2 that is not maintained anymore and doesn't have some nice features available in OAS3. Also, the new generation of OGC APIs doesn't support OAS2. Is it possible to switch from OAS2 to OAS3? This would allow integrating pygeoapi eventually that supports only OAS3

I'd like to discuss the topic during the summit because I think it is strategic for the evolution of GeoNode in the long run.

[tomkralidis]

Valuable and timely discussion here for the Summit next week. Comments based on pygeoapi viewpoint.

• definitely agree with OAS3, which is what we are implementing in support of the OGC API standards
• pygeoapi is built as a Python API which is capable of handling 1..n web frameworks (we support Flask and Django out of the box). For example, see our Flask support which is a very simple web routing layer atop our pygeoapi.api machinery. In similar "multiple API" projects that I am working on, we simply blueprint pygeoapi to, say /oapi and delegate accordingly. Here in GeoNode, perhaps /api/ogc could be a suitable option
• pygeoapi builds the OAS3 document "by hand", i.e. (Python REST framework agnostic), as a build/deployment step. If we want both GeoNode and pygeoapi routes in the same OAS3 document, then we'll need a merge of the two of sorts when building/deploying
• what are the auth considerations (pygeoapi does not do auth yet and would likely delegate to the downstream application)?

Having said this, whichever way the evolving GeoNode API is developed, it should be easy enough to build routes on top of pygeoapi for anything OGC API.

[t-book]

Thanks @francbartoli @tomkralidis

I like the general idea to get rid of the old API system.

From my point of view we shouldn't just disable old api versions for the sake of backward compatibility (but to offer /v1 /v2 /v3 depending on how the api develops.)
Furthermore I do not see the present proposal as a complete redesign but as a technical improvement of version 1. But of course this is open to discussion. I welcome the replacement of Tastypie with Django Rest Framework as it is lightweight, has a great documentation and is productively proven but I understand the question if pygeoapi can be an alternative.

Valuable and timely discussion here for the Summit next week. Comments based on pygeoapi viewpoint.

👍

I would like to know more about how far the development has progressed and what are the possibility with the auth layer.

Nobody would prevent us from considering v2 as a technically improved version of v1 (tastypie -> DRF) and offering a later v3 which is more OpenAPI centric and based on geopyapi.

[kalxas]

First of all, thanks @afabiani for the proof of concept branch! It looks really good.
I also agree with @francbartoli and @tomkralidis that the new API should be based on OAS3 if possible.
@gannebamm yes, we should discuss how to interact with pygeoapi during the summit next week.

On the catalogue perspective I see some possible solution keeping the data model while working with the new OGC APIs at the same time: For resource base, we can keep/extend the model to keep compatibility with the CSW endpoint (which should not go away in my opinion). We can then re-use the resource-base model as a pygeoapi backend to support OGC-API Records, in a similar way we do with pycsw. For the rest of OGC APIs I echo @tomkralidis opinion.

[afabiani]

Dear all,
thanks for the feedbacks.

For the time being the REST APIs are just an improvement which does not
impact the model and GeoNode core structure at all, neither touches the old
apis.

The idea is to transition from an api to another ones, but nothing prevents
the two apis to live together.

About the pygeoapi, as I already said before, my main concern is about the
GeoNode model structure. IMHO the backward compatibility with older
versions of GeoNode should (more or less) always taken into account.
At least there should be a way to move, without struggling too much, from a
model to another one.

The second concern is the amount of work to be done if we decide somehow to
try to get rid of the current GeoNode structure... I know we now have much
more experience, but rewriting GeoNode from scratch is not an option, at
least for us. That would mean that we won't have commercial support anymore
from entities like GeoSolutions.

Il giorno mer 2 dic 2020 alle ore 08:48 Angelos Tzotsos <
[email protected]> ha scritto:

First of all, thanks @afabiani https://github.com/afabiani for the
proof of concept branch! It looks really good.
I also agree with @francbartoli https://github.com/francbartoli and
@tomkralidis https://github.com/tomkralidis that the new API should be
based on OAS3 if possible.
@gannebamm https://github.com/gannebamm yes, we should discuss how to
interact with pygeoapi during the summit next week.

On the catalogue perspective I see some possible solution keeping the data
model while working with the new OGC APIs at the same time: For resource
base, we can keep/extend the model to keep compatibility with the CSW
endpoint (which should not go away in my opinion). We can then re-use the
resource-base model as a pygeoapi backend to support OGC-API Records, in a
similar way we do with pycsw. For the rest of OGC APIs I echo @tomkralidis
https://github.com/tomkralidis opinion.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GeoNode/geonode/issues/6685#issuecomment-737056557,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJYARJNYTBLVLHOWGBOGALSSXWMFANCNFSM4UJDCTKQ
.

--

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V

for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si
precisa che ogni circostanza inerente alla presente email (il suo
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra
operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is
addressed and may contain information that is privileged, confidential or
otherwise protected from disclosure. We remind that - as provided by
European Regulation 2016/679 “GDPR” - copying, dissemination or use of this
e-mail or the information herein by anyone other than the intended
recipient is prohibited. If you have received this email by mistake, please
notify us immediately by telephone or e-mail.

[afabiani]

@Francesco Bartoli [e-mail] @Angelos Tzotsos
[e-mail]

I committed here
https://github.com/GeoNode/geonode/commit/399b420b3c4faf739fa41b6d85475f3a687cb855
an
update to generate the Swagger OAS3 schema

Il giorno mer 2 dic 2020 alle ore 09:35 Alessio Fabiani <
[email protected]> ha scritto:

Dear all,
thanks for the feedbacks.

For the time being the REST APIs are just an improvement which does not
impact the model and GeoNode core structure at all, neither touches the old
apis.

The idea is to transition from an api to another ones, but nothing prevents
the two apis to live together.

About the pygeoapi, as I already said before, my main concern is about the
GeoNode model structure. IMHO the backward compatibility with older
versions of GeoNode should (more or less) always taken into account.
At least there should be a way to move, without struggling too much, from a
model to another one.

The second concern is the amount of work to be done if we decide somehow to
try to get rid of the current GeoNode structure... I know we now have much
more experience, but rewriting GeoNode from scratch is not an option, at
least for us. That would mean that we won't have commercial support anymore
from entities like GeoSolutions.

Il giorno mer 2 dic 2020 alle ore 08:48 Angelos Tzotsos <
[email protected]> ha scritto:

First of all, thanks @afabiani https://github.com/afabiani for the
proof of concept branch! It looks really good.
I also agree with @francbartoli https://github.com/francbartoli and
@tomkralidis https://github.com/tomkralidis that the new API should be
based on OAS3 if possible.
@gannebamm https://github.com/gannebamm yes, we should discuss how to
interact with pygeoapi during the summit next week.

On the catalogue perspective I see some possible solution keeping the
data
model while working with the new OGC APIs at the same time: For resource
base, we can keep/extend the model to keep compatibility with the CSW
endpoint (which should not go away in my opinion). We can then re-use the
resource-base model as a pygeoapi backend to support OGC-API Records, in
a
similar way we do with pycsw. For the rest of OGC APIs I echo
@tomkralidis
https://github.com/tomkralidis opinion.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GeoNode/geonode/issues/6685#issuecomment-737056557,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAJYARJNYTBLVLHOWGBOGALSSXWMFANCNFSM4UJDCTKQ

.

--

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V

for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si
precisa che ogni circostanza inerente alla presente email (il suo
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra
operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is
addressed and may contain information that is privileged, confidential or
otherwise protected from disclosure. We remind that - as provided by
European Regulation 2016/679 “GDPR” - copying, dissemination or use of this
e-mail or the information herein by anyone other than the intended
recipient is prohibited. If you have received this email by mistake, please
notify us immediately by telephone or e-mail.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/GeoNode/geonode/issues/6685#issuecomment-737078042,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJYARJTQP6HF6UR3ZEZVPTSSX337ANCNFSM4UJDCTKQ
.

--

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V

for more information.

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A - 55054 Massarosa (LU) - Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si
precisa che ogni circostanza inerente alla presente email (il suo
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra
operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is
addressed and may contain information that is privileged, confidential or
otherwise protected from disclosure. We remind that - as provided by
European Regulation 2016/679 “GDPR” - copying, dissemination or use of this
e-mail or the information herein by anyone other than the intended
recipient is prohibited. If you have received this email by mistake, please
notify us immediately by telephone or e-mail.

[giohappy]

As already said by @afabiani this is neither a proposal for the API of GeoNode 4, neither for a GeoNode "core". We wanted to make a step beyond the current one, experiment with new use cases as required by our clients which more and more ask for an extende REST interface (including for permissions, users management, etc.). Of course this API reflects somehow the underlying model (including the parts of it that we don't like much).

I don't see any conflict with pygeoapi here. Nothing prevents, now and in the future, haning both the interfaces. We already have pycsw for CSW, we can havd pygeoapi for OGC API. What are your concerns here @francbartoli?

[gannebamm]

I see that the proposal does not interfere with other APIs and therefore voted my +1. Nonetheless we should give developers and users a clear way to interact with GeoNode API-wise. Having many different APIs alongside could cause confusion and convolute the codebase. This is something we should keep in mind.

[giohappy]

I totally agree @gannebamm. Sometime users ask questions or requirements about the GeoNode "API", but then I discover they were talking about the OGC APis. Probably a dedciated section inside the docs might help to clarify the different exposed APIs.