Openfoodnetwork: [API clean up] Document the OFN API

[myriamboure]

Feature candidate description

https://community.openfoodnetwork.org/t/document-our-api/1498

T-shirt size estimate

@luisramos0 said he would need 5 days, like 30 hours. @Matt-Yorkley @sauloperez @HugsDaniel and Luis agreed on T-shirt sizing "L".

Definition of done / Acceptance criteria

Product team / support people are able to answer requests from users who want to have access to the OFN API and the concerned users are able to understand what is the current API.

Description of the feature including mockups

The feature is basically a wiki page that will document the OFN API. No mockups needed.

Story map

No sub stories, only one issue.

Lead team

Product owner:
Tech owner:

[luisramos0]

there are quite a lot of good links on the thread Myriam linked above, the two first ones are the must reads before you start:

• https://community.openfoodnetwork.org/t/road-map-for-ofns-api/931
• https://community.openfoodnetwork.org/t/representing-resources-via-api/1302

This issue is about having some support docs that help making sense of the output of the command "rake routes".

As I revisit this topic, if I was a student :-) I'd learn Swagger https://swagger.io/tools/swagger-editor/
Actually we may have a case for https://swagger.io/tools/swagger-inspector/
https://swagger.io/docs/swagger-inspector/how-to-create-an-openapi-definition-using-swagger/

So, in terms of output of this story we have:

• option 1 - openAPI specification - usually done with swagger
• option 2 - a wiki page like this one -https://github.com/openfoodfoundation/openfoodnetwork/wiki/API-documentation

option 1 is awesomeness, option 2 is easier as you don't have to learn what is swagger and the openAPI.

[kevinchristianson]

Thanks for the info, @luisramos0 I think we'll try to learn Swagger and make it there. What's the best way to set it up? Since Swagger hosts the API docs we're wondering if we should create an account on Swagger and then give it to OFN when we're done or if you want to create your own and give us access to it.

[luisramos0]

I think you can use "login with github" and then share access to the project with other accounts.

[luisramos0]

Even if the API gets based on your account it should be easy later to move it around, it's supposed to be easy to export/import.

[kevinchristianson]

@luisramos0 we found a ruby gem called swagger-blocks that uses Swagger UI to create Swagger documentation connected to ruby code. This would allow us to write documentation within classes themselves and then generate Swagger UI webpages from that. We set it up on our fork with some demo classes and it seems to work well. Would you be okay with this approach to the documentation? We see many positives in this approach both for us and you, but it would mean adding another gem to OFN. Let us know what you think and thank again for working with us on this!

[luisramos0]

ah, brilliant!!! you guys rock!
Your suggestion is a great idea!!! API docs in the code help keep the docs up to date but it's a lot of noise in the controllers...

You will not be able to cover the full API unless you automate the process, right? So, automating the docs to get a good coverage is the most important thing here. My main question is thus: would you be able to use Swagger Inspector and generate these for every API controller in both OFN and Spree?

Also, this tool does not cover openAPI v3 which I think is a big drawback, if we use openAPI we should start we the latest version. v3 was released in 2017 (we could use https://www.apimatic.io/transformer to convert to v3).

But I think the main drawback of this approach is OFN specific: there are many API endpoints in Spree that we want to use and document but we do not want to change the controllers code... it's Spree code.

So, what's the advantages you see in using this approach? Why not just use swagger inspector and generate the json outside the code?

[haseleyi]

RE automation: it’s a strong word; these gems operate by generating json from special code within the code. Swagger-blocks does so via added controller code. Another gem that we’ve succeeded setting up is rswag. It’s similar, but has the advantage of keeping documentation in specs. This could help reduce clutter in the controllers. Swagger blocks has an open PR for covering openAPI V3: https://github.com/fotinakis/swagger-blocks/pull/120. That updated two days ago and tests are passing, so between these two gems, rswag has less (zero) controller clutter while swagger-blocks may soon be V3-compatible.

The advantage to this approach is twofold:

1. Interactivity: people looking to integrate with our API could make actual endpoint calls on an authorized “test” account and see real data come back in the form of json.
2. Maintainability: Whereas having externally-hosted documentation could complicate changes to code, with an integrated gem, each PR that changes the API would have documentation changes encapsulated within.

That said, there are other gem integrations available: a quick search yields rabble, rails-restapi, rspec_api_documentation, rapi_doc. But you raise a good point: with both ‘APIs’ at play here, can one gem coalesce all their endpoints?

Our knowledge of OFN and Spree’s marriage is failing us a little here. Is the same data behind both? Are both APIs accessible from the OFN repo?

[luisramos0]

nice, thanks for your input! your list of advantages is good.

By automation I mean to automate the generation of the docs from the existing API without having to review any endpoint. From what I understand with swagger blocks you would have to generate the "special code" in the controllers. if there are 200 endpoints in our controllers (I think they are more), that means you would have to write and review that "special code" for all 200 endpoints.
By automation I mean swagger inspector type of thing where you just run a script and the existing API has some docs ready to be read and improved manually. These tools just call the existing API, document the endpoints, the methods and the results (by parsing real results of calls to the endpoints).

Thanks for raising this topic, after this conversation I am more convinced that this type of tool like swagger blocks is not a good approach for us, because we would end up with a good approach but with a very incomplete description of our API because we would never have the time to document all the endpoints with that "special code". I had a look at this "special code" (it's a DSL) and it's actually a bit complex with many details to be filled in for each endpoint, isn't it?

In terms of spree vs OFN APIs, yes both APIs are available in a OFN instance, including your local instance. If you run "rake routes" locally you will see the list of endpoints and if you follow some of the endpoints to their controllers you will see that some are in OFN, others are in Spree.
Let me give more details about this:

• this is the magically step where we say "all default spree routes are ofn routes": https://github.com/openfoodfoundation/openfoodnetwork/blob/master/config/routes.rb#L125
• and then we override some of them here: https://github.com/openfoodfoundation/openfoodnetwork/blob/master/config/routes/spree.rb
• the default spree api routes are here:
  https://github.com/openfoodfoundation/spree/blob/step-6a/api/config/routes.rb
• the ofn api routes are these ones:
  https://github.com/openfoodfoundation/openfoodnetwork/blob/master/config/routes.rb#L91

likewise, the default spree api controllers are under app/controllers/spree/api/ (these are overriding these) and the ofn ones are under app/controllers/api/.

I hope these links help!

Have you tried swagger inspector?

[haseleyi]

Luis! You’re the best, thank you so much for your quick and thorough responses. These links are super helpful.

We agree that some level of automation would tip the scale in favor of Inspector over the gem-based alternatives that we found, not to mention its compatibility with openAPI V3. Are you aware of an Inspector feature that has that “run a script” functionality of endpoint detection? We’re playing around with it and haven’t found it.

We do see its potential in detecting types within responses, and even that would sell us. When playing around, Inspector returns good data for localhost:3000/api/products. When we began an API definition, however, that response disappears and we get only 403 Forbiddens, so we're in debugging mode now. We read their troubleshooting guide and just posted on their community forum to solicit advice. We’ll also quickly investigate alternatives in the same family of products. We'll look for openAPI V3 compatibility and the highest degree of automation available.

[luisramos0]

nice, 403... it must be authorization. how are you authenticating before using the api? the user that you use for your test must have the right access to see the data. api keys or rails sessions are two ways to get into the api, I am not the best person to clarify here...
some context: https://community.openfoodnetwork.org/t/road-map-for-ofns-api/931

@mkllnk do you have more info about api auth? thanks

[haseleyi]

It seems to me that Swagger Inspector and Swagger Hub shouldn't differ in the level of authentication they require. I didn't input anything but a URL for the former.

I'm getting good data back from just "http://localhost:3000/api/products" in Inspector here. But when building the API documentation on Swagger Hub, their hosting platform, the same URL returns 403s.

[mkllnk]

Looks like that's a Swagger Hub limitation.

[luisramos0]

I would investigate your authentication, from what I understand swagger inspector will try to ping all endpoints and actions so it will hit all sorts of things including 403s. I would guess that you need to give it credentials for an admin user so the most api endpoints are seen with working and with data.

[kevinchristianson]

@luisramos0 thanks for the tips. We're having a hard time finding the "ping all endpoints" functionality you mentioned for swagger inspector. Do you have a link to it or documentation of it we could look at?

[luisramos0]

I am sorry, it's incorrect, isn't it? I must have dreamed...
With swagger inspector you need to provide the endpoints, methods and parameters.
The endpoints and methods are listed with rake routes.
The challenge is really the parameters. We will have to analyse each controller manually and fetch the list of parameters it takes.
So, one possible course of action would be:

• take rake routes (grep /api)
• create a script to inject this list into swagger inspector
• use swagger inspector to generate docs for these endpoints without info of parameters
• publish results: our api documented in the openAPI format wihtout any endpoint parameters referenced
• afterwards, start the work of going to each controller and add the missing parameters to the docs (I guess this is done in swagger) - probably not a task we can finish right away but we can start and get to a point where we have a good number of endpoints correctly documented

I think this is easier than adding the swagger-blocks DSL to each controller and will result in more endpoints being documented.
But I am not sure this is the best way to go, what do you guys think?

[haseleyi]

I researched ten other technologies that seemed to have the most promise, synopsis here: Documentation Technologies.pdf. I agree with your plan!

Our support case with SwaggerHub is making progress: Hub routes requests be default through its proxy server, and that was causing the 403s. We can change that setting to route through the browser instead, like Inspector does. With that change, we're now getting a TypeError. We'll keep working on this until we've nailed it down.

Here are the /api routes: api_routes.txt. We'll get on writing that next script!

[nozakis]

Hi @luisramos0
In order to use the "Try it out" feature on SwaggerHub we would like to feed requests to the live version of the public OFN API. We noticed that localhost:3000/api/products returns JSON that lists a number of products while accessing the same url on the actual OFN page redirects the user to a product setup guide. Is there a way for us to access the API endpoints on the live version of the site? We thought it would be most useful for us to document the endpoints in a way that directly connects them to the publicly available version of the API, instead of just running requests on localhost. We want people to be able to try out the API without having to go through the hassle of setting up OFN on their local machine.

[luisramos0]

nice research @haseleyi :-)
re TypeError, if you get stuck, pls post here or in slack, we may be able to help.

@nozakis , there's no app on openfoodnetwork.org
only on openfoodnetwork.org.uk or openfoodnetwork.org.au or openfoodfrance.org or app.katuma.org, etc
but I'd not use these apps for trying the API, I believe we are not ready for that, these requests like /api/products are quite heavy on the servers because they return all products in the catalog...
Can you please use a staging server like staging.katuma.org ? thanks!

[kevinchristianson]

@luisramos0 in understanding how routing works in OFN and rails we're wondering why there's such a big discrepancy between the number of API-related results that show up from the rake routes command and the number of api-related routes specified in the routes.rb files you linked above.

[luisramos0]

Thanks for asking, I created a wiki page to clarify routes:
https://github.com/openfoodfoundation/openfoodnetwork/wiki/Tech-Doc:-App-Routes

I think the answer to your question is point 2.2 here: https://guides.rubyonrails.org/v3.2.8/routing.html

[daniellemoorhead]

@luisramos0 am assigning you to this one as the OFN core contributor representative (I can't assign Kevin) 🙂

[RachL]

@luisramos0 is there a deadline or a time frame somewhere for this work? Some of our users in France asked us to access this doc and I was wondering first if it was a good idea to give it to them (I've understood that so far the API was designed to be used inside of OFN and not outside) and also if we are looking at something that would be available this trimester or more likely mid-year.

[luisramos0]

@daniellemoorhead I am not working on this task, I think we can do the same as in other contributions from the community and leave issues unassigned.
Also, I think it's good timing to get @sauloperez and @mkllnk feedback on the discussion above.

@RachL I don't know! it's very difficult to estimate this last part of the v2 project... api comes after.

[mkllnk]

I think we can do the same as in other contributions from the community and leave issues unassigned.

I agree.

I've understood that so far the API was designed to be used inside of OFN and not outside

Yes, it was designed for inside use. That doesn't mean it can't be used outside but we are not making any promise that it will be stable at the moment. If people want to write prototypes, great, they can use the current API as proof-of-concept and maybe it will work for a long time. If people want to build something stable then they need to wait and we don't have a time frame yet.

[daniellemoorhead]

@daniellemoorhead I am not working on this task, I think we can do the same as in other contributions from the community and leave issues unassigned.

Actually @luisramos0, we leave the issues unassigned, but there is always an owner for a feature that is being delivered which is what I saw this being. There isn't an epic with sub-issues, maybe there should be? And I assumed you would be both the product and tech owner for this particular piece of work, thus your assignation. Would this be true?

How shall we move forward with this? Open to suggestions on best approach so that we can track it as an epic (it's not small) but also have it in the delivery pipe.

[luisramos0]

yes, I understand @daniellemoorhead
I think all we need to do now is to wait for the output of these wonderful community contributors
I have marked it as a spike, ok @daniellemoorhead ?

[daniellemoorhead]

Perfect @luisramos0 👌

[nozakis]

Hey all,
We've been working for a bit on this issue and we're interested in getting some feedback from you all!

The documentation is hosted here. We're in the process of updating the listings for PUT and POST requests so the documentation for those is not quite accurate yet, but it will be in the very near future. In addition, there are some requests that only work when using localhost and not when using katuma and we're trying to figure out a pattern there.

[sauloperez]

OMG @nozakis ! this is massive! just the fact that we have these endpoints listed together with their schemas described is huge! Looking forward to having the accurate list!

Also, I think it's good timing to get @sauloperez and @mkllnk feedback on the discussion above.

I can't add anything. I never worked with Swagger and this is looking pretty good so far.

[luisramos0]

yes, this is such a valuable contribution! well done guys.

I think you selected a very good sample list of endpoints! the next ones would be orders and enterprises.

swagger is amazing isn't it? I'd love to hear from you about your experience with it.

I have a rating for you work: 💯

[luisramos0]

Hey guys, can you please remove the super admin user email and password from the web page?
Maybe you can replace with "Some endpoints are public and require no auth, others will require auth. Talk with us to get your credentials setup."

[haseleyi]

The GET /model-name/new endpoint returns a list of attributes and a list of required_attributes. In this example for products, we see that tax_category_id is presumably a required attribute when POSTing a product, but when GETting a product here, this attribute seems to be missing. The required_attributes list for the "property" model is empty, but the controller nonetheless rejects your request if it's missing values for "name" or "presentation". Is there a pattern to determining which attributes are required for a POST request? We could use trial and error for each endpoint, but thought we'd check first!

[sauloperez]

There is no pattern that I know of other than looking at the code. Our API lacks cohesion as things are implemented differently and in different places. You need to check whether it's implemented in OFN itself, as a decorator onto a Spree API endpoint or just a regular Spree API endpoint. Then you can read the controller's action code.

For this particular case of the Property model you mention it is clearly due to a presence validation. You'll find the relevant controller in https://github.com/openfoodfoundation/spree/blob/fe0a1311abb097bf3ac8001a24246c6cff5ecdbb/api/app/controllers/spree/api/properties_controller.rb#L21-L29 and the failing validation in https://github.com/openfoodfoundation/spree/blob/fe0a1311abb097bf3ac8001a24246c6cff5ecdbb/core/app/models/spree/property.rb#L10. Then, it was https://github.com/openfoodfoundation/spree/commit/7f704364dce#diff-a8a3f53a30136f04dad17121b39f5a52 that introduced that required_attributes in the API response but it's not very clear why. You might be able to investigate this further.

[kevinchristianson]

We've found a few routes now that were returned by the rake routes command but don't appear to be valid. For example, GET http://localhost:3000/api/users doesn't seem to be valid since the UsersController doesn't have an index method (or any methods for that matter). We're wondering why rake routes is returning URLs like this and if there's something we should do when we find them.

[kevinchristianson]

A related question to the one above is why in the rake routes output one URL can map to multiple controllers. From what we've seen it looks like each route maps to a specific controller method, so why would a URL be listed under different controllers?

[luisramos0]

the first one is probably related to spree controller and auth devise controllers, they are all included!
see spree:
https://github.com/openfoodfoundation/spree/blob/step-6a/api/app/controllers/spree/api/users_controller.rb
and devise:
https://github.com/openfoodfoundation/spree_auth_devise/blob/spree-upgrade-intermediate/app/controllers/spree

can you give a specific example for the second question?
each URL will have multiple methods GET PUT etc, they can be mapped to diff controllers I believe.

[kevinchristianson]

Example URL that rake routes lists under multiple controllers:

['admin', 'GET', '/api/users/authorise_api(.:format)', 'spree/api/users#authorise_api', '{:format=>"json"}']

['authorise_api_api_users', 'GET', '/api/users/authorise_api(.:format)', 'spree/api/users#authorise_api', '{:format=>"json"}']

These two routes have the same URL and are both GET requests, but one is listed under 'authorise_api_api_users' and the other is under 'admin'.

[luisramos0]

ok, thanks for clarifying.
I am guessing a bit here but I think it's not different controllers, it's just duplicate routes that are named differently. so, I think you can get that route in different ways through those two names but they are the same route with same controller.

I cant find where this particular one you list is coming from...

[sauloperez]

We've found a few routes now that were returned by the rake routes command but don't appear to be valid. For example, GET http://localhost:3000/api/users doesn't seem to be valid since the UsersController doesn't have an index method (or any methods for that matter). We're wondering why rake routes is returning URLs like this and if there's something we should do when we find them.

I haven't checked this particular case (I leave that to you) but in OFN I learned that as long as you provide a template for an action, Rails will render it no matter if there is no action defined in the controller.

I just tried it with app/controllers/spree/admin/reports/enterprise_fee_summaries_controller.rb. From this commit I realized that although the new.html.haml already existed we didn't have the corresponding #new action. If you try deleting it, the page still loads successfully. This is something worth investigating in Rails internals. So I bet the one you mention is one of those cases. I hope it helps!

[lin-d-hop]

Hi API workers,

I just wanted to check in on something....
When OFN extended Spree we added the concept of Enterprises. Spree is designed as a single business ecommerce platform. We essentially modified it to be a many business ecommerce platform.

As a result of this I assume:
1) That the existing Spree API endpoints are general endpoints
2) That if a user rq is authorised response will include ALL relevant data eg all orders for all enterprises
3) That there is no mapping/filtering based on users and enterprises when a user accesses the API using their generated key.

I'm just getting my head around the API and the scale of inception/group decisions to be made to progress. I'll post on discourse but grateful if you could confirm the above for me in the meantime.

Thanks!

[luisramos0]

Awesome to see this discussion happening @lin-d-hop
Your assumptions are not correct in all cases, it needs some clarification.
Can we talk about that in discourse, this issue is too long already isn't it?

[luisramos0]

un-assigning myself from this issue as people think I am working on this, which is not true.

[mkllnk]

@lin-d-hop No, I think we patched a lot of Spree code to consider enterprises. If I remember correctly, Spree endpoints were either patched to show only what the user is allowed to see or they were deactivated. But I'm not sure on this and there may be gaps and some endpoints may still be returning everything. I hope that the API documentation will help us to answer your question.

[kevinchristianson]

Sadly, our group project has come to an end. Here is the file for what we've documented thus far, and you can see it hosted here. We've all enjoyed working on this project and want to thank you all immensely for your support; we would not have gotten far without your help and encouragement. From our experience, this dev community could not have been better and we're all so glad to have been able to contribute to the project!