Hydra: Let's improve the docs!

[aeneasr]

It's awesome to see how Hydra is attracting more and more users, and that we're seeing production deployments pop up! We are really close to releasing the 1.0.0 version (officially stable) as we are observing a steadily declining bug report frequency.

One large issue still open is the documentation. OAuth2 can be very overwhelming and the consent flow is an additional complexity people have to understand. The policy concept adds another mandatory "you have to understand this".

I would like to ask you, the community, what problems you faced when you first read the docs. Maybe you have ideas to improve that? Or better examples?

You could even create a PR - the docs are here. To run them, do:

npm i -g gitbook-cli
gitbook serve -http

To do

• [x] document all special scopes (offline, hydra, openid)
• [x] example with force-consent scope (maybe @waynerobinson could help here)
• [x] write walkthrough for GKE
• [x] document groups

[ysaakpr]

How the warden groups are getting used. I could not find any kind of document related to that. is it related to something the policy subjects .

For example : if subject: "user:1" is belong to group admin
and a policy with subject "group:admin" will allow warden request if the subject is "user:1"

[ysaakpr]

And informations about limitations while using Different Storage implementation.
Which DB supports all features ? Even though RethinkDB which hydra uses by default I did see that few functionalities wont works in RethinkDB and Redis.

Any performance benchmark against using different supported storage (out of this scope, but still will be very helpful for people to decide on what and why to choose hydra with specific storage...again its up to user)
It will be very good to suggest whats suggested storage for hydra, and limitations of using other storage.

[aeneasr]

Even though RethinkDB which hydra uses by default I did see that few functionalities wont works in RethinkDB and Redis.

RethinkDB is not the default storage, MySQL and Postgres are the only officialy supported and maintained adapters.

Any performance benchmark against using different supported storage

Good idea!

[aeneasr]

Outline for blog post on deploying hydra on google container engine with google cloud sql, follow up to
https://github.com/ory/hydra/issues/330#issuecomment-284354389

THIS IS WORK IN PROGRESS! The idea is to have a guide that is specifically targeted at kubernetes, but also includes some general advice on getting hydra up and running.

Notes

• Don't use : in client ids as it will trip basic auth (see so)

Setting up GCE and GCSQL

Pretty straight forward, maybe include screens.

• Allow GCE to access SQL: https://cloud.google.com/sql/docs/mysql/connect-container-engine
• Make sure the database exists

Setting up pods

Hydra

hydra.deploy.yaml

Setting log level to debug helps finding bugs. We use http because of tls edge termination (could be more sophisticated). Also note that replicas could be anything.

apiVersion: v1
kind: Deployment
metadata:
  name: hydra
spec:
  replicas: 1
  template:
    metadata:
      labels:
        service: hydra
    spec:
      containers:
      - env:
        - name: LOG_LEVEL
          value: debug
        - name: CONSENT_URL
          value: !!!!!!!!!!!!!!!!!! TO BE DONE !!!!!!!!!!!!!!!!!!!!!!
        image: oryd/hydra:v0.7.7-http
        name: hydra
        resources: {}
      restartPolicy: Always
status: {}

hydra.secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: hydra
type: Opaque
data:
  SYSTEM_SECRET: somesecret
  DATABASE_URL: !!!!!!!!!!!!!!!!!! TO BE DONE !!!!!!!!!!!!!!!!!!!!!!
  FORCE_ROOT_CLIENT_CREDENTIALS: !!!!!!!!!!!!!!!!!! TO BE DONE !!!!!!!!!!!!!!!!!!!!!!

Consent app

consent.deploy.yaml

TO BE DONE

Deploy secrets & pods

kubectl apply -f hydra.deploy.yaml

Configure hydra

Consent app client

Create

Policies

Resource provider client

Create

Policies

Third-party OAuth2 client

Create

Policies

Protect resource provider

Perform auth code flow

[therebelrobot]

One thing I found was missing documentation on what type of authorization header was needed when swapping out the authorization code for access or refresh token, I finally landed on a base64'd client_id:client_secret (or maybe it was documented somewhere else?)

[therebelrobot]

Also, the format of multiple scopes in the cli, I tried scope1 scope2 scope3, then scope1|scope2|scope3, and finally landed on scope1,scope2,scope3

[therebelrobot]

List and explanation of the available grant types would be extremely helpful, still not sure what every single one does yet.

[therebelrobot]

Also, sometimes when searching in the docs, the page freezes and needs to be killed... kinda weird.

[aeneasr]

Thanks for the feedback, I'll try to incorporate the feedback as best as possible. The freezing pages are unfortunately out of my control :/

By the way, you can always create a PR for the docs, they're located here: https://github.com/ory/hydra/tree/master/docs

[Moggers]

I would like some documentation on the different policy resources and their meanings. What clients should and should not be given access to to complete the different auth flows.

[matteosuppo]

Also what are the default scopes, like offline and hydra

[aeneasr]

Done

[drozzy]

1. Maybe a guide "for dummies"? For people not familiar with crypto? For example, I read oauth2 spec and understand the flows, but this file just overwhelms me:
   https://github.com/ory/hydra-idp-react/blob/master/src/common/service/hydra.js
2. If one is working in a different language to Node.js - they have to _re-implement_ hydra.js themselves. It seems like it is very important thing to understand. The HTTP API reference is not sufficient. I think maybe we need a dedicated section about HTTP API.
3. Perhaps a better documentation for the sample app?
4. Maybe can we drop in an actual identity provider docker container instead of a fake userService.js?

[bhearsum]

More documentation on working with Warden would be greatly appreciated. I've not been able to figure out what exactly I need to provide it to make its endpoints work. The API docs tell me the parameters it needs, but there also appears to be some sort of authorization that's undocumented (and different than what /oauth2/* needs).

[mewalig]

I'm not a novice programmer and am finding it exceedingly difficult to get this set up and running. Maybe because I don't have familiarity with Docker and the instructions in https://ory.gitbooks.io/hydra/content/install.html do not seem to work (at least not on the AWS EC2 Windows or Linux instances I tried) (it breaks at "docker exec -i -t hydra_hydra_1 /bin/sh").

I personally think that the using Docker in the installation+config guide is just horrible, for so many reasons-- not the least of which is that when you are done following the installation steps, you are left with something you can't use for production-- but, for what it's worth, and without trying to debate the worthlessness of using Docker, what I would really, really like to see to get started is a very simple set of commands and then examples:

1. Installation via source or binaries
2. List of external components necessary for a production deployment e.g. postgres. No need for installation instructions assuming the external component is well documented (which postgres is)
3. OS-independent configuration scripts, such as a SQL file that will set up the requisite postgres environment (create database, create table, etc-- this is also helpful because in itself this script helps to document the database structure that hydra uses), and/or sample hydra config files. This should take care of any "seeding" of the database needed to set up e.g. root admin, keys, etc
4. command to start hydra
5. curl commands for each of the critical operations that hydra provides via the REST api (create users, create policies, assign users, query actions, yada yada)

It seems like #1 and #2 are there, but, maybe I'm blind but I can't seem to find #3, #4 and #5 (other than perhaps some Docker-bastardized versions of #4).

Anyway, #1-#5 would be the bare bones to start with. The next phase-- a "nice to have" in my opinion, but very valuable-- would be a "let's imagine a common scenario" and follow with curl examples for each of the steps along that story line.

Admittedly I don't know much about hydra so maybe the above is misguided. But from an ignorant person's perspective, this is what I'd like to see, and if it's not possible or desirable to do that, then I'd be interested to better understand why.

I should add that I really like a lot of the documentation I've seen about how to use hydra-- it's just, I haven't had a chance to actually use any of it because the "Getting started" documentation so far has not been good enough for me to actually get started.

[aeneasr]

I'm not a novice programmer and am finding it exceedingly difficult to get this set up and running. Maybe because I don't have familiarity with Docker and the instructions in https://ory.gitbooks.io/hydra/content/install.html do not seem to work (at least not on the AWS EC2 Windows or Linux instances I tried) (it breaks at "docker exec -i -t hydra_hydra_1 /bin/sh").

You must be reading an old guide. Please make sure to hit F5. If you need a shell in the docker container you can use tags appended with -alpine.

Another possibility is that you are mixing up the lengthy installation guide with the 5 minute tutorial which indeed is broken. Tracked #717

I personally think that the using Docker in the installation+config guide is just horrible, for so many reasons-- not the least of which is that when you are done following the installation steps, you are left with something you can't use for production-- but, for what it's worth, and without trying to debate the worthlessness of using Docker, what I would really, really like to see to get started is a very simple set of commands and then examples:

1. Installation via source or binaries

Again, not sure what guide you are reading, but this is covered here: https://ory.gitbooks.io/hydra/content/install.html#install-ory-hydra-without-docker

1. List of external components necessary for a production deployment e.g. postgres. No need for installation instructions assuming the external component is well documented (which postgres is)

Good point, tracked here: #716

1. OS-independent configuration scripts, such as a SQL file that will set up the requisite postgres environment (create database, create table, etc-- this is also helpful because in itself this script helps to document the database structure that hydra uses), and/or sample hydra config files.

This is probably because you haven't gotten so far yet, but hydra migrate takes care of database migrations, which is pretty standard in software projects today.

1. This should take care of any "seeding" of the database needed to set up e.g. root admin, keys, etc

Hydra creates a root client and required cryptographic keys, which is documented, on first initialization.

1. curl commands for each of the critical operations that hydra provides via the REST api (create users, create policies, assign users, query actions, yada yada)

Not only does Hydra make this easy using a CLI (hydra help), the whole API is documented using swagger. Link is in the Readme (it's http://docs.hydra13.apiary.io/# )

Does this help? If so, why didn't you find these links? Can this be improved somehow? Was the primary reason that the first tutorial is broken which caused frustration and cross-reading? Asking for a friend...so he can improve the docs ;)

[aeneasr]

Actually, mixing up the 5 minute tutorial and the lengthy one has happened twice so far, thus tracked as #718

[mewalig]

Thank you for your comments. Things are indeed clearer in the morning! Yes, I apparently was attempting to use the 5-minute tutorial and the visual look & feel of it is so similar to the tutorial that I was confusing the two. I now have hydra running and in a couple days will return to comment more re #5 (usage examples) later.

In the mean time, re #1-4:

I agree that #1 and #2 are covered. I only included them in comments of what I'd like to see so that the list was comprehensive and all in one place.

My comment re Docker was not just for installation, and also applies to the rest of hydraguide which also relies on Docker at least for the examples. It definitely clutters up the commands and I could be wrong about this next speculation but it seems like the Docker-related pieces would have to be stripped out anyway in the reasonably likely case that someone wanted a non-Docker configuration (e.g. native, Kubernetes, AWS Container etc).

I understand that hydra migrate does the database seeding etc, but would find it more useful to see the actual sql script. I suppose I could just do a database dump to do that so this isn't a big deal.

I had looked at http://docs.hydra13.apiary.io/ already and I think it is fine as a reference document. However, that is not the same, or nearly as useful as, as a guide. The problem with the existing guides, at least for me, is that they do not explain the endpoints in the context of their usage. The closest they come is to show some NodeJS code, which would be great except that I am not going to be using NodeJS. It would be a lot easier for me to just see the analogous curl commands, and even if I was using NodeJS, I'd probably still rather be able to see what's going on under the hood without having to dissect the module code myself.

To summarize the most salient of my thoughts, here are the three resources I used to get set up, along with my preliminary view on how to make the whole experience better, minus any comments re usage examples:

• 5-minute tutorial is broken and should be removed or at least clarified as broken with big bold cap words immediately. I wasted a long time trying to find a fix, chasing tips related to virtualization settings etc. What I thought would save me a few mins ended costing a few hours and I almost gave up on trying this project out at all if it weren't for how much I liked everything else I read about it
• ory.am guide is a GREAT guide in that the visual layout is way better and the examples are much more useful (though I haven't tried them so I can't comment on how well the commands work). Only problem is that the setup commands failed for me.
• installation + configuration: this was the only guide whose commands worked (for me) so far-- though I have not finished getting through the example section.

Instead of all of the above, what I would really like to see is:

• simple install+base config in under 3 commands. you know, like the old school
  ./configure && make install
  Configure could include options such as what db to install (postgres or none), what db connect to (create new or existing), similar re consent app etc (btw I am not actually suggesting a configure bash script, only using as an analogy to a 1-step os-independent command that supports a range of options).
  At this point, no explanations of anything it is doing is necessary. It's too early for that
• A "Getting Started" guide that has steps for setting up common use cases. For example, use case A might be, securing web application where users log into a website that serves each user her own private content (like Gmail). Use case A-2 might be an extension of A, where any user can grant any other user and/or user group privileges for content (like Google Docs, where a doc creator grants access to another user to view/edit the doc). Use case B might be for securing a web API such as Zillow, where the end users use API keys for authentication, and use case B-2 might be an extension analogous to the API version of A-2. I'll probably have more specific thoughts on what this could look like after I dive into the examples more.

[aeneasr]

Thank you for your input, duly noted. The 5 minute tutorial will be fixed, it's actually just replacing the Dockerfile in the docker-compose config. I'm currently working on something bigger which has my focus but since it's tracked I'll get to it afterwards.

The ory.am guide is actually the same as the one from GitBook but for an older version branch (0.9.x). I am a bit confused by:

(though I haven't tried them so I can't comment on how well the commands work). Only problem is that the setup commands failed for me.

I'm not sure if the commands worked or not. They should work though.

Regarding Docker - it's very standard today and we recommend running the software in Docker. Hydra is based on 12 factor principles and thus "cloud-native" - which basically means that the ephermal file system of Docker doesn't mess with Hydra, and that it supports autoscaling. Having Docker installed is very common today and it's actually easier to do than downloading and configuring a bunch of environment dependencies such as PostgreSQL, NodeJS, Golang. If you haven't gotten around to it yet, I highly recommend giving Docker a second try. Not sure where your bad experiences come from, but if used correctly, it's an awesome piece of technology that makes application packaging so much easier.

The Apiary docs support CURL examples, although those might not work completely. Getting swagger to work properly with Apiary is such a headache, writing down those commands manually is insanity. I'm still looking for a better option.

The idea of the 5 minute tutorial with docker-compose is in fact exactly the simple install+base config. You're just out of luck that you where the one to catch the error.

Having said all that your feedback is highly welcomed. If things are hard to understand it is almost never the person reading it, but the person writing it. As part of our efforts to provide a richer suite of API security tools ( https://www.ory.am/products/api-security ) we are also refactoring the docs and putting everything in one place - API docs, guides, examples (which are definitely missing currently). Once our offering is done, it will be even easier (sign up and spin up a test instance) to see this thing in action.

In any case, I hope this experience isn't too frustrating, and you are able to get it working very soon.

[aeneasr]

Forgot two things:

1. The feedback regarding the endpoints is very important. I'm not sure how in-depth we'll go with OAuth2 endpoints, as a basic understanding is required for using Hydra, otherwise it's likely the wrong technology. Everything else has to be documented better, preferably in the guide.
2. Migrations are currently stored in code. This won't change because otherwise we can't ship one binary, but have to ship the SQL files together with the binary, which sucks (bigtime).

[aeneasr]

The tutorial is fixed once #721 is merged and v0.10.10 is tagged. Enjoy!

[mewalig]

Very much appreciated! Will be back to this in a couple of days and as long as the feedback is deemed helpful I am happy to continue to comment. Will actually start to read that Docker book I bought now...