Pipeline: Document running the pipeline tutorial on one's laptop

[a-roberts]

I've noticed https://github.com/knative/build-pipeline/issues/235 too and that mentions an interest in running Knative and the pipelines locally (perhaps some information here could be used there, or this issue should be merged with another?).

Expected Behavior

It should be easy for anyone to follow the build pipeline tutorials on one's laptop - I've had good success using Docker for Desktop using the edge release, 4 CPUs and 8 GB of memory.

Actual Behavior

https://github.com/knative/build-pipeline/blob/master/docs/tutorial.md mentions gcr.io in a few places and the results are stored in stackdriver. Which this is awesome and shows how we could use a real cloud platform (which is where we want knative to be deployed!) I think we'll gain more attention and popularity by having knative more "hackable"; so we should lower the barrier to entry and make it trivial to use on one's laptop.

Steps to Reproduce the Problem

1. Try to follow the build-pipeline tutorial
2. Recognise a few things need to change (the use of registries: we can run our own on Docker for Desktop with a simple docker run command, and push to that), modifying resource.yaml where images are mentioned, and what a user's KO_DOCKER_REPO export should be; mentioning the need for a proper Go source directory or not much happens!).

Additional Info

I think this will be purely a docs contribution as everything works fine (apart from what we know is unimplemented already which is #216). This could be done with comments in the example code (e.g. adding todos where users should be replacing strings, and mentioning what things can/should be change and to what values).

[bobcatfish]

This is super awesome @a-roberts - I was recently on a plane and not thrilled with the idea of trying to upload images constantly via airplane wifi while working on the project - I didn't know we were so close to making it work with Docker for Desktop, now I want to give it a shot myself :)

I've noticed #235 too and that mentions an interest in running Knative and the pipelines locally (perhaps some information here could be used there, or this issue should be merged with another?).

I think I need to put some more detail into #235, but I think one key difference between #235 and this issue is that #235 would include being able to test changes to a git resource without having to push to a remote repo (maybe #235 needs to be more clearly scoped and broken up into some sub issues :thinking: )

[a-roberts]

Thanks for the input @bobcatfish, yep we only need to change a _few_ things really.

I agree it would be a great first issue, it would be awesome if somebody relatively new to knative (but perhaps not only myself as I could have fiddled with other components) were to follow what we come up with.

Here's what I recall doing a few days ago, so we we'll want to either doc this via a new readme or do it via code comments (I like the latter approach, have easy to search for placeholders):

• Install everything using Docker for Desktop (pick the edge version of Docker so we get a new Kubernetes version, 1.13.x IIRC and we need at least 1.11.x for knative). Istio and knative 0.3, as per our existing docs
• Grab the already listed pre-reqs
• Run your own Docker registry on port 5000: e.g. with docker run -d -p 5000:5000 --name registry-srv -e REGISTRY_STORAGE_DELETE_ENABLED=true registry:2
• Clone this repository to $GOPATH/src/github.com/knative
• Change resource.yaml: modifying the image coordinate for where the built image should be. Use localhost:5000/mycoolplace
• Change run/output-pipeline-run.yaml (we'll be modifying the image coordinate again), remove the use of stackdriver for results
• Set your KO_DOCKER_REPO variable to be localhost:5000/mycoolplace
• ko apply -f config and watch as the images get built locally
• Apply more yaml as per the tutorial: actually kicking off some tasks!
• Check the pipeline run succeeds and all is well

I think we should advocate having @mgreau's elasticsearch example deployed too to see the pod logs (useful to see our build logs after the fact, for convenience that's https://github.com/mgreau/knative-elastic-tutorials).

Now that https://github.com/knative/build-pipeline/pull/414 is merged I would like to give the tutorial another go (sanity checking/validating the above steps as well). I did plenty of learning in the one day so may have added steps from other tutorials along the way.

I'll take a look tomorrow if nobody beats me to it, I assume I'll need to build my own knative from source as there hasn't been a new release yet to pick up the changes.

[a-roberts]

I'd like to work on this, @bobcatfish would you do the honours of assigning me please?

I have some progress to share here, will ask a colleague to follow in my footsteps tomorrow 🤞 .

[nadgowdas]

@a-roberts can you please help me get pipeline controller deployed.

I am trying deploying it on 3-node kube cluster. I am following the guidelines here, and set the environment variables properly. I am using local docker registry.

When I run ko apply, it fails with Error: ErrImagePull error for controller.

pulling image "github.com/knative/build-pipeline/cmd/controller"
Failed to pull image "github.com/knative/build-pipeline/cmd/controller": rpc error: code = Unknown desc = Error response from daemon: error parsing HTTP 404 response body: invalid character 'N' looking for beginning of value: "Not Found"
Back-off pulling image "github.com/knative/build-pipeline/cmd/controller"

I believe I need to change registry names at more places beyond just KO_DOCKER_REPO. ?

[a-roberts]

Hey @nadgowdas, you're best suited to asking in the Slack channels we have for this (so for convenience please see https://knative.slack.com, but with regards to the above problem that strikes me as not setting your Gopath and source code paths correctly. You will find useful information clues at the discussion I had on Slack here.

The key part being:

moved my build-pipeline repo to be under github.com/knative under my gopath and now my mac's dying, I'm getting loads of stdout and I see my KO_DOCKER_REPO being mentioned, this command is now taking ages (in a good way)

So to be clear it (build-pipeline) is now at /Users/aroberts/go/src/github.com/knative/build-pipeline, where it should have been all along. Plenty of blob pushing's happened and docker images shows things at localhost:5000/adam/nop now, when I use a ko apply

So with the above steps you should have a local Docker registry running, you set KO_DOCKER_REPO, your folder structures are perfect, your Gopath is awesome and a ko apply goes and builds a bunch of images that are stored in our local registry and are used by build pipeline. Please check this is the case

[nadgowdas]

Thank you @a-roberts , moving this build-pipeline source under GOPATH you mentioned did help.

[bobcatfish]

I _think_ we've basically got this covered in our docs, but if I'm wrong we can re-open :)