Skipper: Create a Github DataClient implementation

[lmineiro]

Github provides a good source of route definitions that can be easily reviewed and versioned

[aryszka]

great idea! could be a nice project. plus the gained experience could be used for considering something like a github based filter repository.

[LappleApple]

If "Help Wanted" can also apply, thumbs-up this comment.

[sarslanhan]

@lmineiro @aryszka What would be the expected functionality for this? I have an early prototype that can load the eskip files in a github repo and initialize the routes with those. I have a question regarding how to select the eskip files to apply. These are the options I can think of:

1. Load all files with .eskip (or something else) extension
2. Have an entry file in the project root (like .skipper), similar to .gitignore files, that contains the list of eskip files to load.

Let me know what you think.

[lmineiro]

Hi @sarslanhan, thanks for offering help.

When we first thought about this I believe we considered a very basic DataClient which would be initialized with a single GitHub object to load routes.

You idea of loading entire sets of eskip files sounds great. I also see how the DataClient could standardize on some well-known manifest file to keep code initialization cleaner. Both sound great to me.

I was also wondering if we could leverage tags and/or branches to load such eskip files.

Feel free to send the PR, I'd love to see those in action.

[aryszka]

agree with @lmineiro that these are great ideas.

I think the DataClient could have some initialization options. One set of these options could control which eskip files to load. It could be sg like this:

1. default: load all eskip files
2. if the repo has a manifest with gob or regexp wildcards, use it to match the eskip files in the repo to be loaded
3. if the dataclient options specify wildcards, use them
   4: if both defined, use the intersection

Another set of options could control authentication. E.g. a reference to a secret storage. This can be a bit more tricky and I think it is cool to address it in a separate iteration, I don't think it would break any backwards compatibility.

Thanks for the help!!!

[hjacobs]

I have some doubts about the "production readiness" of any GitHub data client: do we really want to encourage people to have a direct production dependency to GitHub/GHE? That sounds wrong to me.

[aryszka]

@hjacobs good point. @lmineiro , how about approaching this as a general git client?

[sarslanhan]

@aryszka so in that case we need a local copy for the repo right? if yes this also boils down to the issue related to watching for the file changes and polling the git repository.

[aryszka]

@sarslanhan true, sounds like that. It's possible that a more generic http client and the file watching would cover the use cases without any specific technology. @lmineiro , what do you think?

[sarslanhan]

@aryszka @lmineiro independent of how this issue evolves, i like the idea of having a manifest. maybe it could be extended in another ticket and we can decide on a format.

[szuecs]

I think we should specify what we want to achieve. For a Git integration it could be just a 10 lines bash loop that git pull --rebase to update a routes file and skipper watches from the routes file.
IMHO it's not a feature we should implement and for the http style: PHP had the problem with RFI (remote file inclusion), which I don't want to do either.
If someone finds the issue and think http dataclient would be nice to have please create a new issue, thanks