Api-blueprint: Modularity

[zdne]

Support for multiple API blueprint files within one API.

[rurounijones]

For those situations where there are a large number of APIs, possibly under control of different groups in a company, this would be an absolute godsend.

[zdne]

@rurounijones just for clarification this feature is meant "To be able to split one API description into multiple files".

[zdne]

Anyone has any thoughts on this? As per my previous comment on Drafter https://github.com/apiaryio/snowcrash/issues/57#issuecomment-28538342

The options are either add some include / require facility on the top of blueprint or simply fetch all blueprints form a directory and concatenate them together.

Thoughts?

[BRMatt]

The concatenation is simpler but I feel require would be easier to use as you may want to require, then add something else after.

Will this be just blindly inlining the contents of the file as though it was defined in the patent, or will there be some kind of restriction on the contents of the file(e.g. can they have their own yaml header? Conflicts with parent yaml?) I've got the feeling that no restriction would be better but thought I'd bring it up.

[ecordell]

@zdne

First off, I have a rudimentary system for managing this already with Grunt, using simple file concatenation. Our blueprint was becoming unmanageable, and this was a quick and dirty way to split it into files (of note, arbitrarily many files, because they're concatenated by filesystem order). I posted my Gruntfile with a short explanation here.

Given that the concatenation is so simple to roll yourself, it makes sense to me that an "official" tool should be more powerful. I've been thinking more about this, trying to do some research into how this is handled by other systems.

1. Include/Require
   
   
   
   • Includes would be done at the top of a file, and parts of the file would be referenced within. I think this requires a more robust referencing system than the current [Model Reference][] syntax. Ideally any node in the included file would be addressable - so one might have something like [github.Users.'Create A User'][] or [github][POST /users] or something. (Might also need a prefix like =). Is there a better way to reference a specific node in the AST other than name or url+method?
   
   
   • Given that you have a good way to dereference node contents, this could be done with the C Preprocessor which would come with a lot of additional features at no extra cost.
   
   
2. Templating system/Transclusion
   
   
   
   • Non-markdown - The template would be written in a separate templating language, so a template file would not be parseable by snowcrash. An example of something like this is grunt-readme, which looks like it could work out of the box for templating a blueprint file.
   
   
   • Markdown - This would be something like the =[]() or :[]() syntax we discussed on the drafter issue. It would still be valid markdown and parseable by snowcrash.
   
   
3. Both
   
   
   
   • Given that you define a syntax for referencing nodes ([github]['Create A User']) and direct transclusion (:[/github.md](/github.md)), it seems like you could easily have both. Perhaps they could have the same general syntax like :[]() and the parser determines if it's a node or a file reference.
   
   

Any thoughts on this? Am I missing some options/tools? I'm sort of leaning towards the "both" option with a common syntax for transclusion and node dereferencing, but perhaps that's a little much.

[zdne]

Given that the concatenation is so simple to roll yourself, it makes sense to me that an "official" tool should be more powerful

Well eventually maybe yes, but I would like to not over design it and roll out something lean and simple in the begging.

Looking at the problems present problems that needs to be addressed, it is mainly:

1. Referenced (external) assets - #20 and as discussed in https://github.com/apiaryio/snowcrash/issues/57#issuecomment-28564071
2. Splitting blueprint into more manageable logical parts

For the purpose of splitting I would consider only either a Resource Group or a Resource (e.g. not an action, transaction example, or a request). At least in the beginnings.

Ad 1: I feel _referenced assets_ would be elegantly addressed by the proposal in #20 and discussed https://github.com/apiaryio/snowcrash/issues/57#issuecomment-28564071

Ad 2: I would not complicate it at all. The only thing I would use for transclusion is the regular markdown syntax e.g. [Some Text](path/to/blueprint/file.apib) or [Some Text](path/to/blueprint/file.apib#resource-name). For inclusion I would follow the same principle as with for assets in drafter e.g. use : prior the reference. Something like https://gist.github.com/zdne/8804418#aliens-question-aliens_question

I think this could suffice for the start?

[ecordell]

I think this could suffice for the start?

Agreed. I'm just now realizing that you're making a distinction between including "assets" and including "resources"/"resource groups". I had been thinking of all of them as "nodes" to be included at will.

For the second point, I thought it would be useful to reference the existing MultiMarkdown syntax for transclusion:

This is some text.

{{some_other_file.txt}}

Another paragraph

This seems like a simple way to include resources? (or indeed, arbitrary text/nodes?)

So to clarify my opinion:

1. I think a good syntax for referenced assets would be :[]() or =[]() as discussed
2. I think a good syntax for transclusion would be {{ }} as used by multimarkdown.

And also to clarify some vocabulary: I _think_ that when you say "inclusion" you're referring to what I mean when I say "transclusion", that is, entirely replacing a placeholder with text from another file.

If I'm understanding correctly, the difference between "referencing" and "inclusion"/"transclusion" is simply for the sake of _rendering_, say into html or pdf. To the parser, it doesn't matter if an asset (for example) is referenced or transcluded, the same AST results. Is that correct?

[zdne]

1. I think a good syntax for referenced assets would be : or = as discussed
2. I think a good syntax for transclusion would be {{ }} as used by multi markdown.

Why to have both instead of just one method? Is there any benefit for syntactically distinguish between those two? Personally I prefer number one since it still renders as a you can follow in a plain Markdown?

I think that when you say "inclusion" you're referring to what I mean when I say "transclusion", that is, entirely replacing a placeholder with text from another file.

Agree, will try to use transclusion from now on. Thanks!

If I'm understanding correctly, the difference between "referencing" and "inclusion"/"transclusion" is simply for the sake of rendering

Not really. In my opinion referring (creating a reference) should really only add a link to the output (AST) and leave the rest on the tooling (https://github.com/apiaryio/api-blueprint/issues/20#issuecomment-26766374)

Translcusion should instruct the parser (or preprocessor) to pull the content in...

[ecordell]

Why to have both instead of just one method? Is there any benefit for syntactically distinguish between those two? Personally I prefer number one since it still renders as a you can follow in a plain Markdown?

I could see a case for just having one (and I also prefer the first method, :[][] or =[][]).

The idea behind having two methods for transcluding information was to have two ways to control AST vs. html/md/etc rendering:

1. A method to link to a document. So :[link][link.md] renders from markdown as a link that looks like :link, but in the AST, that node holds a reference to link.md
2. A method to transclude a document. So =[link][link.md] renders from markdown as the _content_ of link.md. The AST representation would look the same as 1.

So I could be totally off here, and this may not be necessary, and we should just go with one syntax. But my reasoning was that the way a blueprint is displayed may need to be different from the way it's parsed into an AST.

In my opinion referring (creating a reference) should really only add a link to the output (AST) [...] Translcusion should instruct the parser (or preprocessor) to pull the content in...

I can't tell if you want to treat these as the same thing (one syntax) or as two separate things (two different syntaxes). Both seem fine to me. (Did I just make things more confusing?)

[vvakame]

:+1:

[zdne]

Just an update here: This is indeed planned and much needed. We are working on the tool that makes this possible. Meanwhile, if you do not rely on Apiary online editor you can you something like https://gist.github.com/danvine/11087404 or another, similar approach (I know at lest of few gulp-based tools for this).

[holic]

Whiskey now supports the .apib extension and makes editing large Markdown files much easier with its outline view: http://vimeo.com/album/3108294/video/110486733

[bcls]

http://9muses.se/erato/ and  http://25.io/mou/ also support .apib if you want a desktop app (for Mac) 

Robert CrooksDirector of Learning Services
Brightcove, Inc

On Dec 29, 2015, at 4:08 PM, Kevin Ingersoll [email protected] wrote:

Whiskey now supports the .apib
extension and makes editing large Markdown files much easier with its outline view: http://vimeo.com/album/3108294/video/110486733

—
Reply to this email directly or view it on GitHub
.

[zdne]

Hey @bcls and @holic, thanks for pointing out these editors!

[zdne]

Related: https://github.com/apiaryio/api-blueprint/issues/20#issuecomment-77108398

[foxx]

Has any decision been made by the core developers on this? It's been outstanding since 2013 and no clear cut answer has been given so far, can we get a line drawn on this please?

[zdne]

Hey @foxx there wasn't any clear decision on this one yet – I am leaning towards the transclusion syntax introduced in Hercule:

FORMAT: 1A

# Gist Fox API
Gist Fox API is a **pastes service** similar to [GitHub's Gist](http://gist.github.com).

# Group Gist

:[Gist](blueprint/gist.md)

:[Gists](blueprint/gists.md)

With the exception of

`````` markdown

• Model

+ Body

  ```
  :[](gist.json)
  ```

``````

Because the code blocks should remain opaque to the parser / preprocessor.

Also I do not think the transclusion should work on any level – the files being transcluded should be valid blueprints itself.

As for commitment to this – it is fairly high on the features roadmap – but no ETA just yet.

I would appreciated any comments or thoughts on this and also on features roadmap visibility.

Thanks!

[foxx]

The transclusion syntax seems like the best proposal so far, it's an elegant and clean approach, and better than any other suggestions I've seen so far. Fwiw, +1 from me.

[fesor]

any news on this?

approach with transclusion seems good to me, i don't see any disadvantages for it.

[zdne]

For the time being I suggest to use Hercule and the :[Gist](blueprint/gist.md) syntax. Will work towards backporting it back into API Blueprint spec and parser toolchain

[adams-sarah]

Has anything been done for this issue? I'd also like to split up a larger apib file into multiple.
Thanks!

[zdne]

Hey @adams-sarah unfortunately not from my side. I do still endorse the use of Hercule (https://github.com/jamesramsay/hercule) as it works pretty well – the only drawback is that you won't be able to edit the multiple files at Apiary.io. Let me know if I can help somehow!

[hendrikmaus]

+1 to the use of hercule https://github.com/jamesramsay/hercule
we have been using it for a while now and hope it will still find its way into the apib specs

[toobulkeh]

:+1: for the addition of this feature.

I also support a single TOC/index methodology. Transclusion is a bit too flexible and has not scaled in large wiki-based documents in my past experience.

[zdne]

We will definitely have to bake this into the parser itself to support source map.

In the mean while, we have prepared a real-example to show how to use Hercule – https://github.com/apiaryio/api.apiblueprint.org

[geemus]

@zdne in our usage of json-schema (via interagent/prmd) we split it along resource lines and basically enforce a particular directory structure to imply inclusions (more or less it builds a top level schema which has all the stuff in the directory as subschemas, so it changes levels of things). This is obviously pretty specialized to what we were doing and not particularly appropriate in your case, but wanted to share our experience. Compared to that, I think transclusion seems much more elegant/flexible, but it would definitely be nice to also have some straightforward means of having a consolidated TOC/index. Maybe that is similar to what we did though, ie you create an uber blueprint to serve as this index and then transclude/reference the sub-blueprints within it. I can see that happening automagically as being nice, but also as being awkward if it doesn't happen to do just what you want. Anyway, long way of saying I'd be happy to discuss/share my experiences further if that would be helpful.

[DavidBM]

What is the status of this feature?
We are using the Pro plan in Apiary.io and we need this feature

[kylewelsby]

@DavidBM I had the same problem. how I got around this is to have two apiary.apib documents.

Uncompiled

apiary-source.apib - a uncompiled source that makes use of the Include Feature

FORMAT: 1A
HOST: https://api.example.com

# Hello World

<!-- include(blueprint/posts.apib) -->

Compile

You can simply run:

aglio --input apiary-source.apib --compile --output apiary.apib

Your apiary.apib should now include the contents of blueprint/posts.apib within.

[DavidBM]

Hi, thanks! Yeah, but then you can't edit the documents in apiary.io

We already had the customized solution with Hercule, but the point of paying apiary pro for us is to let the product managers to update the documentation without entering in git. I'm afraid that this is a completely blocker for us and we will cancel the contract with Apiary if this is not solved.

I wouldn't expect this to be in this state after having Hercule and other tools for so many time available.

This is very very disappointing :(

[clucasalcantara]

Hello everyone, I'm sorry for getting here late and reopening and digging old stuff. If this isn't the right place please forgive me. But, doing a research to develop blueprint mocks I've found this discussion and this is the closest that I've could find until now.

I'm struggling to split my responses by model :(. Isn't it possible to referer a model into a different folder?
I would like to have inside my blueprints folder all my requests, and, for example into a separated models folder all my entities models (MSON/MD/JSON/whatever files, that whose fit) to reuse them into different responses easily. Anyone have success doing this?

[iamareebjamal]

It'd be really great if this issue got some attention. We have 26000 lines in our API Blueprint and it makes it incredibly difficult to manage

[KevinAnthony]

We are implementing Contract testing using Dredd and we are using Aglio, but it would be great if this was added so we wouldn't have to

[Almad]

One of the reasons is that we are iterating on design a lot :) When this issue was opened, there was no MSON and we are still torn on transclusions vs references.

But for the record, what are your needs there? Is it just multifile management and compiling a single document as a result or do you also talk reusability of models and sharing bits and pieces across multiple API description documents?

[KevinAnthony]

One of the reasons is that we are iterating on design a lot :) When this issue was opened, there was no MSON and we are still torn on transclusions vs references.

But for the record, what are your needs there? Is it just multifile management and compiling a single document as a result or do you also talk reusability of models and sharing bits and pieces across multiple API description documents?

We have a standard response that come from a bunch of our endpoints. we would like to be able to define that in a single place, near the library that houses it, and then import it where ever we need to respond with it.