Javalin: Swagger things

[tipsy]

https://bitbucket.org/atlassian/swagger-request-validator
https://github.com/ShikaSD/javalin-swagger
https://github.com/manusant/SparkSwagger
https://github.com/swagger-api/swagger-codegen

[tipsy]

I've added a handler for rendering json/yml now, you can enable it like this:

val app = Javalin.create().apply {
    enableWebJars()
    get("/my-docs", SwaggerRenderer("exampleApiSpec.yaml"))
}.start(0)

[tipsy]

We could generate docs from Javadoc?

/**
 * My very nice description
 *
 * @queryParam my-query-param some description
 * @pathParam user-id some other description
 * @result SerializeableObject object
 */
get("/path") { ctx ->
    val queryParam = ctx.queryParam("my-query-param")
    val userId = ctx.pathParam("user-id")
    ctx.json(SerializeableObject());
}

[grote]

Would this javadoc be picked up by Swagger for an API documentation?

It would be great if specifying all this would enable validation as well. Something like specifying the params and than adding stuff like isInt().nonNull().min(0) or others.

[tipsy]

@grote Would this javadoc be picked up by Swagger for an API documentation?

My thinking was to do something like

SwaggerGenerator.generate("output.yaml").fromProjectFiles() // or list of files

Javalin.create().apply {
    get("/my-docs", SwaggerRenderer("output.yaml"))
}.start(7000)

It would be great if specifying all this would enable validation as well

You should open a separate issue if you want to discuss validation, but a primary goal of Javalin is explicit code, so generating validation rules from comments is pretty much out of the question.

[grote]

And SwaggerGenerator.generate() would then go through the files, read the JavaDoc and then generate the output file from there?

You should open a separate issue if you want to discuss validation, but a primary goal of Javalin is explicit code, so generating validation rules from comments is pretty much out of the question.

I was just bringing it up here, because I would hate having to specify all params two times. The idea was to not use the Javadoc, but some other mechanism that could (optionally) also have validation hooking in later.

Something like this (just brainstorming here):

get("/path",
    QueryParams()
        .add(string("my-query-param").maxLength(50))
        .add(int("my-query-number").optional()),
    PathParams().add(int("user-id").required()),
    Result()
) { ctx ->
    val queryParam = ctx.queryParam("my-query-param")
    val userId = ctx.pathParam("user-id")
    ctx.json(SerializeableObject(SerializeableObject::class))
}

[tipsy]

And SwaggerGenerator.generate() would then go through the files, read the JavaDoc and then generate the output file from there?

Yup, that was the idea.

I was just bringing it up here, because I would hate having to specify all params two times. The idea was to not use the Javadoc, but some other mechanism that could (optionally) also have validation hooking in later. Something like this (just brainstorming here): <pre>...</pre>

I understand not wanting to repeat yourself, but doing this in Javalin is difficult because of the lack of annotations and reflection. You'd have to build a whole swagger DSL, and it looks very noisy compared to a javadoc based generator. @ShikaSD actually started working on it if you want to have a look: https://github.com/ShikaSD/javalin-swagger/blob/master/src/test/kotlin/io/javalin/swagger/ApiTest.kt

[ShikaSD]

@tipsy Like the idea + it won't require that insane amount of reflection to build a reasonable API, which I pretty much like. The only problem could be the IDE support, but is should be fine.
Regarding amount of noise in API, it is hard to compare, as my example is just a prototype which basically copies swagger built-in one, so many constructs there can be easily simplified and enhanced.

[tipsy]

Regarding amount of noise in API, it is hard to compare, as my example is just a prototype which basically copies swagger built-in one, so many constructs there can be easily simplified and enhanced.

I didn't mean to imply that your particular implementation is noisy. I think any amount of of Java/Kotlin code that has to be integrated into the route declarations is going to be noisy compared to a javadoc comment.

With the javadoc approach I could do

app.get("/my-path", Ctrl::myHandler)

And in another file:

/**
 * My very nice description
 *
 * @queryParam my-query-param some description
 * @pathParam user-id some other description
 * @result SerializeableObject object
 */
fun myHandler(ctx: Context) {
    val queryParam = ctx.queryParam("my-query-param")
    val userId = ctx.pathParam("user-id")
    ctx.json(SerializeableObject());
}

In my IDE I could just collapse that comment.

[kyleellman]

What about using the app's final endpoint group? Based on what's in Javalin::handlerMetaInfo, each available endpoint is enumerated and paired with the function that handles the endpoint.

I'm not super familiar with what can be done with the handlerMetaInfo, but maybe that's how a swagger doc generator could access either a javadoc block or some annotations on the function to then process into the JSON for open API spec.

I think this would be particularly useful when using a crud handler since each endpoint isn't explicitly written when using the api builder.

Does that make sense?

[tipsy]

@kyleellman I'm not super familiar with what can be done with the handlerMetaInfo, but maybe that's how a swagger doc generator could access either a javadoc block or some annotations on the function to then process into the JSON for open API spec.

That could work, but I think it's simpler to just loop through all the source files and link things up "manually". If you see a Javadoc comment in a file Ctrl for a handler called myHandler, it should be pretty easy to link that to app.get("/my-path", Ctrl::myHandler).

Edit: It would be nice to keep the implementation independent of other Javalin concepts.

[tipsy]

I wrote a very crude first draft for this:

/*
 * Javalin - https://javalin.io
 * Copyright 2017 David Åse
 * Licensed under Apache 2.0: https://github.com/tipsy/javalin/blob/master/LICENSE
 */

package io.javalin.core.util

import java.io.File

/**
 * My very nice description.
 * It can be multiple lines,
 * because why not!
 *
 * @path /users/:user-id
 * @pathParam user-id a very nice description
 * @queryParam my-query-param some other description
 * @formParam my-form-param a terrible description
 * @result SerializeableObject object
 */

/**
 * My other nice description. It's just a single line.
 *
 * @path /users/:user-id
 * @pathParam user-id a very nice description
 * @queryParam my-query-param some other description
 * @formParam my-form-param a terrible description
 * @result SerializeableObject object
 */

data class SwaggerData(
        var description: MutableList<String> = mutableListOf(),
        var path: String = "",
        val pathParams: MutableList<Pair<String, String>> = mutableListOf(),
        val queryParams: MutableList<Pair<String, String>> = mutableListOf(),
        val formParams: MutableList<Pair<String, String>> = mutableListOf(),
        var result: String = ""
)

fun main(args: Array<String>) {

    fun extractInfo(str: String) = str.split(" ", limit = 3).let { it[1] to it[2] }

    File("src/main/java").walkTopDown().filter { it.isFile }.forEach { currentFile ->

        Regex("/\\*\\*([\\s\\S]*?)\\*/")
                .findAll(currentFile.readText())
                .map { it.groupValues[0] }
                .filter { it.contains("@path ") }
                .forEach { javadoc ->
                    val data = SwaggerData()
                    javadoc.lines().asSequence()
                            .filter { it.length > 3 } // content
                            .map { it.substring(3) } // remove margin
                            .forEach { line ->
                                when {
                                    line.startsWith("@path ") -> data.path = line.split(" ")[1]
                                    line.startsWith("@pathParam ") -> data.pathParams.add(extractInfo(line))
                                    line.startsWith("@queryParam ") -> data.queryParams.add(extractInfo(line))
                                    line.startsWith("@formParam ") -> data.formParams.add(extractInfo(line))
                                    line.startsWith("@result") -> data.result = line.split(" ")[1]
                                    else -> data.description.add(line)
                                }
                            }
                    println(data.description.joinToString(" "))
                    println(data.path)
                    println(data.pathParams)
                    println(data.queryParams)
                    println(data.formParams)
                    println(data.result)
                }
    }

}

It prints

My very nice description. It can be multiple lines, because why not!
/users/:user-id
[(user-id, a very nice description)]
[(my-query-param, some other description)]
[(my-form-param, a terrible description)]
SerializeableObject
My other nice description. It's just a single line.
/users/:user-id
[(user-id, a very nice description)]
[(my-query-param, some other description)]
[(my-form-param, a terrible description)]
SerializeableObject

[tipsy]

There's a PR here now: https://github.com/tipsy/javalin/pull/446/files

It's still very rough and just uses sout. The approach seems okay though.

[OneCricketeer]

Just chiming in.

Might be worth looking at how Proteus offers /openapi out of the box

[mikexliu]

@tipsy
Wondering if it's possible to change SwaggerRenderer to accept a URI instead of file path?

---

Also wondering if it's possible to use static analysis to "figure out" the yaml for swagger, at least for the required parts to get swagger going.

For example, if we see:

javalin.get("/car/:name", ctx -> {
    ctx.queryParam("model");
    // ...
});

It could auto generate (I have no idea what is required or not here):

paths:
  /car
    get:
       parameters:
         - name: name
           in: path
        - name: model
           in: query

I think this could even be detected at run-time if we can do some sort of "pre-processing work" on the endpoint handlers.

[tipsy]

@mikexliu, it could accept both?

Static analysis sounds like a possible solution, but I don't have any experience with it, could you make an initial PR that shows the approach?

[mikexliu]

@tipsy
I'm not sure, I've not used Kotlin before. I'm using Javalin with java and new SwaggerRenderer(...) only accepts a String.

I actually also have zero experience with static analysis. :D

[tipsy]

@TobiasWalle I heard a rumor that you were working on generating Swagger docs for Javalin source code, is this something you'd be interested in contributing?

[TobiasWalle]

@tipsy I'm surprised that the rumor got already out, as I published the repository just yesterday.

I need a swagger generation library for a private project and as I didn't find anything that met my requirements. So I started implementing something by myself. It is currently not a library, but you can find the source code here. In my implementation I was heavily inspired by javalin-swagger, but I use mbknor-jackson-schema to help me generating the Openapi schema, which drastically reduces the amount of the code.

It is already working but it missing a lot of features. So it is currently not possible to most of metadata like descriptions, etc.

As it seems that there is already some interest, I can publish an early version of the library this weekend.

[tipsy]

@TobiasWalle That's great, thank you! The feature has been requested way more than this issue would indicate, there's definitely a lot of interest. Do you plan on publishing the library yourself, or creating a PR?

[TobiasWalle]

@tipsy Sure, I can create a PR. I think this will make it easier to maintain the package in the long run. I'm going to prepare a fork and let you know once it's ready. I'm going to design the initial API but I will be open for changes and discussions.

[tipsy]

Great, looking forward to reviewing!

[TobiasWalle]

I created a PR https://github.com/tipsy/javalin/pull/568

[tipsy]

@TobiasWalle when you make the final PR, could you replace io/javalin/examples/HelloWorldSwagger.kt with an example that shows a few endpoints and annotated methods/classes? I imagine you could also delete/reuse io.javalin.core.util.SwaggerRenderer.

[TobiasWalle]

@tipsy Sounds good. I just started with the PR, but it should be ready today.

[TobiasWalle]

The PR is ready #581

[tipsy]

@TobiasWalle Looks like everything is working great! I've added two small commits:

• Automatically enable WebJars if needed: https://github.com/tipsy/javalin/commit/a0cd50228a61537ac134eaeca28347d299765530
• Print helpful info for missing dependencies: https://github.com/tipsy/javalin/commit/0f588f20bab518ec9fd92319e933444ce450964b

I'll play more with it next week.

[tipsy]

@TobiasWalle it seems setSerializationInclusion(JsonInclude.Include.NON_NULL) is non optional? Would it be possible to not depend on an object mapper for the implementation?

[tipsy]

There also appears to be a hard dependency on Jackson, rather than JavalinJson. Ideally the OpenAPI stuff shouldn't require Jackson, as that is an optional dependency in Javalin.

[tipsy]

@TobiasWalle Annotated methods are not working:

/*
 * Javalin - https://javalin.io
 * Copyright 2017 David Åse
 * Licensed under Apache 2.0: https://github.com/tipsy/javalin/blob/master/LICENSE
 */

package io.javalin.examples;

import com.fasterxml.jackson.annotation.JsonInclude;
import io.javalin.Javalin;
import io.javalin.http.Context;
import io.javalin.plugin.json.JavalinJackson;
import io.javalin.plugin.openapi.OpenApiOptions;
import io.javalin.plugin.openapi.annotations.OpenApi;
import io.javalin.plugin.openapi.annotations.OpenApiParam;
import io.javalin.plugin.openapi.annotations.OpenApiResponse;
import io.javalin.plugin.openapi.ui.ReDocOptions;
import io.javalin.plugin.openapi.ui.SwaggerOptions;
import io.swagger.v3.oas.models.info.Info;

public class HelloWorldSwagger {

    public static void main(String[] args) {

        JavalinJackson.getObjectMapper().setSerializationInclusion(JsonInclude.Include.NON_NULL);

        OpenApiOptions openApiOptions = new OpenApiOptions(new Info().version("1.0").description("My Application"))
            .path("/swagger-docs")
            .swagger(new SwaggerOptions("/swagger").title("My Swagger Documentation"))
            .reDoc(new ReDocOptions("/redoc").title("My ReDoc Documentation"));

        Javalin app = Javalin.create(config -> config.enableOpenApi(openApiOptions)).start(7070);

        app.post("/users", ExampleController::create);

    }

}

class ExampleController {

    @OpenApi(
        queryParams = {
            @OpenApiParam(name = "my-query-param")
        },
        responses = {
            @OpenApiResponse(status = "201", returnType = Void.class)
        }
    )
    public static void create(Context ctx) {

    }

}

[TobiasWalle]

@tipsy

@TobiasWalle it seems setSerializationInclusion(JsonInclude.Include.NON_NULL) is non optional? Would it be possible to not depend on an object mapper for the implementation?

This option should be optional. I just activated it, so the json is more readable.

There also appears to be a hard dependency on Jackson, rather than JavalinJson. Ideally the OpenAPI stuff shouldn't require Jackson, as that is an optional dependency in Javalin.

I don't know if it is possible. Basically we need to replace this line with something mapper independent
https://github.com/tipsy/javalin/blob/master/src/main/java/io/javalin/plugin/openapi/JavalinOpenApi.kt#L54
After work I will look into this.

@TobiasWalle Annotated methods are not working:

This seems to be a bug. I should working if you extend the Handler interface and pass the object directly. I will create a PR which fixes this.

[tipsy]

If I remove setSerializationInclusion(JsonInclude.Include.NON_NULL), I get:

[tipsy]

@TobiasWalle After looking closer at the libraries it seems Swagger Models has a hard dependency on Jackson. That makes things easier. Create your own perfectly sculpted object-mapper in OpenApiHandler and forget about JavalinJackson.

[TobiasWalle]

@tipsy

If I remove setSerializationInclusion(JsonInclude.Include.NON_NULL), I get:

Oh that seems a bug in the ui. If this attribute it not set all properties will be set with "null" instead of getting removed.

@TobiasWalle After looking closer at the libraries it seems Swagger Models has a hard dependency on Jackson. That makes things easier. Create your own perfectly sculpted object-mapper in OpenApiHandler and forget about JavalinJackson.

I created a PR in which I just used the interface of the ModelConverter from OpenApi and let the user override that. I can do the same with the default mapper of the handler (Which I will be separating from the default JavalinJson Handler because of the issues above). In this case the user can at least override the jackson implementation, if he wants to build his own. I think this would be comparable to JavalinJson.

I will fix the other issues as well this evening.

[tipsy]

I created a PR in which I just used the interface of the ModelConverter from OpenApi and let the user override that. I can do the same with the default mapper of the handler (Which I will be separating from the default JavalinJson Handler because of the issues above). In this case the user can at least override the jackson implementation, if he wants to build his own. I think this would be comparable to JavalinJson.

Is that required? If the library already has a hard dependency on Jackson we might as well use that? It's going to be pulled in as a transitive dependency anway.

[TobiasWalle]

Is that required? If the library already has a hard dependency on Jackson we might as well use that? It's going to be pulled in as a transitive dependency anway.

But I think with the PR the user has at least the possibility to use another library (gson, for example) even if jackson will be installed. In my understanding, even if it is a hard dependency of open api, if your replace the implementation jackson won't be used.
The exposed configuration is also pretty convenient if you want to configure the mapper used for the openapi schema (for example to enable pretty formatting in the tests).

I have some problems to fix the annotations with static methods. It seems like I cannot get the annotation from the lambda. The function that is responsible for this is task HandlerMetaInfo.getOpenApiAnnotationFromJava. For kotlin I found a (very hacky) solution, which involves accessing private fields, to get to the original function, but in java it doesn't seems to be possible. At least I didn't found something with the debugger. Maybe you can give it a try?

[tipsy]

@TobiasWalle so for getting the name of method references we used to have this snippet for the route-overview:

private val Any.methodName: String? // broken in jdk9+ since ConstantPool has been removed
    get() {
        val constantPool = Class::class.java.getDeclaredMethod("getConstantPool").apply { isAccessible = true }.invoke(javaClass) as ConstantPool
        for (i in constantPool.size downTo 0) {
            try {
                val name = constantPool.getMemberRefInfoAt(i)[1]
                if (name.contains("(\\$|<init>|checkParameterIsNotNull)".toRegex())) {
                    continue
                } else {
                    return name
                }
            } catch (ignored: Exception) {
            }
        }
        return null
    }

But it'is broken in Java 9+. I haven't found a solution to this yet.

[tipsy]

We had an issue for it: https://github.com/tipsy/javalin/issues/330

[TobiasWalle]

@tipsy Thanks for the tip. In the last hour I tried a lot of stuff, but nothing seems to work. I don't think I will find a solution for this problem.

I think we should document that it is currently not possible to use the annotation api with static java methods. The workarounds are either to define a handler as a class or to use the DSL. Maybe we will find another solution in the future.

[tipsy]

I don't think it's related to the methods being static, just that they're method references. By the way, shouldn't the implementation also work with fields?

@OpenApi(...)
Handler myHandler = ctx -> ctx.result()

[tipsy]

@TobiasWalle there is a method in RouteOverviewUtil which gets the method/field/class names for handlers which might be useful: https://github.com/tipsy/javalin/blob/master/src/main/java/io/javalin/core/util/RouteOverviewUtil.kt#L172-L193

[tipsy]

This is also why I pushed for a pure javadoc/annotation based implementation. It seems way easier to simply specify the verb and path as part of the annotation, then just scan all the classes for annotations. It would be awesome if we could make this work though.

[TobiasWalle]

@tipsy I think with the RouteOverviewUtil I can fix a lot of cases. My recommendation for solving the lambda problem would be creating optional class scanning. The resolving order would be the following:

1. is handler a DocumentedHandler
2. Has handler OpenApi annotation and can it be parsed by reflection
3. Is class scanning activated and does a matching OpenApi annotation with path and method exists -> use the annotation from the path scanning for the documentation

I wouldn't enable the path scanning by default, because it could increase the startup time substantially.

[tipsy]

Sounds like a good plan!

[tipsy]

@TobiasWalle could you give me a rough estimate on when you think this will be ready?

[TobiasWalle]

@tipsy My branch is almost working. I just need to clean up a little bit. I will finish it and create the PR tomorrow evening.

With this implementation it works in all cases. Just for method references in java you need to define which packages should be scanned and set the path in the annotation manually.

[tipsy]

Great, happy to hear that! Would it perhaps work to just scan the parent class of a method reference (in the handler added event)?

[TobiasWalle]

That's definitely worth trying :) Good idea

[tipsy]

Also, you could use this to connect the right handler to the right annotation: https://stackoverflow.com/questions/24095875/is-there-a-way-to-compare-lambdas

Edit: This is really more of a could than a should.

Edit2: I'm starting to remember why I gave up on this...

[TobiasWalle]

@tipsy I got some great news! I found a way to get the annotation of a non static method reference. I could even use this method to fix an ignored test in the RouteOverview. The only case in which path scanning is required is on static method references.

Unfortunately I can only get the package path of the lambda, but not the original method, automatically. So it is still required to specify the path.

[TobiasWalle]

Here is the PR https://github.com/tipsy/javalin/pull/586

[TobiasWalle]

Also, you could use this to connect the right handler to the right annotation: https://stackoverflow.com/questions/24095875/is-there-a-way-to-compare-lambdas

Edit: This is really more of a _could_ than a _should_.

Edit2: I'm starting to remember why I gave up on this...

Unfortunately lambda comparison doesn't seems to help me, because I cannot get another instance of a lambda by reflection to compare with. At least I couldn't get it to work.

[tipsy]

@TobiasWalle That is great news! Thank you for all the effort your putting into this, it's highly appreciated.

[tipsy]

@TobiasWalle I'm looking at things locally now, and I'm wondering if .activateAnnotationScanningFor("...") could be automatically activated when it's needed?

To clarify: If you fail to get the annotation for a handler, fall back to scanning.

[tipsy]

@TobiasWalle I tried to implement this myself, but when I remove options.packagePrefixesToScan.isNotEmpty() other tests start failing.

Does this mean enabling the scanning interferes with the other approaches?

[TobiasWalle]

@tipsy

@TobiasWalle I'm looking at things locally now, and I'm wondering if
.activateAnnotationScanningFor("...") could be automatically activated when it's needed?

To clarify: If you fail to get the annotation for a handler, fall back to scanning.

That's definitely possible, but I do not know if we want this. The reason I wanted the user to explicitly specify the package path is for performance reasons. If you configure an empty string it scans every class in the app (including the classes of packages), which may take a few seconds, which I wanted to avoid.

Does this mean enabling the scanning interferes with the other approaches?

No, in these tests I didn't specify a documentation for the handlers. If you enable the path scanning by default, the handler from the other test will be found and used, as the http path and method are the same.
I added the check to avoid an error if the user does not use the scanning and hasn't the optional dependency installed.

[tipsy]

That's definitely possible, but I do not know if we want this. The reason I wanted the user to explicitly specify the package path is for performance reasons. If you configure an empty string it scans every class in the app (including the classes of packages), which may take a few seconds, which I wanted to avoid.

But it's still a one time cost? Since it only happens for a very specific use case, I think it's fine. It makes the API and implementation simpler, and only affects a small number of people.

No, in these tests I didn't specify a documentation for the handlers. If you enable the path scanning by default, the handler from the other test will be found and used, as the http path and method are the same.
I added the check to avoid an error if the user does not use the scanning and hasn't the optional dependency installed.

I see. That's a bit tricky then 🤔

Realistically it won't be a problem in an actual project, but it still feels a bit clunky.

[TobiasWalle]

But it's still a one time cost? Since it only happens for a very specific use case, I think it's fine. It makes the API and implementation simpler, and only affects a small number of people.

The main problem I see is not that it is automatically activated. It is that the package path wouldn't be limited. (Potentially) there is a scenario in which the user uses a library that also uses javalin. The library uses handlers with the annotation. If we scan the whole package path, these annotations would be found and if the user defines the same handler, the documentation will be used. Maybe this case it not very likely, but if it is happening it would be very confusing. And I think many people use Javalin to avoid such intransparent errors. On the other hand, a user might want to use the documentation of a library. That's why I think it is not avoidable to define the package.

But I agree, that the current approach is also not perfect. Maybe you find another solution that works better. If you could find out the package of static method, we could automate this very easily. But I'm already too frustrated with lambda reflection to continue with my search 😄

[tipsy]

But I agree, that the current approach is also not perfect. Maybe you find another solution that works better. If you could find out the package of static method, we could automate this very easily. But I'm already too frustrated with lambda reflection to continue with my search 😄

That's very understandable, let's keep your current approach. Maybe someone who's affected by this exact edge case will fix it later. I'm going to close this issue now (it's been open for almost a year!). Thanks again for all your hard work.

[tipsy]

@ShikaSD, @kyleellman, @mikexliu, @majorpasza, @ernestas2k, @peterbencze, @grote, @sureshg, @cricket007 I've tagged you all since you have expressed interested in Javalin/Swagger, or because you have contributed significant amounts of code to Javalin recently. I just wanted to let you know that Javalin 3.0.0.RC0 is out now, with OpenAPI/Swagger support:

<dependency>
  <groupId>io.javalin</groupId>
  <artifactId>javalin</artifactId>
  <version>3.0.0.RC0</version>
</dependency>

There are no docs yet, but you can see how it works from the tests in src/test/java/io/javalin/openapi.
You can enable it both via DSL and Annotations, and it supports both Swagger and ReDoc for rendering the docs. Please use this issue if you have any feedback on the implementation.