Openwhisk: CLI spends a long time scanning JAR file to find main for Java actions

[rabbah]

If one creates a Java action from a JAR containing a large number of classes, the time it takes the CLI to "create" the action starts to be come unacceptably long. The CLI runs javap for each class to find main and each of these can take a second or so. So it adds up fast. The CLI could give up once it find 1 match (rather than look for all), or allow/require the user to specify their package.class.method for "main".

Related to #47.
FYI @psuter @mdeuser @dubeejw.

[mbehrendt]

could we put a stake like this into the ground?
given #47 , we don't enforce a particular method name. instead, the user needs to specify the java class, node handler, etc. that should be executed for invoking an action.

[psuter]

Some things we could do:

• for main detection:
  
  
  
  • give up if more than N classes in jar
  
  
  • try with classes named *Main* first
  
  

These will preserve some of the convenience but still fall short in the general case.

The only foolproof solution is to stop detection and force the use of a --mainClass flag for Java actions.

[mbehrendt]

iirc, we had decided to move away from main -- is that correct?

[psuter]

To clarify: for Java, we leave the name of the main _class_ flexible but mandate the name of the main _method_ (and its signature, and that it should be static). I don't think we intent to change that. It's after all the same principle as vanilla Java applications.

The REST call to create a Java action requires specifying the main class. The CLI, for convenience, figures out the main class by analyzing the jar file. This fails (= takes too long) when the jar file is large, and this is the topic of this issue: there is currently no way to bypass the "convenience", hence the need for a CLI flag (whether we keep the automated detection feature is an orthogonal question).

The JavaScript restriction on main is discussed in #47 and is only tangentially related in that the CLI may, e.g., use the same flag to specify the entry point.

[mbehrendt]

let me make sure i understand the proposal correctly: is your suggestion that the user should disable to automatic scanning if it takes too long?

[mdeuser]

The current method signature being search for in the CLI is public static com.google.gson.JsonObject main(com.google.gson.JsonObject). I agree with #47 that this is a bit restrictive.

As for the time it takes for the CLI to validate that the supplied .jar has a method which conforms to the expected signature, we could have a --validate or --novalidate option on the CLI that can be used to control whether or not the .jar is validated.

@starpit - Just wondering, does the GUI validate the action .jar for the expected invocation signature?
/cc @johnsolm

[rabbah]

The GUI doesn't yet support jar-file upload.

[rabbah]

The plan, more concretely (still subject to refinement):

1. add --main to wsk action create|update to allow client to name main().
2. in the absence of a --main command line argument, the client (or API) may assume a default (e.g., main or search for main in the JAR for java).
3. the client may fail to find a main (for example if there are too many classes in a JAR, and fail as suggested in https://github.com/openwhisk/openwhisk/issues/1018#issuecomment-238030048).

I view --validate as an orthogonal feature.

[rabbah]

I agree with #47 that this is a bit restrictive.

The contexts are different for Node vs Java - it's not common in Node to use main() vs module.export. So The spirit of #47 is to make the common case more in line with developer experience for Node.

[mdeuser]

Here's a scenario I was thinking about. The customer has several actions (say, A, B and C) plus all of the common utility code in a single jar. This same jar is used to create actions A, B and C - each having a separate invocation method.

Does the --main take an argument? If so, a --main CLASS.METHOD argument would handle the scenario I described.

Regarding

3. the client may fail to find a main (for example if there are too many classes in a JAR).

Is the CLI still expected to validate that the specified/default invocation method exists in the jar?

[psuter]

@mdeuser yes, the idea is for --main to take a class name. This is familiar to Java developers who run, e.g.:

java -cp ... com.example.MyMain $arg1 $arg2

Regarding the validation, that's orthogonal. The CLI currently runs no check for any of the runtimes. E.g. it doesn't even check that the code parses, let alone that main exists, so I would say we could do it, but as a separate issue.

[glikson]

A possibly user-friendly experience could be to do something similar to the way jar command works, allowing to specify the name of the entrypoint class that has the main() method (jar cfe my-jar-file.jar <my.entrypoint.class.that.has.Java.main.method> <class_files_to_include_in_jar>). So, similarly to what @rabbah suggested, the syntax could be: wsk action create/update <action-name> <my-action.jar> --entrypoint <my.entrypoint.class.that.has.Whisk.main.method>

[mdeuser]

@psuter - Thanks for the clarification. The only "validation" I was referring to is the CLI's examination of the jar file to ensure the expected endpoint (name and signature) exists. In Rodric's proposal, it sounds like the CLI will still attempt this sanity check whether or not the endpoint is explicitly stated via the --main argument. Right? Or, when the endpoint is not specified, will the CLI skip this check and send the jar as-is to the server?

[mbehrendt]

1. in the absence of a --main command line argument, the client (or API) may assume a default (e.g., main or search for main in the JAR for java).

@rabbah if we're making --main optional, users might run into a bad ux, having to wait 30 mins with no clue what the root cause might be. i think we need to protect users from such a bad ux, eg by making --main mandatory.

thoughts?

[glikson]

@mbehrendt, @rabbah another option could be to set a timeout (e.g., 1 minute), to do our best finding the entrypoint (if was not specified), to notify the user that entrypoint has not been found, and suggest setting it explicitly with --entrypoint (or whatever).

[mbehrendt]

hm -- what if more than one class gets found that matches the criteria? we'd have ask the user for input anyway...

[glikson]

@mbehrendt alternatively, we could choose the first one and return a warning (and, of course, document the behavior). If we expect majority of Java actions to have, say, up to 50 classes (including dependencies) - making the entry point parameter optional could be convenient.

By the way, one potentially nice heuristic could be to sort classes by date before applying 'javap' on each -- this would likely minimize the chance of not finding the entrypoint, because dependencies would typically be older than the 'main' project classes.

[rabbah]

eg by making --main mandatory.

Then it's a required (positional) argument and not optional as implied by dash dash. The point of making it optional is for the cli to try and infer it - but as suggested in several comments above the cli doesn't have to try too hard and give up quickly if the cli can't figure it out and ask the user for help. This makes it possible to have a suggested common idiom which makes the need to specify additional parameters unnecessary.

[mbehrendt]

@rabbah what is the suggested exact behavior of this automated detection mechanism, such that we deliver an great user experience. eg i think the timeout shouldn't be longer than 5 seconds. how is the cli going to react if more than one matching classes is found within these 5 seconds?

also, we should be open to treating -main as mandatory parameter, if there is too much that has to be considered / can go wrong with the cli 'magic'

[glikson]

One more option for a heuristic search: check the Main-Class property in the manifest, and if exists - check whether the same class contains also Whisk's main method signature.

Having 'regular' main method in the class is convenient for local execution and testing, so I would assume that it might be a reasonably common practice to have both in the same class.

[csantanapr]

I think explicit provides a better UX. You always know how is going to work.
And have two ways to specify the entry point via manifest file like comment https://github.com/openwhisk/openwhisk/issues/1018#issuecomment-238516867 or via command line argument.

[mbehrendt]

I think explicit provides a better UX. You always know how is going to work.

totally agree. shouldn't try to be fancy on automatic class detection -- it will be totally fine for devs to specify their action class.

[rabbah]

This issue solves #1026 - shall we repurpose it and close the other as duplicate?

[csantanapr]

This issue solves #1026 - shall we repurpose it and close the other as duplicate?
Yes

[glikson]

Well, to be precise, it could be the case that the same solution would address both.. it doesn't mean that the issues are the same. The commit message could mention that it closes both.

[psuter]

Fixed by #1544.