Immutables: Use a standard template engine e.g. moustache?

[Gaff]

Ok I know this is a crazy left-field idea, but I noticed that @elucash said this in another thread:

Unfortunately template syntax and apis are undocumented...

I can relate to this since I've wanted to add to the immutables project before but the arcane template engine put me off.

Would it be possible to replace the templating engine with a standard solution? I'm talking about things such as moustache. I get that this would be a significant amount of work, but the advantages are:

• Much less code to write and maintain.
• Makes it easier for people to contribute modules to the project.

I accept it's probably not worth the effort - but I'd just like to put the idea out there!

[elucash]

Thank you for raising this! It's ok I, guess, to ask for it or at least mention ). I'm not totally fond of current solution, but it has certain benefits which might not be apparent on at a glance: This includes superb whitespace handling, no-reflection access to models with compile-time accessor reference checks, low-friction partials, lexical scoping, extreme good readability for templates full of logic. Of course, you can only enjoy any benefits if familiarize with the syntax, and get to work around insufficient/unclear error reporting (this parts if really arcane, to be sure). The first templates were written in MOFM2T against Eclipse-based modelling toolkits and migrated since to current format. In my view moustache/handlebars/nameit are nowhere near to be called "standard", but they are definitely are thousands times more used/widespread than anything like our taylor made templates. And if "logic-less" templates would be something we would need, moustache would be, probably, a first choice. But the way how Immutables evolved there was a clear need for logic-heavy templates, heavy on partials and, as it needs to generate code, it should also not look like java inside markup (heavy memories of JSP scriptlets generating javascript... oh-my-gosh). The irony is that with templates becoming more and more complex, now it even questionable if templates are needed at all, and maybe model-driven/dsl-building generation might be a better choice (like JavaPoet or something alike). And to this end, we are developing an extension mechanism that is based on plain-java code contributions, not an annotation processor extensions (see #363 and related tickets).
Currently I (an other guys especially) just not ready to invest in either improvements or replacement of templates rendering. But I never actually wanted anybody to "learn" that custom templates (however, number of people are contributing small to medium sized fixes and improvements to Immutable templates without much trouble) in that sense that it should be easy to not use them at all (getFiler() and so on), or, once inside, to escape to any other rendering system.

[output.java 'my.package' 'MyClass']
[-- Any accessor method that return like CharSequence
     or String (or any object, whose toString will be called) will be printed
     in generated my/package/MyClass.java  --]
[myObject.templateText]
[/output.java]

I would gladly provide extension points to easily dive into generation once it will be clear how they could look like/where to insert. Let me know if specific extension points would be useful, like getting rendering control something inside immutable implementation class or creating a side-class etc.

[Gaff]

Ok your experience with templates clearly dwarfs mine! My instinct was that templates that look more java-like and have less logic in them would be a Good Thing. This is based on my limited experience, but for me having java-like templates was far more comprehendable than a JavaPoet-like solution that the stock annotation processor examples seem to go for.

Still my main aim here is to make it easy to contribute ideas to the project. I'm all for any suggestion that allows easy pulgins [to wit: Do we even have to use the same template engine? If one person wants JavaPoet and another wants Handlebars is it /so/ bad?]. Perhaps a few pages of contributors guidance might be a great place to start?

Things that stumped me (notwithstanding that I'm stupid and only put in 30 minutes effort) are:

• How can I access the immutables model structure, whatever that is - for example you must prepare a list of getter methods and return types somehwere, where is it?
• My feature is only going to be useful for some people. If I want to add a companion class for each immutable type in a modular way, how should I do this?

[elucash]

My instinct was that templates that look more java-like and have less logic in them would be a Good Thing.

What I meant about java-like syntax being bad on an eye is when java-like syntax (or javascript) is used inside template directive markup to generate code in java (or alike syntax), like in the following example (inside <% %>)

<% if (request.getParameter("out")) { %>
if (getTransformedIn()) { 
   return '<%= request.getParameter("in").toUpper(); %>'; 
} 

Curiously, looking at the examples you're provided, there's a place where innocent looking template variable is actually backed by conditional logic and string formatting statements. So our decision with Immutables was that we better have formatting/presentational logic in template, rather than swept under the rug of code, that was our trade-off. In reality, of course, we have all sorts of mixtures and proliferations, so there's nothing like that we are nice and clean on either side of models/templates ;)

Still my main aim here is to make it easy to contribute ideas to the project. I'm all for any suggestion that allows easy pulgins [to wit: Do we even have to use the same template engine? If one person wants JavaPoet and another wants Handlebars is it /so/ bad?]

Exactly, that was my point that it should be possible escape arcane lands and proceed with the tool at hand.

• How can I access the immutables model structure, whatever that is - for example you must prepare a list of getter methods and return types somehwere, where is it?
• My feature is only going to be useful for some people. If I want to add a companion class for each immutable type in a modular way, how should I do this?

There are both good and bad news.

How model is integrated (and how templates are invoked) is quite easy to point to:
https://github.com/immutables/immutables/blob/master/value-processor/src/org/immutables/value/processor/Processor.java

values multimap containing ValueType object is the essence of a model. ValueType contains all sorts of accessors and fields, including ValueAttribute models for attributes. We invoke each template and let it iterate the values to figure out .
The actual discovery mechanism is multistep and quite blurry in that sense that it spread across DeclaringType /DeclaringPackage/ Protoclass / Constitution / ValueType / ValueAttribute / AccessorAttributeCollector and much of the other misc stuff which accumulated over time. Templates access mostly ValueType and ValueAttribute objects.

The bad news about that is that resulting model is quite unwieldy and hard to use properly. It is very complicated by the all sorts of styles/customizations and many attribute types that Immutables has built-in support. Let's say you want to create a value object instance in your template from attribute values, how would you do that? Well, you'll need to check what creation path to use if builder is on or off, if it's on (for example), you need to get correct invokation factory object which contains correct path to builder (which is customizable PersonBuilder vs Person.Builder and so on), then you iterates attributes, but you need select special set of settable attributes as not all of them are settable, then depending on a type of attribute you would invoke different builder initializer method, and method names you would use are different depending of a style and attribute type...

Here's how we would create it from Json fields:

@com.fasterxml.jackson.annotation.JsonCreator
static[type.generics.def] [type.typeImmutable.relative] fromJson(Json[type.generics.args] json) {
[let cast][if type.constitution.returnsAbstractValueType]([type.typeImmutable.relative]) [/if][/let]
[if type.useBuilder]
  [type.typeBuilder.relative] builder = [type.factoryBuilder.relative]();
  [for v in type.settableAttributes]
  [if v.requiresTrackIsSet or v.primitive]
  if (json.[disambiguateField type (v.name 'IsSet')]) {
    builder.[v.names.init](json.[v.name]);
  }
  [else]
  if (json.[v.name] != null) {
    [if v.encoding]
    builder.[rr.builderCopyFrom v](json.[v.name]);
    [else if v.collectionType]
    builder.[v.names.addAll](json.[v.name]);
    [else if v.mapType]
    builder.[v.names.putAll](json.[v.name]);
    [else]
    builder.[v.names.init](json.[v.name]);
    [/if]
  }
  [/if]
  [/for]
  return [cast]builder.[type.names.build]();
[else if type.useSingletonOnly]
  return [cast][type.factoryInstance.relative]();
[else]
    [if type.useConstructor]
  [type.typeImmutable.relative] instance = [cast][type.factoryOf.relative]([for v in type.constructorArguments][if not for.first], [/if]json.[v.name][/for]);
    [else if type.useSingleton]
  [type.typeImmutable.relative] instance = [cast][type.factoryInstance.relative]();
    [else]
  [output.error]Cannot generate JSON code when there are no builders, constructors or singletons available[/output.error]
    [/if]
  [for o in type.withSettableAfterConstruction]
  if (json.[o.name] != null) {
    instance = instance.[o.names.with](json.[o.name]);
  }
  [/for]
  return instance;
[/if]
}
[/if]
[/template]

Any third party template that would not follow all the peculiarities would likely not work with other people's styles or for other use cases. Which is not far from just fork Immutables and experiment with/customize whatever we want.

I am open to open up stuff and provide extension points for templates and models, but I'm just afraid we would not be able to adequately document and support it.

I've tried to create quick-and-easy extension mechanism for templates, but in the process there was some many questions, problems and technical obstacles, so that forced me to think harder what we can actually reasonably achieve.

That is the main motivation why we developing "encoding" extensibility #363 via plain java code with annotations rather than templates.

For the quite generic use case of translating immutable objects to/from map-like records or result-sets, I think we can get away with either existing approaches based on Jackson (reflective) / Gson (generated) conversion or can develop another generic generator which would do stuff similar to #370 .

Having been though about it for a while now I have a tendency to believe that using JSON conversion approach is not a bad idea at all. First of all, I hope we would agree that we do need some sort of converter/registry for strings, primitives (and collections of thereof) and that it doesn't make sense to develop our own. So integration with Jackson/Gson is more than likely anyway. The second, JSON libraries are quite optimized and having Immutables generated code for it further it can make runtime performance overhead negligible (and maybe even not measurable) compared to if we generate bare bone mapping logic. This, of course, would not be true for Android, but for Java VM such overhead may go unnoticed. If we take JDBI as example, It would be actually very interesting to compare performance of the generated mapper vs Jackson- or Gson-based one. (I guess this topic can be continued in #97)

Getting back to templates extensibility and reuse of models, my thoughts are:

• It is possible
• The quality of APIs is nowhere near acceptable (to be opened) shape (by me, at least)
• Too time consuming to document and maintain/support
• Common cases may be better served by encodings / generic generators

But I would encourage just to try fork immutables, create additional template and experiment with it (model/template), here's the recipe :

My.java

// Generator_My will be generated using this class and My.generator file
// Generator_My will extend My
@Generator.Template
abstract class My extends ValuesTemplate {
   List<CustomRenderer> getRenderers() {
       List<CustomRenderer> renderers = new ArrayList<>();       
       for (ValueType type : values().values()) {
           renderers.add(new CustomRendered(type));
       }
       return renderers;
   }
   class CustomRenderer {
     final ValueType type;
     CustomRenderer(ValueType type) {
        this.type = type;
     }
     // could be just String toString, but let's say we have CharSequence render()
     CharSequence render() {
         StringBuilder sb = new StringBuilder();
         // load whatever template engine might be useful
         sb.append("package ").append(getPackageName()).append(";");
         sb.append("class ").append(getClassName()).append(" {}");
         return sb;
     }
     String getPackageName() {
         return type.$$package();
     }
     String getClassName() {
         return "My" + type.name();
     }
   }
}

My.generator

[template public generate] [-- defined by ValuesTemplate, no parameters]
  [for r in renderers]   [-- iterate using our getRenderers accessor ]
[output.java r.packageName r.className] [-- output.java directive create java files]
[r.render] [-- just output it here, inside java file ]
[/output.java]
  [/for]
[/template]

add line to org.immutables.value.processor.Processor#process()

  invoke(new Generator_My().usingValues(values).generate());

(it should more or less work, but some minor things might need to be tweaked)

[talios]

Has there been any thought of using javapoet at all? Given that's code based, extension points for processors could just take/return the JavaFile instance etc.

Javapoet is used by Dagger amongst others.

In any case, this would not be a simple change, and I suspect something to be released with care ( almost worthy of a major or maybe just minor version bump maybe if there was concern for any generated peculiarities ).

[elucash]

@talios We thinking hard about all the alternatives. Currently, I don't see a way (nor particular desire) to change templating engine for Immutables, because the amount of things to rewrite is huge and so many mistakes, changes. For some new functionality or spin-off projects we considering using JavaPoet because of large amount of logic/model-driven construction, so that conventional templating will lose any sense: wysiwyg spirit will be destroyed anyway. I don't see templating engine as an obstacle to extending Immutables, in that sense that it should be easy to escape immutable templates in favor of any other approach (as demonstrated above). The biggest problem I see is overall complexity, fuzzyness and lots of intricated details of the model we have. Also we see tremendous value in java-code based templating we are developing with "encodings" #363 . There's an idea to create AutoValue extension to incorporate "encodings" so they could be used with both toolkits (and it is also example where we can use JavaPoet or alike).

[vorburger]

Javapoet looks neat to gen. Java code.. FYI I personally REALLY like Xtend's template expressions as general purpose template engine, incl. code focused (gen. code, no reflection), great whitespace/indentation handling (if you you're not into Xtend for whatever other reason, its template support alone is worth using it, even if just for your templates, IMHO). Have also heard others liking Twirl.