Jsdoc: Support ECMAScript 2015 syntax

[kilianc]

Generators currently cause the following error:

Error: Line 69: Unexpected token *

[hegemonic]

As far as I know, we don't currently support _any_ ECMAScript 6 syntax on Node.js, including generators. (On Rhino, we only support a handful of keywords that Mozilla added in JavaScript 1.8.x.) I've expanded the scope of this issue accordingly.

In general, we can only support ECMAScript 6 to the extent that our JavaScript parser supports it. That means we may never support ECMAScript 6 if you need to use Rhino's built-in JavaScript parser. (You can use Esprima on Rhino, but it's slow.)

Here's a task list, which I'll update over time. It doesn't include every single thing in the ECMAScript 6 spec; some features don't affect us, and Esprima gives us a lot of the new stuff for free. I used https://github.com/lukehoban/es6features and Mozilla's table as a cheatsheet:

• [x] Officially support Node.js 0.11.x.
• [x] Start using the harmony branch of esprima.
• [x] Walk ES6 nodes and their children.
• [x] Switch to the Espree parser.
• [x] Add new doclet properties to capture new ES6 features. (#623)
• [x] Add visitors for new node types. (#624)
• [x] Support class declarations.
• [x] Support class expressions.
• [x] Support modules.
• [x] Support arrow functions.
• [x] Support block scope (for example, when using let and for...of).
• [x] Support rest parameters (for example, function (foo, bar, ...baz)).
• [x] Support default parameters (for example, function (foo, bar = 1)).
• [x] Support binary and octal integer literals.
• [x] Automatically detect constants.
• [ ] Implement new tags for promises. (#509)
• [x] Figure out how to document the value that a generator yields. Maybe a @yield/@yields tag?
• [ ] Infer when something is iterable.
• [x] Figure out what, if anything, to do with template-related nodes.

[briandipalma]

You don't need node 0.11 to write ES6 you can do it right now using traceur https://github.com/google/traceur-compiler

This means people using traceur currently cannot document their code...

[briandipalma]

I think simply using the harmony branch of Esprima might be enough to stop the parsing errors.

[hegemonic]

I don't think I ever said that you can only write ES6 code with Node.js 0.11.

Using the harmony branch of Esprima should prevent Esprima errors. However, the new version of Esprima will return node types that JSDoc does not recognize, which will cause JSDoc errors. Those errors will also need to be ironed out.

Point taken, though, about the usefulness of resolving the parse errors even before we add real ES6 support. For that reason, I'll make it a priority to integrate the harmony branch of Esprima.

[snowmantw]

I know this is purely hack, but I think to let arrow function syntax works is not so complicated.
I've tested it to use the FunctionExpression as ArrowFunctionExpression with the Harmony branch Esprima to let the document be generated as usual, and don't see any problems you mentioned about the unrecognized AST nodes.

This is at least better than throw errors and get nothing from JSDoc.

But I couldn't find supports of spread operator in that Esprima, so some other syntaxes may still cause errors.

[hegemonic]

@snowmantw: Your pull request in #620 proves my point by creating errors in a couple of tests.

EDIT: My apologies! The test errors in #620 were caused by an Esprima bug.

[hegemonic]

On master, JSDoc should now be able to parse ES6 files without throwing an error. For now, when we encounter an ES6 node, we discard it and log a warning. (Hey, it beats crashing!)

[snowmantw]

Great! Now we can have nice doc & syntax!

[flying-sheep]

hi! you might want to add “getters and setters” to the list.

i don’t see them in the list or the test, but they exist (in the very first file i wanted to run JSDoc on ;)):

jsdoc there complains:

js: "…/mozilla-central/browser/components/downloads/content/indicator.js", line 404: Property "hasDownloads" already defined in this object literal.
js:   },
js: ..^

[hegemonic]

@flying-sheep: Getters and setters are an ES5 feature, and JSDoc has supported them for quite a while. I just tested on master, and I'm not seeing the problem you mentioned. If you can reproduce the problem with a current version of JSDoc, please file a new issue.

[flying-sheep]

Hmm, i have JSDoc 3.2.2. I'll update to 3.3

[flying-sheep]

3.3 wasn’t sufficient, but master did the job. sorry.

i don’t get the versioning, though: 3.2.2 is the current stable, 3.3.0alpha5 the current unstable, and git master is _far_ ahead of both?

[hegemonic]

@flying-sheep: git master is the development branch for 3.3.0. I periodically release alpha versions from that branch.

[flying-sheep]

ah! and when is the next one due? i think initial ECMA 6 support is very useful

[raphaelokon]

I second @flying-sheep. ES6 support would be awesome.

[jhorwit2]

+1. ES6 support would be amazing.

[danwoods]

+1

[arielschiavoni]

+1

[gitawego]

+1

[hegemonic]

I'll gladly accept pull requests from anyone who wants to help implement ES 2015 (formerly ES6) support in JSDoc. There's already an open pull request to add support for ES 2015 classes.

Note that I just switched JSDoc to use the Espree parser, a fork of Esprima, so we're limited to the features that Espree supports. However, Espree is moving very rapidly towards full support for ES 2015.

[sashasochka]

So if I understand right, most of the ES6 features are supported now?

[hegemonic]

@sochka The opposite. For most ES 2015 nodes, we log a warning and ignore the node.

[sashasochka]

How do I know what's supported? I'm interested in classes and lambdas in particular.

[hegemonic]

Classes and lambdas are not currently supported.

When I get a chance, I'll clean up the to-do list in this bug so it's more obvious what's done and what isn't done.

[vjsingh]

Probably not super helpful, but +1 for class support in general.

It seems to me that for any projects which use classes (like ours), the lack of support makes jsdoc largely useless? Correct me if I'm wrong.

Obviously, this message should be prefaced with the fact that I appreciate the creator for their work and regret that I don't have time to contribute myself.

[hegemonic]

For some additional context, I should point out that the major parser authors are still haggling over how to represent a variety of ES 2015 nodes in the AST: https://github.com/estree/estree/issues

It's tough to keep up with an AST syntax that's still changing from week to week, so thanks for your patience while I wait for things to settle down a bit.

[lukeapage]

On master, JSDoc should now be able to parse ES6 files without throwing an error.

I'm having trouble with es-6 modules. At first they weren't supported by the version of espress you use (they recently added support). Next I upgraded espress and tried again. Instead of "unexpected reserved word" I now get "Illegal import declaration" and "Illegal export declaration"

It would be really nice to be able to parse and get something, even if I don't get es6 modules integrated into jsdoc yet

[hegemonic]

@lukeapage Yes, I understand that. I'm working to update Espree.

EDIT: Espree has been updated. If you're still seeing parser errors, make sure you're using the current ES 2015 class/module syntax (it changed very late in the spec process).

[lukeapage]

Anything i can do (without having to understand all the code and a limited
amount of time) to help?

[hegemonic]

@lukeapage Nope, at this point it's all about writing code.

[hegemonic]

Classes are now supported! Code like the following now works on master (Node.js only):

/** Point class. */
class Point {
    /**
     * @param {number} x - The x value.
     * @param {number} y - The y value.
     */
    constructor(x, y) {
        // ...
    }
}

Next up: Modules.

[raphaelokon]

Great work. It there an overview which ES 2015 features work as of right now?

[hegemonic]

@interactionist See the checklist at the top of this issue.

[raphaelokon]

@hegemonic How could I miss that! Thanks.

[theasta]

@hegemonic I'd love to help. I saw the check list at the top but I'm guessing you are currently implementing some of the missing items. Which one can I pick?

[hegemonic]

@theasta You could tackle the new promise-related tags in #509 (see this comment for the final syntax) or the new @yield/@yields tag for generators. Both of those will require adding new doclet properties, which means they'll need template changes as well.

[raphaelokon]

@hegemonic I would like to tackle the block scope (let, const, for …) but I don't know where to start from.

[hegemonic]

@interactionist I'm not sure about that one either. There may not be anything to do, but I haven't investigated yet.

[djMax]

I'm having trouble getting this to work on master. I cloned the repo, ran gulp, npm link'ed on both sides, and I get:

WARNING: JSDoc does not currently handle ClassExpression nodes. Source file: /Users/mmetral/dev/js/invoice.js, line 63

/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/astnode.js:112
switch (node.type) {
^
TypeError: Cannot read property 'type' of null
at exports.nodeToValue (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/astnode.js:112:17)
at exports.nodeToValue (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/astnode.js:160:19)
at Object.exports.getInfo (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/astnode.js:317:25)
at Visitor.makeSymbolFoundEvent (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/visitor.js:442:33)
at Visitor.visitNode (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/visitor.js:392:18)
at Visitor.visit (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/visitor.js:314:27)
at Walker.recurse (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/walker.js:546:44)
at Parser._walkAst (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/parser.js:245:18)
at Parser._parseSourceCode (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/parser.js:236:18)
at Parser.parse (/Users/mmetral/dev/node/jsdoc/lib/jsdoc/src/parser.js:157:18)
at Object.module.exports.cli.parseFiles (/Users/mmetral/dev/node/jsdoc/cli.js:359:42)
at module.exports.cli.main (/Users/mmetral/dev/node/jsdoc/cli.js:229:14)
at Object.module.exports.cli.runCommand (/Users/mmetral/dev/node/jsdoc/cli.js:183:5)
at /Users/mmetral/dev/node/jsdoc/jsdoc.js:134:13
at Object. (/Users/mmetral/dev/node/jsdoc/jsdoc.js:136:3)
at Module._compile (module.js:456:26)

[hegemonic]

@djMax Ah, I completely forgot that we also need to handle class expression nodes. Thanks for the heads up. Class declarations (as shown above) are indeed working. (EDIT: As of April 10, class expressions should work as well.)

[gso]

[Symbol.iterator] generators:

*[Symbol.iterator]() {
    ...
}

are cunrrently throwing the following error by the looks:

ERROR: Unable to parse /home/.../class.js: Cannot call method 'toString' of undefined

(Any quick workarounds would be appreciated as it is the only error preventing the documentation for a project from generating.)

[hegemonic]

@gso I don't have a workaround for this. Do you have a reference that demonstrates that [Symbol.iterator] generators are valid ES 2015 code? If so, I can file an upstream bug against the Espree parser.

[lukeapage]

Generators and async/await were dropped from es6/es 2015 and moved to
es7/es2016.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Generator_comprehensions

[briandipalma]

Generators are in ES15, comprehensions were dropped.

Here is an in depth article on them and you will see Symbol.iterator used in the examples.

http://www.2ality.com/2015/03/es6-generators.html

I'd say the problem is * or computed properties not Symbol.iterator though.

[gso]

Symbol, https://people.mozilla.org/~jorendorff/es6-draft.html#sec-symbol-objects.

Symbol.iterator falls under the heading of 'well-known symbols' ('built-in Symbol values') https://people.mozilla.org/~jorendorff/es6-draft.html#table-1.

The former link is sourced from the MDN ref. page for Symbol (specifications section) https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol#Specifications.

The above I guess assumes the MDN ES6 draft is current.

There is an example here https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#Is_a_generator_object_an_iterator_or_an_iterable.3F, however I have not seen an example of use with class syntax and/or generator functions.

[gso]

ClassDeclaration https://people.mozilla.org/~jorendorff/es6-draft.html#sec-class-definitions
ClassElement
MethodDefinition https://people.mozilla.org/~jorendorff/es6-draft.html#sec-method-definitions
GeneratorMethod https://people.mozilla.org/~jorendorff/es6-draft.html#sec-generator-function-definitions

• PropertyName
  PropertyName https://people.mozilla.org/~jorendorff/es6-draft.html#sec-object-initializer
  ComputedPropertyName
  [ AssignmentExpression ]

[sashasochka]

@briandipalma what is ES15? Is it a new name for ES6 or ES7? I can't google the name for some reason

[hegemonic]

@gso I'm familiar with the spec. I'm still not clear on whether *[Symbol.iterator]() {} is legal ES2015 code. Please spell out why you believe that this is legal. (Or you're welcome to file your own bug against the Espree parser.)

@sochka The spec is now called "ECMAScript 2015," not "ECMAScript 6."

[hegemonic]

@gso Never mind--I see that Axel Rauchmeyer's article uses that syntax, so I'm sure it's legit. I'll file the bug against Espree.

[hegemonic]

Modules are now supported! Default exports should work:

/** @module color/mixer */

export default {
    /** Blend two colors together. */
    blend: function(color1, color2) {}
};

As well as all sorts of named exports:

/** @module color/mixer */

/** The name of the module. */
export const name = 'mixer';

/** The most recent blended color. */
export var lastColor = null;

/** Blend two colors together. */
export function blend(color1, color2) {}

// convert color to array of RGB values (0-255)
function rgbify(color) {}

export {
    /**
     * Get the red, green, and blue values of a color.
     * @function
     */
    rgbify as toRgb
}

If you find bugs in JSDoc's support for ES 2015 modules, please file a new issue.

[lukeapage]

@hegemonic its all awesome, thanks for your work. Any chance we can get an alpha release? we're using a github url at the moment which stops npm from shrinkwrapping (because of a mismatch in underscore versions between dependencies further down the chain)

[tkellen]

:+1: thanks for this! About to update http://github.com/endpoints to es6 module syntax. I'll let ya know if anything goes wrong :)

[gso]

FYI if not already aware -- https://github.com/jsdoc3/jsdoc/issues/555#issuecomment-89656059, bug filed against Espree https://github.com/eslint/espree/issues/123 -- iterator expression syntax, closed 3 days ago.

[ndugger]

Are we saying that class expressions are supported? I just installed jsdoc via NPM, and I get errors that state that it's not supported;

JSDoc does not currently handle [...] nodes

• "..." ExportDeclaration
• "..." ClassDeclaration
• "..." ClassBody
• "..." MethodDefinition

[gso]

Ref. https://github.com/jsdoc3/jsdoc/issues/555#issuecomment-39749496, npm git+https://github.com/jsdoc3/jsdoc.git should do it.

[otakustay]

I'm trying to use jsdoc with babel based project, I think it's better for jsdoc to use babel's parser so some ES7 proposal's could be support and not result in parse error

[buzzfeedjoe]

FYI when I try to use shrinkwrap for the latest development version, it fails due to an error related to the version of underscore being used (1.6.0 is needed by underscore-contrib but jsdoc uses 1.7.0). Is there a way to get around this? Thanks for the help!

npm WARN unmet dependency node_modules/jsdoc/node_modules/catharsis/node_modules/underscore-contrib requires underscore@'1.6.0' but will load
npm WARN unmet dependency node_modules/jsdoc/node_modules/underscore,
npm WARN unmet dependency which is version 1.7.0
npm ERR! Error: Problems were encountered
npm ERR! Please correct and try again.
npm ERR! invalid: [email protected]

[lukeapage]

@buzzfeedjoe I had the same problem.

I think Its caused by the github dependency.

From memory, we had to do this (I don't have access to the project any more)

npm install
npm prune

then manually go into node_modules/jsdoc/node_modules/catharsis/node_modules/underscore-contrib, create a node_modules, copy underscore and change the version to 1.6.0, then run npm shrinkwrap again to get a shrinkwrap file, then delete node_modules and install again to get the clean, reproducible install.

It goes without saying it looks like a bug in npm - you might try updating to the latest.

[buzzfeedjoe]

@lukeapage your trick worked! Thanks for the tip :)

[ariporad]

Hey everyone,

Sorry if this is not allowed by the projects contributing rules, but I'm not really sure where JSDoc stands ES6 wise. Could someone summarize it for me?

Thanks!

[h13i32maru]

Hi! Everyone!

If you want ES6 documentation tool, please consider https://github.com/h13i32maru/esdoc that I have developing :smile:

[sashasochka]

@h13i32maru that's nice! Would be even cooler if there was a library like angular-jsdoc with special support for angular services, directives etc. =)

[tenntenn]

https://github.com/h13i32maru/esdoc

It sounds good! :+1:

[hegemonic]

@ariporad See the checklist near the top of this bug. Only the development version of JSDoc (not the version on npm) currently supports ECMAScript 2015.

@h13i32maru Please do not advertise your project on JSDoc's issue tracker. Thanks for your understanding.

[ariporad]

@hegemonic: Sorry if this is dumb, but is that list the list of stuff shipping in npm (which I know from you + experimentation doesn't support ES6) or master?

[hegemonic]

@ariporad That's the list of things shipping on master.

[h13i32maru]

@hegemonic I'm sorry :pensive: If uncomfortable, please delete my comment.

[netpoetica]

Are we saying that class expressions are supported? I just installed jsdoc via NPM, and I get errors that state that it's not supported;

JSDoc does not currently handle [...] nodes

"..." ExportDeclaration
"..." ClassDeclaration
"..." ClassBody
"..." MethodDefinition

I am also still seeing this with my ES6 code in JSDoc 3.3.2.

I also get an additional:

• JSDoc does not currently handle ExportSpecifier nodes

These are just warnings, but generated documentation is essentially empty / incredibly confusing.

I was hoping I could find some sort of example ES6 file using JSDoc in the way the authors of JSDoc would have expected to have their tests pass, but I do not see anything like this.

Considering the size of that list of implemented items, I would think that users (including myself) must be doing something differently than the implementors expected?

[hegemonic]

@netpoetica JSDoc 3.3.x does not support ES2015. You'll need to use the development version of JSDoc for ES2015 support. JSDoc 3.4.0 will support ES2015.

[codecvlt]

Looks like static class variables are not supported with jsdoc 3.4.0-dev
e.g
export default class AClass {
static someVar = 'something';
}

[probablyup]

@codecvlt They're still a stage 1 proposal, so I'd be surprised if any parsers supported it. https://github.com/jeffmo/es-class-static-properties-and-fields

[aweary]

Looks like it has trouble parsing arrow functions used in React components.

class SomeComponent extends React.Component {
  someMethod = () => {
   return false;
 }
}

It throws the Unexpected token = error. This format is a common alternative in to using bind everywhere when using ES6 in React.

[probablyup]

@Aweary Did you see my comment above? Related to the same thing, not supported in the current ES 2015 spec.

[flying-sheep]

@Aweary why would you do that? just leave out the “=” and the “=>” and presto

[probablyup]

@flying-sheep He's trying to autobind a member, which is supported in the stage 1 members proposal, but not in the current class spec.

Though that particular example doesn't make use of any this.*.

[aweary]

@yaycmyk correct, we use this instead of having to call .bind(this) all the time. That was just an example to demonstrate the syntax causing the problem. Thanks for the heads up.

[flying-sheep]

ah, sorry! i missed the part below the code.

[zandaqo]

Guys, detecting default parameters doesn't seem to work with method definitions in 3.4. That is, I can get default values in stand alone functions without documenting them, but not in class/object methods when I use the new method notation:

class A {
  /**
   * @param {number} [a] Some number.
   */
  getA(a = 1) {
    return a;
  }
}

Docs generated for the above do not show 1 as the default value for a param.

[ryaninvents]

The changelog seems to imply that I ought to be able to run JSDoc 3.4.0 on code that uses native modules, yet I'm still getting the error:

/.../node_modules/jsdoc-jsx/node_modules/jstransform/src/jstransform.js:270
    throw e;
    ^

Error: Parse Error: Line 1: Illegal import declaration
    at throwError ...

It failed on the JSX as well, which to my understanding was also resolved. I'm willing to use the jsdoc-jsx plugin for now to resolve that, but I can't seem to get past the modules issue.

Is there a discrepancy between what was published to npm and what's in this repo for 3.4.0? That's the only thing I can think of that would cause this issue.

[Reinmar]

I've spent some time in order to understand how JSDoc handles ES6 modules. I pushed results of my tests to https://github.com/ckeditor/ckeditor5-design/tree/jsdoc-module-test

There are couple of modules which I tried to document.

First of all, due to #1137 I had to trick JSDoc to get anything work: https://github.com/ckeditor/ckeditor5-design/blob/jsdoc-module-test/jsdoc/plugins/export-fix.js

After I got rid of export default and export class classes started to work. However, there was still a problem with export default function which I had to change to export function. In other words, default exports do not work at all.

The result is also a bit confusing, because since default exports don't work, you get links like: module:ckeditor5/a~A instead module:ckeditor5/a~default. Perhaps this is the expected result that exports are documented under their internal names (because the same happens in this case), but it's problematic for us, because the readers of the documentation will not know how to import the right class. We'd need to somehow let readers know that in case of default exports they'll need to do this:

import A from 'ckeditor5/a';

instead of:

import { A } from 'ckeditor5/a';

Right now, JSDoc gives no clue whether an export was the default or not.

Another thing which I don't get is the linking to the exported things. With classes I must use ~ and with other things I must use . as a separator. I wasn't able to find a clarification for that in JSDoc's documentation.

[hyperbrave]

es6 export default is working in my case, but the generated docs are showing require syntax instead of import statements

(require("my/thing"))(options)

I want it to generate using es2015 import statements

import thing from "my/thing"
thing(options)

is this possible with export default?

/**
 * A module for creating thing
 * @module my/thing
 */

/**
 * create thing
 * @function
 */
export default function thing () {
  return "thing"
}

[deltreey]

I'd like to note that backticks are working in master now, so work on this is helping me!

[withinboredom]

FYI, async/await have come out from behind the --harmony flag in node 7.7.1: http://node.green/#ES2017-features-async-functions

[damianobarbati]

Is JSDoc compatible with ES6/ES7 or not?
Or better: is JSDOC compatible just with old aged sh*y JS or not?

macbook:mvp damiano$ ./node_modules/jsdoc/jsdoc.js -a api/*
ERROR: Unable to parse /Users/damiano/Desktop/mvp/api/crud.js: Unexpected token =>

Because it's kinda ridiculous nowadays finding tools not working with current JS specs.

We had ES6 around for 2 years by now and ES7 arrived as well with node/chrome/safari/edge shipping async/await: and here we're still having troubles with async functions?

Keep up the good work guys.

[dancoates]

@damianobarbati take it easy there. No one is being paid to maintain this and you aren't paying to use it so please try and be more constructive with your criticisms.

If you are having trouble with ES2016+ and jsdoc then you have a few options.

• Transpile your code before running it through jsdoc
• Use the jsdoc-babel plugin
• Contribute changes/fixes to jsdoc to improve its ES2016+ support.

That said, I do believe that arrow functions should work with jsdoc, so you may be using an out of date version?

[damianobarbati]

@dancoates you are totally right, my apologies: too much frustration.

Arrow functions are working right, problem is async/await not supported.
For whoever is having same problem, here the current solution:

    "plugins": [
        "node_modules/jsdoc-strip-async-await"
    ],

[dancoates]

@damianobarbati no problemos 🙂

Not sure if you are already using babel but if so you may find that using jsdoc-babel is a more all-purpose solution as it will run your code through the same set of transforms that are specified in your .babelrc before the code is consumed by jsdoc so in theory would work for any other new features like static class properties that you may want to use.

[damianobarbati]

@dancoates I'm using JSDoc to document my APIs for front-end developers who are going to consume them: just latest Node, no transpiling there :)

I'll keep this in mind once I'll start documenting components, thanks!

[hegemonic]

We now support any syntax that Babylon can parse (i.e., far beyond what's in ES2015).