Jsdoc: Move dependencies to be in npm

[lukeapage]

Is there any change to move all dependencies to npm?
https://github.com/jsdoc3/jsdoc/blob/master/package.json#L25

I'm behind a corporate firewall where npm packages have to be requested and then cloned to an internal private npm repo, and breaking out of npm packages means npm goes direct to github.

[hegemonic]

I'll look into this.

One very large caveat: Even if I can switch to an NPM module for this dependency, I can't make any promises that JSDoc will never again use GitHub for dependencies. In the past, there have been times when using GitHub was the only way to pick up essential bug fixes.

I also have no way to guarantee that JSDoc's dependencies will, in turn, only use NPM modules for their dependencies.

[lukeapage]

Thanks, i understand. One thing at a time!

[yamikuronue]

This is crucial to my company's adoption of JSDoc, as we use subversion in-house and do not wish to be backed into installing git just for jsdoc.

[lukeapage]

@yamikuronue its currently http urls to github tarballs, so it does not require git..

[yamikuronue]

I'm definitely getting ENOGIT though...

I'm getting this as a transitive dependency of grunt-jsdoc, which requires version 3.2.2. Would upgrading help?

error   23-Mar-2015 08:53:59    npm ERR! git clone https://github.com/hegemonic/taffydb.git undefined
error   23-Mar-2015 08:53:59    npm ERR! git clone https://github.com/hegemonic/taffydb.git undefined
error   23-Mar-2015 08:54:08    npm ERR! Linux 2.6.32-220.23.1.el6.x86_64
error   23-Mar-2015 08:54:08    npm ERR! argv "node" "/usr/bin/npm" "install"
error   23-Mar-2015 08:54:08    npm ERR! node v0.10.33
error   23-Mar-2015 08:54:08    npm ERR! npm  v2.7.0
error   23-Mar-2015 08:54:08    npm ERR! code ENOGIT
error   23-Mar-2015 08:54:08    
error   23-Mar-2015 08:54:08    npm ERR! not found: git

[lukeapage]

Yes, thats jsdoc 3.2.2, it looks like the alpha 3.3.0 and master 3.4.0 dont suffer that issue (just my issue with a https link to github)

[hegemonic]

This is now fixed on master. I'm not planning to make this change on the 3.3.x branch, just in case the new version of TaffyDB breaks something. That means the fix will be released as part of JSDoc 3.4.0.

[hegemonic]

I forgot to run the test suite after updating TaffyDB, and it looks like the new version breaks a bunch of stuff. :( I'll take a look at some point and see if the breakage is fixable.

[motzke]

Any update on this issue?

[popomore]

Why not use stable version?

By the way, TaffyDB has a strange version.

   { modified: '2013-03-15T04:28:37.330Z',
     created: '2011-11-06T16:55:29.381Z',
     '0.0.1': '2011-11-06T16:55:30.152Z',
     '2.7.2': '2013-03-15T04:28:37.330Z' },

[spawnrider]

Hi,

I'am in the same state. I can't retrieve dependencies from Git due to corporate security settings and the use of an internal Nexus registry.

Could you refer to the NPM package instead?

[redonkulus]

@hegemonic please let us know how we can help. this reared its ugly head when github went down today. affected builds, as npm could not fetch the tarball from github.

[timhemming-fico]

Same issue for our team. Best to stick with NPM dependencies for release builds.

[popomore]

@hegemonic any update?

[hegemonic]

As noted above, the NPM version of TaffyDB breaks JSDoc. TaffyDB is unmaintained, so the odds of getting that breakage fixed are basically nonexistent.

I think the only realistic solution is to publish our fork of TaffyDB to NPM. I'll pursue that option, since this seems to be important to a number of people.

[75lb]

i already published the jsdoc version of Taffy on npm, you can use that if you like.. my personal jsdoc fork uses it.

[popomore]

You can copy the code to jsdoc's repo, as it's not maintained.

[popomore]

I have made a PR that copy taffydb.js to the repo.

[animedbz16]

We are experiecing this same issue.

I would agree with @popomore solution of directly including the taffydb.js into this project if there are no plans to migrate away from taffydb which will resolve this issue.

[ghost]

I have the same problem.

When would this fix be released?

[animedbz16]

NPM has published a blog post that relates to this kind of issue:
http://blog.npmjs.org/post/145724408060/dealing-with-problematic-dependencies-in-a

Please review #961 and merge. It would be best to solve this in the upstream repository, rather than me forking this just to make this change so that my internal CI (which can't directly access GitHub to install TaffyDB) can build properly.

[kaanozcan]

We have also have this problem on our team. We fixed it on our locals by setting proxy but its causing massive performance issues on our continuous integration pipeline where it would be a lot better if we could utilize our private registry.

[fredericrous]

Why is this issue tagged "Feature"!? In my opinion it's a critical bug / show stopper. Everyone working on restricted network environment are affected. This package cannot be used.

[popomore]

Is jsdoc dead? Can anyone review it whether is suitable?

https://github.com/jsdoc3/jsdoc/pull/1202

[mlucool]

Any update on this? It creates a large variability in install times

[atcol]

We are suffering from a flaky proxy that causes our builds to fail. Whilst I'd love to resolve the flaky proxy, the only reason our builds need it is _only_ because of this taffydb dependency from JSDoc. Everything else can be obtained from our internal Nexus instance.

Please, oh please, can we have the dependency updated to point to NPM instead of Github?

[animedbz16]

Lots of discussion surrounding this issue and no response it seems from anyone who maintains this project. @popomore has updated his MR which fixes this. Can someone review and merge to solve this problem.

@hegemonic @jannon @micmath @tschaub

[hegemonic]

I've investigated this issue thoroughly. tl;dr: At this time, I can't find a way to use a TaffyDB package from NPM. Every option I can think of would break ~all JSDoc 3 templates, which is not an acceptable outcome. Please do not comment unless you have a solution that a) I haven't thought of already and b) does not break ~all JSDoc 3 templates.

Here are the gory details for those who are interested, including all the options that I considered:

Background

When JSDoc invokes a template, it passes the doclets to the template as a TaffyDB object. All JSDoc 3 templates expect to receive a TaffyDB object as an argument to their exports.publish method.

JSDoc currently uses a fork of TaffyDB that is known to work correctly with JSDoc. The fork is not published to NPM—it is installed from GitHub when you run npm install.

Almost all JSDoc 3 templates are derived from the default template, which includes the following line in publish.js:

var taffy = require('taffydb').taffy;

As a result, ~all other JSDoc 3 templates include the same line of code. In other words, ~all templates expect that require('taffydb') will work as expected. (The only exception I know of is the Baseline template, which is still experimental and has not been widely adopted.)

Note that this expectation is true _even if the template does not list TaffyDB as one of its dependencies_. That's because, for historical reasons, JSDoc makes it possible for external templates and plugins to require JSDoc's module dependencies, including TaffyDB, without installing them locally.

Alternative 1: Use the version of TaffyDB on NPM

There are two versions of TaffyDB that are published to NPM as taffydb:

• 0.0.1: Fails to run.
• 2.7.2: Affected by the bug typicaljoe/taffydb#114, which causes TaffyDB's isUndefined matcher not to work. The default template depends upon this matcher, which means that ~all other JSDoc 3 templates also depend upon it. As a result, upgrading to version 2.7.2 would break ~all JSDoc 3 templates.

If a new version of TaffyDB were released that fixed typicaljoe/taffydb#114, then it would be possible to use NPM's taffydb module, thus solving the problem. However, TaffyDB is not actively maintained, so this bug is unlikely to be fixed anytime soon.

Alternative 2: Use a fork of TaffyDB published to NPM

I published JSDoc's fork of TaffyDB to NPM as taffydb-jsdoc. JSDoc could be updated to require('taffydb-jsdoc') instead of require('taffydb').

However, this change would break ~all JSDoc 3 templates other than the default template, because ~all JSDoc 3 templates attempt to require('taffydb').

One way to work around this problem would be to use an NPM post-install script that creates a symbolic link from node_modules/taffydb-jsdoc to node_modules/taffydb. However, JSDoc used to include a similar post-install script, and users were very unhappy about it—the post-install script created a large number of installation problems.

Alternative 3: Bundle TaffyDB with JSDoc

In the pull request jsdoc3/jsdoc#1202, @popomore suggested bundling TaffyDB with JSDoc in a deps/ directory. However, this approach has the same problem as Alternative 2—it prevents require('taffydb') from working, so it breaks ~all JSDoc 3 templates.

Alternative 4: Stop using TaffyDB

If JSDoc did not use TaffyDB, we wouldn't have to worry about this problem at all. However, migrating from TaffyDB is impractical for a few reasons:

• All JSDoc templates expect to receive a TaffyDB object as one of their inputs. We could try to provide an object that behaved like a TaffyDB object, but that would essentially require reimplementing TaffyDB as part of JSDoc.
• Even if we solved the previous problem somehow, ~all JSDoc 3 templates would still break, because they attempt to require('taffydb').

Conclusion

For now, we're stuck with the current system of installing TaffyDB from GitHub.

If you _really, really_ care about being able to install JSDoc from NPM, please put your energy into fixing typicaljoe/taffydb#114 and convincing the TaffyDB maintainers to release a new version with your fix.

[animedbz16]

The only other alternative that I can thnk of is what if a retroactive version of TaffyDB can be published to NPM for this specific commit that JSDoc is dependent on if there are in fact no plans for TaffyDB to correct that issue on their end preventing JSDoc from updated to the latest TaffyDB.

Obviously this is still dependent on someone from TaffyDB, but including @mcwhittemore since he seems to maintain the NPM TaffyDB for thoughts.

From my review JSDoc is currently dependent upon the forked TaffDB:

"taffydb": "https://github.com/hegemonic/taffydb/tarball/7d100bcee0e997ee4977e273cdce60bd8933050e",

https://github.com/jsdoc3/jsdoc/blob/37d0f91515e42cd3a568e3bf111d8d565b02de9e/package.json#L26

Looking at the commits, it seems that this fork is originally based off of this commit in TaffyDB:
https://github.com/typicaljoe/taffydb/commit/fa2a2c9ef79634ddecba845de45c5b79f6148943

Looking at the commits made in the fork based on the TaffyDB commit above, @hegemonic seems to have listed TaffyDB with a version of v2.6.2.
https://github.com/hegemonic/taffydb/commit/544496fb8df9b91239126c814f6faa212e0e0c5f

I'm not really familar with the history of TaffyDB, but was this an official version at one point in time, which became unpublished from NPM?

If this commit could be republished to NPM within the official TaffyDB namespace, would this then allow JSDoc base itself onto this new retroactive version? Thoughts?

[lextiz]

@mcwhittemore @chambery Could you please upload to npm taffydb v2.6.2 from https://github.com/hegemonic/taffydb? It is functionally the same as typicaljoe/taffydb@fa2a2c9 only has package.json added. This would help a lot to some jsDoc users. Thanks.

[mcwhittemore]

Hi all. Sorry for the slow reply. I just opened an issue on taffy about getting people who work on the project to maintain the npm module. https://github.com/typicaljoe/taffydb/issues/127

I've also published typicaljoe/taffydb@fa2a2c9 as 2.6.2 on npm. As this version is older than the current one on npm, it is not what will be downloaded via npm install taffydb. Hope that helps.

[hegemonic]

@animedbz16 Thanks for the great suggestion, and @mcwhittemore many thanks for publishing version 2.6.2 to NPM!

I'll soon release JSDoc 3.4.1, which will include this change.

[popomore]

Cool, can you release a new version now？

[lextiz]

@mcwhittemore Thank you! @hegemonic Thanks for your attention to this issue. May I kindly ask you when do you plan to release 3.4.1?