Aws-cdk: Windows setup is broken

[eladb]

[eladb]

@mpiroc wrote:

So npm is smart and automatically adds a batch file wrapper for bin commands when a package is installed on Windows. But we don't distribute as an npm back, but as a pre-built zip. The build container is not Windows, so it doesn't add the batch file wrappers, and there doesn't seem to be an option to force it to always add them.

[eladb]

I think the fact that we vend a pre-built node_modules closure has other potential issues hidden in it. For example, if there are any native modules there, they won't work on Mac/Windows (since they are compiled during "npm install").

Let's recap why we are vending the prebuilt closure:

1. There's an operational/security requirement to allow customers who wish to consume a self-contained and signed bundle which includes all of our dependencies in order to mitigate any risk related to 3rd party packages.
2. Setup is very simple - unzip and play.

But... There's a silver lining. @RomainMuller recently added to our distribution an npm wrapper called [cdk-beta-npm] (I think we should rename it to npm-cdk so it will begin with "npm" and also be a little more concise). It uses [verdaccio], which is a local npm repository service, to allow developers to install CDK dependencies idiomatically. This means, that we can also install the CDK itself like that. Here's an alternative setup story:

$ mkdir -p ~/.cdk
$ unzip aws-cdk-0.x.y-beta+bla.zip -d ~/.cdk
$ ~/.cdk/bin/cdk-beta-npm install -g aws-cdk # say this is is in the root of the bundle now
$ cdk --version # the toolkit is now installed on your system!

This works! (and even compiles a native module...). Let's see if our getting started experience holds:

cdk docs

I am getting an error:

Unable to open CDK documentation - the aws-cdk-docs package appears to be missing. Please run `npm install -g aws-cdk-docs`

However, when I followed the instructions (cdk-beta-npm install -g aws-cdk-docs), it worked. Perhaps we can automatically do that if the docs cannot be found and solve this issue.

cdk init --language typescript

Just Works:tm:, and uses cdk-beta-npm to work it magic so users can continue to install CDK modules this way in their projects.

cdk init --language

Java is still WIP (#100) but I think will just need something like an mvn-cdk wrapper which will behave similarly to npm-cdk. We have the building blocks to do that.

Self-contained Bundle?

We still need to somehow distribute a self-contained bundle. This could be a completely different bundle or we can include a cache of all of our dependencies into the cdk-beta-npm repository. This means that if you use the npm wrapper, you’ll consume the dependency from the local cache.

Summary

I think I really like this approach. There are a few benefits:

• __Idiomatic!__: this will ensure we are ready to publish to npm. All the issues above will be issues we would have to solve when we publish to npm. Let's solve them today instead and make sure the experience is tight.
• __Cross platform__: solves the issue @mpiroc brought up.

Issues to resolve:

• cdk docs should auto-install aws-cdk-docs (easy!)
• We need to come up with the right way to vend a self-contained bundle that contains all our deps (I'd like to think that we can just include that .tgzs of all our deps into the bundled cdk-beta-npm repository).
• Setup instructions will slightly change, but I believe not dramatically.

[RomainMuller]

For the self-contained bundle, in the case of verdaccio, we can enable local caching of proxied dependencies, so we can build a local verdaccio registry (aka what cdk-beta-npm uses) that includes all packages that are transitively used -- obviously if the customer runs the cdk-beta-npm update command, they'll be upgraded to the non-local registry versions.

[eladb]

local caching of proxied dependencies

Does that mean that when we “publish” during our bundling process, all dependencies will be also kept under repo/npm?

[RomainMuller]

I believe so. This would require a slightly different handling of verdaccio's configuration (if an upstream is configured for a package, it will always check the upstream registry for a newer version -- so we'd have to disable this behavior for all packages we cached).

[sasaraf]

Thanks for the lib. looks like great initiative, as we did search for good dev way to deploy CF stacks. following the getting started guide i get the following error (for init App object) on windows.
Thanks for any lead on this.

software.amazon.jsii.JsiiException: ENOENT: no such file or directory, mkdtemp '/tmp/jsii-kernel-XXXXXX'
Error: ENOENT: no such file or directory, mkdtemp '/tmp/jsii-kernel-XXXXXX'

at software.amazon.jsii.JsiiRuntime.processErrorResponse(JsiiRuntime.java:118)
at software.amazon.jsii.JsiiRuntime.requestResponse(JsiiRuntime.java:90)
at software.amazon.jsii.JsiiClient.loadModule(JsiiClient.java:68)
at software.amazon.jsii.JsiiEngine.loadModule(JsiiEngine.java:109)
at software.amazon.jsii.JsiiEngine.loadModule(JsiiEngine.java:106)
at software.amazon.jsii.JsiiEngine.createNewObject(JsiiEngine.java:444)
at software.amazon.awscdk.App.<init>(App.java:16)
at com.myorg.HelloStackTest.testStack(HelloStackTest.java:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

[eladb]

@sasaraf thanks for reporting. We'll let you know when this is fixed and merged.

[eladb]

@sasaraf I wonder, if you create a directory C:\tmp, does that work around the bug?

[rix0rrr]

Just found another /tmp in the toolkit assets file. Fixing it.

[sasaraf]

@eladb thanks for the solving the tmpdir issue. the C:\tmp helps to workaround the above issue. sorry for the delay in response.

I now get a missing module exception related the kernel?
[software.amazon.jsii.JsiiException: Cannot find module ' mpjsii-kernel-VcnlCZ
ode_modules@aws-cdkcx-api'
module.js:549
throw err;
]

i didn't find it in the list of requirement in the getting started manual. I guess , once both fixed there will probably be issue with the the clock synchronization between my windows machine to aws as part of the 'signature' of cli commans?, or should i apply chronyc/ntp to solve it. For now i moved to centos 7 instance as i'm in early steps to use your cdk for infrastructure-as-code lib for CF pipeline/deployment for different environment. So the above don't block me , just thought this thread is the right place to target windows issues with the lib. hope ok.

[eladb]

@sasaraf Do you still see the issue with cx-api?

[sasaraf]

it still has this issue with mpjsii module. couldn't find such module. what npm install should be used for that? (i moved to the latest released cdk maven dependencies 0.8.2, perhaps the fix is in some snapshot version?)

Error: Cannot find module ' mpjsii-kernel-2DjNHC ode_modules@aws-cdkcx-api' at Function.Module._resolveFilename (module.js:547:15) at Module._load (module.js:474:25) at require (C:\Users\siasaraf\AppData\Local\Temp\jsii-java-runtime4231062144825626203\jsii-runtime.js:1:80961) at \tmp\jsii-kernel-2DjNHC\node_modules\@aws-cdk\cx-api:1:1