Shadow-cljs: Standalone script for grabbing shadow-cljs-2.x.x-aot.jar file?

[jiyinyiyong]

Updates... turned out this script is enough:

VERSION=2.8.8
PROXY=proxychains4
JAR_HOST=https://clojars.org/repo/thheller/shadow-cljs

mkdir ~/.m2/repository/thheller/shadow-cljs/$VERSION/
cd ~/.m2/repository/thheller/shadow-cljs/$VERSION/
$PROXY wget $JAR_HOST/$VERSION/shadow-cljs-$VERSION-aot.jar
$PROXY wget $JAR_HOST/$VERSION/shadow-cljs-$VERSION.pom

cd -

---

There are still some occasions I can't not download jar files from official registry and meanwhile requests to the mirror just fail.

Could you provide some scripts that we can easily fallback to manual downloads of the jar files and place them into the right path?

=>> proxychains4 yarn watch
[proxychains] config file found: /Users/chen/.proxychains/proxychains.conf
[proxychains] preloading /usr/local/Cellar/proxychains-ng/4.13/lib/libproxychains4.dylib
yarn run v1.13.0
$ shadow-cljs watch client
shadow-cljs - config: /Users/chen/repo/mvc-works/calcit-workflow/shadow-cljs.edn  cli version: 2.7.16  node: v11.6.0
shadow-cljs - updating dependencies
shadow-cljs - dependency update failed - Could not find artifact thheller:shadow-cljs:jar:aot:2.7.16 in central (https://maven.aliyun.com/nexus/content/groups/public/)
DependencyResolutionException Could not find artifact thheller:shadow-cljs:jar:aot:2.7.16 in central (https://maven.aliyun.com/nexus/content/groups/public/)
    org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:393)
    sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java:-2)
    sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke (Method.java:498)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    cemerick.pomegranate.aether/resolve-dependencies* (aether.clj:806)
    cemerick.pomegranate.aether/resolve-dependencies* (aether.clj:707)
    clojure.core/apply (core.clj:657)
    clojure.core/apply (core.clj:652)
    cemerick.pomegranate.aether/resolve-dependencies (aether.clj:815)
Caused by:
ArtifactResolutionException Could not find artifact thheller:shadow-cljs:jar:aot:2.7.16 in central (https://maven.aliyun.com/nexus/content/groups/public/)
    org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:453)
    org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:255)
    org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:376)
    sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java:-2)
Caused by:
ArtifactNotFoundException Could not find artifact thheller:shadow-cljs:jar:aot:2.7.16 in central (https://maven.aliyun.com/nexus/content/groups/public/)
    org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:48)
    org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:364)
    org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:76)
    org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute (BasicRepositoryConnector.java:590)
error Command failed with exit code 1.

[thheller]

That looks like maybe the mirror is not up to date and didn't sync yet?

Did you try a different mirror or the default clojars?

I'm not quite sure what a script would do that downloading directly from clojars wouldn't also accomplish as that is what the script would download from?

[jiyinyiyong]

I think it's not synced yet.

In the script it can be the address the file on Clojars. I can use proxychains, I can copy the link and download with Chrome, or even download with my server on Aliyun. Not perfect, but I when I get the address and the path, there might be various of ways.

[thheller]

But why can you not do that with what we already have? All the maven download does is exactly that?

You can find all the download links here:
https://clojars.org/repo/thheller/shadow-cljs/

[jiyinyiyong]

and which path to put the file?

But why can you not do that with what we already have? All the maven download does is exactly that?

I don't not understood the question. For the most of the time in the past month, it was not concerned problem since I could always download the jars from mirrors of clojars. But in the past days I noticed it took far longer time to download. Better we have fallback solutions in such cases.

[thheller]

All maven does is request the file via HTTP and put it into the correct location, so exactly what your suggested script would also do.

The problem clearly is that your mirror is out of date, so it should work if you just don't use the mirror and download from the default instead. Or just wait until your mirror syncs.

I'm open to other suggestions but a script to replicate what maven already does seems pointless to me.

[jiyinyiyong]

I'm only taking such a script as a rescue method. Taking version 2.7.16 for example, it sounds like:

cd ~/.m2/repository/thheller/shadow-cljs/2.8.7/
wget https://clojars.org/repo/thheller/shadow-cljs/2.8.7/shadow-cljs-2.8.7-aot.jar
wget https://clojars.org/repo/thheller/shadow-cljs/2.8.7/shadow-cljs-2.8.7.pom
cd -

is shadow-cljs-2.8.7-aot.jar the only file that is really necessary to be downloaded, right?

---

Feelings are different:

=>> time wget https://clojars.org/repo/thheller/shadow-cljs/2.7.16/shadow-cljs-2.7.16-aot.jar
--2019-01-21 01:21:50--  https://clojars.org/repo/thheller/shadow-cljs/2.7.16/shadow-cljs-2.7.16-aot.jar

Resolving clojars.org (clojars.org)... 23.253.149.7
Connecting to clojars.org (clojars.org)|23.253.149.7|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8783788 (8.4M) [application/java-archive]
Saving to: 'shadow-cljs-2.7.16-aot.jar'

shadow-cljs-2.7.16-aot 100%[=========================>]   8.38M  99.5KB/s    in 92s

2019-01-21 01:23:25 (93.5 KB/s) - 'shadow-cljs-2.7.16-aot.jar' saved [8783788/8783788]


real    1m34.835s
user    0m0.152s
sys 0m0.290s
=>>
=>> time proxychains4 wget https://clojars.org/repo/thheller/shadow-cljs/2.7.16/shadow-cljs-2.7.16-aot.jar
[proxychains] config file found: /Users/chen/.proxychains/proxychains.conf
[proxychains] preloading /usr/local/Cellar/proxychains-ng/4.13/lib/libproxychains4.dylib
--2019-01-21 01:23:35--  https://clojars.org/repo/thheller/shadow-cljs/2.7.16/shadow-cljs-2.7.16-aot.jar
Resolving clojars.org (clojars.org)... 224.0.0.1
Connecting to clojars.org (clojars.org)|224.0.0.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8783788 (8.4M) [application/java-archive]
Saving to: 'shadow-cljs-2.7.16-aot.jar.1'

shadow-cljs-2.7.16-aot 100%[=========================>]   8.38M  2.13MB/s    in 3.9s

2019-01-21 01:23:41 (2.13 MB/s) - 'shadow-cljs-2.7.16-aot.jar.1' saved [8783788/8783788]


real    0m5.972s
user    0m0.121s
sys 0m0.154s

[thheller]

That depends on your :dependencies and other deps are also still required, they just rarely change and downloaded less frequently.

It is also not enough to download the .jar, at least I think that the .pom is also required.

I still don't understand why you are treating this as a rescue method? Not using a mirror should have identical results without any additional work/script on your part.

I still have no idea what proxychains does? Does it not work with the default maven downloader?

[jiyinyiyong]

It is also not enough to download the .jar, at least I think that the .pom is also required.

okay.

I still don't understand why you are treating this as a rescue method? Not using a mirror should have identical results without any additional work/script on your part.

Mirrors may not have synced the latest version when I try it. I'm still using the configs picked in previous discussions:

 :repositories {"central" {:url "https://maven.aliyun.com/nexus/content/groups/public/"}
                "clojars" {:url "https://mirrors.ustc.edu.cn/clojars/"}}

"rescue method" might not be an accurate description. But it's best way I can think of that help me fetch the jar file in a predictable period of time.

I still have no idea what proxychains does? Does it not work with the default maven downloader?

I would guess it's the problem mentioned in the docs, https://github.com/haad/proxychains#known-limitations-of-the-current-version Not sure since I have to little knowledge on Unix network programming.

[thheller]

So what happens if you just comment out that mirror config? It just takes 1m34.835s then right? Seems ok to me? 1m30s is not the end of the world.

[jiyinyiyong]

It's unpredicatable. There's not a progress bar. If I found it not finished in 2mins or 4mins, I would guess there's something wrong. Then I would prefer downloading the files in 10s and take some time to move them to where they should be.

[chenyong]

Some random thoughts... would it be nice if shadow-cljs adds some extra warning messages when this error happens:

shadow-cljs - dependency update failed - Could not find artifact thheller:shadow-cljs:jar:aot:2.7.16 in central (https://maven.aliyun.com/nexus/content/groups/public/)
DependencyResolutionException Could not find artifact thheller:shadow-cljs:jar:aot:2.7.16 in central (https://maven.aliyun.com/nexus/content/groups/public/)

The message offers help to redirect user a gist or a script like shown above... Then he/she has a way to download manually.

Probably only people in mainland China need that message...

[thheller]

Did you try adding more repos so that it checks the "original repos" as well?

 :repositories {"central" {:url "https://maven.aliyun.com/nexus/content/groups/public/"}
                "clojars" {:url "https://mirrors.ustc.edu.cn/clojars/"}
                "maven-central" {:url "https://repo1.maven.org/maven2/"}
                "clojars-default" {:url "https://clojars.org/repo"}}

[chenyong]

Not known I can configure that. Will try.

Another thought about that case is displaying a message when it takes more than 20s to download AOT file...