Shadow-cljs: Possible to download shadow-cljs from maven mirror?

[jiyinyiyong]

Heard someone raised question on WebChat. Downloading aot file is slow in China. If we can make a use of mirrors, it might be faster.

[thheller]

You can set :mirrors in shadow-cljs.edn, similar to lein.

https://github.com/technomancy/leiningen/blob/stable/sample.project.clj#L130-L138

[thheller]

@jiyinyiyong how fast are download speeds from github in China? I'm currently reworking the "launcher" and currently plan on having the npm package download that from github instead of clojars.

The location will change again but is it faster for you to download the .jar from here:
https://github.com/shadow-cljs/launcher/releases
https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar

or maven/clojars?

[jiyinyiyong]

I think GitHub might be even slower sometimes. I can bare the time downloading jars from maven at current from Shanghai. GitHub or Amazon might be slower.

=>> time wget https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
--2018-08-12 05:59:54--  https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
Resolving github.com... 52.74.223.119, 13.250.177.223, 13.229.188.59
Connecting to github.com|52.74.223.119|:443... connected.
^C

real    0m50.005s
user    0m0.027s
sys 0m0.009s
=>> time wget https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
--2018-08-12 06:00:45--  https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
Resolving github.com... 13.250.177.223, 13.229.188.59, 52.74.223.119
Connecting to github.com|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220047Z&X-Amz-Expires=300&X-Amz-Signature=e361c80900a34dc6d229f623c3ef06a38653f50ab7dfb978727c6199bd00cf4e&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream [following]
--2018-08-12 06:00:47--  https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220047Z&X-Amz-Expires=300&X-Amz-Signature=e361c80900a34dc6d229f623c3ef06a38653f50ab7dfb978727c6199bd00cf4e&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.102.67
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.102.67|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10834551 (10M) [application/octet-stream]
Saving to: 'launcher-1.2.0-standalone.jar'

launcher-1.2.0-standalone.jar                 100%[==============================================================================================>]  10.33M   170KB/s    in 58s

2018-08-12 06:01:49 (182 KB/s) - 'launcher-1.2.0-standalone.jar' saved [10834551/10834551]


real    1m4.260s
user    0m0.155s
sys 0m0.297s
=>> time wget https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
--2018-08-12 06:01:51--  https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
Resolving github.com... 13.250.177.223, 13.229.188.59, 52.74.223.119
Connecting to github.com|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220152Z&X-Amz-Expires=300&X-Amz-Signature=8239301cfc845572d511d933c739619597e69116558b91d32523f1761947b1f7&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream [following]
--2018-08-12 06:01:52--  https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220152Z&X-Amz-Expires=300&X-Amz-Signature=8239301cfc845572d511d933c739619597e69116558b91d32523f1761947b1f7&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.102.67
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.102.67|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10834551 (10M) [application/octet-stream]
Saving to: 'launcher-1.2.0-standalone.jar.1'

launcher-1.2.0-standalone.jar.1                 4%[==>                                                                                            ] 432.56K  34.3KB/s    eta 5m 18s ^C

real    0m16.157s
user    0m0.041s
sys 0m0.030s
=>>

Failed to run asciinema in 5 mins due to pip issues BTW.

[jiyinyiyong]

I think GitHub might be even slower sometimes. I can bare the time downloading jars from maven at current from Shanghai. GitHub or Amazon might be slower.

=>> time wget https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
--2018-08-12 05:59:54--  https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
Resolving github.com... 52.74.223.119, 13.250.177.223, 13.229.188.59
Connecting to github.com|52.74.223.119|:443... connected.
^C

real    0m50.005s
user    0m0.027s
sys 0m0.009s
=>> time wget https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
--2018-08-12 06:00:45--  https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
Resolving github.com... 13.250.177.223, 13.229.188.59, 52.74.223.119
Connecting to github.com|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220047Z&X-Amz-Expires=300&X-Amz-Signature=e361c80900a34dc6d229f623c3ef06a38653f50ab7dfb978727c6199bd00cf4e&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream [following]
--2018-08-12 06:00:47--  https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220047Z&X-Amz-Expires=300&X-Amz-Signature=e361c80900a34dc6d229f623c3ef06a38653f50ab7dfb978727c6199bd00cf4e&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.102.67
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.102.67|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10834551 (10M) [application/octet-stream]
Saving to: 'launcher-1.2.0-standalone.jar'

launcher-1.2.0-standalone.jar                 100%[==============================================================================================>]  10.33M   170KB/s    in 58s

2018-08-12 06:01:49 (182 KB/s) - 'launcher-1.2.0-standalone.jar' saved [10834551/10834551]


real    1m4.260s
user    0m0.155s
sys 0m0.297s
=>> time wget https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
--2018-08-12 06:01:51--  https://github.com/shadow-cljs/launcher/releases/download/1.2.0/launcher-1.2.0-standalone.jar
Resolving github.com... 13.250.177.223, 13.229.188.59, 52.74.223.119
Connecting to github.com|13.250.177.223|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220152Z&X-Amz-Expires=300&X-Amz-Signature=8239301cfc845572d511d933c739619597e69116558b91d32523f1761947b1f7&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream [following]
--2018-08-12 06:01:52--  https://github-production-release-asset-2e65be.s3.amazonaws.com/144060303/8d460bf6-9b54-11e8-8d5c-87fc83338667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180811%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180811T220152Z&X-Amz-Expires=300&X-Amz-Signature=8239301cfc845572d511d933c739619597e69116558b91d32523f1761947b1f7&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlauncher-1.2.0-standalone.jar&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.102.67
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.102.67|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10834551 (10M) [application/octet-stream]
Saving to: 'launcher-1.2.0-standalone.jar.1'

launcher-1.2.0-standalone.jar.1                 4%[==>                                                                                            ] 432.56K  34.3KB/s    eta 5m 18s ^C

real    0m16.157s
user    0m0.041s
sys 0m0.030s
=>>

Failed to run asciinema in 5 mins due to pip issues BTW.

[thheller]

Hmm thats really slow but npm was slow too right?

Overall I think its better since the launcher will be downloaded much less frequently now and only once per system not once per project.

[thheller]

Would http://code.thheller.com/data/shadow-cljs-launcher-2.0.0.jar be faster?

[yqrashawn]

There are multiple npm mirror in china, speed won't be a problem if we put the .jar file into npm package. But it may cause other problem.

https://npm.taobao.org/browse/keyword/shadow-cljs

[jiyinyiyong]

=>> wget http://code.thheller.com/data/shadow-cljs-launcher-2.0.0.jar
--2018-08-12 23:37:47--  http://code.thheller.com/data/shadow-cljs-launcher-2.0.0.jar
Resolving code.thheller.com... 138.201.193.44
Connecting to code.thheller.com|138.201.193.44|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://code.thheller.com/data/shadow-cljs-launcher-2.0.0.jar [following]
--2018-08-12 23:37:48--  https://code.thheller.com/data/shadow-cljs-launcher-2.0.0.jar
Connecting to code.thheller.com|138.201.193.44|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23719331 (23M) [application/java-archive]
Saving to: 'shadow-cljs-launcher-2.0.0.jar'

   shadow-cljs   4% 967.63K  18.2KB/s    eta 8m 19s       ^C
=>>

[zcaudate]

@yqrashawn

But it may cause other problem.

Why would it be a problem?

[yqrashawn]

@zcaudate I don't know if it's right to put large file in npm package. Cause many npm packages that with big binary files has their own mirror (phantomjs, ChromeDriver, electron etc). These packages download their binary files from other place instead of directly from npm while runing npm install. Maybe file in packages must go into git? I'm not sure.

I found that when I run shadow-cljs the first time, it start a node process and download the launcher jar file. But the node process does not go through http_proxy https_proxy. The process stuck there for a while. I have to use proxychains to force the request go through proxy. Maybe that's the problem.

What's more, there are maven mirror and clojars mirror in china as well.

There are preinstall postinstall npm scripts. Maybe we can use those to read mirror config and download jar file while running npm install or yarn add?

[thheller]

The https package from node does not support proxies out of the box. Coding my own proxy support is probably too much work for now. The request package supports proxies but has too many dependencies to use it for one single download.

I'm currently considering adding an optional :launcher-download-cmd "proxychains wget {launcher-url} -O {output-file}" setting in shadow-cljs.edn. If found the launcher download would be attempted by running the specified command. I think that should cover all possible scenarios?

The :mirror support kinda got lost since I thought tools.deps supported it but doesn't. Might switch back to pomegranate or see if I can get a patch into TDEPS.

[jiyinyiyong]

proxychains4 wget {launcher-url} -O {output-file}

in my case.

[thheller]

Point is that you can set it to whatever makes sense for you.

Someone should verify if setting

:jvm-opts
["-Dhttps.proxyHost=<ip>"
 "-Dhttps.proxyPort=<port>"]

in shadow-cljs.edn actually affects the maven downloads? Assuming you have an accessible HTTPS proxy. It won't affect the launcher download but might be a short term alternative until :mirrors support is back.

[thheller]

FWIW you can currently set alternate maven repos via :repositories. No need for mirrors.

 :repositories
   {"central" {:url "http://mirrors.163.com/maven/repository/maven-public/"}
    "clojars" {:url "https://mirrors.ustc.edu.cn/clojars/"}}

This does not affect the loader download however. Only dependencies.