Pluto.jl: Artifacts to proxy&cache CDNs

[staticfloat]

It would be nice for startup time (especially on a slow connection) to bundle all web assets locally; if this is a relatively slowly-moving target, this can be done easily by using Artifacts; simply write a script that collects all files into a single location inside of create_artifact(), archive the artifact into a tarball, upload that tarball somewhere, bind the location of the artifact to a name in your package, and you'll be able to serve the files from a tarball that gets downloaded at the same time as your package.

As a reference, these are the assets that my browser requests after a force-refresh on Pluto v0.11.1:

[fonsp]

All these assets are cached by a service worker, so the second load should even work offline! So the uncompressed files and HTTP1 overhead for the larger number of files isn't really a problem.

Improvements to the first load time are welcome of course, but they might be overshadowed by Julia compiling Pluto's methods. There is still the use case of deploying Pluto notebooks online - this would be useful there.

[fonsp]

But I am very interested in the artifacts system - do you think we can use it to make our project independent of the CDNs? Ideally you could run an old version of Pluto 20 years from now, when (JS hype has moved on to something else and) some of these CDNs might have gone offline.

Because Pluto is scientific software - these CDNs are commercial.

[staticfloat]

Improvements to the first load time are welcome of course, but they might be overshadowed by Julia compiling Pluto's methods. There is still the use case of deploying Pluto notebooks online - this would be useful there.

Ah, that makes sense, yes. I was working on a tethered phone, so it took many seconds to fetch those resources the first time. A pathological case for sure.

do you think we can use it to make our project independent of the CDNs?

Yes, definitely! I'd be happy to help you walk through it; the short version is that you would run a script to bundle together your content and upload it as a tarball. Then in Pluto.jl, you would have an Artifacts.toml that points a name (such as javascript_bundle) to that file, and when you want to access, for instance, js/foo.js, you'll just do something like joinpath(artifact"javascript_bundle", "js", "foo.js") and it will reach inside of the artifact (which will have been installed by Pkg automatically when Pluto.jl itself was installed, and it will have been installed by the Pkg server, so the tarball will have been cached for all eternity in the Julia Storage Servers, not to mention hosted wherever you chose to host it (I recommend a GitHub release somewhere). So you can then manually serve out those files. I imagine you could still use a service worker but attempt to load from the server it's being stored on first, and in the event of a 404, then you could grab it from a CDN, and in that way you would get the best of both worlds.

[c42f]

It would be great to use artifacts for this for the reasons mentioned but there might be some other surprise benefits as well. For example, making Pluto work better in places like China which like to block some CDNs (including google fonts, at least in the past.)

[fonsp]

Thanks everyone! I am definitely interested, but I still have some questions:

• Does this mean changing Pluto's compatibility from v1.0 to v1.3? How many people do you think would be affected? (The Julia version is not collected as part of Pluto's telemetry unfortunately.)
• My tagbot lags behind the general registry, so if I generate the bundle using the tagbot, it will not exist for users who install during the inbetween time period. Even if I increase the tagbot interval, creating the bundle will still take time. Should I bundle for each commit instead?

[staticfloat]

Does this mean changing Pluto's compatibility from v1.0 to v1.3?

Yes. It's possible to build a "hybrid system" that downloads the assets via Artifacts on 1.3 and through some other mechanism on 1.0-1.2, but IMO it's not worth the effort and maintenance burden.

How many people do you think would be affected?

Hard to say, but the volume of users asking for help with binary issues on Julia 1.3 has dropped off significantly, so my guess is that the majority of users are 1.3+ at this point.

My tagbot lags behind the general registry, so if I generate the bundle using the tagbot

How often does the bundle change? I was under the impression that this was a "generate once, forget about it for a few months" kind of thing. If it changes rapidly, then perhaps we'll need a different solution; a Travis job to rebuild it after every commit and automatically update the Artifacts.toml file if it detects a change, perhaps.

[fonsp]

How often does the bundle change?

That's true, only about 1-2 times per month. What would a manual solution look like? A second repository that just contains releases, that I upload myself?

[staticfloat]

Precisely; the way I usually do it is to have a script somewhere that, when run, bundles everything together then uploads it to a GitHub release somewhere. It could be Pluto.jl, (And you just create named releases where the gitsha that the release is associated with doesn't matter, like these auto-generated ones for GCC shards on Yggdrasil; the gitsha the "release" is tagged upon has nothing to do with the contents of the release, it's literally just free binary hosting for me) or it could be a totally separate repository.

I tend to use ghr to upload binaries to github releases; here's an example invocation from BinaryBuilder itself:

function upload_to_github_releases(repo, tag, path; gh_auth=Wizard.github_auth(;allow_anonymous=false),
                                   attempts::Int = 3, verbose::Bool = false)
    for attempt in 1:attempts
        try
            ghr() do ghr_path
                run(`$ghr_path -u $(dirname(repo)) -r $(basename(repo)) -t $(gh_auth.token) $(tag) $(path)`)
            end
            return
        catch
            if verbose
                @info("`ghr` upload step failed, beginning attempt #$(attempt)...")
            end
        end
    end
    error("Unable to upload $(path) to GitHub repo $(repo) on tag $(tag)")
end

And heres's how an artifact is created, archived (e.g. turned into a tarball) and bound to an Artifacts.toml file:

# Create artifact (which is a folder in `~/.julia/artifacts`)
treehash = create_artifact() do dir
    # write your files into <dir> here
end

# Archive artifact into tarball and upload it to GHR
local tarball_hash
filename = "WebAssets-1.0.0.tar.gz"
mktempdir() do dir
    tarball_hash = archive_artifact(hash, joinpath(dir, filename))
    BinaryBuilder.upload_to_github_releases(repo, tag, dir)
end

# Bind artifact to Artifacts.toml that needs to go into your package:
bind_artifact!(
    "./Artifacts.toml",
    # This is the name of the artifact; you'll get the path of the locally-installed files via `artifact"WebAssets"` in your package
    "WebAssets",
    treehash,
    download_info = Tuple[
        ("https://github.com/$(repo)/releases/download/$(tag)/$(filename)", tarball_hash),
    ],
)

[fonsp]

Thanks so much! Your examples are very helpful. I am motivated to incorporate this to ensure that old versions of Pluto can stay online for "ever".

(In response to @c42f: thank you for bringing this up, I hadn't thought about access in China. I did find that the jsdelivr.com CDN is available inside China (https://www.jsdelivr.com/network). I have already switched all resources to that CDN, except Google Fonts.)

EDIT: Google Fonts are now also served over jsdelivr.

[ArrowRaider]

My laptop is giving errors when trying to load these files from the cdn. I have no idea what this means. Please see this screenshot: https://i.imgur.com/k3u64Qc.png

[fonsp]

@ArrowRaider have a look here: https://github.com/fonsp/Pluto.jl/issues/607#issuecomment-717754836

And you are right that browser extension problems are solved by a proxy.