Wrangler: Always upload `.well-known` dotfiles

[ndarilek]

Wrangler doesn't appear to publish hidden files. In particular, I'm attempting to publish a Hugo site with public/.well-known/* (I.e. .well-known/keybase.txt, .well-known/matrix/server, etc.) These files are definitely in public/, but don't appear in the key list for the created namespace.

[ndarilek]

Figured out that include = ["*"] in wrangler.toml gets me a bit further with this. But while /.well-known/keybase.txt is served up fine, /.well-known/matrix/client and /.well-known/matrix/server aren't. I see the following in kv:list:

  {
    "name": ".well-known/keybase.8c602ecfc951c47ee9ad9845116e2082d5999a988f145dde543a7a1d1e75ab23.txt"
  }
  {
    "name": ".well-known/matrix/server.f9a5bcb8e11c9fc83a11d31e40daf3083d228ed0396d3abcbd45e8c4ea5ebfac"
  },
  {
    "name": ".well-known/matrix/client.c8ca63cb67a167114e46fc4b5ce38c70b751572b83038dd1ea79cf5bc1c30a6a"
  },

So they seem to be there, but I can't get them via Curl as I can /.well-known/keybase.txt.

What are the hash-looking things after the file's base and before the extension? Wondering if something is confused by the lack of extension on my /.well-known/matrix/* files.

This is kind of unintuitive. Every other platform I've worked with happily assumes that anything in public/ is meant for serving. I may still have some header massaging to do to ensure that, say, /.well-known/matrix/server gets the correct content-type, but it shouldn't be this hard to just ship my directory content and count on it arriving intact.

Thanks.

[exvuma]

Thanks for the feedback. We don't upload hidden files by default because if there is any sensitive info, it will be served globally on our network.

What are the hash-looking things after the file's base and before the extension?

Hashes of the content contained in the file to ensure unique content is cached on Cloudflare properly. It's pretty fascinating how we implemented this, read more here: https://blog.cloudflare.com/extending-the-workers-platform/

Wondering if something is confused by the lack of extension on my /.well-known/matrix/* files.

A mime library is used to determine the content type to serve from the extension. Would you mind sharing the actual results of the curl with the -v flag?

[ndarilek]

Thanks. It's a 404. The URLs:

https://lightsoutgames.nolan.workers.dev/.well-known/keybase.txt works

https://lightsoutgames.nolan.workers.dev/.well-known/matrix/server doesn't

Both files appear in my public/ directory.

My wrangler.toml site config:

[site]
bucket = "./public"
include = [
   "*",
   ".well-known/matrix/*",
]

Not sure if I need the additional .well-known path, or if it needs to be
relative to the bucket. I think I tried both.

Thanks.

[exvuma]

A custom 404 is caching:

curl -v https://lightsoutgames.nolan.workers.dev/.well-known/matrix/server  2>&1 | egrep -i "404|cache"
< HTTP/2 404
< cache-control: max-age=8640000
< cf-cache-status: HIT

Did you adjust the cache settings on getAssetFromKV ? Or adjust any other internals of workers-site/index.js? Can you try purging your Cloudflare cache?

[ndarilek]

No changes to index.js. I do eventually need to tweak the content-types
of files in .well-known, but I thought I'd get them serving first.

I ran rangler kv:namespace delete on both existing namespace IDs
(preview and deployed site) and verified that wrangler kv:namespace list is empty. Same result.

Thanks for the help so far.

[exvuma]

Sorry for the back and forth, we actually don't serve files without extensions. After speaking with the team, we agree the kv asset handler should though, so we are moving this issue to that repo.

In the meantime, you can work around this for your project by passing in a customer modifier to getAssetFromKV. Instead of using the default mapRequestToAsset which maps requests w/o extensions to an index.html, you'd want to have the request just be the root.

Code would look something like

const mapRequestToAsset = (request: Request) => {
  return request //  to NOT append index.html to files with no extensions
  } 

See more docs on this at https://github.com/cloudflare/kv-asset-handler#maprequesttoasset

[ndarilek]

Thanks. Where can I find the newly-opened issue? Poked around
https://github.com/cloudflare/kv-asset-handler/issues but nothing looked
immediately promising.

[ashleymichal]

@ndarilek I'm either going to file a new one today or transfer this one there. Need to decide if there is also work we want to do in Wrangler to include dotfiles.

[ndarilek]

Thanks! I'd like to subscribe, so if it's a new issue, please link it here.

[ashleymichal]

@ndarilek see https://github.com/cloudflare/kv-asset-handler/issues/71

[gabbifish]

Following up on the original issue, which is allowing .well-known to be uploaded by default:
Given that the include field is a whitelist that, if specified, overrides ALL upload ignore conditions and only permits the upload of whatever matches whitelist entries, we're stuck with instructing users to specify include = ["*"] or include = [".well-known", <all other directories specified in bucket>]. These solutions are workable but honestly not sastisfactory; since .well-known is an IETF standard, we should support it by default.

The problem with supporting it by default in our preexisting code is that the Ignore crate adopts the include rules that are consistent with cargo; that is, if any whitelist exists, the only files that are uploaded MUST match whatever patterns are in the whitelist. This leaves us in the same situation as before--asking users to specify every directory that gets uploaded.

This leaves us with other options to consider:

1. Introduce a hidden bool flag to [sites] in wrangler.toml that toggles the upload of hidden files. Potentially more ergonomic than specifying include = ["*"], plus this won't mess with users' preexisting include and exclude fields (conflicts will be resolved with standard gitignore conflict resolution).
2. Do nothing and just have users specify include = ["*"].

I could gravitate towards either of these options at this point. I don't really love either of them, though, because they simply just allow for the upload of hidden files. I'd prefer more granularity, where only .well-known is uploaded and other hidden files are ignored by default.

[ndarilek]

From an end user perspective, I don't particularly mind which option
you go with. It's been a couple weeks since I worked with this so my
memory of the specifics isn't great, but I remember it being very
difficult to notice that these files were being ignored to begin with.
It only occurred to me after doing the upload to try hitting them via
curl, and when that failed, that wrangler may not be uploading dotfiles.

If you'd just make it clearer that certain files aren't being uploaded,
I think that'd help immensely. Or, given that dotfiles are used to
contain sensitive information, make it an error to initiate an upload
without explicitly including or excluding them. Again, I don't mind
which, I just wish it didn't fail quietly, or so quietly that I have to
use --verbose or equivalent to spot it.

Thanks.

[gabbifish]

Thank you for the feedback, @ndarilek! This is super helpful. I can definitely add a warning message about hidden files being ignored (and not have this lumped in under verbose output).

[chris-erickson]

I'd second automatically uploading .well-known - I got bit by this as well, and everything about that directory really says _I know that this is a directory with some stuff I deliberately put here for deploy!_

On the second issue, files w/o an extension, how should I update this so that I serve only these files w/o extensions? I use Gatsby and I'm not exactly sure if I need the file name appending logic. Perhaps there's something a bit more precise than just removing the default behavior of Wrangler setups.

[ndarilek]

I'd like to second, or third/fourth/whatever,  including .well-known by
default as a fix for this issue. I've got a few sites I want to migrate,
and this is a big blocker since two include .well-known/matrix/server.
Right now these sit right at the sweet spot where they're mostly static
but need some dynamic behavior. I don't want to make them full-blown
SPAs, but at some point I'm going to want to do a bit of
header/cookie-based auth. Workers seems like a great fit, but I can't
lose Matrix server discovery support.

Thanks.

[ndarilek]

Here's a hacky potential workaround for some .well-known routes. In workers-site/index.js:

    if (url.pathname == "/.well-known/matrix/server")
      return new Response(
        '{"m.server": "matrix.my.domain:443"}',
        {
          headers: {
            "Access-Control-Allow-Origin": "*",
            "Content-Type": "application/json"
          }
        }
      )
    else
      return await getAssetFromKV(event, options)

Would love to continue serving that as a static file, but for now it seems to work. Naturally, tweak for your own .well-known routes/path matches.

[chris-erickson]

For future people, I ended up with this:

if (url.pathname == applePayPath) return new Response(applePayBody);

It’s not really great and adding a lot more of these could get onerous and error-prone. I hope the team allows some built-in mechanism for this someday.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ndarilek]

So this isn't getting fixed? This whole trend of projects marking issues
as stale automatically has always baffled me. Someone else is just going
to have this issue in a few more months unless we routinely poke it to
keep it open.

FWIW, the lack of this feature put me off of using Cloudflare Workers to
host a static site. Not trying to seem harsh, but if Cloudflare's tools
are just going to arbitrarily not upload files from a directory, that's
kind of not what I want from anything advertising itself as a static
site hosting tool, and it might be good to rework the branding. I'll
still use Workers to intercept requests where needed, but if I can't
host .well-known files then there are several needs I just can't meet.

[ashleymichal]

@ndarilek i'm going to update this issue to always upload .well-known files within the bucket directory. hopefully we can add it to the milestone for 1.11.0, which should go out in a few weeks.

Thank you for re-upping.

[nataliescottdavidson]

I think @gabbifish 's comment still stands. The overrides truly do override all the existing paths in the crate we are depending on to walk the directory. This behavior seems really weird, so I've asked a question on the repository page.

[nataliescottdavidson]

It looks like getting a fix for ignore::WalkBuilder not traversing directories recursively when overrides are given is unlikely. My understanding is that the maintainer of ripgrep isn't accepting changes on the include crate right now.. So, do we want to introduce a hidden bool flag to [sites] in wrangler.toml that toggles the upload of hidden files, or mark this as a wontfix?

[chris-erickson]

That would be a sad outcome 😔. We could make use of uploading invisibles to remove a bunch of really dumb hacks.

[exvuma]

Why not an explicit opt-in/opt-out field rather than toggling a boolean for all hidden directories? I'd like one sensical default behavior for what gets uploaded with the ability to opt in or out by directory pattern or file (e.g. include: [.well-known/*, .my-speicla-file]

[osdiab]

Just wanted to chime in that my company is also trying to use Cloudflare Worker Sites because it was marketed as a static site hosting service, but issues like this make it hard for me to continue advocating for it... Our bundle doesn't currently have anything sensitive so I'll do combination of the above hacks for now - but this is pretty rough.

[ispivey]

Next step: we'll evaluate just doing this ourselves natively without a dependency on the Walkdir/Glob crates, since our use case is rather narrow.