Codimd: 'urlpath' not working for webpack-generated components

[Joe136]

When I use a urlpath (e.g. http://www.example.org/hackmd/) I would set this config.json:

{
"production": {
"domain": "www.example.org",
"urlpath": "hackmd"
}
}

(respectively public/js/config.js)

This will generate this result public/views/build/cover-scripts.ejs:

The browser will then try to download http://www.example.org/build/7.cover.9ca8b804729321d2fe23.js instead of http://www.example.org/hackmd/build/7.cover.9ca8b804729321d2fe23.js (with 'hackmd').

webpack.production.js does not import config.json AND does not prefix publicPath with urlpath.

[jackycute]

Hi @Joe136,
This issue has been confirmed.
Thanks for report.

[Yukaii]

Fix in https://github.com/hackmdio/hackmd/commit/62c703baa18d5455261bf1dc18e218914c383f12

[Joe136]

Thanks,
you have to add the baseUrl line to webpack.config.js and webpack.production.js (two times), too; then it works.
Best regards Joe

[jackycute]

@Joe136 Thanks!
We only need baseUrl appear when generating templates, not all outputs.
Fixed in 8095f8cc983bbed532db7eee9f7f5a64e910b731

[ionphractal]

Hi @jackycute .
This problem is in current master (1ca39d9c8e47ae2ce5a524504beb0aaed221a97c), too.
I think it came back with this commit: https://github.com/hackmdio/hackmd/commit/c2a8911b9c7872ea0d085020aa17e82162130f3d#diff-575abd8291ec3d1aa52d2d348cb50bc0R1
However I haven't build it with an earlier commit to verify.

But what happens now if you run bin/setup (which generates a default config.json) and then npm run build is that webpack sees the import statement and compiles that config file in. Hence it is not read at runtime nor are ENV variables honored (because there were no ENV variables at compile time).
This causes an error in the console loading the page (connection error localhost/me) and history links to localhost/:noteId instead of the domain.

In my opinion the configuration should be passed to the template when you render the front end and should be used throughout the different components subsequently.
I tried to find a way for this myself but as I'm still a beginner with JS and the front end code looks quite complicated to me, I wasn't successful.

Could you please check this issue?

[jackycute]

Hi @ionphractal
I just test about this issue but it didn't show up.
This might be something wrong with your config.
Please paste your config.json or your ENV vars (if you're using them).
BTW, if you're using the ENV vars, you should also pass them when you do npm run build (you could do it, if you can't please use config.json). We'll need configs to compile frontend, too.

[ionphractal]

BTW, if you're using the ENV vars, you should also pass them when you do npm run build (you could do it, if you can't please use config.json). We'll need configs to compile frontend, too.

I guess that's exactly my problem as I am not setting any configuration before I create a container.
Requiring ENV variables or any other configuration before runtime breaks the whole concept of having images / execution droplets. Or else I would have to stage (~=build) the whole app over and over again if I want to create more sites with a single container - or even if I just want to add a new authentication method. That is not much fun to me as it already compiles round about 10 minutes on my server machine, causing a lot of workload...

Anyway, here's my configuration and how I build the docker image.
As you can see, configuration is done at Docker compose level or (which is because I came from older hackmd that didn't have much configuration by ENV) by injecting the config file into the container via volume. Currently neither one works for me.

config.json:

{
  "production": {
    "debug": "true",
    "port": 3000,
    "domain": "hack.dev.local",
    "alloworigin": ["localhost", "hack.dev.local"],
    "protocolusessl": "true",
    "db": {
      "username": "hackmd",
      "password": "redacted",
      "database": "hackmd",
      "host": "db.local",
      "port": "3306",
      "dialect": "mysql"
    },
    "github": {
      "clientID": "redacted",
      "clientSecret": "redacted"
    }
  }
}

I guess I don't need this anymore, yet it is still present in bin/setup (the comment at the end).
config.js:

var domain = 'hack.dev.local'
var urlpath = ''
var debug = true

var GOOGLE_API_KEY = ''
var GOOGLE_CLIENT_ID = ''
var DROPBOX_APP_KEY = ''

Dockerfile:

FROM node:alpine
LABEL maintainer "Michael Both <root>"

# this branch/tag will be checked out later
ENV HACKMD_VERSION="0.5.0" \
  HACKMD_SRC_URL="https://github.com/hackmdio/hackmd.git" \
  HACKMD_COMMIT="1ca39d9c8e47ae2ce5a524504beb0aaed221a97c"
# Commit #1116, 17.02.2017

# install necessary packages
#   bash: for scripting
#   gettext: for substituing env variables in configs
#   git: required for nodejs to be able to download packages
#   python: requirement for npm packages (gyp build)
#   build-base: requirement for npm packages
RUN apk update \
 && apk add bash gettext supervisor \
 && apk add --virtual .build-deps build-base git python \
 && git clone ${HACKMD_SRC_URL} /app/hackmd \
 && cd /app/hackmd \
 && git fetch \
 && git checkout ${HACKMD_COMMIT}

RUN cd /app/hackmd \
 && bash bin/setup \
 && npm update \
 && npm run build \
 && npm prune --production \
 && chown -R nobody:nogroup /app/hackmd/

# cleanup
RUN apk del .build-deps \
 && rm -rf /var/cache/apk/* \
 && rm /app/hackmd/.sequelizerc
# I generate .sequelizerc at runtime

ADD config/ /app/hackmd
ADD bin/hackmd.sh /bin/
ADD bin/container /opt/bin/
ADD bin/init-hackmd.sh /opt/bin/
ADD supervisor/ /etc/supervisor.d/

CMD ["/opt/bin/container","start"]

docker-compose.yml:

version: '2'
services:
  db:
    restart: unless-stopped
    image: mariadb:10.1
    labels:
      SERVICE_IGNORE: "true"
    environment:
      MYSQL_ROOT_PASSWORD: redacted
      MYSQL_USER: hackmd
      MYSQL_PASSWORD: redacted
      MYSQL_DATABASE: hackmd
    volumes:
      - /storage/docker/volumes/hackmd/db:/var/lib/mysql

  hackmd:
    depends_on:
      - db
    restart: unless-stopped
    image: ionphractal/hackmd:0.5-0
    hostname: hack.dev.local
    environment:
      NODE_ENV: "production"
      DEBUG: "true"
      HMD_DOMAIN: "hack.dev.local"
      HMD_PROTOCOL_USESSL: "true"
      HMD_ALLOW_ORIGIN: "hack.dev.local"
      HMD_USECDN: "false"
      HMD_ALLOW_ANONYMOUS: "true"
      HMD_DEFAULT_PERMISSION: "limited"
      HMD_DB_URL: "mysql://hackmd:[email protected]:3306/hackmd"
      HMD_IMAGE_UPLOAD_TYPE: "filesystem"
      HMD_GITHUB_CLIENTID: "redacted"
      HMD_GITHUB_CLIENTSECRET: "redacted"
      HMD_EMAIL: "true"
      HMD_ALLOW_EMAIL_REGISTER: "true"
    links:
      - db:db.local
    volumes:
      - /storage/docker/volumes/hackmd/config/config.json:/app/hackmd/config.json
      - /storage/docker/volumes/hackmd/config/config.js:/app/hackmd/public/js/config.js
      - /storage/docker/volumes/hackmd/uploads:/app/hackmd/public/uploads
    ports:
      - "172.17.0.1::3000"

[ionphractal]

@jackycute btw thanks for the fast response :)

[jackycute]

@ionphractal you should consider using docker-hackmd instead
https://github.com/hackmdio/docker-hackmd
Our contributors have done tons of works to make it simpler.

The compile of the front-end package is inevitable.
Since we're using ajax and the endpoint might contain urlpath which you'll never parse it correctly only from the browser location.
And why are you trying to run multiple sites on one container?
Our resources path are sent from the server, so you won't able to do that.
It's very unsafe to trust all data from the client-side.

[jackycute]

You only need to re-run the npm run build when you change below configs (because frontend use them):

domain
urlpath
debug
google.apiKey
google.clientID
dropbox.appKey

But you still should set them or pass them before the first compile.

[ionphractal]

@jackycute I saw that repo, but I'd have to mess with it anyway to fit it into my environment. And I'm not trying to push multiples sites into one container but based on one image I want to spawn multiple container with different configs. Of couse you have to compile the frontend but this way you have to create one image for every site. Besides, one hackmd docker image already takes up about or over 1GB of disk space, depending on the configuration. I would love to save those resources.

I don't quite understand why passing the configuration (of course only those parts the client really needs) freshly from the ENV to the frontend/client in the rendered output would be less secure than pre-building the the configuration and then passing it to the client (as it is right now). The server should not give the client sensitive information like API keys in any case. But what's wrong with giving it the right domain/path and debug "on the fly"?

[jackycute]

@ionphractal We already pushed the boundary of the smallest image we could make (so as you did) and 1GB sounds reasonable to current average disk storage I think.
So our first thinking is - we won't need to change those configs (domain, urlpath) that often.

There is nothing wrong to pass the domain and path "on the fly", but it's better to compile them into the code because you won't need to pass these variables to templates or js files (dirty way) on every request.
And since we have dropped use of config.js (will release in 0.6.0), the only setting file is config.json which is easier to manage them now.

But I do understand you're trying to avoid recompile for every image, but static resources always faster than dynamic generated content and easier to be cached (and may have lower possibility to be faked).
In your situation, I would suggest to put a script with npm run build and restart server into the image and re-run it whenever you change the configs then maybe it's possible to reuse the image.

[SISheogorath]

Wait,

You only need to re-run the npm run build when you change below configs (because frontend use them)

we have to rerun npm run build in docker entrypoint if we use another URL than localhost/default settings from build? O.o

That's not a good idea. Or did I misunderstood something?

[jackycute]

@SISheogorath No, totally correct.
You'll need to re-run npm run build if you change above mentioned configs in the next release.
Because we dropped the config.js which was copied to container on the build before.
Is this bad? @Yukaii @bananaappletw

[jackycute]

There is a workaround that I could think of.
Copy out frontend related configs on the server startup, and the template refer to it.

[ionphractal]

@jackycute If you want to have hackmd cloud compatible (-> honor 12 factor model), the current way is a bad idea (violating "III. Config").
And I highly doubt that adding domain/URL reference at runtime is noticeably slower than this static approach - except it came from a slow/congested media like a database. Loading it startup is more than welcome for me! 👍

[jackycute]

@ionphractal @SISheogorath
I've tried to lure this issue in 448b0061945b65dc36c08eaf6752060600d5895d

[ionphractal]

@jackycute Cool! Thanks! Will try this as soon as I can and report back.

[ionphractal]

@jackycute Just tried the latest commit. Works great!
Thanks a lot!
👍

[haccht]

Since upgrading CodiMD to webpack 4 in #932, baseUrl property was removed and so this problem has come back.