Flynn: Replace Procfile with Flynnfile

[lmars]

We currently use a Procfile to declare process types, but this only supports declaring how to start the job, whereas process types are more complex than that.

We should replace this with a Flynnfile which can be parsed by the receiver and supports declaring all aspects of a process type (e.g. mounted volumes, ports, services, healthchecks etc.).

The Flynnfile could also declare pre / post deploy hooks once supported.

[lmars]

We should ensure that we keep backwards compatibility.

[thomasfedb]

I would be careful about removing the Procfile altogether, unless you can get foreman to support your format.

[lmars]

@thomasfedb

I would be careful about removing the Procfile altogether, unless you can get foreman to support your format.

Flynn doesn't use Foreman, it parses the Procfile directly and extracts the process types (see here), so by "Replace Procfile" I mean "Replace the file which Flynn uses to define processes". Users can still have a Procfile in their project for use locally (which will be a fallback), but by default Flynn will look for a Flynnfile.

I don't think trying to redefine the Procfile format to support Flynn is going to be useful to a lot of people that use Procfiles (I'm pretty sure the current format is simply TYPE: COMMAND).

[thomasfedb]

Right, so it was more hijacking to begin with anyway.

[lmars]

The Flynnfile could have a packages section which uses heroku-buildpack-apt to install packages using apt-get.

[geovanisouza92]

@lmars I'd recent problems with heroku-buildpack-apt on flynn buildpack caching, and I think that one session exclusively for this can be nice, although I feel that this special case is not so special.

Anyway, creating another file format can be nice to fit with flynn ecosystem, but could detour flynn of nice existing tools like Foreman/Honcho or even Docker Compose, that I particularly think that is not so awesome substitute for Fig ('cause is just a rebrand and rewrite of the tool), without any additions.

So I think it may be nice to have a new file format to configure everything in one place, but I worry that this new format can accumulate so many others like Procfile, [fig | docker-compose].yml, terraform.yml , etc, etc, etc.

[lmars]

From @thomasfedb in #1503:

The Flynnfile should allow a developer to specify a list of environment variables that are required at each of buildtime and runtime.

Flynn should prompt the user to specify these variables before attempting to build and/or run the application.

[aeneasr]

Here are some example for application manifests:

• http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#optional-attributes
• https://blog.heroku.com/archives/2014/5/22/introducing_the_app_json_application_manifest

Also, I'm working on a manifest for gitdeploy.io:

• https://github.com/ory-am/gitdeploy/blob/master/GITDEPLOY-PROPOSAL.md

[lmars]

We should consider adding this soon to help users who want to do more than just have a routable web service (e.g. see this user in IRC who wants to run an SMTP service).

Some example formats:

TOML

[processes.web]
command     = "start web"
service     = "my-app-web"
port        = 8080
healthcheck = { type = "http", path = "/ping", match = "pong" }

[resources]
postgres = true
redis    = true

YAML

processes:
  web:
    command: "start web"
    service: my-app-web
    port:    8080
    healthcheck:
      type:  "http"
      path:  "/ping"
      match: "pong"
resources:
  postgres: true
  redis:    true

JSON

{
  "processes": {
    "web": {
      "command": "start web",
      "service": "my-app-web",
      "port":    8080,
      "healthcheck": {
        "type":  "http",
        "path":  "/ping",
        "match": "pong"
      }
    }
  },
  "resources": {
    "postgres": true,
    "redis":    true,
  }
}

Lua

processes = {
  web = {
    command = "start web"
    service = "my-app-web"
    port    = 8080
    healthcheck = {
      type = "http"
      path = "ping"
      match = "pong"
    }
  }
}

resources = {
  postgres = true
  redis    = true
}

I included Lua as it is a scripting language and could give more flexibility, for example exposing functions to declare complex dependencies more concisely.

Just some thoughts to get the discussion going.

[josephglanville]

I think TOML would be great. Lua might err on the side of a bit too powerful.
If it can be easily executed in a sandbox though maybe not a bad idea, just concerned about security in multi-tenant environments (when we get there).

[temujin9]

-1 for JSON: definitely not user friendly. I abstract away from it every chance I get.

TOML or YAML could both work.

I'm actually quite fond of scriptable interfaces, especially at this level. I also see the security concerns, however. Not as sure about Lua specifically, mostly due to unfamiliarity. I have no perfect suggestions for replacement, and could cope with learning / pushing Lua at developers.

[thomasfedb]

I'd go for more expressive power if possible, mostly because I want to be able to use my Flynnfile to hook up a few things, to avoid magical numbers/strings.

In particular, I'd like a way to choose my own ENV variables for PG info, something like:

processes:
  web:
    cmd: "unicorn_rails ..."
    env:
      THREADS: 4
      DB_HOST: resources.pg1.hostname
      DB_USER: resources.pg1.username
      DB_PASS: resources.pg1.password

...

resources:
  pg1:
    type: "postgres"

[aeneasr]

I'm sorry for being so blunt here, but YAML is a bad idea. I don't want my environment to fail because I used tabs instead of spaces (this happens to all of the internet). YAML is a nice thing if you have 5-10 LOCs but for application critical stuff it should not really be an option. I know that many services are using it but it's the wrong decision imho.

I'd rather write JSON which by the way works wonderful with npm, composer and all those other package managers, than debug my code because the IDE replaced spaces with tabs or similar crazyness.

It's still developers and DevOps using Flynn and they should be comfortable with reading and writing JSON. LUA looks like a valid option to me too.

[titanous]

Don't worry, we won't be using YAML and JSON will very likely be used by the API.

[temujin9]

JSON is a good human-readable interface language, and I'm not saying don't use it. It is, however, a godawful annoying thing to have to write regularly, and as the guy spearheading Flynn at my office, I will have to write whatever we choose here regularly.

A JSON interface and a script interface (even if it just outputs JSON to the other interface) would be fine. I'll pick up whatever language you pick: I'm vaguely familiar with Lua from your Tupfile setup, and it looks fine. I'd only ever use the script interface myself, but having the other (and permissions on whether the script interface is available, and what it can access, etc.) makes sense as a compromise for security concerns. But pretty please: don't make me type lots of JSON directly.

(Also: Procfile is a well known interface, and seems like it would be simple to keep as a backup option. That makes migration from Heroku to Flynn much easier, not to mention supporting my existing clients without code change.)

[titanous]

Understood. We won't be getting rid of Procfile support, and we won't make people hand-write JSON.

[spinus]

@temujin9 @arekkas @titanous but you know that JSON is subset of YAML, right :-)? If you implement YAML, JSON is already there.

[aeneasr]

Yes I know that, but YAML has various flaws which are well-known and well-documented and make it inferior to JSON imho.

[titanous]

Let's leave the discussion of YAML alone for now. I think all of the viewpoints have been represented on this thread and we'll evaluate when we move forward on the Flynnfile specification.

[titanous]

My current leaning on this is to use JSON5 for the end-user config format. It is 1:1 compatible with JSON but has a lot of the niceties of the YAML without going overboard on whitespace sensitivity, etc. The format is just the ES5 object format, so standard JS highlighters and syntax awareness will work in editors. Any future scripting system can be layered on top and emit JSON.

[philiplb]

Maybe this would also be a good place for the GC setting instead of the Meta Data?

[titanous]

@philiplb Yep.

[titanous]

I have implemented a Go JSON5 decoder: https://github.com/flynn/json5

[titanous]

I believe that Jsonnet will be a good long-term solution for dynamic generation of Flynnfiles. I'm currently not sure if we're going to jump straight to it or use JSON5 as an intermediary step.

[kckrinke]

Just want to mention that heroku-buildpack-apt does not actually install .deb files. All that it does do is download them via apt-get -d from the apt mirror (or URL given) and then extract to /app/.apt/... using dpkg -x. Clearly this is not installation in the debian sense and very misleading when packages are indicated as installed and yet the actual resources are not configured or otherwise not in fully working order.

It also doesn't cleanup the cached files at all so if you remove a package from Aptfile and deploy; you're still going to get that package mysteriously "installed" again with the next push.

Again, just adding my $0.02 note on the topic of heroku-buildpack-apt as a formal solution to anything.

[temujin9]

With the advent of a Docker interface, heroku-buildpack-apt is probably best deprecated in favor of just using a Dockerfile.

It might be cool if I could use the Dockerfile to set up the build environment, and then use conventional buildpacks to run a further build. Not sure how easy that would be, in practice.

[evantahler]

Since we are talking both about process start/stop and monitoring, I would like to add the age-old monit file format, which the monit tool has been using for years to do exactly this. It has clear directives for starting and stopping the process, _how_ to monitor it (proicess, fork, pidfile, etc), and alerting thresholds and checks.

Lots of examples here: https://mmonit.com/wiki/Monit/ConfigurationExamples

[titanous]

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.