Dhall-haskell: Creating multiple file and directory structure using dhall

[tsutsarin]

How can I create multiple files and directories with names as dhall variables/result of dhall computation?

For example:

$ dhall <<< './create-files.dhall'
...creating files (no stdout)
$ tree
.
├── dir1
│   ├── file1.json
│   └── file2.json
├── dir2
│   ├── file3.json
│   └── file4.json
└── dir3

If this task can't be accomplished by dhall executable or it is not intended by dhall-lang, are there any alternatives/helpers/workarounds that can use dhall config for multiple file generation?

[f-f]

@tsutsarin Dhall won't perform similar side-effects itself (it won't perform any side effect, except for resolving the imports).

What you could do instead is to have Dhall generate a record that is structured in the same way as you would like the content to be structured, and then have another tool "apply" the configuration.

E.g. in your case you'd have a Dhall file like

{ dir1 =
  { file1 = "some content.."
  , file2 = "other content"
  }
, ...
}

What's your use case? (as in, why would you like to create multiple files?)

[tsutsarin]

@f-f, I have a project consisting of multiple files (yaml, json, hcl) and tools (Ansible, Terraform, etc) and entities (like a service name) that are interleaved throughout this configs.
I want to structure config in one central way, so changes from one place would affect all related tools, aka single source of truth.

What are your suggestions for kind of tool that will "apply" dhall generated configuration?
I could come up with following - generate data from dhall for Ansible to create required directory structure; use language-specific bindings to dhall and program it with side-effects (like dhall-eta with Java bindings).

Also, I think this should be documented, that dhall has no side-effects with file generation and some approaches recommended for this use case.

[joneshf]

We do this at work. We use dhall-to-json, dhall-to-text, and dhall-to-yaml to generate the appropriate file from Dhall, and a build system to orchestrate the commands. dhall-to-text is used for those file formats that don't have a direct binary (e.g. HCL, ini, conf). Hopefully these examples help.

HCL

Let's say you wanted to create this HCL file foo/bar.tf (from https://learn.hashicorp.com/terraform/getting-started/build):

provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "example" {
  ami = "ami-2757f631"
  instance_type = "t2.micro"
}

and you wanted to template out the AMI value.

You might create a template infrastructure.dhall:

  λ(ami : Text)
→ ''
  provider "aws" {
    region = "us-west-2"
  }

  resource "aws_instance" "example" {
    ami = ${ami}
    instance_type = "t2.micro"
  }
  ''

The template uses multi-line strings to template out the file we want, since there's no dhall-to-tf or dhall-to-hcl or something similar.

Then, you might have a foo/bar.tf.dhall (convention being that if you drop the .dhall, you know what format you're outputting):

let template = ./../infrastructure.dhall in template "ami-2757f631"

To orchestrate this in a way that scales easily, you might use Make:

%.tf: %.tf.dhall infrastructure.dhall
    dhall-to-text <<< ./$< > $@

When you call make foo/bar.tf, you should get something like:

provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "example" {
  ami = ami-2757f631
  instance_type = "t2.micro"
}

YAML

YAML is a little easier, because you don't have to template in strings, you can use Dhall code and retain types until the end.

Let's say you wanted to create this playbook baz/qux/gar.yaml (simplified a bit from https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html#playbook-language-example):

- hosts: webservers
  vars:
    http_port: 80
    max_clients: 200
  remote_user: root
  tasks:
  - name: ensure apache is at the latest version
    yum:
      name: httpd
      state: latest
  handlers:
    - name: restart apache
      service:
        name: httpd
        state: restarted

and you want to template out the hosts and max_clients values.

You might create a template for the playbook using plain old Dhall playbook.dhall:

  λ(config : { hosts : Text, max-clients : Natural })
→ [ { hosts =
        config.hosts
    , vars =
        { http_port = 80, max_clients = config.max-clients }
    , remote_user =
        "root"
    , tasks =
        [ { name =
              "ensure apache is at the latest version"
          , yum =
              { name = "httpd", state = "latest" }
          }
        ]
    , handlers =
        [ { name =
              "restart apache"
          , service =
              { name = "httpd", state = "restarted" }
          }
        ]
    }
  ]

Since we can use dhall-to-yaml, we can use regular old Dhall types like lists and records.

Then, you might have baz/qux/gar.yaml.dhall (again, following the convention that dropping the .dhall tells you the file type):

    let template = ./../../playbook.dhall

in  template { hosts = "webservers", max-clients = 200 }

Finally, you might again use Make to orchestrate things:

%.yaml: %.yaml.dhall playbook.dhall
    dhall-to-yaml <<< ./$< > $@

When you call make baz/qux/gar.yaml, you should get something like:

- handlers:
  - service:
      state: restarted
      name: httpd
    name: restart apache
  hosts: webservers
  tasks:
  - name: ensure apache is at the latest version
    yum:
      state: latest
      name: httpd
  remote_user: root
  vars:
    http_port: 80
    max_clients: 200

Takeaways/Suggestions

• You can use whatever naming conventions you want. It doesn't have to be foo.json.dhall or whatever.
• You can template out anything with dhall-to-text. We use it at work to template all sorts of config files, HTML files, JS files, whatever we need that doesn't have a dhall-to-$FORMAT executable.
• You can use whatever directory structure you need. Where the files are located is mostly irrelevant to a build system. The examples have the source dhall files sitting next to their destination, but you can structure the files anyway you want.
• You can use whatever build system you want. If you want to use shell scripts, that might work for a handful of files. At some point, shell scripts can't keep up and you might think about make. These examples are intentionally contrived. Make can keep up with them because they're simple. At some point Make can't keep up. Shake is a great alternative for mitigating complexity as build systems grow.

[tsutsarin]

@joneshf Awesome! Thanks for your detailed response.

Do you have any open-source work using dhall that I can look as examples?

[joneshf]

I believe I do, but I'm not sure where they live these days. I'll try to find them and circle back.

[Gabriel439]

I think that it would make sense to have a dhall-fs tool that took nested records of strings and turned them into equivalent files. In other words, for a Dhall expression like this:

{ dir1 =
    { `file1.json` =
        ''
        [ 1, true ]
        ''
    , `file2.json` =
        ''
        { "foo": 1
        , "bar": 2
        }
        ''
    }
}

... it would create a directory named dir1 with two files named file1.json and files2.json with those contents.

However, if we go down that route then that would imply implementing dhall-to-json and dhall-to-yaml support within the Dhall language (rather than as an out-of-band executable) in order to avoid string templating errors. This is something that I've already suggested in a different context here:

https://github.com/dhall-lang/dhall-lang/issues/336#issuecomment-451503204

If you had that, then you could do this:

let JSON =
      https://raw.githubusercontent.com/dhall-lang/dhall-lang/gabriel/json_pure_dhall/Prelude/JSON/package.dhall

in  { dir1 =
        { `file1.json` =
            JSON.render (JSON.array [ JSON.number 1.0, JSON.bool True ])
        , `file2.json` =
            JSON.render
            ( JSON.object
              [ { mapKey = "foo", mapValue = JSON.number 1.0 }
              , { mapKey = "bar", mapValue = JSON.number 2.0 }
              ]
            )
        }
    }

[tsutsarin]

@Gabriel439 that's what I initially thought about when created issue.

[Gabriel439]

Just an update on this. I do plan to implement this, but I would like to first finish implementing dhall-to-json and dhall-to-yaml in pure Dhall so that this is more useful out-of-the-box

[Nadrieril]

I would generate all the data in one json output and use a small script that uses jq to put everything in the right file. Like:

dhall-to-json ./file.dhall | jq .dir1.file1 > dir1/file1.json
dhall-to-json ./file.dhall | jq .dir1.file2 > dir1/file2.json

Maybe with a bash loop if there are a lot of files, or a Makefile. Since yaml is a superset of json, this works for yaml files too. If one of the files needs to get a text output instead of a json one, just use jq -r.
I'm not a fan of a dhall-fs tool, because it wouldn't be clear where to draw the boundary between what defines the desired file structure and what should be the content. Having to render to json from inside dhall seems to partially defeat the purpose of dhall.

[timbertson-zd]

I would like to first finish implementing dhall-to-json and dhall-to-yaml in pure Dhall so that this is more useful out-of-the-box

I heard that the yaml part is done as of a recent release, is json also ready? I'm pretty keen to use dhall to centralise the logic for template/generating a number of tedious yaml/text files at once.

The idea is to reuse some central dhall functions from a number of independent repos, so having a single tool like dhall-fs that can do this with a oneliner is important for bootstrapping (otherwise you have to replicate the generation code/script in every repo that uses it).

[Gabriel439]

@timbertson-zd: Yes, rendering JSON and YAML in pure Dhall are both ready, so this is viable now

[Gabriel439]

Also, I'm going to move this issue to the dhall-haskell repository since the plan I have is to implement this as a subcommand of the dhall executable (e.g. dhall fs or something like that)