Toml: `nil` or `null` values

[benolee]

(moved discussion from #11)

It seems like nil or null values must be allowed. For example,

# is this equivalent to {"foo":null,"bar":{"baz":null}} or {} ?
[foo]
[bar.baz]

in this case, it seems like it would make sense to be able to set them with the normal key = value syntax. Here are some alternatives for thought:

key = nil
key = null
key = # empty value ala bash

[aaronblohowiak]

you don't need nil or null, just leave out that assignment.

[mojombo]

Yeah, I'm not convinced of the usefulness of nil. TOML is intended for configuration, at which point @aaronblohowiak is right: just leave it out. I'm open to use cases and further convincing, though.

[benolee]

no further arguments here. I think it might be important to implementers to know if an empty "key group" should result in a key with no value (ie. nil or null depending on the language) or no key at all

[mojombo]

@benolee Ah, good question. I'm going to say it should be an empty hash table.

[Iron-E]

Yeah, I'm not convinced of the usefulness of nil. TOML is intended for configuration, at which point @aaronblohowiak is right: just leave it out. I'm open to use cases and further convincing, though.

It's possible I'm using TOML for the wrong thing here, but I was going to use TOML as a bridge for query languages in my CLInvoice crate since I already have toml as a dependency for parsing user configurations, and it is a dead-simple markup language which I believe users would be able to learn without much trouble.

The reason for this is that CLInvoice is designed to be able to handle any permanent storage facility, whether or not it actually has a Structured Query Language of its own. Because of this, I needed to create a unified query 'language' based on the model and what operations made sense for it. Writing an adapter for CLInvoice explicitly provides support for this query 'language'.

Querying in CLInvoice is built on the backbone of the Match type, which can accept a list of types for HasNone, HasAny, or HasAll operations. Some types give Match values which may be None. For example, querying an InvoiceDate requires specifying an Option<chrono::DateTIme<chrono::Local>> for its paid field. If I were using toml, that means that TOML would have to be able to accept a list containing None/Nil _and_/or a concrete date, like so:

# rest of `InvoiceDate` query left out for simplicity's sake

[paid]
condition = 'HasAny'
value = [
  2020-04-01T03:00:00Z,
  None
]

The above would be quivalent to the English statement "match InvoiceDates that are either unpaid or were paid on 2020/04/01 at 3:00 UTC."

Obviously, for Match operations such as EqualTo that only accept one value, just leaving the value out is good enough to imply a None. But in list types there isn't a good way to specify a None in a given position.

Right now I'm thinking of switching to YAML I've switched to YAML, but YAML has some of its own issues (such as its fear of tabs and embedded types using way too much indentation). Accepting None in TOML would be very handy for the odd case such as this!

---

I had considered nom but writing a DSL for this project seems like it would lead to less ROI than serializing / deserializing a model + helpful errors in this case.

[eksortso]

@Iron-E Although there won't be a None or Nil added to TOML (as far as I can see), you do have an option to use within the TOML syntax that would fit the bill. If you would never need to represent a hashmap value (and few relational database columns in this world store whole hashmaps), you could use an empty inline table to express a NULL value in your value list. For example, the sample you provided could be changed to look like this:

[paid]
condition = 'HasAny'
value = [
  2020-04-01T03:00:00Z,
  {}  # This represents NULL in the value list.
]

It's not a literal null, but it would do the trick. You could also use any value that isn't a datetime, like false, if you'd rather have something more lightweight than a table here.

In any case, you would need to handle non-datetime values gracefully, but you would need to do that with any hypothetical NULL anyway.

[danhje]

Yeah, I'm not convinced of the usefulness of nil. TOML is intended for configuration, at which point @aaronblohowiak is right: just leave it out. I'm open to use cases and further convincing, though.

Here's a use case: You want to read config from a TOML file, interpreting values as defaults, but you want environment variables with the same name to be able to override the defaults. An empty field thus indicates to your code that it must look in the environment, and it indicates to the user that an environment variable must be set.

[config]
my_db_host = '127.0.0.1'
my_db_user = 'user'
my_db_pass

[eksortso]

@danhje Based on what you wrote, this use case has no real "defaults," a.k.a. values that are used in the absence of all other settings. Everything is set by environment variables first and foremost, followed by the settings in the TOML configuration file. Any missing setting must certainly lead to an error.

What this use case needs is just documentation. No value, not even an explicit null, would indicate that my_db_pass must be assigned by an environment variable. Worse, users may consider an explicit null to be a legitimate value for a password. An explicit null is equivalent to a missing setting, so why use an explicit null? In any case, you must explain your intention for password assignment, which is what comments are for. Or external documentation, if you don't want configuration comments.

Here's a pattern for this use case. This configuration would come with the installation for the users to fill out. All equivalent environment variable settings appear next to the configuration setting.

[config]
# Environment variable settings override the values here.

# Database host (Env: MY_DB_HOST)
my_db_host = '127.0.0.1'

# Database user account (Env: MY_DB_USER)
my_db_user = 'user'

# Database password cannot be set here.
# Required Env: MY_DB_PASS

[danhje]

Nobody reads documentation, and config comments are ugly. But more importantly, in my use case it’s not just about signaling to the user what variables are expected, I also want access to “empty” variables in code.

Consider how docker-compose interprets empty variables to mean that the variable should be mirrored from the host’s environment. In that case, leaving out the variable or using a comment isn’t an option. Docker composes uses yaml, and my understand is that leaving the value out really just results in an empty string, not a null, which I suppose is fine for my use case.

Here’s my use case, in a little bit more detail:

I want to create a variable / secret managing library for Python. The library is meant to be used for app development in large teams, where it’s difficult for each developer to keep track of all the environment variables that have to be set in order for the code to work. I want the users of my library to be able to centrally manage all these variables in a config file that could either be included or excluded from version control. I want the library to be able to give a friendly warning to a developer if a variable doesn’t have a default and isn’t found in the environment. So if you as a developer pulls down some commit where a fellow developer, unbeknownst to you, have introduced new variables that need to be set, you’ll find out about it right away rather than when the app fails unexpectedly, possible with a not so helpful error.

When working in interactive mode, I also want tab completion to present you with all variables from the config file, both set and unset.

I could force the users of my library to list expected variables in code rather than a config file, including default values, but this breaks the separation of config and code. In a project with hundreds of code files it also makes it harder to track down those expected variables, and it’s hard to enforce a central location for them.

If there’s a clever solution I haven’t thought of, I’d love to hear about it. But I think I’ll just use yaml instead. Which is a shame, since I was hoping to allow using pyproject.toml.

[albertotb]

What about arrays such as

key = [1, 2, 3, null, 5]

I also find it a little weird that

key = [1, 2, 3, , 5]

is parsed just fine, ignoring the extra ",". This also parses fine

key = [1, 2, 3, "", 5]

[marzer]

@albertotb if that second example parses OK in some implementation you're using you should file a bug report because it should not

[albertotb]

@albertotb if that second example parses OK in some implementation you're using you should file a bug report because it should not

It seems it was fixed in the latest Python implementation (0.10.2)

[nyuszika7h]

Nobody reads documentation, and config comments are ugly. But more importantly, in my use case it’s not just about signaling to the user what variables are expected, I also want access to “empty” variables in code.

Can't you just use config.get('value') which will automatically fall back to None? Or does your use case require differentiating between missing and null values?