Json: Allow comments in JSON

[selaux]

Hi we are trying to use serde-json in a C++/Rust. The C++ library we use allows for comments in the JSON files. We would like to have the same in rust. Could that be a PR for this library? Or is that not possible because it's not part of the Spec? How would we go about writing our own parser without copying code from serde-json?

[dtolnay]

I would strongly prefer not to support this because it is not part of the spec. Hjson may be better for your use case. It should be able to parse "JSON with comments."

[dtolnay]

Let me know if Hjson does not work for your use case and we can reconsider.

[nicoburns]

I'd like to make a case for supporting // line comments only in serde_json:

• Rightly or wrongly, JSON is frequently used for shared config files in multi-language environments. And for this use case, allowing comments in the file is often desirable. Notably there isn't really a good alternative to JSON+comments for this use case. TOML doesn't have good cross-language support, and YAML is considerably more complex.
• The other parts of "lax JSON" specs such as JSON5 or HJSON are often not desirable in this situation. For example, if a config file is missing quotes around it's key, the correct solution is to modify the config file to include the quotes.
• The same is not true for comments, which it is often desriable to keep for documentation purposes.
• Adding support for // single-line comments is a very minor change to the parser, and is unlikely to meaningfully affect parsing performance.
• serde_json is much more actively developed than other projects such as Hjson (For example, at the time of writing, Hjson hasn't yet been updated to be compatible with serde 0.9. serde_json also has ). Has much more attention paid to it for things like performance and semantics around serializing enums, which it would be a shame to lose just for comment support.

TL;DR: I this use case is very common, has clear motivation, and has only a small cost.

[oli-obk]

One big issue is that roundtripping won't work, because comments can't be deserialized in any useful manner.

this use case is very common

That's subjective. I've never seen commented json.

and has only a small cost.

If the cost is small enough (both in performance degradation and code complexity), I can live with it. Maybe it could be a feature.

[erickt]

Hello @nicoburns. I'm also not in favor of this. This feature was explicitly excluded from JSON, so it opens up our users to incompatibility problems across JSON parsers, which also feels like a show stopper to me. I think it'd be better to encourage development of Hjson. It should be pretty simple to upgrade it to serde 0.9 and sync with serde_json's enum semantics.

TOML doesn't have good cross-language support

It has pretty good coverage to me. Is there a language missing you want to support?

That's subjective. I've never seen commented json.

Me neither. The only case I can think of is when people are using a JavaScript to store a config.

[dtolnay]

If your input is not JSON, you should not expect to use a JSON parser to parse it.

If you are concerned about the other parts of "lax JSON" specs, let's add a feature to the hjson crate to opt out of those parts. Commented JSON is HJSON so it may make sense for the hjson crate to support that mode. Commented JSON is not JSON so we won't.

this use case is very common

I think if this were the case, the hjson crate would be more actively maintained.

[LinusCDE]

For anyone interested, to solve this issue for myself, I made this (probably rather ugly) function to turn jsonc (very rare to find but basically json with comments) into regular json.

It's really basic, not fancy and removes single line as well as block/multiline comments and supports nesting which is my personal preference.

So if you need comments in your json, just pass it through this function first before using serde_json on it.

Code to turn JSONC (loosly) into JSON

/// Takes a string of jsonc content and returns a comment free version
/// which should parse fine as regular json.
/// Nested block comments are supported.
/// preserve_locations will replace most comments with spaces, so that JSON parsing
/// errors should point to the right location.
pub fn strip_jsonc_comments(jsonc_input: &str, preserve_locations: bool) -> String {
    let mut json_output = String::new();

    let mut block_comment_depth: u8 = 0;
    let mut is_in_string: bool = false; // Comments cannot be in strings

    for line in jsonc_input.split('\n') {
        let mut last_char: Option<char> = None;
        for cur_char in line.chars() {
            // Check whether we're in a string
            if block_comment_depth == 0 && last_char != Some('\\') && cur_char == '"' {
                is_in_string = !is_in_string;
            }

            // Check for line comment start
            if !is_in_string && last_char == Some('/') && cur_char == '/' {
                last_char = None;
                if preserve_locations {
                    json_output.push_str("  ");
                }
                break; // Stop outputting or parsing this line
            }
            // Check for block comment start
            if !is_in_string && last_char == Some('/') && cur_char == '*' {
                block_comment_depth += 1;
                last_char = None;
                if preserve_locations {
                    json_output.push_str("  ");
                }
            // Check for block comment end
            } else if !is_in_string && last_char == Some('*') && cur_char == '/' {
                if block_comment_depth > 0 {
                    block_comment_depth -= 1;
                }
                last_char = None;
                if preserve_locations {
                    json_output.push_str("  ");
                }
            // Output last char if not in any block comment
            } else {
                if block_comment_depth == 0 {
                    if let Some(last_char) = last_char {
                        json_output.push(last_char);
                    }
                } else {
                    if preserve_locations {
                        json_output.push_str(" ");
                    }
                }
                last_char = Some(cur_char);
            }
        }

        // Add last char and newline if not in any block comment
        if let Some(last_char) = last_char {
            if block_comment_depth == 0 {
                json_output.push(last_char);
            } else if preserve_locations {
                json_output.push(' ');
            }
        }

        // Remove trailing whitespace from line
        while json_output.ends_with(' ') {
            json_output.pop();
        }
        json_output.push('\n');
    }

    json_output
}