Browser-compat-data: Provide utils to save in-memory BCD to files

[foolip]

The BCD module itself (index.js) reads all of the data files into an in-memory object which consumers can use.

When making scripted changes to the data, one has to load, edit and save some JSON files. If one loads each file individually it can be done, but unfortunately then one can't just require('.') and work with the merged data.

It would be helpful if the file structure could be written back exactly from the in-memory BCD and there's an in-repo utility for doing this.

[foolip]

Here's a save.js that almost works:

'use strict';

const fs = require('fs');
const path = require('path');

function isDirectory(fp) {
  try {
    return fs.statSync(fp).isDirectory();
  } catch (e) {
    return false;
  }
}

function save(bcd, dirname) {
  function processObject(object, keypath) {
    for (const [key, value] of Object.entries(object)) {
      const candidate = path.join(dirname, ...keypath, key);

      if (isDirectory(candidate)) {
        // If the path is a directory, recurse.
        processObject(value, keypath.concat(key));
      } else {
        // Otherwise, write data to file.
        const filepath = `${candidate}.json`;
        // Add wrapping objects with keys as in keypath.
        let wrappedValue = value;
        const keys = keypath.concat(key).reverse();
        for (const key of keys) {
          const wrapper = {};
          wrapper[key] = wrappedValue;
          wrappedValue = wrapper;
        }
        const json = JSON.stringify(wrappedValue, null, '  ') + '\n';
        fs.writeFileSync(filepath, json);
      }
    }
  }

  processObject(bcd, []);
}

module.exports = save;

Unfortunately, the file structure doesn't exactly line up with expectations. This is what git status says after running the script:

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   browsers/firefox.json
    modified:   browsers/ie.json
    modified:   browsers/opera.json
    modified:   browsers/qq_android.json
    modified:   browsers/safari.json
    modified:   browsers/safari_ios.json
    modified:   browsers/webview_android.json
    modified:   css/types/frequency-percentage.json
    modified:   css/types/frequency.json

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    css/types/hz.json
    css/types/khz.json
    html/elements/input/__compat.json
    html/elements/input/input-button.json
    html/elements/input/input-checkbox.json
    html/elements/input/input-color.json
    html/elements/input/input-date.json
    html/elements/input/input-datetime-local.json
    html/elements/input/input-email.json
    html/elements/input/input-file.json
    html/elements/input/input-hidden.json
    html/elements/input/input-image.json
    html/elements/input/input-month.json
    html/elements/input/input-number.json
    html/elements/input/input-password.json
    html/elements/input/input-radio.json
    html/elements/input/input-range.json
    html/elements/input/input-reset.json
    html/elements/input/input-search.json
    html/elements/input/input-submit.json
    html/elements/input/input-tel.json
    html/elements/input/input-text.json
    html/elements/input/input-time.json
    html/elements/input/input-url.json
    html/elements/input/input-week.json
    html/elements/input/x-moz-errormessage.json
    http/headers/csp.json
    javascript/builtins/Infinity.json
    javascript/builtins/NaN.json
    javascript/builtins/decodeURI.json
    javascript/builtins/decodeURIComponent.json
    javascript/builtins/encodeURI.json
    javascript/builtins/encodeURIComponent.json
    javascript/builtins/escape.json
    javascript/builtins/eval.json
    javascript/builtins/globalThis.json
    javascript/builtins/isFinite.json
    javascript/builtins/isNaN.json
    javascript/builtins/null.json
    javascript/builtins/parseFloat.json
    javascript/builtins/parseInt.json
    javascript/builtins/undefined.json
    javascript/builtins/unescape.json
    javascript/builtins/uneval.json

Still, it already matches api/ exactly, confirmed by changing a single support statement and seeing just that change written back. css/ is also very close to matching.

@Elchi3 @jpmedley do you think aligning the file structure with the data structure until there are no differences is worthwhile? Would you use such a utility to make edits yourself?

[foolip]

I sent https://github.com/mdn/browser-compat-data/pull/3618 to resolve the only issue in css/.

[jpmedley]

I have no opinion on the first question. On the second question, as soon as I say no, I'll think of something to use it for.

[Elchi3]

So far I was worried that aligning the file structure with the data structure will make us uncomfortable when editing or that it will be inflexible for adding more data.
For example, sometimes you might want to split very large files (looking at you document.json), or organize things in a way that is friendly for the human editor (see the input element). However, it seems that this authoring comfort hasn't been a thing very much and people just deal with large files and potential merge conflicts and so forth (we now have 11,000 features in here and there are only a handful of cases where it isn't aligned).

For machines editing the data there is obviously a benefit if we're aligned.
Further, it could help with https://github.com/mdn/browser-compat-data/issues/2985 as you currently can't get to the data source from a data query directly.

All this said, I could imagine enforcing these more strict file rules, so that there's an even better programmatic data access (which is the whole purpose of this project anyway).

[jpmedley]

If you need the source file for a data point, I don't think you need to restructure. If as BCD is being loaded, the module could attach it to the tree somewhere as a symbol. This would keep it out of the way of iterators, but would still be accessible for the kumascript macro to generate an accurate link.

My point is that if there's another way to solve 2985 then it shouldn't be regarded as an argument for restructuring files.

I'd also be nervous about changing the file structure at this late date. It seems straight forward, but you'll be stomping out bugs for weeks.

[Elchi3]

I don't think we're talking about restructuring, but just aligning the very few non-conforming cases as found by @foolip to a file structure that can already be observed otherwise.

[foolip]

I've filed https://github.com/mdn/browser-compat-data/issues/6585 which supersedes this issue.