Redwood: Add option to read/output to memory from rw generate (instead of writing files directly to disk)

[aldonline]

• Current behavior: When runningrw generate..., the CLI reads and writes directly to disk
• This makes it hard to integrate the generators with IDE environments. For example, if the user has open/dirty/unsaved files, having the CLI writing directly to disk can lead to conflicts (the dreaded: "this file was modified on disk, do you want to overwrite?" scenario).
• Proposed behavior: Add an option (ex: --json or --dry-run) that tells the generator to output a JSON map representing the files that were modified/generated, instead of applying those changes directly.
• This would enable seamless integration with IDEs (taking into account unsaved files, undo/redo, etc) and it can open the door for post-processing generated files, for example

Example

$ yarn rw g page About /about --json

will print:

{
  "/foo/web/src/pages/About/About.js": "import ...",
  "/foo/web/src/Routes.js": "..."
}

JSON Format

The JSON format could be very simple:

• Added files are represented by entries with content
• Modified files are represented by entries with the new content (if IDEs/tools want to be smart about calculating diffs, it is up to them)
• Deleted files are represented by entries with "null" value.

{
  "/foo/new-file.js": "content.",
  "/foo/modified-file.js": "new content",
  "/foo/deleted-file.js": null,
}

Passing File Overrides / Reading from Memory

• In order to fully eliminate the "this file was modified on disk, do you want to overwrite?" error, a second feature must be implemented: Generators must accept a set of file overrides.
• This is necessary since some generators operate on existing files (they read a file from disk, run a regular expression or some transformation, and then write the new file). If Routes.js is open in an IDE and unsaved (with changes), for example, running rw g page will result in a conflict.
• The solution is to enable passing "file overrides" when running generators.

Example

$ rw g page About /about --json --override '{"foo/web/src/Routes.js": "..."}'

[Tobbe]

Clearly you're the IDE expert here, and I have no idea how the integration would work, but wouldn't it be easier/better if the generators also had an API, and were importable as modules?

Modified files are represented by entries with the new content

How would you indicate where in the file this new content should go?

The solution is to enable passing "file overrides" when running generators.

What would this do exactly? Revert all unsaved changes, and then apply the changes from the generator?

[aldonline]

Clearly you're the IDE expert here, and I have no idea how the integration would work, but wouldn't it be easier/better if the generators also had an API, and were importable as modules?

Yes absolutely. Having a programmatic API for generators would open even more doors.
But this one is necessary, and should be easy to implement.

Modified files are represented by entries with the new content

How would you indicate _where_ in the file this new content should go?

I was suggesting the easiest way: Adding the "complete" content to the map. Always the complete content. No need to worry about diffs, insertion points, edits, etc. A new file and an edited file look the same in the map. The only way to know which is which is to compare against the existing working tree.

The solution is to enable passing "file overrides" when running generators.

What would this do exactly? Revert all unsaved changes, and then apply the changes from the generator?

The simplest use case:

• You open Routes.js
• Make a small edit (for example, add a comment) and DON'T save
• Run rw g page...
• Today, this will result in a conflict. The generated file will have a line added by the generator, but it won't have your unsaved change. This is because the generator logic did a readFileSync("...Routes.js") somewhere. So it doesn't know about your change.
• With the "overrides" option, the IDE would be able to pass the "unsaved" content of Routes.js (and any other open files)

[thedavidprice]

@aldonline this problem and approach make sense to me. And personally I'd vote --json.

Looping in @peterp

[aldonline]

There are two ways to implement this.

1. Edit the generator code and add an abstraction that handles all access to the filesystem. In other words, extract all the calls to the "fs" module. This way we can pass our own "fs" abstraction. This is the most solid way, but it requires significant effort. (<-- this is the preferred way)
2. Leave the code as-is and use a library such as proxyquire to "mock" access to the filesystem. This is what I do in Decoupled Studio. It works for most generators, but it adds yet another factor to the big build mess. I attach some code to give you an idea.

This is roughly what we do today:

• Instead of calling the CLI directly, Decoupled Studio generates a JS file that serves as entry point
• This file loads the CLI code, but before doing that, it intercepts some require() calls
  Here's the code:

const proxyquire = require("proxyquire")
const fs = require("fs")
const files = []
const fileOverrides = { FILE: "OVERRIDES" } // <--- this gets replaced with the overrides map
proxyquire("@redwoodjs/cli/dist", {
  fs: {
    mkdir() {},
    mkdirSync(...args) {},
    writeFile(...args) {
      files.push(args)
    },
    writeFileSync(...args) {
      files.push(args)
    },
    readFileSync(...args) {
      const path = args[0]
      if (typeof path === "string")
        if (fileOverrides[path]) return fileOverrides[path]
      return fs.readFileSync.apply(fs, args)
    },
    "@global": true,
  },
})
process.on("exit", () => {
  console.log("__SEPARATOR__") // <--- this gets replaced with a unique string that serves as a boundary
  console.log(JSON.stringify(files, null, 2))
})

[Tobbe]

Just wanted to link this forum topic reply from Tom where he said he wasn't a fan of json on the command line
https://community.redwoodjs.com/t/scaffold-textarea-s/800/11?u=tobbe

Not sure if it's relevant here or not... And no, I don't have a better idea, sorry 🙁

[aldonline]

Just wanted to link this forum topic reply from Tom where he said he wasn't a fan of json on the command line
https://community.redwoodjs.com/t/scaffold-textarea-s/800/11?u=tobbe

The hard part is intercepting the filesystem calls. The rest is just the API to surface this feature. I'm just trying to provide a starting point that lets us get this done quickly.
A lower level API to generators is even better, but it will take longer.

In the end, the point where the IDE will call this is roughly here. So it can be via importing an API, or an exec call.

[Tobbe]

Sorry if I was sounding negative and/or annoying. That was not my intention. I'm super excited about what this will bring to my IDE. Can't wait to generate pages, cells etc straight from VS Code 👍 😃

[aldonline]

Sorry if I was sounding negative and/or annoying. That was not my intention. I'm super excited about what this will bring to my IDE. Can't wait to generate pages, cells etc straight from VS Code 👍 😃

Hey Tobbe! You're not sounding negative at all. This debate is great for the definition of the feature. The specification is starting to take shape as the thread evolves, and forcing me to defend some of these ideas helps me externalize some thoughts that I might have skipped over. In the long run, this will save us time ;)

[peterp]

+1 for --json! This would also make testing much easier.