Freecodecamp: Challenge Seed Editor discussion

[tchaffee]

Some of us have agreed to work on a new Challenge Seed Editor to make it easier to create, update, delete, re-order, and in general maintain the challenge seeds. This issue is to here to discuss the goals and make some decisions on approach before we get started with the coding.

Before getting into implementation details, I think it would be useful to agree on some user stories. Just for example, editing the tests is currently difficult because it's JavaScript inside of a string. I would like to edit that JavaScript inside of a code editor. Another user story I've already seen is the possibility of using Markdown to edit the Challenge description.

[tchaffee]

Here's another feature important to me: I should be able to use the Challenge Seed Editor to maintain any curriculum, not just the freeCodeCamp official curriculum.

[bigkatspence]

Another thing is to keep in mind it's maintaining user and site security to
Todd

On Jun 9, 2018 1:45 PM, "Todd Chaffee" [e-mail] wrote:

Here's another feature important to me: I should be able to use the
Challenge Seed Editor to maintain any curriculum, not just the freeCodeCamp
official curriculum.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/freeCodeCamp/freeCodeCamp/issues/17511#issuecomment-395990470,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGl_C-TqcMt1XhE2I17a2gxJNPS54LSRks5t7BexgaJpZM4UhcV7
.

[SaintPeter]

Having worked on/in COM1000, there were the main features:
1) Online code editor, using the same Code Mirror Code editor that freeCodeCamp uses.
2) Ability to edit tests and have them properly formatted for the JSON file.
3) Ability to write out the source file.

Also very helpful was a live-refresh of the site so you can see how they look in real-time. I don't know if the learn/Gatsby has a "watch" type feature, but if it does then maybe this is straightforward to do - once the files are saved/changed, they will update?

I guess that the editor should be able to parse the standard challenge structure using the getChallenges() code to determine the hierarchy of the challenges in the same way that the learn environment does.

In terms of dependencies, is it just a matter of pointing a config or .env variable to the challenge root folder? As long as that folder has the challenges module, then you're good to go? Or maybe have it as a Submodule? We had mixed luck with submodules for the old Gatsby/Guide stuff.

---

I think it's a given that you'll need a local development environment, since you need node or equivalent to edit local files. Ideally we keep the resource dependencies low.

[frontendwizard]

Important features:

1. Ideally there would be a dashboard with overall stats for each track. Maybe even reported issues.
2. We should be able to easily navigate to each challenge and see it's content and tests.
3. We should be able to edit both its contents and it's tests.
4. We should be able to edit it's translations, with the original content side by side.
5. Our changes should be persisted back into the seed files, ready for commit and push.

I think the main challenge lies on persisting the changes. Can gatsby solve this issue easily? I'm not familiar with it.

EDIT:
Also, edit tests adds a bit of complexity since only editing and saving would not suffice. We'll need also to be able to provide the ability to execute the tests on some sample code.

[tchaffee]

Ideally there would be a dashboard with overall stats for each track. Maybe even reported issues.

I'm not sure what you mean by this feature? What is a track? What kind of stats are you interested in and why would those stats help when editing a challenge seed?

The other features make sense and I agree, but I do have a question about number 5:

Our changes should be persisted back into the seed files, ready for commit and push.

I think for now it makes sense to keep the existing workflow of opening a PR for review when someone wants to improve a challenge seed. But the workflow is pretty technical and I wonder if we could eventually make it easier for non-technical users who might want to just volunteer to do translations for example. I don't think we should put this as a high priority, just keep it in mind as a future possible goal to make the Edit -> Stage -> Review -> Publish workflow require less technical training and skills.

[SaintPeter]

Our changes should be persisted back into the seed files, ready for commit and push.

This should not be hard. We're already reading/parsing the input JSON data. We just have to make changes to that data and write it back out.

Assuming we're editing from a clone of the curriculum repo, all you need to do is git add/git commit/git push. This is no more difficult than the current process.

I would consider as a base requirement for editing challenges the ability to use Git. I don't want to accept changes to our curriculum from someone who is unable to follow this process. I personally learned more about Git by contributing to FCC than by any other process before or since. I don't have a problem with documenting the process, I just don't see any value with making it "easier".

[frontendwizard]

What is a track?

I'm sorry, that wasn't clear. By track I mean each group of challenges. Maybe grouped by certifications.

What kind of stats are you interested in and why would those stats help when editing a challenge seed?

I'd like to keep track of percentage of translations somewhere. Tests stats could be shown here too. We need an overall view on how is the translation work on each track/topic/certification. Definitely not a top priority feature, but it'd be really good to have it.

I think for now it makes sense to keep the existing workflow of opening a PR for review when someone wants to improve a challenge seed. But the workflow is pretty technical and I wonder if we could eventually make it easier for non-technical users who might want to just volunteer to do translations for example. I don't think we should put this as a high priority, just keep it in mind as a future possible goal to make the Edit -> Stage -> Review -> Publish workflow require less technical training and skills.

I guess we can strive to make this process easier, but I'm not sure how easier can it get without losing some organisation. It is necessary to maintain a fork in order to be able to submit and create a PR without polluting the repo with unnecessary branches. Maybe some nom scripts could improve the experience, but it is hard to do it without losing some flexibility. Anyway, I don't think this is a primary goal.

[frontendwizard]

I would consider as a base requirement for editing challenges the ability to use Git. I don't want to accept changes to our curriculum from someone who is unable to follow this process. I personally learned more about Git by contributing to FCC than by any other process before or since. I don't have a problem with documenting the process, I just don't see any value with making it "easier".

This is a fair point. We want to make challenge editing and translations easier without removing Git from the workflow. People should have to fork the repo and maintain this fork to be able to submit PRs and contribute to the project.

[raisedadead]

I should be able to use the Challenge Seed Editor to maintain any curriculum, not just the freeCodeCamp official curriculum.

I think this is fairly easier said than done. The reason being, the curriculum has a schema that is tied to it. So, unless any other curriculum is NOT following the same structure as the freeCodeCamp curriculum, it would be difficult to avoid issues in the editor which will be modeled after the schema.

Its fair to say that the editor will HAVE to be custom, taking inspiration and logic from previous versions of the editor we have had.

We have re-structured the curriculum to be able to be distributed via channels like, npm, open-api and the learn platform (which make up the eco-system for freeCodeCamp).

And its not going to change anytime soon.

Regarding the format of the source files:

For all its worth and to have excellent DX, the source files in the repo need not be JSON at all! They are actually the output that we need for the npm package. They maybe generated as a build step and published to npm.

@SaintPeter If it becomes tedious to parse symbols we could even have these output encoded as Base64 for transmitting.

For easy "Copy" editing of the challenges including translating them:

1. The source files can be simply markdown/html files with the challenge editor that we are discussing (and potentially for edition over GitHub interface as well).

2. The existing toolset and workflow currently on curriculum repo:

   • We have unpack scripts npm run unpack that generate HTML files out of the JSON.
   • These are viewable by any text editor etc.
   • These are editable.
   • These are ignored in Git.
   • Output of the tests is available. (using browser's dev console, i.e. simply open the HTML in chrome and check the console tab).
   • Once ready to commit you can npm run repack to covert the HTML back to JSON.
   • Then you add them to git and npm run commit to make a commit.

3. The editor component:

   We want to make the toolset more powerful and simpler for "copy editing". The planned editor can use Monaco as a component in lieu of now outdated CodeMirror. This is already in use on Learn. It is accessible by default and has full AST support, so you can use its powerful api to get what we need from the markdown.

   See https://microsoft.github.io/monaco-editor/monarch.html and select markdown from the dropdown on the Language Syntax Definition Panel.

   There is already a react component for it as well.

   This should give you all the access to calculating the stats that you need for translations.

4. Evaluation of tests for the challenge:

   What we want is stripped down version of the Learn's challenge execution logic.

   Currently its very tightly coupled to the application itself (updating user properties, etc). We do not want that with the challenge editor.

   What we want is to be able to execute the tests for challenges in assert statements for simple challenges.

   And for react or other UI rendering challenges to use dev tools like Jest, puppeteer etc.

5. As for the stats for the challenge tracks (which BTW we call certification superBlocks for some weird reason IDK), they should show no of challenges with NO solutions as an addition as well. I would agree to whatever you already have in mind for translations.

6. Persisting the changes:

   Gatsby is just a static site generator. But at the end of the day, its a Node process doing stuff. It has got tons of community curated plugins to do almost anything fancy.

   Netlify CMS is what we want to use with it to make our lives easier https://www.gatsbyjs.org/docs/netlify-cms/. It was originally meant to solve a complex problem of tying a headless CMS with GitHub (any git remote) etc. as a backend. But it ties really well with Gatsby in local development too. Meaning your local Gatsby app is running at localhost:8000 with the CMS as a frontend.

   We may need to "Hack" it for our needs though.
   Please check it out and let me know what you think.

Thanks a lot everyone for you thoughts!! This is indeed a huge feature that we are looking forward to.

[tchaffee]

One thing that would help this discussion - and I know it's very tough for devs - is to avoid worrying about implementation and to just brainstorm the user stories about how the editor should work if it did everything you want, and exactly the way you want it to. Once we come up with a prioritized list of features, we can start to evaluate which is the best implementation for those features. We should be deciding "what" in this discussion, not "how".

For sure we will want to get as close to existing FCC tech stacks as possible, but that in itself is just another user story and I think we can leave it at that very general statement for the moment.

I'll be happy to summarize everything when we stop getting new comments for a few days and it's clear we've collected all the user stories.

Here's some feedback on a few of the comments above that are important to me:

unless any other curriculum is NOT following the same structure as the freeCodeCamp curriculum, it would be difficult to avoid issues in the editor which will be modeled after the schema.

I would only expect the editor to work with a curriculum that follows the freeCodeCamp schema.

It is necessary to maintain a fork in order to be able to submit and create a PR without polluting the repo with unnecessary branches. Maybe some nom scripts could improve the experience, but it is hard to do it without losing some flexibility. Anyway, I don't think this is a primary goal.

Agreed it's not a primary goal. But my vision is that anyone good at translation could easily do translations without knowing anything about git, forks, PRs, etc. Let's stay open minded about improving the workflow long term. We can park this for later discussion though since it's not a priority.

the source files in the repo need not be JSON at all! They are actually the output that we need for the npm package. They maybe generated as a build step and published to npm.

I wanted to point out that this is an example of a user story we need, and not an implementation detail. The output of the editor needs to eventually be published to npm as a JSON file. This is a fixed requirement we can't change because other components rely on this output. As @raisedadead mentioned, it also frees us up to re-imagine what the input is. For the description part of a challenge, markup was already mentioned and makes the most sense to me, so unless someone comes up with a better idea we can finalize that as a user story. I think that also means any translations would be in markup too?

Its fair to say that the editor will HAVE to be custom, taking inspiration and logic from previous versions of the editor we have had.

Does anyone have screenshots or anything that would help us newbies understand what the previous versions did?

[tchaffee]

It would also be good to start defining some roles more explicitly. I've guessed the following roles from the user stories so far:

• Author - Writes lesson description, the tests, and challenge initial code. Probably an experienced dev. But less experienced devs will certainly want to fix problems with descriptions and maybe even tests.
• Reviewer - Reviews changes to challenge seeds, including new challenges. Asks for improvements, and gives final approval to publish the changes.
• Apps - The Learn platform uses the output of the editor.
• Translator - Translates the challenge seed description. What about the seed code? I don't think we translate the starting code for Campers? Example below, the lesson would be to change "Hello" to "Hello World".

<h1>Hello</h1>

Please feel free to correct me or suggest additional roles I might have missed.

[tchaffee]

In the spirit of brainstorming features, here are some long term ideas that aren't priorities that you can safely ignore. I'm just hoping they might inspire other ideas:

• The tests are JavaScript, right? Would it add any value to store these as plain .js files for the input? Or maybe store all the tests in one .js file? That would allow me to open the tests in my favorite IDE to work on them there. It would also allow comments so I can document what the test is supposed to do, and document the sometimes very complex regexes.

• If I fix a test it's sometimes hard to tell if I broke something else. Let's say I'm testing for a certain style property but I accidentally remove the test for a semicolon at the end of the style. This is really hard to detect. Would it help to have examples of code that will definitely not pass the tests along with an example solution that will pass all the tests?

Again, these are really long term ideas and low priority, so please feel free to totally ignore.

[tchaffee]

Can I assume the editor will be used only locally and does not need to be available via a public URL? I'm assuming that because so far we have talked about editing challenges that are already on your box in a folder somewhere. Probably from forking the fcc curriculum repo?

If that's the case, would it make sense to be able to start the editor using a simple command? Something like:

fcc-challenge-editor --curriculum-path=./challenges

At which point I would guess you would get a browser running the editor app, or perhaps it could be an Electron app - the implementation isn't important at the moment. I'm just trying to make sure we are all on the same page about what the developer experience would look like. Does that sound like the kind of developer experience everyone else was envisioning or did you have something else in mind?

[frontendwizard]

Can I assume the editor will be used only locally and does not need to be available via a public URL?

Yes. It makes total sense since it's a tool for contributors. Making it a electron app might make it even better because that way we can directly manipulate the json files.

[raisedadead]

Yeah.. it will be local.

But instead of a full blown electron app, will it make sense to then make it a plugin for say atom or vs code? I know this leaves out other editors. But it would be light weight?

Or if you think an electron app is better, then I would go ahead and incline on making it a separate repo like @tchaffee originally intended.

[tchaffee]

But instead of a full blown electron app, will it make sense to then make it a plugin for say atom or vs code? I know this leaves out other editors. But it would be light weight?

I'm not stuck on any one implementation yet. I just wanted to use that as an example to better illustrate what I meant by running locally. I really don't know what the best choice is until I (or someone else) looks a bit closer at pros and cons. Starting with a simple local browser based solution might be the easiest since we are all familiar with that.

[tchaffee]

We've already talked about the possibility of changing the input format of the challenge seeds. Since changes to the curriculum will be reviewed by PR for now, would it make sense to make the input format as human readable as possible? By making the description input use markup, we will have already achieved that for the description. Should we be trying to achieve the same for the other data associated with a challenge seed?

[raisedadead]

Starting with a simple local browser based solution might be the easiest since we are all familiar with that.

Yes I agree that would be totally ideal.

The only thing is we would need access to the File System. While this should be possible using Gatsby with plugins. But would be easiest with a local editor?

..., would it make sense to make the input format as human readable as possible?

Yes, that is what we would need for easier QA/review.

[SaintPeter]

What we did with COM1000 was make a local node server which hosted the client. Node has access to the local file system.

Honestly, we could probably just tune up COM1000 to accept the new structure and it would work just fine. Check out the repo. It's reasonably well written React stuff.

[raisedadead]

Yeah, sure that would be a great place to start and seek inspiration.

https://github.com/freeCodeCamp/COM1000

Our curriculum has grown (exploded).

Not that I am biased with Gatsby (which is also a Node app), its just optimised to work with large no files. So, we would have a lot of the system internals available to us? That is we would not have to work on problems that an abstraction can handle for us.

Please note we are already breaking the Gatsby and Netlify's performance limits!! And they are actually in the core business of managing huge content sources. 😅 Guide and learn take 15-20mins to build the final static content on the production side.

But on the local development side, Gatsby's in memory building is incredibly fast. Its near instant, auto reloads and has MarkDown to HTML rendering from its basic starter template, out of the box.

All we would need to do is, add our Schema-to-Markdown-to-packing logic.

[SaintPeter]

I'm not sure that I understand the obsession with Gatsby. There is almost nothing about it that it helpful to a challenge editor. We're not going to be serving all bazillion challenge files. We're just going to be editing one challenge at a time. Not everything is a nail for this hammer.

We can enumerate the JSON files with a root path and recursive fs.readdir() calls. We present those files to the user and they choose which one to open. When we make a change to that JSON file, we save it off. This is just not rocket science.

I wrote 63 new challenges with COM1000 and I edited probably 100 more. It wasn't the most feature rich app, but it was two orders of magnitude faster than editing the raw files.

The only challenge I see right now is that we need a way to "live edit" files, or at least pull in changes made to the source .JSON files into the learn environment so they can be viewed/tested. If I understand the current process, this looks like:
JSON Files --> Gatsby Transformer Thing --> Learn Environment

It is not clear to me if the Gastby -> Learn can be automated or set up as a "Watch" or Hotload type environment.

[tchaffee]

I'm not sure that I understand the obsession with Gatsby.

Doesn't make sense to me either, but hard to tell what the ideal stack is until we've agreed on all the initial requirements, both functional and non-functional. The reason I asked about only using the app locally is because that's an important non-functional requirement around performance. Pretty clear now that we don't have to optimize the app for thousands of online users. Really only one person at a time would be using the app locally based on what we've discussed so far.

I wrote 63 new challenges with COM1000

What happened to this? Is there source? Or screenshots? It would be great to see how it worked.

If I understand the current process, this looks like

I think it could look more like this based on what we've discussed so far?

Human readable input files --> Challenge Editor --> PR using changed human readable input files --> Output to final JSON format for Learn platform

[tchaffee]

Oh, now I see the repo now for COM1000. I didn't realize the source was still available. It makes a whole lot of sense to me to just get that working again with the new schema with no new features and then decide if a rewrite is even worth it.

[SaintPeter]

I'd like to better understand the "Human Readable" requirement.

My instinct is that a JSON format - a format specifically designed to be machine/code readable - is always going to be the best way to store this data. There is a hard line as to if a file is properly encoded as JSON or not, there is no ambiguity. It can also constantly be used to generate output which, in turn, is going to be used by a Javascript based machine.

The purpose of the challenge editing tool is to act as an intermediary in transforming to and from the "machine readable" format. We can do this selectively on a field by field basis, with different requirements on a per-field basis. That way we can ensure that the field specific data is rendered and re-encoded consistently.

In engineering we have a principle of having a "single source of truth", or a "master record". All data flows from there. That's what I see the JSON format as.

[SaintPeter]

For everyone on the thread, here is the old COM1000 Repo:
https://github.com/freeCodeCamp/COM1000

It has been archived, but can still be downloaded. The name, BTW, is a bit of a joke. It stands for "Challenge-o-Matic 1000".

[tchaffee]

The purpose of the challenge editing tool is to act as an intermediary in transforming to and from the "machine readable" format

Since someone still has to review the PR in whatever form the raw input is in, it makes sense to consider something more readable than noisy JSON with all the quotes and properties. Don't you think we can do much better than one gigantic JSON file? I don't know if I'm keen on putting each challenge in its own dir... but if we did that we could have description.md (along with description-es.md for each translation, etc.), tests.js, and meta.json files. The meta.json file could have all the properties like id, isRequired and so on. Github would even recognize tests.js as JavaScript, making it easier to review. Just a very rough idea. I'm sure someone could improve upon it, and it would even be good to submit some example PRs to test out our ideas.

In engineering we have a principle of having a "single source of truth", or a "master record".

Agreed. That should be the input, whatever we agree on what that's going to be.

But maybe this discussion is all overkill now? What does everyone think about just "fixing" COM1000 first?

[SaintPeter]

I don't know if I'm keen on putting each challenge in its own dir... but if we did that we could have description.md (along with description-es.md for each translation, etc.), tests.js, and meta.json files. The meta.json file could have all the properties like id, isRequired and so on. Github would even recognize tests.js as JavaScript, making it easier to review.

That actually sounds pretty reasonable. I don't know if it'd be too messy to have a different folder for each challenge, but it becomes really self organized. You could have description-en.md and description-XX.md the different languages.

Order would be an issue, though. Right now they're in the order that they occur in the array in the JSON file. What happens if you want to "insert" a challenge? I'd hate to have to have a separate "index" file that gave the order. You could prefix folders with 00010 or something. Then you might end up with 00010A, 00010B, etc.

It's certainly an interesting idea for longer term.

It does seem like fixing COM1000 is a good way to start.

[tchaffee]

It does seem like fixing COM1000 is a good way to start.

Unless someone objects, let's try that out and see how it goes. I have other commitments for the next week, but after that I can give it a go unless someone else wants to take it on.

[raisedadead]

I'm not sure that I understand the obsession with Gatsby. There is almost nothing about it that it helpful to a challenge editor.

As I already said:

1. Gatsby's in memory building is incredibly fast. Its near instant, auto reloads and...
2. ...has MarkDown to HTML rendering from its basic starter template, out of the box.

Anyways I understand the friction. And I'll let you move ahead with COM1000. Todd please proceed with the prototyping, as you seem fit and comfortable.

My only concern is we might be getting into re-inventing the wheel a little. But as long as someone can get an functional editor working we are good.

Thanks :)

[tchaffee]

I'm back from travelling and will start this in the next couple of days.

My idea is that we should spend minimal effort on COM1000 and the new format. If it goes quickly then it's a good stop gap everyone can start using right away. At which point we can decide on how best to move forward, and take as much time as needed to discuss it properly since the pressure will be off. If there are loads of problems with getting COM1000 working with the new format, then it might be time to scrap it and start over. I'll report back once I've had a few days to give COM1000 a shot.

[frontendwizard]

Agreed. I'm gonna give COM1000 a shot too, but I'm biased towards starting over clean using it as a reference check.

[tchaffee]

@raisedadead or really anyone, is there any documentation anywhere about the differences between the old challenge seed format and the new one? Or can someone explain the differences? Maybe a call would work?

[raisedadead]

@Bouncey I guess you are the best person to guide on this. Also we would want to take this opportunity to possibly do the directory restructuring (but more on that later)... lets hop on a call. @tchaffee @julianorafael and anyone else interested in a call, please DM me your emails on Gitter.

I'll be sending out the invite, and we can confirm on the time.

[tchaffee]

As a start to our discussion, I've tried to guess most of the differences myself by comparing the two formats.

It seems like id, title, description, and challengeType haven't changed?

The tests property is largely the same in terms of content, it's just better broken into properties now instead of the properties being sort of hidden in the test string.

Example of tests property for the old format:

"tests": [
  "assert(($(\"h2\").length > 0), 'message: Create an <code>h2</code> element.');",
  "assert(code.match(/<\\/h2>/g) && code.match(/<\\/h2>/g).length === code.match(/<h2>/g).length, 'message: Make sure your <code>h2</code> element has a closing tag.');"
]

And the tests property for the new format:

"tests": [
  {
    "text": "Create an <code>h2</code> element.",
    "testString":
      "assert(($(\"h2\").length > 0), 'Create an <code>h2</code> element.');"
  },
  {
    "text": "Make sure your <code>h2</code> element has a closing tag.",
    "testString":
      "assert(code.match(/<\\/h2>/g) && code.match(/<\\/h2>/g).length === code.match(/<h2>/g).length, 'Make sure your <code>h2</code> element has a closing tag.');"
  }
]

Is translations unchanged too?

The biggest change I can see is with the challengeSeed property which has moved to a files property. The way that property works is a mystery to me and that's where I think I could use the most help.

Old challengeSeed example:

"challengeSeed": [
  "<h1>Hello World</h1>"
]

New files property:

"files": {
  "indexhtml": {
    "key": "indexhtml",
    "ext": "html",
    "name": "index",
    "contents": ["<h1>Hello World</h1>"],
    "head": [],
    "tail": []
  }
}

Have I missed anything? Hopefully this saves us some time on the call if we can get all the differences documented ahead of time.

[tchaffee]

In the new format, what types of challengeType are supported? So far I only see '0' (html) and '1' (javascript). I'm hoping we haven't done away with '7' (step) as I use them all over the place in my custom curriculum.

[tchaffee]

Actually using grep I found challengeTypes of 0 through 7, although it seems like type 7 is now being used for something different?

Is challengeTypes.js still accurate?

Copy pasted here for reference:

export const html = 0;
export const js = 1;
export const backend = 2;
export const zipline = 3;
export const frontEndProject = 3;
export const basejump = 4;
export const backEndProject = 4;
export const bonfire = 5;
export const modern = 6;
export const step = 7;
export const quiz = 8;
export const invalid = 9;