P5.js: List of p5.js keywords

[catarak]

Nature of issue?

• [X] New feature request

Most appropriate sub-area of p5.js?

• [ ] Color
• [ ] Core/Environment/Rendering
• [ ] Data
• [ ] Events
• [ ] Image
• [ ] IO
• [ ] Math
• [ ] Typography
• [X] Utilities
• [ ] WebGL
• [ ] Other (specify if possible)

New feature details:

Hi! I was looking at processing/p5.js-web-editor#978, and I was wondering if there's a way to get a list of p5 functions/variables? I looked at processing/p5.js-web-editor#211 and I could not find the list I originally used. I'm thinking I pulled it from the desktop editor? Anyway, it would be cool if this list could be automatically generated so that I would be easy to keep the syntax highlighting file up to date.

[limzykenneth]

Would it be a case of just getting all the p5 function and variable names in a list? I think this list can probably be generated from data.json with a bit of script as long as there are no problematic edge cases.

[AdilRabbani]

I would like to work on this, though, I am still unsure how to get started. I believe the API/function to do this should run every time the library is compiled? So that the list of all functions/variables are updated whenever a change is introduced to p5.js (making this a part of the build process). This gives some idea on getting functions from a javascript object :

function getAllMethods(obj)
{
    return Object.getOwnPropertyNames(obj)
        .filter(function(prop) {
            return typeof obj[prop] == 'function';
        });
}

Can the same be applicable for p5 as well? Any tips/suggestions @limzykenneth? Also how do you propose on getting the functions/variables from data.json? Is it by iterating over all files, going to the respective line as indicated in data.json and fetching the function name?

[limzykenneth]

@AdilRabbani Running it on every build is rather excessive, running it on every release should be enough. Also I'm not sure if such a step would necessarily belong here or it would be better over at the web editor's repo? @catarak What's your opinion on the last point?

I'm thinking to interate through data.json to fetch all the documented function and variable names instead of either going through the source code or looking through the p5 object's property at runtime so that we are not dealing with a potentially really complex object that have keywords that are not reserved in the global namespace (Graphics in p5.Graphics for instance is not exposed globally).

If you look at data.json, it's a big file but I'll just pull an excerpt:

        {
            "file": "src/color/creating_reading.js",
            "line": 91,
            "description": "<p>Extracts the HSB brightness value from a color or pixel array.</p>\n",
            "itemtype": "method",
            "name": "brightness",
            "params": [
                {
                    "name": "color",
                    "description": "<p><a href=\"#/p5.Color\">p5.Color</a> object, color components,\n                                        or CSS color</p>\n",
                    "type": "p5.Color|Number[]|String"
                }
            ],
            "return": {
                "description": "the brightness value",
                "type": "Number"
            },
            "example": [
                "\n<div>\n<code>\nnoStroke();\ncolorMode(HSB, 255);\nlet c = color(0, 126, 255);\nfill(c);\nrect(15, 20, 35, 60);\nlet value = brightness(c); // Sets 'value' to 255\nfill(value);\nrect(50, 20, 35, 60);\n</code>\n</div>"
            ],
            "alt": "Left half of canvas salmon pink and the right half white.",
            "class": "p5",
            "module": "Color",
            "submodule": "Creating & Reading"
        }

This is the entry for brightness() and you can see immediately we can simply pull the name property from it to populate the list. Same goes for the other functions and variables. Then the only challenge is to traverse the somewhat complex JSON file which I think require effort but not hard. How FES notice errors that users are trying to use p5 functions outside setup() or draw() may be worth looking into as well, not sure if useful though.

[catarak]

@limzykenneth i think writing a script that parses data.json and puts it into a list would be the right path! the only distinction i need for CodeMirror syntax highlighting is a p5 variable vs a p5 function.

unless there's other use cases for this list, if only the web editor needs it, i think it makes sense to put the script that generates that list in there. maybe the script could even generate the syntax highlighting file.

a related question—can you think of a way to trigger an event on a p5.js release? it would be cool if this could all happen automagically ✨🔮✨

[lmccart]

I think it makes sense to host this file in the p5.js repo so people can make use of it with other editors or things they're building. I agree, triggering on release makes sense. This could either be a webhook or a grunt task that's part of the release tasks.

[Spongman]

the typescript definitions would have been perfect for this.

[limzykenneth]

@Spongman Typescript definitions would work but it still needs to be compiled or generated so the process for creating it vs creating a list of function names and variables would be comparable no? Then when it is just the outpu is different, I think in this case wouldn't just a simple word list be easier to work with? Also Typescript definitions needs to be maintained as an entity while data.json is something we will need anyway in order to have documentation so there is less risk of it going out of sync.

@catarak The data.json entries have the types as well (variable vs function) under the itemtype key.

[Spongman]

yeah, typescript definitions require a little more processing than just a list of keywords (which has a limited use), but they offer the most most semantically rich format we have. the definitions can be simply parsed using a library, such as https://www.npmjs.com/package/typescript-parser. and generating them is reasonably simple. unfortunately, it's almost impossible to reliably release them in a separate repository. that's why i initially had them generated as part of the p5 build.

[lmccart]

Yes, thinking back to our reasoning for move typescript to a separate repository, maybe we should decouple p5.js derived files like this, and this keyword list actually belongs in the p5.js web editor repo. Since the Processing Foundation is maintaining both, we can reliably direct people to pull that file if they'd like to use it for their own purposes.

[Spongman]

one issue that might be relevant here is that it's currently quite difficult to programmatically extract the data.json from any given p5.js release. the two options that come to mind are:
1) use something like nodegit to clone the sources from github, spawn a p5.js build and then extract the data.json file from the docs/reference directory. this is what the p5.ts project has to do and is obviously quite heavy-handed and some of the dependencies do not play well cross-platform.
2) download the p5.js file from the release, use something like esprima to parse the javascript, reverse-engineer the embedded require implementation in order to find the exported module in the AST and render it back out as JSON. this is really fiddly and difficult to write in a way that's resilient to changes in the p5 build process.

[catarak]

for the purposes of the web editor, it's totally okay if data.json is just hosted on the p5.js website, as it is now—a script can be written that just assumes its at a url.

[lmccart]

Just checking in on this one, is it sufficient to pull https://p5js.org/reference/data.json for the p5 keywords? It should have all the keywords and also some additional metadata / definitions if we ever want to build any of that into the editor.

[stalgiag]

@lmccart that seems sufficient to me unless I am missing something. That file appears to have all of the publicly available properties and methods already categorized.

[catarak]

@lmccart yes, i think that is sufficient! i think we can close this issue.