Marked: Update testing harness to allow for unit, integration, and application tests

[joshbruce]

Marked version: https://github.com/markedjs/marked/pull/1102/commits/fb28a9538e27c5c97988bf2423cd257def243a60

Proposal type: other

Sorry, one more, then I'll be done...sometimes my brain latches on to things and I can't do anything else until I get it done (think @chjj has that type o' thing going on as well 😊).

What pain point are you perceiving?

I typically bucket test types into three major areas:

1. Unit: Object is able to be tested in complete (or darn close) isolation from other objects and environments...at the public API level (individual public properties and methods).
2. Integration: Objects can be tested to make sure they play well with each other.
3. Application: The entire application (all objects) are leveraged to test things.

I know there's like 500 different labels for testing types, which is why I tend to just bucket them this way.

I think marked.js has a lot of large functions and methods that can be broken down using the extract method refactoring. However, our current test harness only seems to be able to run what I would consider application tests (some might even call it end-to-end).

Markdown + marked() + HTML + are they equal? pass/fail

It would be nice if I could test each method on the marked object though. Really beat the heck out of that one method to make sure it's clean.

What solution are you suggesting?

1. Add ability to have unit and integration style testing (think we have application testing pretty well covered).
2. Have ability to run the one or two types of tests separately (being able to run a single test would be awesome, but not required, our tests run pretty quickly right now); maybe something like npm test:unit or something.
3. Add to Travis checks.
4. Maybe update CONTRIBUTING with the test types and instructions (technically, I think that would be part of the checklist but maybe we want to test it out first - make sure it has the ROI I think it will).

Tagging @UziTech as Dr. DevOps and @davisjam as a Node Nerd (new title?? I think you're a NodeJS cat, yeah??)

[joshbruce]

Oh! And @UziTech, here's the Angular 5 Circle CI config thing I mentioned somewhere when we got on the subject of Travis supporting separate builds and things...not sure if it helps with Travis: https://github.com/joshbruce/sam-design-system-site/blob/master/.circleci/config.yml

But, think you're on to something with that...though not sure exactly what you were thinking - maybe separate out the lint checks and the test tests?? Maybe have a build for each one - first to fail - fails the PR?? Anyway, yeah that's the config and project.

[joshbruce]

Just want to make the implicit explicit...sorry for the lack of emoticon and intent behind the comment:

Node Nerd

Would be a term of endearment. And, I thought y'all might be able to help each other out on it maybe. And I thought an article you linked me to was from NodeJS; so, I'm assuming you know a lot about NodeJS and what's going on there to have been able to find that so quickly. Sorry, meant no offense. 😐

This is why text-based communication kills me. Ruins all my jokes. 😃

[UziTech]

Do we want to use mocha, jasmine or jest?

I am most familiar with jasmine but I have used all three.

[davisjam]

Do we want to use mocha, jasmine or jest?

Any sounds good to me.

Node Nerd

I have had no trouble interpreting your tone so far. Fear not.

[joshbruce]

@UziTech: Fair question. I'm not exceedingly familiar with any of them, to be honest. Whatever you think is the easiest for a new contributor (with limited JS experience) to pick up. I trust your judgment (I think?? 😉 ) (This is kind of our target audience right now, to be honest. We are so few. There are so many issues. And none of us really has the time to dedicate to making any of this happen beyond fixing the odd security issue when it comes up, which seems to have been the MO for the last few years.)

I'm also the Agile Software Development guy; so, individuals and interactions over processes and tools. 😊

I'll let you know if I think it's got too much friction compared to what I think should be the case?? Is that fair??

@davisjam: Good. It's always a concern I have (being misinterpreted to the point of causing someone pain or discomfort).

If something I did ever lands oddly for you, just call me out through some impact feedback, and I'll try to explain my intent and we can take it from there (heck, even if I do something that lands in a good way for you, feel free); totally have my consent on that score.

(Buddhism, trying to reduce suffering in the world, not increase it. I reference it a lot but I'm not actually Buddhist...I wonder if that actually makes me Buddhist in and of itself sometimes. 😆)

Speaking of which, I'm working on game mechanics for authors related to the badges (think it should establish some conflict resolution norms in a nice way while being primarily intended to give praise and contributor active status).

We also have #1094 to figure out...no rush...but it's something we should probably get in place before too many more Curious Contributors show up.

Speaking of which, again, @intcreator, totally contribute your thoughts on #1094 as well. It effects you too, after all. 😉 Do you have a preference between the three options @UziTech mentioned? And, what's your benefit analysis as to why to choose one over the other?

[intcreator]

I don't know enough about the testing frameworks to have a recommendation. As long as it's pretty easy to write tests, see what's going on, and debug/fix errors when the tests fail I think it'll be fine.

How are we going to test cross-browser compatibility (especially IE8+)?

[joshbruce]

@intcreator: Good question. I got nothing. I think Selenium might have the ability to use different browsers. We could also use something like Browserstack (though would be better to have it in our own test bed for CI purposes) - they do have a free option for open source projects: https://www.browserstack.com/pricing?tab=browser-plans-tab

@UziTech: Might also be nice to set up a --watch on lib, if it's not too difficult to implement. Change -> save -> light is green. That way contributors don't have to explicitly run them. From the UX world: Make things you want people to do, easy to do and the things you don't want people to do, difficult to do. (Probably why my hard argument on making lint part of the base npm test - would probably do the same with the --watch - again, not if it's too difficult.)

[davisjam]

How are we going to test cross-browser compatibility (especially IE8+)?

What specifically is the concern here? The generated HTML, or the JS in marked?

I am most familiar with jasmine but I have used all three.

Sounds like jasmine is a good choice then.

Probably why my hard argument on making lint part of the base npm test

+1

[styfle]

I really like tap/tape because it doesn’t introduce globals and you can run it without any magic test runner...the tests are just JS.

https://ponyfoo.com/articles/testing-javascript-modules-with-tape

Although I admit that I don't know much about testing beside simple unit tests.
My second vote would be for jest since it has an auto-mocking feature.

[UziTech]

1. I've heard of tap but I don't think this is a big enough project to prioritize flexibility over simplicity like it does.

2. From first-hand experience Jests auto-mocking feature is more of a pain to work around then it is worth. (Which is why they changed it to be off by default.)

I am going to implement Jasmine 3. In my experience new developers seem to get the hang of it the easiest.

[joshbruce]

What specifically is the concern here? The generated HTML, or the JS in marked?

I think it's mainly that Marked works in the browser.

Thinking our unit tests should cover the JS. Our MD + HTML test suite should cover the input output question (believe this was initially written for the CLI prior to the client-side capability, which was a good design decision, imho - but when we went client-side, maybe we didn't consider the browser question from @intcreator). Thinking having one or two tests per supported browser would suffice.

SomeTool.launchBrowser('IE8').marked('Hello, World!').didNotExplode();

I am going to implement Jasmine 3. In my experience new developers seem to get the hang of it the easiest.

Jasmine sounds good. I appreciate @styfle's comment regarding the tests just being JS though. But, putting on the developer hat - I just want to make sure I can trust them, they don't read like calculus, and they help me understand the API I'm using. 😊

[UziTech]

I appreciate @styfle's comment regarding the tests just being JS though.

All of those test frameworks are written in JS for JS

[styfle]

All of those test frameworks are written in JS for JS

Correct.

What I mean by "just JS" is that you can run node tests/my-first-test.js and it will run because there are no magic globals. Or you can run all tests with the test runner and a glob pattern tape test/**/*.js.

This makes it really easy to debug. But I'm sure jasmine has any easy way to debug so I will defer to your knowledge on the subject. Let's use jasmine.

[intcreator]

My concern with the browser compatibility is in-browser usage of the library. For example, I have two sites which use Marked to parse Markdown files on the fly client side. If people access my sites in IE8 (assuming my audience includes people who use IE8) I don't want JavaScript errors that result from using the parser.

[UziTech]

I reached out to BrowserStack to see if we can get free browser testing. Just waiting to hear back.

[joshbruce]

I will defer to your knowledge on the subject.

Ditto. You da man! Thank you.

ps. That using front-matter for configuration was a stroke of genius in my book.

[UziTech]

Sumit Banga (BrowserStack Support)

Mar 2, 09:43 IST

Hi Tony,

We are happy you reached out! BrowserStack loves open source, and has sponsored thousands of projects. Here is more information about our offering and activation process.

What you get for free: Unlimited manual and automated testing for open source websites and mobile apps. Full platform access to Live, Automate, App Live and App Automate (coming soon). Total of 5 user licenses and parallels

What we request in return: Give us a shoutout on your GitHub project page or website. Post and hyperlink our attached logo, and a line about how you use BrowserStack. We are counting on you to help us spread the word to the open source community.

Next steps:

1. Sign up for a BrowserStack free trial account.
2. Add a status badge to your project, if you're using Automate.
3. Send us your free trial account email ID and the updated GitHub page.

We look forward to being a part of your open source journey. Have a great day!

Regards,
Sumit Banga
BrowserStack Support

Got this response back from browserstack. Looks like we just need to add a badge to the readme and sign up for a free trial.

I will get this done after I get Jasmine setup. What browsers do we want to enable starting out?

[UziTech]

@joshbruce do we want to use your email to sign up for this or should we create an email?

[joshbruce]

@UziTech: Hmmmmm...good question. Reading that email, I'm almost thinking we may want to slow down on achieving browser testing. I had an idea (dangerous, I know) for something else and I think we might be able to kill two birds with one stone.

But I'm not sure how the following could fit in with BS and CI.

@styfle: You mentioned somewhere else about putting things like the CONTRIBUTING into a /docs folder to generate a GitHub Pages site. As I was watching the USWDS evolution, they started with their package and docs site in one repo; over time users started giving feedback that they couldn't contribute easily because they couldn't discern what was site-related and what was documentation related.

I was thinking, we could move the all-caps to /docs. Create a separate repo for a GH Pages site. Then use the GitHub API to grab the various markdown docs and actually use marked to render the appropriate pages (this is kind of how I'm doing the 8fold UI Kit project site).

If we have that and BS...we have browser testing and we're eating our own dog food.

Again, don't think we have all the necessary information and project life yet to prioritize and maybe go this far with it. Maybe something we all can marinate on and save browser testing for later in the 0.x release cycles?

[davisjam]

save browser testing for later

This sounds good to me. Getting the test suite revamped seems higher priority.

[styfle]

I wouldn't mind working on the docs.
We could even have a version picker that would allow you to select a specific version of the docs based on a tag which will be helpful when we start introducing breaking changes.

My first thought was to utilize Jekyll with GitHub pages since that works out of the box to render markdown files but eating our own dogfood is a good idea too.

Let me know when you think all the PRs are merged for the md files and I'll start on moving stuff to the /docs folder ⚡️

[davisjam]

eating our own dogfood is a good idea too.

+1

[joshbruce]

ps. @UziTech, thank you for reaching out to them. Really appreciate it, sorry I didn't mention it in my first response. It seems like there is also the possibility of a lot of unknowns there and didn't think we should chase the squirrel much further than we already had. Thanks again, brother.

[UziTech]

I'm not sure dogfooding is going to do a very good job replacing automated browser testing. unless one of us makes it a habit to check our docs regularly with IE8

[joshbruce]

Agreed. Think that's the unknown BrowserStack + CI + question.

1. dogfood for our site
2. Browserstack checks for us against IE8 (we ain't doing nothing) ;)

We get a safe place to test potentially breaking things (our site). We get a site Browserstack can do its "Unlimited manual and automated testing for open source websites and mobile apps." And we can say things like, "This site brought to you by Marked."

Think the unknown - and totally not worth pursuing actively right now - question is: Can we get Travis and Browserstack to play nice together against a PR? And, can that happen across two repos, or would it have to be int one?

Complex question. Lots of variables. Totally not needed - but good brainstorming and coming up with ideas to consider for later, I think. Just throwing copper there.