Problem-specifications: Insights - what data would be helpful in improving the tracks?

[kytrinyx]

The question has come up a few times, most recently in https://github.com/exercism/xjavascript/pull/103 by @ZacharyRSmith.

The ordering in the tracks is mostly arbitrary when starting out. It would be incredibly helpful to see where we people are falling off so that we can move harder exercises farther down in the list, or improve boring ones. Eventually, the idea of paths or clusters or topics would be a big improvement (see #63), but that might be a bit of a ways off, as I'm trying to do work around the onboarding user experience at the moment.

If we were to create a quick metrics-dashboard, what data should we have?

[soniakeys]

I was thinking recently about time-solve. It would be rough, after all you can fetch a problem and then not even look at it for months, but it should be common to fetch and solve promptly. If you made a histogram of submit time - fetch time, the first peak should represent the typical time to solve by people who fetched and solved promptly.

Abandonment rate for a problem might be (number of people that fetched the problem - number that submitted a solution) / number that fetched. It might not make sense for the first problem of a language track, but after that it might be interesting. Not sure though how to differentiate between problems abandoned because they are boring or because they are hard.

What would a graph look like of number of people submitting by exercise in config.json order? Would it look linear? Would there be cliffs? A cliff might indicate an exercise that causes people to abandon the whole language track. If time to solve on that exercise was still in line with previous exercises, it might just be boring. If time to solve was high, it might be too hard.

[kytrinyx]

This sounds like a really good data set for a first-pass analysis.

Eventually we would need to have the ability to narrow down the timeframe of the analysis so that we can see if the changes we make (removing/reordering/updating exercises) have made a useful difference.

What would the best way be to generate graphs? Are there any good graphing libraries that could integrate into the site? Or should this be a standalone task that could be run to export data to some format and then imported into some other tool?

[sshine]

should this be a standalone task

As a first iteration, generating these statistics for each track once may prove valuable to track maintainers. We can eventually think about exercises that change difficulty over time (I migrated some OCaml exercises from Jane Street "Core" to "Base", which made the built-in Map type a lot more difficult to grok).

If it has to be Ruby, there's a page on graph toolkits:

https://www.ruby-toolbox.com/categories/graphing

I don't know how the website looked like in 2015, but now that most tracks have core exercises, perhaps:

For each exercise:

• How many times has it been started
• How many times has something been submitted
• (If mentor-requested, how many times has it been approved)
• How many times has it been abandoned (@soniakeys)
• Average solve time (submit time - fetch time) sans outliers (@soniakeys)
• Average waiting time before first mentor feedback (feedback time - submit time)

For averages, perhaps only go a fixed amount of time back (e.g. some months).

[kytrinyx]

Ah, great find @sshine :) This one has been around for a while. We've been doing a lot of work on this lately, and will likely formalize some of these queries over the next 6-or-so months as we get a better grasp on what metrics are actually useful to us.

We've been tracking some of these already (informally).

@iHiD FYI for a list of questions that are of interest to maintainers with respect to usage stats.