Runtime: System.Text.Json usability study results

[terrajobst]

We've performed a usability study on the new System.Text.Json APIs which will replace JSON.NET as the default JSON experience in .NET Core 3.0:

• We focused on the primary "getting started" scenarios to validate that we picked the right type- and method names.
• We simulated what we believe to be common error cases to validate that our exception messages are understandable and actionable.
• Since we were short on time, we couldn't use the recruiting provided by the UX lab but instead brought in our own developers, which we recruited from the .NET engineering team

This has worked out well for us as we were able to get short term feedback which resulted in significant improvements to the overall API shape and usability. In the future, we should do more API focused UX studies 😊

Tasks

| # | Description | Code
|:--|:---------------------------------------|:-----
| 1 | Serialize and pretty print | Link
| 2 | Deserialize with trailing comma | Link
| 3 | Deserialize with custom property names | Link
| 4 | Read with DOM | Link
| 5 | Pretty print an existing JSON file | Link

Hypotheses

We expect developers to...

| # | Description | Result
|:--|:-------------------------------------------------------|:---------------
| 1 | ...assume the APIs are similar/identical to JSON.NET | Validated
| 2 | ...complete tasks 1-4 successfully | Validated
| 3 | ...struggle/unable to complete the task 5, because it requires working with two different components which is very different from JSON.NET | Invalidated
| 4 | ...not look at documentation for most of their coding as IntelliSense is good enough | Invalidated
| 5 | ...be happy with the experience of using the API | Validated
| 6 | ...plan on using System.Text.Json for their next JSON task | Partially Validated

Key Observations

• Everyone completed scenarios dotnet/corefx#1-#3, 2 didn't complete dotnet/corefx#4, 6 didn't complete dotnet/corefx#5
  
  
  
  • The reason was always "out of time". Every person who started a scenario successfully finished it.
  
  
  • We feel this validates that we picked the right scope for a 60 minute study of people from various backgrounds
  
  
• Overall people liked the new API
  
  
  
  • Key value prop for most people was the fact that "it's now built in"
  
  
  • Very few mentioned perf and the ones that did thought that it probably wouldn't make a difference in their scenarios.
  
  
  • Most people said they would prefer System.Text.Json for new APIs, but would stick to JSON.NET in code that already uses it.
  
  
  • [Action] None, this is in line with what we expected.
  
  
• Virtually everyone tried to google for sample code in the documentation
  
  
  
  • Everyone found our blog post and online documentation
  
  
  • People new to .NET were confused that JSON.NET and System.Text.Json weren't the same thing
  
  
  • Everyone complained that the current docs are lacking, especially sample code.
  
  
  • [Action] Docs are work in progress. We plan on shipping with a fully documented API surface and sample code.
  
  
• Virtually everyone found the key type relatively quickly
  
  
  
  • Some people were confused when types were static, rather than something that one can new up
  
  
  • [Action] None. This is by-design as we try to reduce the number of allocations.
  
  
• Virtually everyone had trouble selecting the correct method on
  JsonSerializer
  
  
  
  • We chose a very complicated naming convention
  
  
  • [Action] We simplified it by using names that people in the study told us were looking for.
  
  
• Most of our error messages were well received because they told people what
  the problem is & how they can fix them.
  
  
  
  • One error message was unclear because we reported an unexpected character when we could have reported that we didn't expect a comment.
  
  
  • [Action] We've reworded the error message.
  
  
  • We had one error message that was generated by a lower layer and provided an action item that doesn't make sense when people receive it when using the higher layer component.
  
  
  • [Action] We've reworded the error message to be more generic so it makes sense for both kind of consumers.
  
  
• Virtually everyone used the new Roslyn completion feature which will also show
  types from namespaces not imported yet.
  
  
  
  • However, virtually nobody realized that they were doing it.
  
  
  • This caused confusion when people found other APIs that happen to include Json in them, specifically the old data contract serialization.
  
  
  • [Action] Talk to the Roslyn IDE team to see if this something more people will run into. This could prove to be especially problematic for technologies that share type names (e.g. UserControl in WinForms and WPF) and both are referenced by the project.
  
  
• Optional parameters and async/await are usability cliffs
  
  
  
  • Many were confused by optional parameters and tried to pass them in, even though they didn't have to.
  
  
  • [Action] Talk to the Roslyn IDE team to see if we can visualize them better in VS.
  
  
  • [Action] Try to avoid heavy use of optional parameters for mainline APIs.
  
  
  • Async is a very steep cliff. People have no idea what to do in their code to make async work. While IntelliSense shows the usage by prefixing it with await, people didn't notice that.
  
  
  • [Action] Unclear; this is a broader issue.
  
  
• Virtually nobody used F1 to get help for APIs
  
  
  
  • Surprise. I expected more people to use F1.
  
  
• Virtually everyone used F12 to explore the API ("metadata view")
  
  
  
  • Surprise. I expected most people to explore the type via the
    
    IntelliSense completion
  
  
• Nobody used Object Browser to explorer the APIs
  
  
  
  • Surprise. I expected at least some people to.
  
  

[manigandham]

Glad to see the progress. I didn't know till now that F1 goes to the docs page for the type that's actually under the cursor, never saw this mentioned anywhere. F12 is probably used more because it's easier to dig through the code, especially for 3rd party libraries. Also lots of docs pages are nothing more than a print of the XML comments so looking at the code is just faster.

[benaadams]

Nobody used Object Browser to explorer the APIs

Object Browser is good; however the usability to get to it is hard. You have to open it and then type in and search for the class or method. Am sure in a previous version of VS there was a right-click option that took you directly to it. It would good if this returned as object browser gives the most context and holistic overview.

[crozone]

F1 goes to the docs page for known types??? I've been using VS for like 10 years and didn't know this. I always thought it just went to the generic VS help page. Maybe this should be in the right click menu for better discoverability?

Also I get why everyone used F12 to view the API, Alt-F12 "Peek" works wonders too. Reading method signatures with doc strings is usually the easiest and fastest way to tell exactly what a library expects. Intellisense is fine, but it doesn't always have the concise detail developers crave. Object Browser is a little clunky and hidden away compared to F12 which is a frequently used tool.

[Robot1que]

That's good news! Had a headache in few projects trying to keep JSON.NET dependency version the same.
Any chance of getting YAML parser as a part of .NET Core? In particular it would be nice to be able to use it in PowerShell Core without adding third party modules (because of the company policies).

[jods4]

Just tried asp.net core 3.0 (w/ the new default JSON formatter), so here's a piece of feedback:

• Add support for serializing dynamic/expando objects.

I understand that you are not aiming for full compatibility with JSON.NET but I feel it's a scenario worth covering.

I created an OData endpoint and when the client sends a query with $select (i.e. a custom projection), the returned shape is an Expando object.

[terrajobst]

Any chance of getting YAML parser as a part of .NET Core?

Not any time soon. This isn't mainstream enough warranting us having to build one. NuGet packages already exist :-)

[terrajobst]

I'm closing this to get the release PMs off my back who freak out about open bugs right now. However, please keep the comments/questions coming :-)

[nbarbettini]

Object Browser is good; however the usability to get to it is hard.

+1, I've tried and always end up frustrated because I can't quite remember how to.

In the future, we should do more API focused UX studies

💯 this was a great read @terrajobst!