Problem-specifications: Standardize test case nesting

[cmccandless]

Consistent nesting level

There is not standard for how test cases should be nested in canonical-data.json. For example, suppose an exercise with depth=2 looks like this:

{
    "exercise": "foo",
    "cases": [
        {
            "description": "foo-group",
            "cases": [...]
        }
    ]
}

Here is every current exercise, sorted by depth descending, name ascending:
| Exercise | Depth |
| --- | --- |
| beer-song | 3 |
| complex-numbers | 3 |
| kindergarten-garden | 3 |
| rational-numbers | 3 |
| acronym | 2 |
| affine-cipher | 2 |
| allergies | 2 |
| alphametics | 2 |
| atbash-cipher | 2 |
| binary-search-tree | 2 |
| book-store | 2 |
| clock | 2 |
| custom-set | 2 |
| difference-of-squares | 2 |
| dnd-character | 2 |
| etl | 2 |
| food-chain | 2 |
| forth | 2 |
| gigasecond | 2 |
| grains | 2 |
| grep | 2 |
| high-scores | 2 |
| house | 2 |
| isogram | 2 |
| list-ops | 2 |
| micro-blog | 2 |
| nucleotide-count | 2 |
| ocr-numbers | 2 |
| pascals-triangle | 2 |
| perfect-numbers | 2 |
| phone-number | 2 |
| pig-latin | 2 |
| pov | 2 |
| prime-factors | 2 |
| protein-translation | 2 |
| queen-attack | 2 |
| rail-fence-cipher | 2 |
| resistor-color | 2 |
| rest-api | 2 |
| robot-simulator | 2 |
| rotational-cipher | 2 |
| run-length-encoding | 2 |
| scale-generator | 2 |
| secret-handshake | 2 |
| simple-cipher | 2 |
| triangle | 2 |
| trinary | 2 |
| twelve-days | 2 |
| variable-length-quantity | 2 |
| all-your-base | 1 |
| anagram | 1 |
| armstrong-numbers | 1 |
| binary | 1 |
| binary-search | 1 |
| bob | 1 |
| bowling | 1 |
| change | 1 |
| circular-buffer | 1 |
| collatz-conjecture | 1 |
| connect | 1 |
| crypto-square | 1 |
| darts | 1 |
| diamond | 1 |
| diffie-hellman | 1 |
| dominoes | 1 |
| flatten-array | 1 |
| go-counting | 1 |
| grade-school | 1 |
| hamming | 1 |
| hello-world | 1 |
| isbn-verifier | 1 |
| knapsack | 1 |
| largest-series-product | 1 |
| leap | 1 |
| luhn | 1 |
| markdown | 1 |
| matching-brackets | 1 |
| matrix | 1 |
| meetup | 1 |
| minesweeper | 1 |
| nth-prime | 1 |
| palindrome-products | 1 |
| pangram | 1 |
| poker | 1 |
| proverb | 1 |
| pythagorean-triplet | 1 |
| raindrops | 1 |
| react | 1 |
| rectangles | 1 |
| resistor-color-duo | 1 |
| resistor-color-trio | 1 |
| reverse-string | 1 |
| rna-transcription | 1 |
| roman-numerals | 1 |
| saddle-points | 1 |
| satellite | 1 |
| say | 1 |
| scrabble-score | 1 |
| series | 1 |
| sgf-parsing | 1 |
| sieve | 1 |
| space-age | 1 |
| spiral-matrix | 1 |
| sublist | 1 |
| sum-of-multiples | 1 |
| tournament | 1 |
| transpose | 1 |
| two-bucket | 1 |
| two-fer | 1 |
| word-count | 1 |
| word-search | 1 |
| wordy | 1 |
| yacht | 1 |
| zebra-puzzle | 1 |
| zipper | 1 |

This unnecessarily complicates test generators

Inconsistent nesting level within a single exercise

Additionally, there are a handful of exercises that use nested cases, but do not use them in a uniform way, like so:

{
  "exercise": "resistor-color",
  "version": "1.0.0",
  "cases": [
    {
      "description": "Color codes",
      "cases": [
        {
          "description": "Black",
          "property": "colorCode",
          "input": {
            "color": "black"
          },
          "expected": 0
        },
        {
          "description": "White",
          "property": "colorCode",
          "input": {
            "color": "white"
          },
          "expected": 9
        },
        {
          "description": "Orange",
          "property": "colorCode",
          "input": {
            "color": "orange"
          },
          "expected": 3
        }
      ]
    },
    {
      "description": "Colors",
      "property": "colors",
      "input": {},
      "expected": ["black", "brown", "red", "orange", "yellow", "green", "blue", "violet", "grey", "white"]
    }
  ]
}

These exercises all have inconsistent nesting:

• binary-search-tree
• complex-numbers
• dnd-character
• grains
• high-scores
• kindergarten-garden
• rational-numbers
• resistor-color

This practice is especially difficult to parse for generators.

Conclusion

In my opinion, the inconsistent nesting depth from exercise to exercise may be acceptable, if a "caseDepth" property were added at the top level. However, the exercises that use inconsistent nesting within the same data file should be corrected.

[coriolinus]

It's not obvious to me why this matters. It's clear from the spec that any cases node may contain other cases nodes; any recursive parser/generator should be able to handle arbitrary nesting in arbitrary patterns. Moreover, limiting to a depth of, say, 2, wouldn't substantially simplify the parser or generator.

Things would get simpler if we eliminated nesting entirely, but there's value to humans in being able to group and document a set of tests. I would oppose eliminating nesting entirely.

[cmccandless]

Things would get simpler if we eliminated nesting entirely, but there's value to humans in being able to group and document a set of tests. I would oppose eliminating nesting entirely.

Agreed. And I also agree that maybe using a single standard depth across all exercises is probably unnecessary. As a contributor implementing a generator for a new exercise, I can look at canonical-data and surmise how to structure the template based on how many layers deep cases are nested.

However, that structure is not clear when cases are not nested at a consistent level within that same exercise.

[ErikSchierboom]

I've also hit this issue when writing the C# and F# test generator. I do like the idea of grouping test cases, which can really help when reading the canonical data to see which test cases are related to each other.

I think the main issue is that the same key is used for defining a group of test cases and for listing test cases. It might be worth considering using a different key for a grouping of test cases versus a listing of test cases.

[SaschaMann]

Could you explain what you mean with "inconsistent nesting depth"?

I took a look at the complex-numbers example and can't find any inconsistencies. There are ungrouped top level tests, and then (nested) grouped tests. But they all look consistent with each other to me.

[cmccandless]

There are ungrouped top level tests, and then (nested) grouped tests.

That's exactly what I mean, and you've found a better way to describe it: a mixture of ungrouped and grouped cases (within the same exercise), opposed to sticking to one or the other.

[coriolinus]

There are ungrouped top level tests, and then (nested) grouped tests.

I feel dumb for asking, but why is this a problem? It feels like a natural
use of the spec.

On Tue, Sep 15, 2020 at 8:49 PM Corey McCandless [e-mail]
wrote:

There are ungrouped top level tests, and then (nested) grouped tests.

That's exactly what I mean, and you've found a better way to describe it:
a mixture of ungrouped and grouped cases (within the same exercise),
opposed to sticking to one or the other.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/exercism/problem-specifications/issues/1671#issuecomment-692905684,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3V4TSJPEDEPKSAH32ZDQLSF6ZL5ANCNFSM4RIE26QQ
.

[cmccandless]

If the consensus is that it's _not_ a problem, then that's fine. It was a hurdle I faced with the Python test generator, but if the other maintainers feel it serves a purpose, then no change is needed.

[ErikSchierboom]

Just to be clear, do people think there's any merit in the following suggestion:

It might be worth considering using a different key for a grouping of test cases versus a listing of test cases.

Something like:

{
  "exercise": "foo",
  "caseGroup": [
    {
      "description": "foo-group",
      "cases": [...]
    }
  ]
}

or

{
  "exercise": "foo",
  "group": [
    {
      "description": "foo-group",
      "cases": [...]
    }
  ]
}

[coriolinus]

If we were starting from scratch, I could see some value in defining a group object which contained both an optional sub-groups key and an optional cases key: it makes it absolutely clear what is a test case, and what is a group.

However, we're not starting from scratch; across the tracks, a fairly sizeable number of people have written a fairly substantial amount of software assuming the current setup, all of which would be broken if we introduced a new format. We'd also have to transform all the existing specifications; that could be automated, but the automation itself would take some development. I don't think the benefits outweigh the costs.

[ErikSchierboom]

Yep, that makes sense.

[ErikSchierboom]

If the consensus is that it's not a problem, then that's fine. It was a hurdle I faced with the Python test generator, but if the other maintainers feel it serves a purpose, then no change is needed.

@cmccandless The above seems to suggest this can be closed? It has been a very useful question and we have the above discussion for future reference.

[ErikSchierboom]

Thanks everybody for chiming in!