Dhall-lang: Tracking issue for missing tests and corner cases

[Nadrieril]

The test suite in this repo is extraordinarily useful and quite extensive. @singpolyma has also been doing an awesome job recently of adding a lot of unit tests (thanks !).
I thought it would be useful to have a central issue where we list which bits of the spec are or aren't fully tested, and corner cases of the semantics that aren't resolved one way or another yet.

I have two main goals with this: have unit tests for everything to help new implementors, and maximize test coverage for robustness. Another benefit of unit tests is that we're less likely to forget adding tests for every new feature.
I used a coverage tool to find missing corner cases with my implementation, so YMMV. If anyone else has coverage data to contribute, that would be awesome.

Feel free to edit this list to add entries or edit existing ones.

• [ ] parser
  
  
  
  • [x] success unit tests
  
  
  • [x] failure unit tests
  
  
  • [x] import unit tests https://github.com/dhall-lang/dhall-lang/pull/517
  
  
  • [x] Unions variants without type
  
  
  • [x] Some x
  
  
  • [x] None
  
  
  • [x] Kind
  
  
  • [x] fail on Some on its own
  
  
  • [x] fail on merge and merge x
  
  
  • [x] We currently allow Some x y z. Decide what we want and test it. https://github.com/dhall-lang/dhall-lang/pull/467
  
  
  • [x] We currently disallow merge x y z. Decide what we want and test it. https://github.com/dhall-lang/dhall-lang/pull/467
  
  
  • [x] We currently allow True@2. Decide what we want and test it. Discussed in https://github.com/dhall-lang/dhall-lang/issues/462
  
  
  • [x] (merge x y): t
  
  
  • [x] merge x y: t u
  
  
  • [x] [] : List x y
  
  
  • [ ] Case sensitivity tests for builtins and keywords
  
  
  • [ ] IPvFuture addresses
  
  
• [ ] import
  
  
  
  • [x] success unit tests
  
  
  • [x] failures unit tests
  
  
  • [ ] add an import cycle that can only be detected if path canonicalization is done correctly (if that's possible)
  
  
  • [ ] A failure test for a cached protected import where the the bytes of the cached expression don't match the hash in the filename.
  
  
• [ ] typecheck
  
  
  
  • [x] success unit tests
  
  
  • [x] failures unit tests
  
  
  • [x] interesting equivalence cases, like alpha-equivalence
  
  
  • [x] variable shadowing, like \(x: Type) -> \(x: x) -> x
  
  
  • [x] check that typechecking respects import boundaries for variables
  
  
  • [ ] (let union = < x | y > in union).x (from https://github.com/dhall-lang/dhall-lang/pull/538)
  
  
• [ ] normalization
  
  
  
  • [x] unit tests
  
  
  • [x] interesting equivalence cases, like if x then NaN else NaN
  
  
  • [x] variable shadowing, like \(x: Type) -> \(x: x) -> x
  
  
  • [ ] merge {} <x = y>
  
  
  • [ ] {x = y}.z
  
  
  • [ ] "${""}${x}", "${"${x}"}"
  
  
  • [ ] check that normalization respects import boundaries for variables
  
  
• [x] alpha-normalization
  
  
  
  • [ ] unit tests
    
    
    
    
    
    • [x] lambda
    
    
    
    • [x] forall
    
    
    
    • [ ] let
    
    
    
  
  
  • [x] variable shadowing, like \(x: Type) -> \(x: x) -> x
  
  
• [ ] binary (binary and parser success tests will probably be shared)
  
  
  
  • [ ] encoding/decoding success unit tests
  
  
  • [ ] decoding failures unit tests
  
  
  • [x] variable named _
  
  
  • [x] \(_: x) -> y
  
  
  • [x] Some x
  
  
  • [x] -1
  
  
  • [x] ../path #517
  
  
  • [x] variable named as a quoted builtin like `Bool` https://github.com/dhall-lang/dhall-lang/issues/462
  
  
• [x] hash
  
  
  
  • [x] unit tests
  
  

[Gabriel439]

The Haskell project also has a coverage report it generates (although you have to build it locally for the links to highlighted source code to work): https://hydra.dhall-lang.org/job/dhall-haskell/master/coverage-dhall/latest/download/1/hpc_index.html

Once I update the Haskell package to run the latest set of tests I can report back with anything else that I find

[f-f]

And here is the coverage report for dhall-clj's master branch: https://coveralls.io/github/f-f/dhall-clj?branch=master

I'm in the process of catching up on dhall-lang's master, so the new test suites are still missing

[philandstuff]

Something I noticed today: the import unit tests don't have a test for successfully importing from an environment variable. I guess that's hard without requiring the test harness to set up the environment variable as expected.

[Nadrieril]

I checked quite a few boxes thanks to https://github.com/dhall-lang/dhall-lang/pull/523 :tada:

[sjakobi]

The Haskell project also has a coverage report it generates (although you have to build it locally for the links to highlighted source code to work): https://hydra.dhall-lang.org/job/dhall-haskell/master/coverage-dhall/latest/download/1/hpc_index.html

@Gabriel439 The links for the per-module reports don't work. Do you know how to fix that?

EDIT: I've since discovered stack test dhall --coverage. Neat! :)

[Gabriel439]

@sjakobi: Unfortunately, I do not know how to fix it. You have to run the coverage check locally