Atmosphere: [Meta] Get some framework in place so that we can unit test Atmosphere code.

[SciresM]

This would be really nice, one-bit debugging really sucks (especially for early boot bugs).

[misson20000]

A few possible approaches (may be good to combine a few):

• Abstract away platform-specific stuff and compile certain pieces for PC, use a regular C++ testing library like GoogleTest.
  
  
  
  • Probably good for testing things like loader content override logic.
  
  
• Develop a test suite (GoogleTest again?) that can run on emulator.
  
  
  
  • Variation: run on hardware.
  
  
• Using some unicorn magic, build a blackbox testing framework that can run the same tests against N code and AMS code.
  
  
  
  • Good for verifying that we match N behavior.
  
  

[Crissal1995]

You can setup Travis CI or Jenkins and then use Codecov (compatible with both C and C++)

[jporsay]

@misson20000 Have you tried Catch2? I've used both and found it to be a better choice than GoogleTest. Simpler to set up, allows for both BDD or TDD test types and writing tests for it is a breeze =D.

[misson20000]

I have not! Thanks for telling me about it, I'll take a look. I'm not particularly married to GoogleTest, that's just the one I'm vaguely familiar with.

[Optiroc]

Since Catch2 was mentioned, and I was on the lookout for a new testing framework myself recently, may I also suggest DocTest? Extremely similar to Catch2 but faster by orders of magnitude.

[misson20000]

I'm going to have to make a table for this

[misson20000]

My personal favorite is the unicorn magic test rig, since it can:

• Run on release binaries. No worries about things behaving differently when compiled for testing vs. release.
• Run on Nintendo binaries, verifying that our test suite is asserting the correct behavior (I think we'd be particularly prone to errors here concerning behavior that's changed between versions).
• Test things like MMIO writes for boot.
• Estimate test suite coverage for not only stratosphere implementations, but also Nintendo binaries.

As far as design is concerned, I think it's best to write a new emulator specifically for testing that's designed to be easy to mock important things (services, SVCs, memory-mapped IO, etc.). We don't need implementations of most services, and don't really want them either since we're probably going to want to mock them anyway to test sad-paths. My personal favorite for these things is Ruby/RSpec. Imagine writing tests like these:

describe "nn::sm::detail::IUserInterface" do
  # binaries and version config loaded from cmdline/environment
  INITIALIZE = 0
  GET_SERVICE = 1
  REGISTER_SERVICE = 2
  UNREGISTER_SERVICE = 3

  before do
    # run until all threads blocked
    kernel.startup
    @session = kernel.managed_ports["sm:"].connect 
  end

  describe "GetService" do
    it "fails with 0x415 when not initialized" do
      expect(@session.send_message(GET_SERVICE) do
        data "example"
      end).to fail_with(0x415)
    end

    it "fails with 0x1015 when not allowed" do
      expect(@session.send_message(INITIALIZE) do
        pid
        u64 0
      end).to succeed
      expect(@session.send_message(GET_SERVICE) do
        data "example"
      end).to fail_with(0x1015)
    end
  end
end

describe "ro startup" do
  it "aborts if spl is inaccessible" do
    hle.sm.unregister("spl")
    expect do
      kernel.startup
    end.to raise_error(AbortError)
  end

  it "aborts if splIsDevelopment fails" do
    hle.spl = double("spl", :is_development => CMIFMessage.new(:result => 0xdead))
    expect do
      kernel.startup
    end.to raise_error(AbortError)
  end
end

describe "loader" do
  CREATE_PROCESS = 0

  it "registers SAC when loading a new process" do
    hle.sm_m = spy("sm:m")
    # mock out filesystem services, etc.
    kernel.startup
    session = hle.sm.get_port("ldr:pm").connect
    expect(session.send_message(CREATE_PROCESS) do
      ...
    end).to succeed
    expect(hle.sm_m).to have_received(:register_process).with(...)
  end
end

Could then set this up on Travis to run tests for every supported firmware version, and to download official binaries from a secret filestore and run against them too. This is something that, now that I have some free time, I'd be happy to get the ball rolling on, but would like design feedback first since it seems like I could easily pour lots of time into something that'd never get upstreamed. @SciresM ?

[neobrain]

I'm chiming in for Catch2 as well - it's trivial to set up, has a rich feature set, and if you wanted it can implement testing on the actual Switch too. (I've set it up on my 3DS previously, for instance.) Catch2 optionally supports the above-mentioned RSpec-syntax, too.

The only point against Catch2 is that it only covers unit testing, and hence doesn't provide mocking utilities. In my projects, that turned out not to be a big issue, but you'll have to decide that for yourself.

I would go with implementing the unit tests in C++ either way fwiw - especially for libstratosphere, the existing IPC definitions can be reused to write much more reliable and automated testing code.

[neobrain]

As far as actual testing methodology goes, I think abstracting away all direct hardware-interfacing, mocking it out, and then running the tests on PC would be best way to go. That's the only way you can reliably simulate corner cases in hardware behavior (e.g. booting with low battery).

For things like libstratosphere this is going to be exceptionally easy, whereas for things that interact with nontrivial MMIO I could see a lot of work being involved.

@misson20000's suggestion of using Unicorn could work too, and probably is easier to set up. I'm not sure we could achieve the same flexibility as the other approach though (imagine trying to run 1000000 tests that each require to be started from a clean start - Unicorn's startup overhead could make that infeasible), and it seems like more of an integration-testing focuses approach rather than unit-testing (with all the merits/drawbacks that involves).

[misson20000]

For my own reference, stuff to do before PR'ing:

• [x] Launch Nintendian 1.0.0 SM.
• [x] Launch Stratosphere SM targeting 1.0.0.
• [x] Launch all Nintendian SMs.
• [x] Launch Stratosphere SM targeting all firmware versions.
• [ ] Write some tests for SM.
• [x] Set up Makefile targets for running tests.
• [ ] Set up Travis.
  
  
  
  • [ ] Infrastructure for passing build artifacts between stages.
  
  
  • [ ] Infrastructure for downloading Nintendian reference implementations.
  
  
  • [ ] Travis config wrangling.
  
  
• [ ] Write docs for emulator.

Once I've finished all of these, I think it'll be presentable. After that, (I will copy this list to the PR), more things to do:

• [ ] Resolve symbols in backtraces using debug info from ELF files or IDA map files.
• [ ] Show what each thread is blocked on when deadlock occurs.
• [ ] Design a faster (forking?) test runner.
• [ ] PM
  
  
  
  • [ ] Bringup
  
  
  • [ ] Tests
  
  
• [ ] RO
  
  
  
  • [ ] Bringup
  
  
  • [ ] Tests
  
  
• [ ] SPL
  
  
  
  • [ ] Bringup
  
  
  • [ ] Tests
  
  
• [ ] Loader
  
  
  
  • [ ] Bringup
  
  
  • [ ] Tests
  
  
• [ ] Boot
  
  
  
  • [ ] Bringup
  
  
  • [ ] Tests
  
  
• [ ] Produce code coverage results by tracing blocks.