Azure-webjobs-sdk: In-Memory Integration Testing

[jaredcnance]

The problem

I would like to be able to run integration tests using the Azure Function runtime.

Workaround

I can use a FunctionRunner like this to fire up the runtime and then validate the results after some specified timeout.

There are several problems:

• If I attempt to debug the test, I won't be able to set breakpoints within the actual function code since they are not in the same process. I have to run the test from the command line and add some while(Debugger.IsAttached == false) {/*...*/} hack.
• I am required to set some arbitrary timeout and hope the function completes execution during that time
• I cannot run a single function, all functions within the compiled function project will be started when the test host starts
• requires func.exe to be installed on the testing server in a hardcoded location

Desired Solution

It would be great if we had a solution similar to AspNetCore that might look something like:

// arrange
var testHost = new TestHost(relativePathToCsproj);

// act
await testHost.RunFunctionAsync("MyFunction", maxInvocations: 1);

// assert...

A few things I would like to see included:

• specify which function should be executed (and probably the binding as well)
• specify the max number of times the function should be invoked before closing the host
• a reasonable debugging story
• assembly/type load errors should reproduce in tests. This probably means the test project should be executed by the function runtime

If the tools to do this already exist (within this project or any other), please let me know.

[paulbatum]

I would start by looking at our own e2e tests. For example we have this test fixture that should get you started.

If you end up building a test harness along the lines of what you outlined above that could be pretty useful in general, you should let us know.

Reopen this if you're still blocked.

[jaredcnance]

It looks like that fixture is dependent upon System.Web.Http, will that be changing with the movement to dotnet core?

[paulbatum]

Good question. I'm not sure what test fixture updates are planned, @fabiocav should be able to answer.

[jaredcnance]

I don't want to try and craft a solution based on volatile infrastructure, but this is something that is sorely needed, especially when the risk of runtime errors is so high because of https://github.com/Azure/azure-webjobs-sdk-script/issues/992

It looks like JobHost may be the thing I'm looking for in the future?

But then there is also this FunctionsLocalServer which seems to just start a new AspNetCore web host by spawning a new process to run the host via StartHostAction.

Can anyone provide me with the current design/direction so I don't do work that boxes me into a corner in future versions? Between this repository and the azure-functions-cli repo, I'm thoroughly confused as to how the runtime actually works.

[jaredcnance]

Can we re-open this issue? I'm not sure why it is closed. I don't think a proper solution has been provided nor do I have enough information to produce one.

[charisk]

I'm also facing the same challenge and agree with @jaredcnance - it would be great if there was some guidance here.

[jeffhollan]

Spent some time this weekend playing with this during some travel. I shared my project here. As @paulbatum suggested it follows the same patterns used by the azure functions host project. It's a proof of concept but does allow me to execute the Azure Functions Host inside the test suite and not as an external executable. Has some kinks (wasn't able to ever get the breakpoints inside the function to fire when debugging) but should enable you to verify assembly loading as part of a test

https://github.com/jeffhollan/functions-test-helper

[s-krawczyk]

+1

[danhamlin]

Curious if @fabiocav has anything he is able to share about a proper test fixture for function apps.

[riezebosch]

Checkout my solution: https://github.com/riezebosch/AzureFunctions.TestHelpers. It indeed uses the JobsHost from the SDK with some helper functions & classes aside to invoke http-triggered functions and wait for orchestration completion.

[jeffhollan]

@riezebosch this is awesome! So glad someone took some time to look into this as my sample is very stale at this point. Will check this out

[matt-lethargic]

3.5 years on, is there any official advice on how to achieve this?

[fabiocav]

@matt-lethargic there have been recommendations from @jeffhollan and @paulbatum on this issue. Aside from that, that has been no other work done.

Please do understand that, though the original question/request was made a while ago, there have been no commitments to bring anything beyond what we've been able to share. This is not something that have seen enough demand to be prioritized over other items with higher impact.

We decided to keep this open, as the idea is good and valid, but I want to make sure that wasn't sending the wrong message. In the meantime, I'll add a help wanted label to this as it would be a good opportunity for community engagement.