Pythonnet: Can't reenter Python via passed C# function

[lostmsu]

Environment

• Pythonnet version: 2.3.0
• Python version: 3.6
• Operating System: Win 10 x64 1809

Details

• Describe what you were trying to get done.

I am trying to pass a C# function, that calls back into Python interpreter to Python code.

• What commands did you run?

int passed = 0;
void CallMe(int value) => passed = (dynamic)PythonEngine.Eval("42");
var callMeAction = new Action<int>(CallMe);
using (Py.GIL()) {
    dynamic callWith42 = PythonEngine.Eval("lambda f: f(42)");
    callWith42(callMeAction.ToPython());
}
Assert.AreEqual(expected: 42, actual: passed);

• If there was a crash, please include the traceback here.

It throws System.AccessViolationException in Eval inside CallMe:

   at Python.Runtime.Runtime.PyEval_GetGlobals()
   at Python.Runtime.PythonEngine.RunString(String code, Nullable`1 globals, Nullable`1 locals, RunFlagType flag)
   at Python.Runtime.PythonEngine.Eval(String code, Nullable`1 globals, Nullable`1 locals)

If I replace CallMe's body with passed = value, no exception is thrown

[lostmsu]

I also tried to add PythonEngine.BeginAllowThreads() above this code where I set up the PythonEngine in case MSTest does funky things with threading.

[denfromufa]

confirmed on the master branch:

scriptcs (ctrl-c to exit or :help for help)

> #r "C:\Python\Anaconda3\Lib\site-packages\Python.Runtime.dll"
> using Python.Runtime;
> int passed = 0;
> void CallMe(int value) => passed = (dynamic)PythonEngine.Eval("42");
> var callMeAction = new Action<int>(CallMe);
> using (Py.GIL()) {
*     dynamic callWith42 = PythonEngine.Eval("lambda f: f(42)");
*     callWith42(callMeAction.ToPython());
* }

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Python.Runtime.Runtime.PyEval_GetGlobals()
   at Python.Runtime.PythonEngine.RunString(String code, Nullable`1 globals, Nullable`1 locals, RunFlagType flag)
   at Python.Runtime.PythonEngine.Eval(String code, Nullable`1 globals, Nullable`1 locals)
   at Submission#2.CallMe(Int32 value)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Python.Runtime.MethodBinder.Invoke(IntPtr inst, IntPtr args, IntPtr kw, MethodBase info, MethodInfo[] methodinfo)
   at Python.Runtime.MethodBinder.Invoke(IntPtr inst, IntPtr args, IntPtr kw)
   at Python.Runtime.Runtime.PyObject_Call(IntPtr pointer, IntPtr args, IntPtr kw)
   at Python.Runtime.PyObject.Invoke(PyTuple args, PyDict kw)
   at Python.Runtime.PyObject.TryInvoke(InvokeBinder binder, Object[] args, Object& result)
   at CallSite.Target(Closure , CallSite , Object , PyObject )
   at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid2[T0,T1](CallSite site, T0 arg0, T1 arg1)
   at Submission#4..ctor(Object[] submissionArray, Object& submissionResult)
   at Submission#4.<Factory>(Object[] submissionArray)
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmission(Func`2 submissionRunner)
   at Microsoft.CodeAnalysis.Scripting.Script.TryRunFrom(ScriptState state, ScriptExecutionState& executionState, Object& value)
   at Microsoft.CodeAnalysis.Scripting.Script.Run(Object globals)
   at Microsoft.CodeAnalysis.Scripting.Script.Run(Object globals)
   at ScriptCs.Engine.Roslyn.CSharpScriptEngine.GetScriptState(String code, Object globals)
   at ScriptCs.Engine.Roslyn.CommonScriptEngine.Execute(String code, Object globals, SessionState`1 sessionState)
   at ScriptCs.Engine.Roslyn.CommonScriptEngine.Execute(String code, String[] scriptArgs, AssemblyReferences references, IEnumerable`1 namespaces, ScriptPackSession scriptPackSession)
   at ScriptCs.Repl.Execute(String script, String[] scriptArgs)
   at ScriptCs.Command.ExecuteReplCommand.ExecuteLine(IRepl repl)
   at ScriptCs.Command.ExecuteReplCommand.Execute()
   at ScriptCs.Program.Main(String[] args)

[lostmsu]

@denfromufa so I just debugged this issue. It seems that GIL is not locked upon reentry. And PyThreadState_GET returns NULL. Changing CallMe to

void CallMe(int value) {
  using (Py.GIL()) {
    passed = (dynamic)PythonEngine.Eval("42");
  }
}

works.

Now I am not an expert on Python embedding. Is it by design? As you can see, the topmost function (in my sample - callWith42) is wrapped in Py.GIL(), and does not release it until exception happens.

[dasMulli]

@lostmsu thanks that helped me as well when trying to call scipy.optimize.least_squares() with a C# error function.. (PyObject ErrorFunction(PyObject p)) VS debugger would crash just evaluating p.ToString(), which it does by default for all variables.
Wrapping the error function into a using(Py.GIL) helped - and not setting a breakpoint outside of that using :)

[lostmsu]

Seems weird though. According to my understanding, this should only be required when the thread executing callback is different from the thread, that started Python interpreter, which was not the case for me. Perhaps BeginAllowThreads changes that.