Clearscript: "RangeError: Maximum call stack size exceeded" with default V8ScriptEngine constructor

[acatr]

We are using ClearScript to run some JavaScript on the back end of our web service, however for a small portion of the calls (<1%) it is throwing "RangeError: Maximum call stack size exceeded" when we are creating the V8ScriptEngine with its default constructor. Does anyone know what might be going wrong or how it could be resolved?

Unhandled Exception: Microsoft.ClearScript.ScriptEngineException: RangeError: Maximum call stack size exceeded
    at V8Exception.ThrowScriptEngineException(V8Exception* )
    at Microsoft.ClearScript.V8.V8ContextProxyImpl.Execute(DocumentInfo documentInfo, String gcCode, Boolean evaluate)
    at Microsoft.ClearScript.V8.V8ScriptEngine.<>c__DisplayClass71_0.<Execute>b__0()
    at Microsoft.ClearScript.ScriptEngine.ScriptInvokeInternal[T](Func`1 func)
    at Microsoft.ClearScript.V8.V8ScriptEngine.<>c__DisplayClass76_1`1.<ScriptInvoke>b__0()
    at Microsoft.ClearScript.V8.?A0xb8863588.LockCallback(Void* pvArg)
    at Microsoft.ClearScript.V8.V8ContextProxyImpl.InvokeWithLock(Action gcAction)
    at Microsoft.ClearScript.V8.V8ScriptEngine.ScriptInvoke[T](Func`1 func)
    at Microsoft.ClearScript.V8.V8ScriptEngine.Execute(DocumentInfo documentInfo, String code, Boolean evaluate)
    at Microsoft.ClearScript.ScriptEngine.Evaluate(DocumentInfo documentInfo, String code, Boolean marshalResult)
    at Microsoft.ClearScript.ScriptEngine.Evaluate(DocumentInfo documentInfo, String code)
    at Microsoft.ClearScript.ScriptEngine.Evaluate(String documentName, Boolean discard, String code)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(V8Runtime runtime, String name, V8RuntimeConstraints constraints, V8ScriptEngineFlags flags, Int32 debugPort)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(String name, V8RuntimeConstraints constraints, V8ScriptEngineFlags flags, Int32 debugPort)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(String name, V8RuntimeConstraints constraints, V8ScriptEngineFlags flags)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(String name, V8RuntimeConstraints constraints)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor()
    ...

[ClearScriptLib]

Hi @acatr,

The V8ScriptEngine constructor runs a small script to set up some data for internal use, and it looks like V8 is detecting a stack overflow while executing that script.

We haven't encountered this behavior and can only guess at its cause. Some questions:

1. What ClearScript version are you using?
2. Are you using a public NuGet package, or did you build ClearScript yourself?
3. You posted a partial stack above. Is the full stack unusually long?
4. Can you estimate the number of V8ScriptEngine instances your application uses concurrently?
5. Are you constructing V8ScriptEngine in a 32-bit process?
6. Can you post a minimal code sample that reproduces the issue?

Thanks!

[acatr]

1. Version 5.5.5
2. The public NuGet package
3. No, it wasn't unusually long, I just cut it off to what I thought was applicable. Here is the full stack:

    at V8Exception.ThrowScriptEngineException(V8Exception* )
    at Microsoft.ClearScript.V8.V8ContextProxyImpl.Execute(DocumentInfo documentInfo, String gcCode, Boolean evaluate)
    at Microsoft.ClearScript.V8.V8ScriptEngine.<>c__DisplayClass71_0.<Execute>b__0()
    at Microsoft.ClearScript.ScriptEngine.ScriptInvokeInternal[T](Func`1 func)
    at Microsoft.ClearScript.V8.V8ScriptEngine.<>c__DisplayClass76_1`1.<ScriptInvoke>b__0()
    at Microsoft.ClearScript.V8.?A0xb8863588.LockCallback(Void* pvArg)
    at Microsoft.ClearScript.V8.V8ContextProxyImpl.InvokeWithLock(Action gcAction)
    at Microsoft.ClearScript.V8.V8ScriptEngine.ScriptInvoke[T](Func`1 func)
    at Microsoft.ClearScript.V8.V8ScriptEngine.Execute(DocumentInfo documentInfo, String code, Boolean evaluate)
    at Microsoft.ClearScript.ScriptEngine.Evaluate(DocumentInfo documentInfo, String code, Boolean marshalResult)
    at Microsoft.ClearScript.ScriptEngine.Evaluate(DocumentInfo documentInfo, String code)
    at Microsoft.ClearScript.ScriptEngine.Evaluate(String documentName, Boolean discard, String code)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(V8Runtime runtime, String name, V8RuntimeConstraints constraints, V8ScriptEngineFlags flags, Int32 debugPort)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(String name, V8RuntimeConstraints constraints, V8ScriptEngineFlags flags, Int32 debugPort)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(String name, V8RuntimeConstraints constraints, V8ScriptEngineFlags flags)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(String name, V8RuntimeConstraints constraints)
    at Microsoft.ClearScript.V8.V8ScriptEngine..ctor()
    at Callbacks.Services.ClearScriptService..ctor(HttpRequestMessage request)
    at lambda_method(Closure , IBuilderContext )
    at Microsoft.Practices.ObjectBuilder2.DynamicBuildPlanGenerationContext.<>c__DisplayClass1.<GetBuildMethod>b__0(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.DynamicMethodBuildPlan.BuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.BuilderContext.NewBuildUp(NamedTypeBuildKey newBuildKey)
    at Microsoft.Practices.Unity.ObjectBuilder.NamedTypeDependencyResolverPolicy.Resolve(IBuilderContext context)
    at lambda_method(Closure , IBuilderContext )
    at Microsoft.Practices.ObjectBuilder2.DynamicBuildPlanGenerationContext.<>c__DisplayClass1.<GetBuildMethod>b__0(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.DynamicMethodBuildPlan.BuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.BuilderContext.NewBuildUp(NamedTypeBuildKey newBuildKey)
    at Microsoft.Practices.Unity.ObjectBuilder.NamedTypeDependencyResolverPolicy.Resolve(IBuilderContext context)
    at lambda_method(Closure , IBuilderContext )
    at Microsoft.Practices.ObjectBuilder2.DynamicBuildPlanGenerationContext.<>c__DisplayClass1.<GetBuildMethod>b__0(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.DynamicMethodBuildPlan.BuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context)
    at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)
    at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)
    --- End of inner exception stack trace ---
    at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)
    at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, String name, IEnumerable`1 resolverOverrides)
    at Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name, ResolverOverride[] resolverOverrides)
    at Microsoft.Practices.Unity.UnityContainerExtensions.Resolve(IUnityContainer container, Type t, ResolverOverride[] overrides)
    at Unity.WebApi.UnityDependencyScope.GetService(Type serviceType)
    at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.GetInstanceOrActivator(HttpRequestMessage request, Type controllerType, Func`1& activator)
    at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)
    --- End of inner exception stack trace ---
    at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)
    at System.Web.Http.Controllers.HttpControllerDescriptor.CreateController(HttpRequestMessage request)
    at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()

1. Most of the time it should be a fairly small amount (<10), but getting up to 100 is possible.
2. Yes, it is a 32 bit process.
3. Unfortunately we haven't been able to intentionally reproduce it, we just see it occasionally in our logs. Here is a simple example of what our code looks though:
   ```C#
   using System.Net;
   using System.Net.Http;
   using System.Threading;
   using System.Threading.Tasks;
   using System.Web.Http;
   using Microsoft.ClearScript.V8;
   using Microsoft.Practices.Unity;
   using Unity.WebApi;

namespace WebApplication
{
public class WebApiApplication : System.Web.HttpApplication
{
protected void Application_Start()
{
GlobalConfiguration.Configure(WebApiConfig.Register);
}
}

public static class WebApiConfig
{
    public static void Register(HttpConfiguration config)
    {
        var unityContainer = new UnityContainer();
        unityContainer.RegisterType<IFooService, FooService>();
        config.DependencyResolver = new UnityDependencyResolver(unityContainer);
        config.MessageHandlers.Add(new RequestRegistrationHandler());
        config.MapHttpAttributeRoutes();
    }
}

public class RequestRegistrationHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var unityContainer = (IUnityContainer)request.GetDependencyScope().GetService(typeof(IUnityContainer));
        unityContainer.RegisterInstance(request);
        return base.SendAsync(request, cancellationToken);
    }
}

public class FooController : ApiController
{
    private IFooService service;

    public FooController(IFooService service)
    {
        this.service = service;
    }

    [HttpGet, Route("foo")]
    public HttpResponseMessage Foo()
    {
        var result = service.Foo();
        return Request.CreateResponse(HttpStatusCode.OK, result.ToString());
    }
}

public interface IFooService
{
    object Foo();
}

public class FooService : IFooService
{
    V8ScriptEngine scriptEngine;

    public FooService(HttpRequestMessage request)
    {
        scriptEngine = new V8ScriptEngine();
        request.RegisterForDispose(scriptEngine);
    }

    public object Foo()
    {
        return scriptEngine.Evaluate("1+1");
    }
}

}
```

[ClearScriptLib]

Hi @acatr,

Thanks for posting all that information. Unfortunately the cause remains elusive.

You aren't specifying script stack limits, the script being executed is tiny, and thread stack overflow wouldn't throw a script exception. The longer stack indicates some recursion, but the script engine wouldn't take that into account unless you invoked it recursively, which isn't the case here as you're just constructing it.

All this would seem to point at a V8 bug. Can you try ClearScript 5.5.6? It comes with a newer V8 version.

Thanks again!

[acatr]

Unfortunately updating to 5.5.6 did not help, we are still seeing the issue.

[ClearScriptLib]

Hi @acatr,

Hmm. We've used a 32-bit test program to create millions of V8ScriptEngine instances and run a small script in each, but we still can't reproduce this issue.

Can you estimate how often this happens, on average? That is, how many scripts does your application run successfully before encountering this issue?

Also, does your application recover from this situation? Can it run scripts after encountering the exception?

Thanks!

[acatr]

We have had it happen on 349 out of 189,710 calls the past 30 days. No real pattern that we can see for when it happens (every n calls, high traffic, high cpu, high memory usage).

It does recover in the sense that subsequent calls to the endpoint succeed, but the calls that get the exception are not able to recover. We tried adding a try/catch for it and retry a couple times with a small sleep between but that didn't help.

We are running on AWS t3.large instances if that might trigger any other thoughts about what might be happening.

I do appreciate the help with this, I know it's not easy when I'm not able to give you a way to reproduce the issue.

[ClearScriptLib]

Hi @acatr,

We tried adding a try/catch for it and retry a couple times with a small sleep between but that didn't help.

Please clarify. Do you mean that retrying V8ScriptEngine construction threw the same exception? If so, how did you recover? Did you have to recycle the process, or were you able to abandon the current request or operation and continue successfully?

Thanks!

[acatr]

During a request we will try to construct the V8ScriptEngine up to three times with a 50 ms sleep between but all the retries throw the same exception. These requests just end up responding with a 500 while other requests can continue to be handled successfully during this time.

[ClearScriptLib]

Hi @acatr,

These requests just end up responding with a 500 while other requests can continue to be handled successfully during this time.

Do subsequent requests succeed within the same process?

A few more questions if you don't mind:

1. Is the exception always thrown from the V8ScriptEngine constructor, as opposed to the Evaluate method?
2. Do you always evaluate JavaScript from source code, or do you use compiled scripts?
3. Your code sample above indicates that you use a new V8ScriptEngine instance to process each request. Is that always the case?
4. How do you prepare each instance for script execution? Do you expose large .NET APIs, run large boilerplate scripts, etc.?

Sorry about all these questions, but with no known way to reproduce the issue, any information could help us diagnose this issue.

Thanks!

[acatr]

Yes, subsequent requests succeed within the same process.

1. Yes, we only see the exception coming from the constructor.
2. We do not use the Compile method. We are only calling Execute, Evaluate, and AddHostObject. And then we also set MaxRuntimeStackUsage to 200,000 once we successfully construct the V8ScriptEngine.
3. Yes, each request will always create its own V8ScriptEngine.
4. For each request, we execute some of our scripts (~10,000 lines of js) for initial setup and then add ~30 functions that will call back into our C# code. Then we just Execute whatever the request is calling for. This always seems to work fine once we make it there, it is just the rare exceptions during construction that are giving us issues.

[ClearScriptLib]

@acatr Thanks, that's very useful info. Apparently multiple instantiation attempts fail, but only when issued from the same thread/stack environment as the original failure. Once you abandon the operation and let the stack unwind, subsequent attempts succeed, as do operations being processed on other threads. Is that correct?

Also, are you aware of any scenarios in which your application _re-enters_ V8? That is, does your application expose host APIs that synchronously run other scripts?

[acatr]

Apparently multiple instantiation attempts fail, but only when issued from the same thread/stack environment as the original failure. Once you abandon the operation and let the stack unwind, subsequent attempts succeed, as do operations being processed on other threads. Is that correct?

Yes, that is correct.

Also, are you aware of any scenarios in which your application re-enters V8? That is, does your application expose host APIs that synchronously run other scripts?

Yes, this does happen, on pretty much every request.

[ClearScriptLib]

Hi @acatr,

We've identified a set of conditions that leads to the behavior you're seeing. We believe the root cause is a V8 bug that we've reported here.

Unfortunately, because we can't reproduce your issue, we don't know whether fixing that bug would resolve it. If you'd like to test a potential fix and wouldn't mind building ClearScript yourself, please let us know.

Thanks again for reporting this issue!

[acatr]

Yeah, we can test the potential fix.

[ClearScriptLib]

Hi @acatr,

We've committed a potential fix here.

If you haven't built ClearScript before, please see the instructions here.

If you've already built ClearScript, pull the fix, rerun V8Update without the /N flag, and do a full rebuild of ClearScript.sln.

Please let us know if this fix works for you.

Thank you!

[acatr]

Been a bit busy but I was finally able to get everything built and consumed in our service. We haven't seen the issue yet so it looks good so far. We want to give it a few more days of testing to truly confirm that it is fixed though. I'll reply back back with our results sometime next week.

[acatr]

Everything is still looking good so it does seem to be fixed. Thanks for the work on this.

Do you have an estimated date for when a new version of the NuGet package will be published?

[ClearScriptLib]

Hi @acatr,

Thank you very much for testing the fix! ClearScript 5.6.0 should be out later this month.

Cheers!

[acatr]

Awesome, thanks again.

[ClearScriptLib]

ClearScript 5.6 incorporates this patch.