Machinelearning: InvalidProtocolBufferException: Protocol message was too large.

[jonathan-taina]

System information

• OS version/distro: Windows 10 Pro v1803, build 17134.885
• .NET Version (eg., dotnet --info): .NET Core 2.2.108

Issue

• What did you do? Attempted to load an ONNX model
• What happened? It failed: InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.SetSizeLimit() to increase the size limit.
• What did you expect? For the model to load successfully.

Source code / logs

The model is a pre-trained resnet34 trained on the MNIST handwritten digits database. The model was exported from pytorch.

I assume the .onnx file is valid, as I can view it without any issues at https://lutzroeder.github.io/netron/

I'd attach the file itself but github won't allow me, the filesize is 83.3MB; 73MB zipped.

var modelPath = $"{Directory.GetCurrentDirectory()}/Models/mnist_digits.onnx";
var pipeline = mlContext.Transforms.LoadImages("image", "", nameof(ImageNetData.ImagePath));
pipeline.Append(mlContext.Transforms.ResizeImages("image", ImageNetSettings.imageWidth, ImageNetSettings.imageHeight, "image"));
pipeline.Append(mlContext.Transforms.ExtractPixels("image"));
pipeline.Append(mlContext.Transforms.ApplyOnnxModel(modelPath));

Exception occurs on the final line above.

Stacktrace

InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.SetSizeLimit() to increase the size limit.
Google.Protobuf.CodedInputStream.RefillBuffer(bool mustSucceed)
Google.Protobuf.CodedInputStream.get_IsAtEnd()
Google.Protobuf.CodedInputStream.ReadTag()
Google.Protobuf.CodedInputStream.PeekTag()
Google.Protobuf.Collections.RepeatedField.AddEntriesFrom(CodedInputStream input, FieldCodec codec)
Microsoft.ML.Model.OnnxConverter.OnnxCSharpToProtoWrapper+GraphProto.MergeFrom(CodedInputStream input)
Google.Protobuf.CodedInputStream.ReadMessage(IMessage builder)
Microsoft.ML.Model.OnnxConverter.OnnxCSharpToProtoWrapper+ModelProto.MergeFrom(CodedInputStream input)
Google.Protobuf.MessageExtensions.MergeFrom(IMessage message, Stream input)
Google.Protobuf.MessageParser.ParseFrom(Stream input)
Microsoft.ML.Transforms.Onnx.OnnxModel..ctor(string modelFile, Nullable gpuDeviceId, bool fallbackToCpu, bool ownModelFile)
Microsoft.ML.Transforms.Onnx.OnnxTransformer..ctor(IHostEnvironment env, Options options, byte[] modelBytes)
Microsoft.ML.Transforms.Onnx.OnnxScoringEstimator..ctor(IHostEnvironment env, string modelFile, Nullable gpuDeviceId, bool fallbackToCpu)
Microsoft.ML.OnnxCatalog.ApplyOnnxModel(TransformsCatalog catalog, string modelFile, Nullable gpuDeviceId, bool fallbackToCpu)
ONNX_backend.Services.OnnxService.LoadHandwrittenDigitsModel(MLContext mlContext) in OnnxService.cs
+
pipeline.Append(mlContext.Transforms.ApplyOnnxModel(modelPath));
ONNX_backend.Services.OnnxService.Start() in OnnxService.cs
+
var model = LoadHandwrittenDigitsModel(_mlContext);
ONNX_backend.Controllers.ValuesController.Get() in ValuesController.cs
+
_onnxService.Start();
lambda_method(Closure , object , object[] )
Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(object target, object[] parameters)
Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor+SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, object controller, object[] arguments)
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

[jonathan-taina]

I can confirm that a resnet18 model prepared exactly the same as the resnet34 above does load. Seems to purely be to do with the size of the model.

[pabloppp]

I am finding the same issue. Did you find a solution other than using a smaller model size? Thanks 🙇

[jonathan-taina]

Nope. I've also had difficulty loading various onnx models from the onnx model zoo, for varying reasons. Looks like onnx support in ml.net isn't very robust at the moment I'm afraid.

[toras9000]

When I encountered this problem, I did a survey and found a workaround.

Although it appears in the error message, this problem seems to be due to the implementation of the Google.Protobuf version 3.5.1 package.
This seems to be a safety feature, but the default limit seems small.
This default value has been changed in package version 3.6.
https://github.com/protocolbuffers/protobuf/blob/66dc42d891a4fc8e9190c524fd67961688a37bbe/csharp/src/Google.Protobuf/CodedInputStream.cs#L97

Users of ML.NET 1.2 can work around this issue by explicitly referencing Google.Protobuf 3.6 or higher in their own projects.
However, I have not checked if there are any other inconsistencies in the new version.

_This sentence relies on translation. I am sorry if it is a strange sentence._

[harishsk]

The note above clarifies that this is no longer a problem in version Google.Protobuf 3.6 and higher. ML.NET now takes a dependency on v3.10.1.