Machinelearning: Cannot select accuracy as target metric

[daholste]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): .NET Core 2.1

Issue

Cannot select 'Accuracy' as target metric

• What did you do?
  Ran MML command line: dotnet MML.dll execgraph "C:\Benchmarking\automl_graph.json

Contents of automl_.graph.json:

{
    "Inputs": {
        "file_train": "D:\\SplitDatasets\\ExcitementFG2_train.csv",
        "file_test": "D:\\SplitDatasets\\ExcitementFG2_valid.csv"
    },
    "Nodes": [{
            "Inputs": {
                "CustomSchema": "sep=, col=Label:R4:78 col=Features1:R4:0-77 col=Features2:R4:79-202 header=+",
                "InputFile": "$file_train"
            },
            "Name": "Data.CustomTextLoader",
            "Outputs": {
                "Data": "$data_train"
            }
        },
        {
            "Inputs": {
                "CustomSchema": "sep=, col=Label:R4:78 col=Features1:R4:0-77 col=Features2:R4:79-202 header=+",
                "InputFile": "$file_test"
            },
            "Name": "Data.CustomTextLoader",
            "Outputs": {
                "Data": "$data_test"
            }
        },
        {
            "Inputs": {
                "BatchSize": 3,
                "StateArguments": {
                    "Name": "AutoMlState",
                    "Settings": {
                        "Engine": {
                            "Name": "Rocket",
                            "Settings": {}
                        },
                        "Metric": "Accuracy",
                        "TerminatorArgs": {
                            "Name": "IterationLimited",
                            "Settings": {
                                "FinalHistoryLength": 100
                            }
                        },
                        "TrainerKind": "SignatureBinaryClassifierTrainer"
                    }
                },
                "TestingData": "$data_test",
                "TrainingData": "$data_train"
            },
            "Name": "Models.PipelineSweeper",
            "Outputs": {
                "Results": "$output_data",
                "State": "$xyz"
            }
        }
    ],
    "Outputs": {
        "output_data": "C:\\Benchmarking\\01-ResultsOut.csv"
    }
}

• What happened?
  An exception was thrown (below)

• What did you expect?
  A successful run

Source code / logs

--- Command line args ---
dotnet MML.dll execgraph C:\Benchmarking\automl_graph.json

--- Exception message ---

(1) Unexpected exception: Requested value 'Accuracy' is not a member of the Enum type 'Metrics', 'System.InvalidOperationException'

Exception context:
    Throwing component: Environment

   at Microsoft.ML.Runtime.EntryPoints.JsonUtils.InputBuilder.ParseJsonValue(IExceptionContext ectx, Type type, Attributes attributes, JToken value, ComponentCatalog catalog) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\InputBuilder.cs:line 441
   at Microsoft.ML.Runtime.EntryPoints.JsonUtils.InputBuilder.TrySetValueJson(String name, JToken value) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\InputBuilder.cs:line 181
   at Microsoft.ML.Runtime.EntryPoints.JsonUtils.InputBuilder.GetComponentJson(IExceptionContext ectx, Type signatureType, String name, JObject settings, ComponentCatalog catalog) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\InputBuilder.cs:line 559
   at Microsoft.ML.Runtime.EntryPoints.JsonUtils.InputBuilder.ParseJsonValue(IExceptionContext ectx, Type type, Attributes attributes, JToken value, ComponentCatalog catalog) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\InputBuilder.cs:line 485
   at Microsoft.ML.Runtime.EntryPoints.JsonUtils.InputBuilder.TrySetValueJson(String name, JToken value) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\InputBuilder.cs:line 181
   at Microsoft.ML.Runtime.EntryPoints.EntryPointNode.CheckAndSetInputValue(KeyValuePair`2 pair) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\EntryPointNode.cs:line 680
   at Microsoft.ML.Runtime.EntryPoints.EntryPointNode..ctor(IHostEnvironment env, IChannel ch, RunContext context, String id, String entryPointName, JObject inputs, JObject outputs, Boolean checkpoint, String stageId, Single cost, String label, String group, String weight, String name) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\EntryPointNode.cs:line 504
   at Microsoft.ML.Runtime.EntryPoints.EntryPointNode.ValidateNodes(IHostEnvironment env, RunContext context, JArray nodes, String label, String group, String weight, String name) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\EntryPointNode.cs:line 929
   at Microsoft.ML.Runtime.EntryPoints.EntryPointGraph..ctor(IHostEnvironment env, JArray nodes) in C:\MLDotNet\src\Microsoft.ML.Data\EntryPoints\EntryPointNode.cs:line 1003
   at Microsoft.ML.Runtime.EntryPoints.JsonUtils.GraphRunner..ctor(IHostEnvironment env, JArray nodes) in C:\MLDotNet\src\Microsoft.ML.Legacy\Runtime\EntryPoints\JsonUtils\GraphRunner.cs:line 32
   at Microsoft.ML.Runtime.EntryPoints.JsonUtils.ExecuteGraphCommand.Run() in C:\MLDotNet\src\Microsoft.ML.Legacy\Runtime\EntryPoints\JsonUtils\ExecuteGraphCommand.cs:line 62
   at Microsoft.ML.Runtime.Tools.Maml.MainCore(ConsoleEnvironment env, String args, Boolean alwaysPrintStacktrace) in C:\MLDotNet\src\Microsoft.ML.Maml\MAML.cs:line 139

[najeeb-kazmi]

Hi @daholste, for some reason that eludes me, Accuracy is not one of the metrics supported for PipelineSweeper. You can take a look here for the full list of supported metrics.

That said, this is an experimental feature in ML.NET that is not officially supported or documented, and we do not plan on working on this before v1.0. In fact, I'm told that much of Microsoft.ML.PipelineInference will possibly be removed from the repo in the near future. See #689 for the discussion.

[daholste]

Hey! Unfortunately, I do not think that list of supported metrics may work. When I type 'AccuracyMicro', it still crashes?

[najeeb-kazmi]

AccuracyMacro and AccuracyMicro are specific to the Multi-class classification domain. They will not work with Binary classification trainers.

[daholste]

Good to know, thanks Najeeb!

[rogancarr]

Closing: Microsoft.ML.PipelineInference has been removed from the repository.