Google-cloud-node: Vision api imageProperties dominant color score meaning

[mik115]

Hi,
we are using vision api for image analysis.
In the image properties, the dominantColor field is an array containing objects like this:

| field | meaning |
| --- | --- |
| color | rgb color representation |
| score | Image-specific score for this color. Value in range [0, 1] |
| pixelFraction | Stores the fraction of pixels the color occupies in the image. Value in range [0, 1]. ! |

I can't understand the score property meaning and how it is calculated.
In many cases the score value is not related to the pixelFraction value.

thanks in advance

[stephenplusplus]

(link: https://cloud.google.com/vision/reference/rest/v1/images/annotate#ColorInfo)

@jgeewax anyone from the Vision team that can clarify?

[mik115]

hi, any news??

[jgeewax]

Hey @mik115 - going to dig on this for you and hopefully update our docs to be a bit more clear. Score is typically a "confidence level" (ie, "I'm 90% sure this picture is of a dog"), but I can see the confusion this might be causing. Hang tight -- going to figure it out :)

[mik115]

hi, any news about my question?
Thanks

[maxdelia]

Hello guys!
Any answer here?
In my honest opinion this is a key question and should be answered asap!

[stephenplusplus]

@jgeewax anyone from the Vision team who can help?

[stephenplusplus]

Going to close since this isn't an issue with our library. But feel free to stay tuned if @jgeewax or @omaray can find an answer.

[zubairshokh]

What google vision API provides us is the color, score and pixel fraction. Can we have a formulla to get the percentage of the color.

[stephenplusplus]

@zubairshokh do you mean the percentage of pixels of a color across the entire image? I believe that is what we calculate as coverage:

var img = 'https://camo.githubusercontent.com/e7653451a451dd7219ffbfddccab39daac4aead4/687474703a2f2f636f6f6c626c61636b7070742e636f6d2f77702d636f6e74656e742f75706c6f6164732f3235363078313434302d636f6e676f2d70696e6b2d636f6f6c2d626c61636b2d616e642d636f6f6c2d677265792d74687265652d636f6c6f722d6261636b67726f756e642d32303135303230353031303935362d353464326332653432613935362e6a7067'

vision.detectProperties(img, { verbose: true }, function(err, props) {
  if (err) throw err
  props = [
    {
      score: 97.700495,
      hex: '002e62',
      red: undefined,
      green: 46,
      blue: 98,
      coverage: 32
    },
    {
      score: 0.76268637,
      hex: '8990ab',
      red: 137,
      green: 144,
      blue: 171,
      coverage: 33.333334
    },
    {
      score: 1.5368186000000001,
      hex: '0a3567',
      red: 10,
      green: 53,
      blue: 103,
      coverage: 1.3333334000000001
    },
    {
      score: 5.019082e-7,
      hex: 'f88379',
      red: 248,
      green: 131,
      blue: 121,
      coverage: 32.666665
    },
    {
      score: 2.0494233e-8,
      hex: 'e87e77',
      red: 232,
      green: 126,
      blue: 119,
      coverage: 0.6666666800000001
    }
  ]  
})

[zubairshokh]

@stephenplusplus But here is what I get .

`Vision.Images.Annotate annotate =
            vision.images()
                    .annotate(new BatchAnnotateImagesRequest().setRequests(ImmutableList.of(request)));

   BatchAnnotateImagesResponse batchResponse = annotate.execute();`



`{
"dominantColors" : {
    "colors" : [ 
        {
            "color" : {
                "blue" : 171.0,
                "green" : 233.0,
                "red" : 241.0
            },
            "pixelFraction" : 0.0413596630096436,
            "score" : 0.33846253156662
        }, 
        {
            "color" : {
                "blue" : 96.0,
                "green" : 50.0,
                "red" : 48.0
            },
            "pixelFraction" : 0.208271935582161,
            "score" : 0.0438975356519222
        }, 
        {
            "color" : {
                "blue" : 121.0,
                "green" : 119.0,
                "red" : 128.0
            },
            "pixelFraction" : 0.0259357504546642,
            "score" : 0.00990148447453976
        }, 
        {
            "color" : {
                "blue" : 193.0,
                "green" : 238.0,
                "red" : 241.0
            },
            "pixelFraction" : 0.0325179286301136,
            "score" : 0.182118445634842
        }, 
        {
            "color" : {
                "blue" : 162.0,
                "green" : 235.0,
                "red" : 239.0
            },
            "pixelFraction" : 0.015227428637445,
            "score" : 0.129081636667252
        }, 
        {
            "color" : {
                "blue" : 141.0,
                "green" : 204.0,
                "red" : 218.0
            },
            "pixelFraction" : 0.020827192813158,
            "score" : 0.0650643929839134
        }, 
               ]
}

}`

[stephenplusplus]

Oh, I see. Is that another client library from a different language?

All we do is pixelFraction * 100: https://github.com/GoogleCloudPlatform/google-cloud-node/blob/6c1ebc0a43feb21279808304b515d139ef29ab41/packages/vision/src/index.js#L1525

[zubairshokh]

@stephenplusplus
Thank you for your response.
Though what you are saying is about the coverage,
but please have a look at the attached screenshot of what we get on the cloud vision website.
I need the value of percentage.

[stephenplusplus]

That's actually a decorated response that the front end is doing, but I can't find the exact calculation. The raw response from the JSON API is just this:

[zubairshokh]

Thank you @stephenplusplus for your responses.
I needed the percentage calculation.
Thank you

[stephenplusplus]

Alright, found it!

function decorateColors(colors) {
  var scoresSum = colors.reduce(function(sum, color) {
    return sum + color.score;
  }, 0) / 100;

  return colors.map(function(color) {
    color.percent = color.score / scoresSum;
    return color;
  });
}

vision.detectProperties(img, { verbose: true }, function(err, props) {
  if (err) throw err
  props.colors = decorateColors(props.colors);
})

If this gives the results you expect, I'll change how we calculate coverage to match this formula.

[zubairshokh]

@stephenplusplus
Perfect. However I believe 'score' here creates confusion. As its definition here do not provide with the exact idea. @jgeewax in the above comment, defines it as the "confidence level", which rather creates more confusion. It is a misnomer and to avoid confusion should have been named differently. However it cant be changed now, but rather properly defining it will help everyone in future.
Thank you.

[jgeewax]

@zubairshokh : I agree that calling this thing (whatever it is) the "score" is confusing at first, but I suspect that's because I've been taking the view that the color annotations are "dumb" annotations rather than intelligent ones.

In other words, I've been assuming that Color annotations blindly look at the pixels and assign a number based on how many pixels have "similar colors". After some research here, it turns out that's not the case.

My current understanding of a color's score is a combination of two things:

1. What is the focus of the image?
2. What is the color of that focus?

For example, given the following image:

The focus is clearly the cat, and therefore the color annotation for this image with the highest score (0.15) will be RGB = (232, 183, 135) which is the beige color:

The green of the grass (despite having more pixels in the image dedicated to it) has a much lower score by virtue of the algorithm's detection that it's the background and not the focus of the image.

In other words, higher "scores" means higher confidence that the color in question is prominent in the central focus of the image.

[zubairshokh]

@jgeewax Thank you. Lot of confusions are clear by your explanation. :+1:

[stephenplusplus]

Given that explanation, we probably shouldn't change our formula to consider the score of an annotation. I imagine most use cases will be technical, and the precise number of pixels is what's wanted. As always, the score and raw API response is available in the same callback as our decorated response for the other use cases.

[00hello]

@jgeewax Thanks for this explanation. I'm trying to find more info on the type of RGB space used for the colors. Can you tell me what code you looked at that allowed you to figure out what "score" meant. It may be what I'm looking for