Handson-ml: Chapter 2 Exercise 2, log of a distribution

[eliaperantoni]

In ex. 2 of chapter 2 you're using this params dict

param_distribs = {
        'kernel': ['linear', 'rbf'],
        'C': reciprocal(20, 200000),
        'gamma': expon(scale=1.0),
    }

and you later state

The reciprocal distribution is useful when you have no idea what the scale of the hyperparameter should be (indeed, as you can see on the figure on the right, all scales are equally likely, within the given range), whereas the exponential distribution is best when you know (more or less) what the scale of the hyperparameter should be.

These is the hist plot for the reciprocal distribution

I don't understand why you're using such a skewed distribution and state that all scales are euqally likely. What does that mean? Why are we looking at the log of this distribution, what does it mean?

Thanks a lot, loving your book so far. Am at chapter 11 but I've been having this dilemma forever

[ageron]

Hi @eliaperantoni ,

Thanks for your kind words, I'm glad you enjoy my book. :)

The reciprocal distribution is useful when you really don't know much about the correct value of the hyperparameter, not even its approximate scale. The best value may be 0.00001, or 0.01, 100 or 100,000, you just don't know. Before you can fine-tune this hyperparameter, you have to discover at least its approximate scale. Suppose you just tried picking random values for this hyperparameter, between 0 and 100,000, using a uniform distribution: it would be highly unlikely that any value smaller than 1.0 would be tried, let alone any value smaller than 0.01. But perhaps a very small hyperparameter is what will work best. So instead of randomly picking a hyperparameter value uniformly between 0 and 100,000, a better option is to pick its _scale_ uniformly (i.e., its exponent). That's what the reciprocal distribution does. If you wanted to do this manually, it would look like this:

import numpy as np

scale = np.random.uniform(-5, 5)
hyperparameter_value = 10 ** scale

This will pick a random hyperparameter value between 0.00001 and 100,000 using the reciprocal distribution: small values will be much much more likely than if you used the uniform distribution, but large values may be explored as well.

In other words, reciprocal(20, 200000) is equivalent to picking a scale randomly using a uniform distribution between log(20) and log(200000), then exponentiating it. This is why I plotted the log of the reciprocal distribution: you can see that it is (more or less) uniform between about 3 (since log(20)=2.9957) and 12.2 (since log(20000)=12.2061).

Hope this helps,
Aurélien

[eliaperantoni]

@ageron Absolutely crystal clear, thanks so much

[FezTheImmigrant]

I spent a few days really trying to understand expon and reciprocal. (I hate using math functions when I don't understand how they work) I believe I have explanations for the inner workings of expon and reciprocal that is staisfactory to me atleast.

• Reciprocal (20,20000) will choose a value from 20 to 20000, but it will favor lower values over larger ones
  
  
  
  • Steps To Reproduce:
  
  
  • Choose value from log(20) == 1.3 to log(20000) == 4.3 from a normal distribution (X)
  
  
  • exponentiate the value ( Y = 10 ** X)
  
  
  • Note: Y should be between 20 and 20000, but it should favor smaller values
  
  
  • The reason why this favors smaller values is because we are effectively scaling down random number generation on a normal distribution from [20,20000] to [1.3,4.3]. This means that lower numbers have a better chance at being chosen since the range to pick from has gotten significantly smaller. 10 ^ 1.3 == 20 and 10 ^ 4.0 == 10,000. The likelihood of choosing a value above 4.0 is smaller than choosing a value below 4.0 ;therefore, having a value above 10,000 is way smaller that having a value below 10,000. (This is just an example of how lower numbers are favored from this distribution)
  
  
• Expon(scale=1.0) will choose a value with an infinite range, but it will favor numbers below 1 (scale == mean)
  
  
  
  • Steps To Reproduce:
  
  
  • Choose value from 0 to 1 using a normal distribution (X)
  
  
  • Calculate Y using Y = ln(1 - X ) / -(1/scale) [ We'll call this function g(x) ]
  
  
  • Note Y should be between 0 and infinity, but it will favor numbers below 1 since mean == 1
  
  
  • Using the above function g(x):
    
    
    
    
    
    • g(0.5) = 0.69 . This means that 50 % of numbers from this function will be below 0.69
    
    
    
    • g(0.9999999) = 16.12 . This means that less that 0.0000001% of number will be above 16