Handson-ml: chap 5 Support Vector Machine : How to get Support vector in LinearSVC

[nitml]

can you please explain me Support Vector in LinearSVC and Unscaling part of this code( chap.5 jupyter notebook cell 09):

scaler = StandardScaler()
svm_clf1 = LinearSVC(C=1, loss="hinge", random_state=42)
scaled_svm_clf1 = Pipeline([
        ("scaler", scaler),
        ("linear_svc", svm_clf1),
    ])
scaled_svm_clf1.fit(X, y)
# Convert to unscaled parameters
b1 = svm_clf1.decision_function([-scaler.mean_ / scaler.scale_])
w1 = svm_clf1.coef_[0] / scaler.scale_
svm_clf1.intercept_ = np.array([b1])
svm_clf1.coef_ = np.array([w1])

# Find support vectors (LinearSVC does not do this automatically)
t = y * 2 - 1
support_vectors_idx1 = (t * (X.dot(w1) + b1) < 1).ravel()
svm_clf1.support_vectors_ = X[support_vectors_idx1]

[ageron]

Hi @nitml ,

A trained LinearSVC model basically computes decision_function = w1.x1 + w2.x2 + ...+ wn.xn + b, and if decision_function ≥ 0, then the instance is classified as positive, or else it's classified as negative. x1, x2, ..., xn are the scaled input features and w1, w2, ..., wn are the corresponding weights, and b is the bias term.

It is important to scale the input features before training a support vector machine. That's why we use the StandardScaler. But if we want to plot the decision boundary of the trained LinearSVC on a plot whose axes show the original (unscaled) features, then we need to "unscale" the weights and bias term before plotting the decision boundary. In other words, once we have trained the LinearSVC, we can compute the decision function with w1.x1 + ...+ wn.xn + b, but we would like to be able to compute that same decision function as a function of the unscaled input features (let's call them x'1, x'2, ..., x'n).
So we are looking for the "unscaled" weights w'1, ..., w'n and "unscaled" bias b' such that:

decision_function = w1.x1 + w2.x2 + ...+ wn.xn + b = w'1.x'1 + w'2.x'2 + ... + w'n.x'n + b'

The standard scaler computes xi = (x'i - mean)/scale, so x'i = xi.scale + mean

Therefore:
decision_function = w1.x1 + w2.x2 + ...+ wn.xn + b
= w'1.(x1.scale + mean) + ... + w'n.(xn.scale + mean) + b'
= w'1.scale.x1 + ... + w'n.scale.xn + w'1.mean + ... + w'n.mean + b'

Looking at each input feature xi's weight, we see that: wi = w'i.scale, so w'i = wi/scale.
Moreover, by setting x1, ..., xn to zero, we see that:
b = w'1.mean + ... + w'n.mean + b'
So:
b' = - w'1.mean - ... - w'n.mean + b
= - w1.mean/scale - ... - wn.mean/scale + b
= decision_function(-mean/scale)

This explains:

b1 = svm_clf1.decision_function([-scaler.mean_ / scaler.scale_])
w1 = svm_clf1.coef_[0] / scaler.scale_

Hope this helps,
Aurélien

[nitml]

Great Explanation 👍 ....But I'm still not getting how to find Support Vector in LinearSVC .
Can u please also explain the other part of code.

also i'm having doubt about chap 7 jupyter notebook cell 32 code:

list(m for m in dir(ada_clf) if not m.startswith("_") and m.endswith("_"))

also something similar was used in chap 8 cell 72:

def learned_parameters(model):
    return [m for m in dir(model)
            if m.endswith("_") and not m.startswith("_")]

Thanks for your Time !!

[ageron]

The support vectors are the ones that are not "on the right side of the road". In other words, it's any positive instance whose score is <1, or any negative instance whose score is >-1. It's a linear model, the score is simply w1 x1 + w2 x2 + ... + wn xn + b, and we can compute the scores for many instances at once by computing X.dot(w) + b, where X is the input matrix and w is the weight vector. If y is the target vector, 0 for each negative instance and 1 for each positive instance, then t = 2 * y - 1 defines a new target vector, with -1 for each negative instance, and 1 for each positive instance.

We want:

• X.dot(w) + b < -1 for negative instances
• X.dot(w) + b > 1 for positive instances

We can rewrite this as:

• -1 * X.dot(w) + b) > 1 for negative instances
• +1 * X.dot(w) + b > 1 for positive instances

So this simplifies to:

We can rewrite this as:

• t * X.dot(w) + b) > 1 for all instances

So the code just finds these instances. ravel() just reshapes the array to a one-dimensional array.

Regarding:

a = list(m for m in dir(ada_clf) if not m.startswith("_") and m.endswith("_"))

This is equivalent to:

a = []
for m in dir(ada_clf):
    if not m.startswith("_") and m.endswith("_"):
        a.append(m)

I'm just getting the list of attributes of the ada_clf object whose name ends with an underscore, but does not start with an underscore. This is a convention in Scikit-Learn: all learned parameters have a name with this format.

Cheers,
Aurélien

[amansingh9097]

as per your approach, when calculating the support vectors and passing them back to svm_clf.coef_ = np.array([w1] it basically throws an error as AttributeError: can't set attribute

Is there any other work around, because this one doesn't seem to work now!

[ageron]

Hi @amansingh9097 ,
Thanks for your feedback. That's odd, I just tried setting coef_ on a LinearSVC, and it works fine.

Which version of Scikit-Learn, NumPy and Python are you using?

[aloosh89]

Hello @ageron

Thank you for the additional explanations you have provided here, it was very insightful to mathematically show how the beta primes formulation is obtained.

I have struggled a bit with wrapping my head around the t = 2 * y - 1 transformation you did so I took an alternative and longer approach to ensure I am grabbing this right here is what I did:

1) I want all points which the SVM classifies incorrectly as false positives and false negatives and for that I have to find all points:
(((X.dot(w1) + b1) > 0) & (y==0)) as well as all points (((X.dot(w1) + b1) < 0) & (y==1)).

2) additionally I want to find all the points that are classified correctly but violate the dashed boundary which defines the width of our road. Hence I want to find the points which are True positives and true negatives but with a score that falls above the dashed line and below the dashed line. Combining this with 1 above I ended up with this long logical statement:

supporter_vecotr = ((((X.dot(w1) + b1) > 0) & (y==0)) | (((X.dot(w1) + b1) < 1) & (y==1)))| ((((X.dot(w1) + b1) < 0) & (y==1)) | (((X.dot(w1) + b1) > -1) & (y==0)))

It works perfectly and gives the same result as yours which was a good step for me to ensure I grasp how the SVM was working. The reason I am writing this post is there is one thing I still don't grasp: What is it that is driving the border violation scores to be less than 1 and more than -1 for each class? Why not -2 and 2 or other arbitrary number. I think the answer lies in how the SVM loss function minimization is formulated mathematically but I can't quite grasp it. Perhaps if you have time to touch on this point would be great to enhance my understanding before moving on to non linear classifiers.

Thanks!
Ali