Handson-ml: request solution chapter 3 - standardScaler()

[alberthasudungan]

Hai I request solution for chapter 3 practice, I wrote as this:

from sklearn.preprocessing import StandardScaler 
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring="accuracy")

and then the output:

MemoryError                               Traceback (most recent call last)
<ipython-input-70-96d9dde478da> in <module>()
      1 from sklearn.preprocessing import StandardScaler
      2 scaler = StandardScaler()
----> 3 X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
      4 cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring="accuracy")

MemoryError:

I try to simulate to transform X_train first and then performing cross value afterwards, but still error. I audited again sgd_clf, X_train, y_train, and they are all good. Error occurred when I display X_train_scaled. I request if there is any solution for this issue.

[ageron]

Hi @alberthasudungan ,

Thanks for your question. You are getting an memory error, this is quite weird because your code looks good, and the MNIST dataset is not large at all, so it should easily fit in memory multiple times. How much RAM does your computer have? Perhaps you have some other application using up all the RAM? Try to reboot your computer, look at your system's memory monitor to ensure that very little memory is used, then only start this Jupyter notebook. If the problem persists, then you may have a problem with your hardware. Alternatively, it may be because you loaded a different dataset than MNIST? Can you confirm that it is MNIST? Could you please tell me what X_train.shape is? Or perhaps it's a bug in your Python/NumPy/Scikit-Learn installation. What versions are you using? Could you please try to update all your libraries?

Hope this helps,
Aurélien

[ageron]

Hi @alberthasudungan , are you still experiencing the OOM issue?

[alberthasudungan]

@ageron i try to check again actually my computer was on windows version x86. I tried to run in version x64, it is all good. Thanks

[luvaihassanali]

@ageron I am having same issue on x64. On task manager memory is 22% (16GB RAM) and X_train.shape = (60000, 784). Python version 3.7.0, NumPy 1.15.1 and SciKit-Learn 0.19.2. Any help would be appreciated.

[ageron]

This sounds like a bug in one of these libraries. The code looks good to me. Could you please create a small python file that just loads MNIST and applies the StandardScaler, and run it independently to see if the same issue occurs? I just want to rule out Jupyter.

[luvaihassanali]

This sounds like a bug in one of these libraries. The code looks good to me. Could you please create a small python file that just loads MNIST and applies the StandardScaler, and run it independently to see if the same issue occurs? I just want to rule out Jupyter.

Thanks for the reply, I created this python file:

import numpy as np
from sklearn.datasets import fetch_mldata
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import cross_val_score, cross_val_predict
from sklearn.preprocessing import StandardScaler

mnist = fetch_mldata('MNIST original')
X, y = mnist["data"], mnist["target"]
X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]

shuffle_index = np.random.permutation(60000)
X_train, y_train = X_train[shuffle_index], y_train[shuffle_index]
print("X_train.shape: ", X_train.shape)

y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)

sgd_clf = SGDClassifier(random_state=42, max_iter=5, tol=None)
sgd_clf.fit(X_train, y_train_5)
sgd_clf.predict([X[36000]])

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
print(cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring = 'accuracy'))

This was the output:

>python mnist_test.py
>env\lib\site-packages\sklearn\feature_extraction\text.py:17: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping, defaultdict
X_train.shape:  (60000, 784)
[0.91086783 0.91124556 0.90748612]

In Jupyter this is the error message:

MemoryError                               Traceback (most recent call last)
<ipython-input-26-a5a54dfd1dad> in <module>()
      2 
      3 scaler = StandardScaler()
----> 4 X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
      5 cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring = 'accuracy')

>env\lib\site-packages\sklearn\base.py in fit_transform(self, X, y, **fit_params)
    515         if y is None:
    516             # fit method of arity 1 (unsupervised transformation)
--> 517             return self.fit(X, **fit_params).transform(X)
    518         else:
    519             # fit method of arity 2 (supervised transformation)

>env\lib\site-packages\sklearn\preprocessing\data.py in fit(self, X, y)
    588         # Reset internal state before fitting
    589         self._reset()
--> 590         return self.partial_fit(X, y)
    591 
    592     def partial_fit(self, X, y=None):

>env\lib\site-packages\sklearn\preprocessing\data.py in partial_fit(self, X, y)
    610         """
    611         X = check_array(X, accept_sparse=('csr', 'csc'), copy=self.copy,
--> 612                         warn_on_dtype=True, estimator=self, dtype=FLOAT_DTYPES)
    613 
    614         # Even in the case of `with_mean=False`, we update the mean anyway

>env\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    442             array = np.atleast_2d(array)
    443             # To ensure that array flags are maintained
--> 444             array = np.array(array, dtype=dtype, order=order, copy=copy)
    445 
    446         # make sure we actually converted to numeric:

MemoryError:

[ageron]

Thanks @luvaihassanali ,
Ouch, this seems to confirm that the problem comes from Jupyter since the same code works fine outside of it. Could you please run the following command in a terminal and copy/paste the output here?

jupyter troubleshoot

(make sure no private information is displayed)

Thanks,
Aurélien

[luvaihassanali]

(env) >env\Scripts>jupyter troubleshoot
$PATH:
        >env\Scripts
        C:\Program Files (x86)\Common Files\Oracle\Java\javapath
        C:\WINDOWS\system32
        C:\WINDOWS
        C:\WINDOWS\System32\Wbem
        C:\WINDOWS\System32\WindowsPowerShell\v1.0\
        C:\WINDOWS\System32\OpenSSH\
        C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
        C:\Program Files\nodejs\
        C:\Program Files\Microsoft VS Code\bin
        C:\Program Files (x86)\Java\javacc-5.0\javacc-5.0\bin
        C:\Program Files (x86)\Java\jdk1.8.0_181\bin
        C:\Users\User\AppData\Local\Programs\Python\Python37-32
        C:\Users\User\AppData\Local\Programs\Python\Python37-32\Scripts
        C:\Users\User\AppData\Roaming\Python\Python37\Scripts


sys.path:
        >env\Scripts\jupyter-troubleshoot.EXE
        >env\scripts\python37.zip
        >env\DLLs
        >env\lib
        >env\scripts
        c:\users\User\appdata\local\programs\python\python37-32\Lib
        c:\users\User\appdata\local\programs\python\python37-32\DLLs
        >env
        >env\lib\site-packages

sys.executable:
        >env\scripts\python.exe

sys.version:
        3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:06:47) [MSC v.1914 32 bit (Intel)]

platform.platform():
        Windows-10-10.0.17134-SP0

where jupyter:
        >env\Scripts\jupyter.exe

pip list:
        Package            Version
        ------------------ -------
        backcall           0.1.0
        bleach             2.1.4
        colorama           0.3.9
        cycler             0.10.0
        decorator          4.3.0
        entrypoints        0.2.3
        html5lib           1.0.1
        ipykernel          4.8.2
        ipython            6.5.0
        ipython-genutils   0.2.0
        ipywidgets         7.4.0
        jedi               0.12.1
        Jinja2             2.10
        jsonschema         2.6.0
        jupyter            1.0.0
        jupyter-client     5.2.3
        jupyter-console    5.2.0
        jupyter-core       4.4.0
        kiwisolver         1.0.1
        MarkupSafe         1.0
        matplotlib         2.2.3
        mistune            0.8.3
        nbconvert          5.3.1
        nbformat           4.4.0
        notebook           5.6.0
        numpy              1.15.1
        pandas             0.23.4
        pandocfilters      1.4.2
        parso              0.3.1
        pickleshare        0.7.4
        pip                18.0
        prometheus-client  0.3.1
        prompt-toolkit     1.0.15
        Pygments           2.2.0
        pyparsing          2.2.0
        python-dateutil    2.7.3
        pytz               2018.5
        pywinpty           0.5.4
        pyzmq              17.1.2
        qtconsole          4.4.1
        scikit-learn       0.19.2
        scipy              1.1.0
        Send2Trash         1.5.0
        setuptools         40.2.0
        simplegeneric      0.8.1
        six                1.11.0
        terminado          0.8.1
        testpath           0.3.1
        tornado            5.1
        traitlets          4.3.2
        wcwidth            0.1.7
        webencodings       0.5.1
        wheel              0.31.1
        widgetsnbextension 3.4.0

[BerzinsU]

I managed to get same memory error.
What was the solution here?

[luvaihassanali]

¯_(ツ)_/¯... I just completed it in a separate .py file

[BerzinsU]

I got it running by switching to MacOS. Seems there's specific problem with Jupyter on Win

[Lisi92]

I got the same error, but i found my mistake. The problem was the version of Python. The installation file in the topic of "https://www.python.org/downloads/ " is an 32bit win version. But under windows a 32 bit application is limited to 2 GB RAM.

I see u installed the 32 bit version too ("sys.version:
3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:06:47) [MSC v.1914 32 bit (Intel)]").

So i unistalled python and the packages and cleared the folder ".ipynb_checkpoints", "datasets" and "env". Afterwards i downloaded from "https://www.python.org/downloads/release/python-370/" the 64bit file "Windows x86-64 executable installer" . Installed all again and now it works.

I hope that this may help you.

[samaujs]

Hi Ageron,

I have tried to create a separate python script file and execute as per "luvaihassanali" comments above on a "Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1032-aws x86_64)" machine. But, I have still have the same MemoryError problem with outputs as follow :

Traceback (most recent call last):
File "mnist_test.py", line 35, in
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
File "/home/ubuntu/ml/VirtualEnv/env/lib/python3.6/site-packages/sklearn/base.py", line 462, in fit_transform
return self.fit(X, **fit_params).transform(X)
File "/home/ubuntu/ml/VirtualEnv/env/lib/python3.6/site-packages/sklearn/preprocessing/data.py", line 625, in fit
return self.partial_fit(X, y)
File "/home/ubuntu/ml/VirtualEnv/env/lib/python3.6/site-packages/sklearn/preprocessing/data.py", line 649, in partial_fit
force_all_finite='allow-nan')
File "/home/ubuntu/ml/VirtualEnv/env/lib/python3.6/site-packages/sklearn/utils/validation.py", line 598, in check_array
array = np.array(array, dtype=dtype, order=order)
MemoryError

[horizonSky]

Maybe there is not enough memory. Use the 'free -h' command to see.

[samaujs]

Got it. Thanks.