Pysyft: Model accuracy is too low

[keenlykeenly]

Describe the bug
In tutorials Part 13a - Secure Classification with Syft Keras and TFE - Public Training, the model test accuracy is quite low: Test accuracy: 0.0925. I cannot reproduce the high accuracy as your notebook writes: 93%. The code and result are as follows.

I have not changed anything of your notebook. And I try running it many times to remove the effect of randomness but the accuracy is about 0.09. How can I solve this question?

[robert-wagner]

@mortendahl

[mortendahl]

Hi @keenlykeenly, could you provide a few details about the system you're on:

• OS (macOS, Linux, Windows)?
• TensorFlow version (eg via pip list)?

[keenlykeenly]

Hi @keenlykeenly, could you provide a few details about the system you're on:

• OS (macOS, Linux, Windows)?
• TensorFlow version (eg via pip list)?

@mortendahl Thank you for your reply.
It is Windows system. And I use the anaconda virtual environment. The tensorflow version is 1.14.0 rc0.
When I enter 'pip list' in the anaconda prompt, I get:
Package Version Location

---

absl-py 0.7.1
alabaster 0.7.12
anaconda-client 1.7.2
anaconda-navigator 1.9.7
anaconda-project 0.8.2
asn1crypto 0.24.0
astor 0.8.0
astroid 2.1.0
astropy 3.1.2
atomicwrites 1.3.0
attrs 19.1.0
Babel 2.7.0
backcall 0.1.0
backports.os 0.1.1
backports.shutil-get-terminal-size 1.0.0
beautifulsoup4 4.7.1
bitarray 0.9.3
bkcharts 0.2
blaze 0.11.3
bleach 3.1.0
bokeh 1.2.0
boto 2.49.0
Bottleneck 1.2.1
certifi 2019.3.9
cffi 1.12.3
chardet 3.0.4
Click 7.0
cloudpickle 1.1.1
clyent 1.2.2
colorama 0.4.1
comtypes 1.1.7
conda 4.6.14
conda-build 3.17.8
conda-package-handling 0+unknown
conda-verify 3.3.0
contextlib2 0.5.5
cryptography 2.7
cycler 0.10.0
Cython 0.29.10
cytoolz 0.9.0.1
dask 1.2.2
datashape 0.5.4
decorator 4.4.0
defusedxml 0.6.0
distributed 1.28.1
docutils 0.14
entrypoints 0.3
et-xmlfile 1.0.1
fastcache 1.1.0
filelock 3.0.12
Flask 1.0.3
Flask-Cors 3.0.7
flask-socketio 4.0.0
future 0.17.1
gast 0.2.2
gevent 1.4.0
glob2 0.6
google-pasta 0.1.7
greenlet 0.4.15
grpcio 1.21.1
h5py 2.9.0
heapdict 1.0.0
html5lib 1.0.1
idna 2.8
imageio 2.5.0
imagesize 1.1.0
importlib-metadata 0.17
ipykernel 5.1.1
ipython 7.5.0
ipython-genutils 0.2.0
ipywidgets 7.4.2
isort 4.3.4
itsdangerous 1.1.0
jdcal 1.4.1
jedi 0.13.3
Jinja2 2.10.1
jsonschema 3.0.1
jupyter 1.0.0
jupyter-client 5.2.4
jupyter-console 6.0.0
jupyter-core 4.4.0
jupyterlab 0.35.4
jupyterlab-launcher 0.13.1
jupyterlab-server 0.2.0
keras-applications 1.0.8
keras-preprocessing 1.0.9
keyring 18.0.0
kiwisolver 1.1.0
lazy-object-proxy 1.3.1
libarchive-c 2.8
llvmlite 0.28.0
locket 0.2.0
lxml 4.3.3
lz4 2.1.6
markdown 3.1.1
MarkupSafe 1.1.1
matplotlib 2.1.0
mccabe 0.6.1
menuinst 1.4.16
mistune 0.8.4
mkl-fft 1.0.12
mkl-random 1.0.2
mkl-service 2.0.2
more-itertools 7.0.0
mpmath 1.1.0
msgpack 0.6.1
multipledispatch 0.6.0
navigator-updater 0.2.1
nbconvert 5.5.0
nbformat 4.4.0
networkx 2.3
nltk 3.4.1
nose 1.3.7
notebook 5.7.8
numba 0.36.2
numexpr 2.6.4
numpy 1.16.4
numpydoc 0.9.1
odo 0.5.1
olefile 0.46
openpyxl 2.6.2
packaging 19.0
pandas 0.24.2
pandocfilters 1.4.2
parso 0.4.0
partd 0.3.10
path.py 12.0.1
pathlib2 2.3.3
patsy 0.5.0
pep8 1.7.1
phe 1.4.0
pickleshare 0.7.5
Pillow 6.0.0
pip 19.1.1
pkginfo 1.5.0.1
pluggy 0.12.0
ply 3.11
prometheus-client 0.6.0
prompt-toolkit 2.0.9
protobuf 3.8.0
psutil 5.6.2
py 1.8.0
pycodestyle 2.5.0
pycosat 0.6.3
pycparser 2.19
pycrypto 2.6.1
pycurl 7.43.0.2
pyflakes 2.1.1
Pygments 2.4.2
pylint 2.2.2
pyodbc 4.0.26
pyOpenSSL 19.0.0
pyparsing 2.2.0
pyreadline 2.1
pyrsistent 0.14.11
PySocks 1.7.0
pytest 4.6.2
pytest-arraydiff 0.3
pytest-astropy 0.5.0
pytest-doctestplus 0.3.0
pytest-openfiles 0.3.2
pytest-remotedata 0.3.1
python-dateutil 2.6.1
python-engineio 3.7.0
python-socketio 4.0.3
pytz 2017.3
PyWavelets 1.0.3
pywin32 223
pywinpty 0.5.5
PyYAML 5.1
pyzmq 18.0.0
QtAwesome 0.5.7
qtconsole 4.5.1
QtPy 1.7.1
requests 2.22.0
rope 0.14.0
ruamel-yaml 0.15.46
scikit-image 0.13.1
scikit-learn 0.19.1
scipy 1.0.0
seaborn 0.8.1
Send2Trash 1.5.0
setuptools 41.0.1
simplegeneric 0.8.1
singledispatch 3.4.0.3
six 1.11.0
sklearn 0.0
snowballstemmer 1.2.1
sortedcollections 1.1.2
sortedcontainers 2.1.0
soupsieve 1.8
Sphinx 2.1.0
sphinxcontrib-applehelp 1.0.1
sphinxcontrib-devhelp 1.0.1
sphinxcontrib-htmlhelp 1.0.2
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.2
sphinxcontrib-serializinghtml 1.1.3
sphinxcontrib-websupport 1.1.2
spyder 3.3.4
spyder-kernels 0.4.4
SQLAlchemy 1.3.4
statsmodels 0.8.0
style 1.1.0
syft 0.1.17
sympy 1.4
tables 3.4.2
tblib 1.4.0
tensorboard 1.13.1
tensorflow 1.14.0rc0
termcolor 1.1.0
terminado 0.8.2
testpath 0.4.2
tf-encrypted 0.5.5 d:\programdata\anaconda3\tf-encrypted
tf-estimator-nightly 1.14.0.dev2019042301
toolz 0.9.0
torch 1.0.1
torchvision 0.2.2
tornado 6.0.2
tqdm 4.32.1
traitlets 4.3.2
typed-ast 1.1.1
typing 3.6.4
unicodecsv 0.14.1
update 0.0.1
urllib3 1.24.2
vboxapi 1.0
wcwidth 0.1.7
webencodings 0.5.1
websocket-client 0.56.0
websockets 7.0
Werkzeug 0.15.4
wheel 0.33.4
widgetsnbextension 3.4.2
win-inet-pton 1.1.0
win-unicode-console 0.5
wincertstore 0.2
wrapt 1.11.1
xlrd 1.2.0
XlsxWriter 1.1.8
xlwings 0.15.8
xlwt 1.3.0
zict 0.1.4
zipp 0.5.1
zstd 1.4.0.0

Thanks a lot.

[lc0]

I found the issue, from TensorFlow 1.13 to TensorFlow 1.14 seems like default learning rate for Adadelta was changed from 1.0 to 0.001, that's why currently it's so much also slower to learn

[lc0]

Here is an example - https://colab.research.google.com/drive/1l_dLRxws29wqBJeVSmXdGOOXnuTx8VK2#scrollTo=vicyresvIesY

@iamtrask would it make sense to add a version of executed environment(tensorflow/pytorch) in the beginning of such tutorials, so people do not get confused?

[keenlykeenly]

@lc0 Thanks very much. And what can I do to fix it? Change the version of tensorflow to 1.13 or specify the learning rate of Adadelta? And how to do it? I am sorry that I cannot open the link you gave above.

[keenlykeenly]

@lc0 Yes! It is the problem of learning rate! When I specify the learning rate of Adadelta as optimizer=keras.optimizers.Adadelta(lr=1.0), the accuracy is 94.6%.

Thank you so much!

[keenlykeenly]

@mortendahl @lc0 But there still exits a problem in tutorial Part 13c. In order to remove the randomness,
I test 100 samples, but all of them are classified as 0. Even though the model test accuracy is 98.21% in Part 13a when I increase the epochs to 12.

[mortendahl]

Thanks for the system information @keenlykeenly!

all of them are classified as 0.

This has to do with a Windows bug we're currently investigating: https://github.com/tf-encrypted/tf-encrypted/issues/577!

[mortendahl]

Issue should be solved but happy to have third party verification as well. Would you be available to give it a try @keenlykeenly?

cc @robert-wagner

[keenlykeenly]

Thank you very much. I will try it as soon as possible. And do I need to re-install pysyft from the beginning as your guide in Readme of Pysyft on my computer? @mortendahl

[mortendahl]

I will try it as soon as possible.

Thank you, please keep us posted!

And do I need to re-install pysyft from the beginning as your guide in Readme of Pysyft on my computer?

You will need the latest version of TF Encrypted (>= 0.5.7). You will also need the latest version of Syft but I believe part of the fix (https://github.com/OpenMined/PySyft/commit/4045ef26d214d7ea35b2acc4e947b5632648863e) has not made it into an official release yet, meaning you will likely need to run Syft from a source code installation of the dev branch.

[keenlykeenly]

If I execute the codes on a virtual machine of VMware which is Ubuntu 16.04, will the error disappear?

At 2019-07-25 20:45:17, "Morten Dahl" [e-mail] wrote:

I will try it as soon as possible.

Thank you, please keep us posted!

And do I need to re-install pysyft from the beginning as your guide in Readme of Pysyft on my computer?

You will need the latest version of TF Encrypted (>= 0.5.7). You will also need the latest version of Syft but I believe part of the fix (4045ef2) has not not made it into an official release yet, meaning you will likely need to run Syft from a source code installation of the dev branch.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[keenlykeenly]

@mortendahl Hi, I install the tf-encrypted 0.5.7 via pip and the latest version of Syft 0.1.22a1 via git from a source code installation of the dev branch:

Then the accuracy of the trained model in tutorial Part 13a is 0.9394. But the classification results in tutorial Part 13c are wrong too ( I test 100 cases and all of them are classified as 0):

Maybe the windows system is not suitable for the codes.

[keenlykeenly]

@mortendahl When I print the result of client.query_model in Part 13c, they are the same:

So when execute "predicted_label = np.argmax(res)", the predicted_label is 0 all the time because the elements in the array res are the same.
Maybe the question is here.

[github-actions[bot]]

This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the stale label to unmark it. Otherwise, this will be closed in 7 days.