Deeplearning4j: Spark dataframe input to the dl4j model
There should be a method through which we can give feature columns and target column from the spark dataframe to the dl4j model.
daksh-agarwal
All 7 comments
https://github.com/deeplearning4j/deeplearning4j/tree/master/deeplearning4j-scaleout/spark/dl4j-spark-ml This should be what you are looking for. It isn't released to maven central, yet. Open to more ideas, if this doesn't fit exactly what you are looking for.
It includes an autoencoder than can transform dataset/dataframe features, and a general SparkDl4jNetwork that can handle Spark networks from a dataset/dataframe. It works with both spark 1 and 2. In the future, I would like to add word2vec, ParagraphVectors, and SparkComputationGraph, just haven't had the time yet.
dmmiller612
on 18 Jun 2017
Hi thanks for that! SparkComputationGraph would be great.
oguzserbetci
on 5 Apr 2018
It has been a long time since I have worked with this, but that does sound like a nice feature. I'll try to get something together.
dmmiller612
on 5 Apr 2018
@dmmiller612 we'lll be doing a release within the next 24 hours. expect it soon thanks for the help!
agibsonccc
on 5 Apr 2018
Implemented.
raver119
on 27 Apr 2018
Could you please provided the link for this implemented part and examples ?
Thanks for your great efforts
rickyhai11
on 12 Jul 2018
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
![lock[bot] picture](https://avatars1.githubusercontent.com/in/6672?v=4&s=40)
lock[bot]
on 21 Sep 2018
Related issues
AlexDBlack
路
4Comments
AlexDBlack
路
4Comments
AlexDBlack
路
4Comments
Paranormaly
路
5Comments
sshepel
路
4Comments
Most helpful comment
https://github.com/deeplearning4j/deeplearning4j/tree/master/deeplearning4j-scaleout/spark/dl4j-spark-ml This should be what you are looking for. It isn't released to maven central, yet. Open to more ideas, if this doesn't fit exactly what you are looking for.
It includes an autoencoder than can transform dataset/dataframe features, and a general SparkDl4jNetwork that can handle Spark networks from a dataset/dataframe. It works with both spark 1 and 2. In the future, I would like to add word2vec, ParagraphVectors, and SparkComputationGraph, just haven't had the time yet.