Spark: [FEATURE REQUEST]: Implement ML Features

[GoEddie]

This is to track implementation of the ML-Features: https://spark.apache.org/docs/latest/ml-features

Bucketizer has been implemented in https://github.com/dotnet/spark/pull/378 but there are more features that should be implemented.

• [ ] Feature Extractors
  
  
  
  • [x] TF-IDF
  
  
  • [x] Word2Vec (https://github.com/dotnet/spark/pull/491)
  
  
  • [x] CountVectorizer (https://github.com/dotnet/spark/pull/608)
  
  
  • [x] FeatureHasher (https://github.com/dotnet/spark/pull/652)
  
  
• [ ] Feature Transformers
  
  
  
  • [x] Tokenizer (https://github.com/dotnet/spark/pull/574)
  
  
  • [x] StopWordsRemover (https://github.com/dotnet/spark/pull/726 thanks @SARAVANA1501 )
  
  
  • [x] n-gram (in-progress #734)
  
  
  • [x] Binarizer (in-progress #744)
  
  
  • [] PCA (in-progress)
  
  
  • [ ] PolynormalExpansion
  
  
  • [ ] Dicrete Cosine Transform (DCT)
  
  
  • [ ] StringIndexer (in-progress)
  
  
  • [ ] IndexToString
  
  
  • [ ] OneHotEncoderEstimator
  
  
  • [ ] VectorIndexer
  
  
  • [ ] Normalizer
  
  
  • [ ] StandardScaler
  
  
  • [ ] MinMaxScaler
  
  
  • [ ] MaxAbsScaler
  
  
  • [X] Bucketizer
  
  
  • [ ] ElementwiseProduct
  
  
  • [x] SQLTransformer (https://github.com/dotnet/spark/pull/781 @ramanathanv)
  
  
  • [ ] VectorAssembler
  
  
  • [ ] VectorSizeHint
  
  
  • [ ] QuantileDiscretizer
  
  
  • [ ] Imputer
  
  
• [ ] Feature Selectors
  
  
  
  • [ ] VectorSlicer
  
  
  • [ ] RFormula
  
  
  • [ ] ChiSqSelector
  
  
• [ ] Locality Sensitive Hashing
  
  
  
  • [ ] LSH Operations
  
  
  • [ ] Feature Transformation
  
  
  • [ ] Approximate Similarity Join
  
  
  • [ ] Approximate Nearest Neighbour Search
  
  
  • [ ] LSH Algorithms
  
  
  • [ ] Bucketed Random Projection for Euclidean Distance
  
  
  • [ ] MinHash for Jaccard Distance
  
  

If anyone else is going to implement probably best to put a comment here and I'll keep the list up to date.

[imback82]

Thanks @GoEddie for compiling the list.

If anyone else is going to implement probably best to put a comment here or something??

This sounds good to me. We will review #378 in the next few days (some of us are out on vacation).

[GoEddie]

Great thanks, @imback82 I didn’t expect anyone to reply until the new year :)

[Niharikadutta]

@GoEddie Thanks so much for your work on this! Would you be willing to write a blog post on these ML-features currently supported on .NET for Apache Spark? Please let us know if that is something that could interest you, and if you might require any help from us. Thanks! :)

[GoEddie]

Hi @Niharikadutta sure no problem I would be happy to do this.

Where is best to publish it? My site or is there a ms one I could guest write it for?

[Niharikadutta]

Hi @GoEddie Awesome, thanks so much! I will get back to you with the details about where to publish and how to get you guest access for it. Could you provide me with a preferred email account you would wish to have this associated with, and I will work on getting that access. Thanks again! :)

[GoEddie]

Great, ed.elliott AT outlook.com is good :)

[Niharikadutta]

Perfect, thank you! Will get back to you soon with the details. Also, is this email good to also have the monthly office hour Teams meeting on, as discussed in a separate thread?

[GoEddie]

Yes great (sorry forgot about that issue!)

[Niharikadutta]

Awesome, thanks for confirming! Will set that one up starting sometime next month if that works for you, or please let me know when you would like to have it start, along with your day of the week/time preference.

[GoEddie]

Just whatever suits you really, I didn’t want to make work for anyone

[Niharikadutta]

oh no we are happy to do this :) I just want to make sure I have your timezone/work hour constraints in mind before setting a meeting up.

[GoEddie]

Great, I’m in the UK so morning US time works better for me

[Niharikadutta]

Sounds good, will set up a cadence accordingly, thanks!

[Niharikadutta]

Hi @GoEddie , We had a discussion with the .NET blog folks and looks like the current process of doing this is to have the guest publish on their own blog if they have one, which then gets amplified by us through social media. Do you have a blog you can publish this on? If no, we can work on co-authoring one on the .NET blog.

[GoEddie]

Hi @Niharikadutta yes no problem, I have a blog https://the.agilesql.club/ - what are you looking for in the post?

[SARAVANA1501]

@Niharikadutta @GoEddie I saw a list of features, quite a lot and amazing, I would like to contribute this, Can you help me to get start, gain context?

[Niharikadutta]

Hi @SARAVANA1501 ! Thank you so much for your interest in contributing to this project. You can go through the following docs to familiarize yourself with the project and it's coding guidelines:

1. Coding guidelines
2. Getting started guides
3. Contributing guide
4. Developer guide

Also you can checkout a few PRs related to ML features as implemented by @GoEddie:

1. FeatureHasher
2. ML/CountVectorizer and ML/CountVectorizerModel
3. ML Features: Word2Vec

Please feel free to reach out in case of any questions or doubts and let us know if you have any feedback about the on-boarding docs or process. Thanks again!

[SARAVANA1501]

@GoEddie update list NGram is in progress

[ramanathanv]

@GoEddie As part of #381, SQLTransformer class is in progress

[GoEddie]

@GoEddie As part of #381, SQLTransformer class is in progress

Awesome - have updated list :)

[ramanathanv]

Hi @GoEddie . I have started implementation for StringIndexer class.

[GoEddie]

@ramanathanv awesome, have updated the list