Kedro: [KED-925] Add Apache Sqoop IO Class

[sw6714]

Description

I would like to use Kedro not only for Data Science pipelines but also for Data Ingestions. For large data ingestion, I usually use Apache Sqoop to speed up the job. Would it be possible to implement an operator in Kedro for end-to-end pipeline?

Context

I will use it as source of my pipeline, which will help me (and surely others) to migrate all my current data pipelines to Kedro

Possible Implementation

Not sure about this but I assume that wrapping pysqoop class (https://pypi.org/project/pysqoop/) in a IO class should be enough

[yetudada]

@sw6714 This is great to read! Could you tell us more about Apache Sqoop? And what you like about Kedro.

Adding support for Apache Sqoop is something we would love to have as part of kedro.contrib.io and we will be more than happy to welcome contributions for the IO class. I will mark this with a good first issue label, so anyone interested in doing can pick it up.

[sw6714]

@yetudada Apache Sqoop is a tool that allows bulk data transfer between Apache Hadoop to relational databases. We personally use for data ingestion from different sources to our data lakes where we process them and export them for Analytics or Data Science engines.
If we could use Kedro to generalize our pipeline definition within the team, visualize all the pipelines and schedule runs with Airflow, that would be fantastic for maintaining flexible and modular pipelines :)

Thank you so much for putting this issue forward as I am sure other would benefit from it to :)
Another thing I am thinking is to also look at some Kafka IO class. I might raise another issue for this though.

[shril]

@yetudada Can I take this issue?

[yetudada]

@shril Go for it! We would love your help here.

[lorenabalan]

I've updated the title with our internal ticket number to keep track of this more easily. :)
@shril how is it coming along? Do you need any help from our side?