Tfx: Consider adding Docker-based TFX workshop tutorial

[wjarek2]

context: https://twitter.com/uncontainer/status/1119979796224561153

[zhitaoli]

While I liked this idea and have mentioned it from times to times, we need to determine how much we will keep a docker based tutorial similar to the step by step workshop . Specifically, if we want to preserve the step oriented progression, how will the workflow looks like for the user?

@rcrowe-google

[zhitaoli]

Also to think about how the development flow will look like if user started the tutorial w/ a simplified docker based tutorial, but decided to proceed to use TFX to solve their own problem (on their own data and code). We want to make sure that the tutorial serves as a step stone towards that.

[Bumbleblo]

I'm interested in working on this tutorial. Is anybody already working on it?

I believe that to solve the issues mentioned by the @zhitaoli would need to separate the installation section from the usage example. Thinking as a developer start using the application he could look for installation guide that would suit him best and then move on to the get started page. That way we could maintain the installation and get started documentation separately.

[zhitaoli]

No one is working on this so go for it.

I agree with the separation of installation and usage in tutorial doc. If
there can be a way to "smoke test" the installation it'll be even better,
but that's optional and with an pre-built docker image maybe we don't
really need it.

On Wed, Jun 19, 2019 at 12:54 AM Felipe Borges [e-mail]
wrote:

I'm interested in working on this tutorial. Is anybody already working on
it?

I believe that to solve the issues mentioned by the @zhitaoli
https://github.com/zhitaoli would need to separate the installation
section from the usage example. Thinking as a developer start using the
application he could look for installation guide that would suit him best
and then move on to the get started page. That way we could maintain the
installation documentation and get started separately.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tfx/issues/51?email_source=notifications&email_token=AAY6AZSOIPARCOYR3LJPOQ3P3HQ4TA5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYBAV2I#issuecomment-503450345,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAY6AZQ4FCF7TWGRYHFXOMLP3HQ4TANCNFSM4HH3TOUQ
.

--
Cheers,

Zhitao Li

[rcrowe-google]

+1 for all of the above, and thanks Felipe for stepping forward!

Robert Crowe | TensorFlow Developer Advocate | [email protected] |
@robert_crowe https://twitter.com/robert_crowe

On Thu, Jun 20, 2019 at 12:25 AM Zhitao [e-mail] wrote:

No one is working on this so go for it.

I agree with the separation of installation and usage in tutorial doc. If
there can be a way to "smoke test" the installation it'll be even better,
but that's optional and with an pre-built docker image maybe we don't
really need it.

On Wed, Jun 19, 2019 at 12:54 AM Felipe Borges [e-mail]
wrote:

I'm interested in working on this tutorial. Is anybody already working on
it?

I believe that to solve the issues mentioned by the @zhitaoli
https://github.com/zhitaoli would need to separate the installation
section from the usage example. Thinking as a developer start using the
application he could look for installation guide that would suit him best
and then move on to the get started page. That way we could maintain the
installation documentation and get started separately.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/tensorflow/tfx/issues/51?email_source=notifications&email_token=AAY6AZSOIPARCOYR3LJPOQ3P3HQ4TA5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYBAV2I#issuecomment-503450345
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAY6AZQ4FCF7TWGRYHFXOMLP3HQ4TANCNFSM4HH3TOUQ

.

--
Cheers,

Zhitao Li

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tfx/issues/51?email_source=notifications&email_token=AKVWSW3X7UZLUXZIB7BBMJTP3KW7BA5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYDORGA#issuecomment-503769240,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKVWSWZ2OR3H5QRR7YNL2W3P3KW7BANCNFSM4HH3TOUQ
.

[Bumbleblo]

Searching in dockerhub I find that image (that I believe it's official). The image is prepared to run the run executor script. Since that the image is not referenced by the official documentation should I use it as a base ? I mean, that is a official image ?

If that is not recognized as a official, it's a goal of the project run the system in a docker image ?

[zhitaoli]

Hi @Bumbleblo,

Yes this image is the official TFX image. It currently runs the run_executor script and will later be converted to a (WIP) run_component.py which runs driver/executor/

I think it's the right thing to extend this image (through "FROM tensorflow/tfx" docker file clause), install apache-airflow/mysql/etc necessary to run the demo, and create another image (i.e, "tensorflow/tfx:latest-airflow-demo"). I can help once we have this image built.

[Bumbleblo]

I already create a initial version for that image (that version is using sqlite). Now I'm trying to understand what is important in default setup script of the workshop.

Is there any other step that was not in that script?

[zhitaoli]

Great! Can you post the Dockerfile for the initial version?

IMHI, high level things we need from the default setup script should be:

• install jupyter notebook;
• install notebook related dependencies;
• start notebook on a different port and expose that port to browser;
• make sure pipeline data and sqlite DB file is on a volume (which can be
  mounted to the container);

The part of copying python files around should be more or less optional as
long as you can get Airflow running.

On Fri, Jun 28, 2019 at 7:32 AM Felipe Borges [e-mail]
wrote:

I already create a initial version for that image
https://hub.docker.com/r/bumbleblo/tfx (that version is using sqlite).
Now I'm trying to understand what is important in default setup script of
the workshop
https://github.com/tensorflow/tfx/blob/master/tfx/examples/workshop/setup/setup_demo.sh
.

With that complete I believe I can start writing the tutorial.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tfx/issues/51?email_source=notifications&email_token=AAY6AZVJWEBVH3G4WGFVPCDP4YOG7A5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2HU6Q#issuecomment-506755706,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAY6AZUR54DRGM7HH3RQGDLP4YOG7ANCNFSM4HH3TOUQ
.

--
Cheers,

Zhitao Li

[Bumbleblo]

Great! Can you post the Dockerfile for the initial version? IMHI, high level things we need from the default setup script should be: - install jupyter notebook; - install notebook related dependencies; - start notebook on a different port and expose that port to browser; - make sure pipeline data and sqlite DB file is on a volume (which can be mounted to the container); The part of copying python files around should be more or less optional as long as you can get Airflow running.
…
On Fri, Jun 28, 2019 at 7:32 AM Felipe Borges @.*> wrote: I already create a initial version for that image https://hub.docker.com/r/bumbleblo/tfx (that version is using sqlite). Now I'm trying to understand what is important in default setup script of the workshop https://github.com/tensorflow/tfx/blob/master/tfx/examples/workshop/setup/setup_demo.sh . With that complete I believe I can start writing the tutorial. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AAY6AZVJWEBVH3G4WGFVPCDP4YOG7A5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2HU6Q#issuecomment-506755706>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAY6AZUR54DRGM7HH3RQGDLP4YOG7ANCNFSM4HH3TOUQ .
-- Cheers, Zhitao Li

Yes, that is the Initial Dockerfile

FROM tensorflow/tfx

# write that in a file later
RUN pip install --no-cache \
    tensorflow==1.13.1 \
    tfx==0.13.0 \
    flask==1.0.3 \
    apache-airflow 

# removing base script
RUN rm -rf * 

WORKDIR /home

RUN airflow initdb

COPY setup.sh . 

ENTRYPOINT bash -c "./setup.sh"

setup.sh script:

#!/bin/bash
airflow webserver -p 8080  &
airflow scheduler  

[Bumbleblo]

Recently a new release of Flask broke the Dockerfile above. I already did an update on the post with the repair

[Bumbleblo]

Great! Can you post the Dockerfile for the initial version? IMHI, high level things we need from the default setup script should be: - install jupyter notebook; - install notebook related dependencies; - start notebook on a different port and expose that port to browser; - make sure pipeline data and sqlite DB file is on a volume (which can be mounted to the container); The part of copying python files around should be more or less optional as long as you can get Airflow running.
…
On Fri, Jun 28, 2019 at 7:32 AM Felipe Borges @.*> wrote: I already create a initial version for that image https://hub.docker.com/r/bumbleblo/tfx (that version is using sqlite). Now I'm trying to understand what is important in default setup script of the workshop https://github.com/tensorflow/tfx/blob/master/tfx/examples/workshop/setup/setup_demo.sh . With that complete I believe I can start writing the tutorial. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AAY6AZVJWEBVH3G4WGFVPCDP4YOG7A5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2HU6Q#issuecomment-506755706>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAY6AZUR54DRGM7HH3RQGDLP4YOG7ANCNFSM4HH3TOUQ .
-- Cheers, Zhitao Li

I pushed a new version of the image with that volumes (one for user data and other with airflow data). Also I already update the dockerhub page description with a new usage command. All files used for build that image it's here.

[zhitaoli]

@Bumbleblo Thanks. I'll try the image out later today and provide feedback. Since I cannot comment on your branch, consider send a PR so I can comment?

@rcrowe-google

[Bumbleblo]

For sure, I'll open a pull request for that docker files and link here.

[Bumbleblo]

Hey @zhitaoli, can you send me your feedback about the latest version of tutorial image ? I made some changes to python 3 and that new version is based in the Dockerfile on master branch.

[SuperCorks]

Any update on this? I'm building a similar image right now and could use this.

[Bumbleblo]

Any update on this? I'm building a similar image right now and could use this.

I believe that this idea has become obsolete as reported in that pr. If that is the situation maybe we could consider closing this issue.