Recommenders: Unable to create an appropriately versioned cluster per instructions in reference architecture and als_movie_o16n

[jreynolds01]

What is affected by this bug?

• Creating an appropriate cluster
• Running a notebook on that cluster
• Unit tests.

In which platform does it happen?

• Azure Databricks.

How do we replicate the issue?

1. Create a databricks workspace
2. Navigate to Clusters
3. Click [+Create Cluster] In the Databricks Runtime Version, there is no longer an option for DB 4.1, Spark 2.3.0. It was deprecated on 2019-01-17. See deprecation schedule here.

Expected behavior (i.e. solution)

Workarounds:

• It is still possible to create a cluster by cloning a cluster of the recommended version.
• It is still possible to create a cluster through the API. Happy to do PR with appropriate json for creating with databricks CLI or directly through the REST API.

Other Comments

Have we tested whether the cosmosdb connector jar works with more current versions of ADB and spark?

[yueguoguo]

It is still possible to create a cluster through the API. Happy to do PR with appropriate json for creating with databricks CLI or directly through the REST API.

This would be awesome.

Have we tested whether the cosmosdb connector jar works with more current versions of ADB and spark?

It was mentioned here that which of the versions are supported.

[jreynolds01]

ok, have an initial draft working here. https://github.com/jreynolds01/Recommenders/blob/jeremr-refarch-review/notebooks/04_operationalize/create_and_configure_cluster.ipynb

Will file a PR after going through the walkthrough again to make sure that's clear.

[jreynolds01]

Do you have a recommended way of writing a test for something like this? I can use papermill to run the notebook, but the biggest issue I don't know how to deal with is in dealing with the secrets associated with the DB workspace.

[miguelgfierro]

As of now we don't have an example of testing something on databricks, I know that devops recently added a way to do CICD with it https://docs.azuredatabricks.net/user-guide/notebooks/azure-devops-services-version-control.html. That should allow one to manage secrets

[yueguoguo]

One approach is to go with the DevOps setup for Databricks. Another is to hide the secrets by using dbutils in Databricks. See here.

[jreynolds01]

@miguelgfierro - this is the issue I mentioned earlier today that raised the question of what version the cosmosdb jar works with.

[jreynolds01]

We should differentiate the dependencies of the reco_utils module and the dependencies of the java jar.

The conda file we create for reco_utils installs pyspark=2.3.1, and that conda file is used to create the testing environment, so we know that the pyspark tests are run with 2.3.1.

So, there seems to be a little bit of version skew between our testing environment, and our recommended azure databricks env (which is due to the cosmos connector).

We need to decide which side of the trade-off we want to be on:

• Keep using the version that has been tested by the connector group per https://github.com/Azure/azure-cosmosdb-spark with the knowledge that we're recommending a deprecated version of Databricks and force creation through the API.

or

• Update the version and knowingly go to a situation where we may be able to get it to work, but it's not thoroughly tested, so it may produce unexpected behavior that we don't control and can't really fix. If we decide this route, do we recommend an incremental change (e.g. spark 2.3.x) vs. something that is more likely to have an impact (e.g. 2.4.x).

[miguelgfierro]

I think for spark we should stick with databricks which is the reference for our customers. Can we choose a new version (stable enough) and refactor our code and reference architecture to stick with it?

Commented by @nikhilrj in PR #438: I tested the O16N notebook with Databricks 5.0 / Spark 2.4 and it worked oob. Would this be a good version to take?

[jreynolds01]

Update per our sync this morning:

Current Plan: update tests to 2.4.x, with plan to update to a newer version of databricks while pushing the cosmosdb connector team to update their tests. In addition to our normal feedback process, I've also filed a public issue with that team here: https://github.com/Azure/azure-cosmosdb-spark/issues/286

[jreynolds01]

@miguelgfierro - I would prefer to try 5.1, because it allows for notebook-scoped libraries, which would dramatically simplify things.

I will test that real fast now (ignoring the work to update libraries).

[jreynolds01]

It ran successfully on databricks 5.1, spark 2.4 with these versions:

PySpark version: 2.4.0.dev0
Azure SDK version: 1.0.8

There doesn't look like there's a long-term support (LTS) version yet with 2.3 or 2.4.

[miguelgfierro]

Should we move everything to databricks 5.1 and spark 2.4. We should then also check all the tests for stand alone spark: https://github.com/Microsoft/Recommenders/blob/staging/scripts/generate_conda_file.sh#L85

[jreynolds01]

For that script, should we update the spark version to be a variable that the build agent can set to facilitate version testing?

[miguelgfierro]

For that script, should we update the spark version to be a variable that the build agent can set to facilitate version testing?
sounds good, maybe we could add an optional input and put as default the version we think is best

[jreynolds01]

Created issue: https://github.com/Microsoft/Recommenders/issues/460 and pull request: https://github.com/Microsoft/Recommenders/pull/461

[jreynolds01]

updated ref arch. so this is solved for now.