Great_expectations: spark_context yaml key does not pass configuration to spark

[EricSteg]

Describe the bug
The spark_context option for a spark datasource in the .yml config file does not work. Configuration options passed through it will not take effect on the resulting SparkContext object.

This is problematic for a number of spark usecases, eg: if you want to read files from S3 you need to have spark install hadoop-aws libraries. Typically this is done through SparkSession.builder.config, which is what spark_context should be passing its arguments to.

To Reproduce
Steps to reproduce the behavior:

1. Create a spark datasource with non-default spark_context arguments eg:

  spark_dataframe:
    data_asset_type:
      class_name: SparkDFDataset
      module_name: great_expectations.dataset
    class_name: SparkDFDatasource
    module_name: great_expectations.datasource
    spark_config:
      spark.jars.packages: 'org.apache.hadoop:hadoop-aws:2.7.3'
      spark.hadoop.fs.s3a.impl: 'org.apache.hadoop.fs.s3a.S3AFileSystem'
      spark.hadoop.fs.s3.impl: 'org.apache.hadoop.fs.s3a.S3AFileSystem'

1. Start a great expectations action in the CLI that uses the spark datasource eg: great_expectations suite new
2. Select the spark datasource you configured and hit enter. This will start the spark server locally.
3. Visit the spark UI (typically at localhost:4040) and go to the Environment tab
4. Search for spark.jars.packages, it will not be found.

Expected behavior
spark.jars.packages to exist in the spark.properties section in the environment tab and be populated with the argument org.apache.hadoop:hadoop-aws:2.7.3

Environment (please complete the following information):

• OS: Ubuntu 18.04
• GE Version: 0.12.9
• Spark Version 2.4.5

[EricSteg]

I have a working fork of GE where I've fixed this issue. Should I open a PR for it?

[ryanaustincarlson]

I can confirm I ran into the same issue, and that it was fixed by updating the relevant Schema definition

[eugmandel]

@EricSteg @ryanaustincarlson Thank you for reporting this! I would love to see both of your fixes - could you please create PRs? Thank you!

[ryanaustincarlson]

Opened a PR (https://github.com/great-expectations/great_expectations/pull/2125) -- not confident this is the best solution, but I'm also not sure I have enough context to know what the right solution is 😅