Here's what the entry country looks like:
# conf/base/catalog.yml
my_partitioned_dataset:
type: "PartitionedDataSet"
path: "s3://my-bucket-name/path/to/folder"
dataset: # full dataset config notation
type: "pandas.CSVDataSet"
load_args:
delimiter: ","
save_args:
index: false
credentials: "my_credentials"
load_args:
load_arg1: "value1"
load_arg2: "value2"
filepath_arg: "filepath" # the argument of the dataset to pass the filepath to
filename_suffix: ".csv"
Here's what we'd like it to look like, note that if the entry has some sort of punctuation then the quotation marks remain:
# conf/base/catalog.yml
my_partitioned_dataset:
type: PartitionedDataSet
path: s3://my-bucket-name/path/to/folder
dataset: # full dataset config notation
type: pandas.CSVDataSet
load_args:
delimiter: ","
save_args:
index: false
credentials: my_credentials
load_args:
load_arg1: value1
load_arg2: value2
filepath_arg: filepath # the argument of the dataset to pass the filepath to
filename_suffix: ".csv"
I'll work on this
Should this change include removing the quotation marks found in the "Checkpoint configuration" section?
my_partitioned_dataset:
type: "IncrementalDataSet"
path: "s3://my-bucket-name/path/to/folder"
dataset: "pandas.CSVDataSet"
checkpoint: "2020-01-01/data.csv"
It would be great is you could handle this section too @s-mawjee!