Spark-on-k8s-operator: Allow sparkctl to create SparkApplication from a ScheduledSparkApplication

[dharmeshkakadia]

This is similar to one time job support from cronjob in kubectl. (kubectl create job one-time --from=cronjob/hello).

[liyinan926]

I have some proposal on extending sparkctl to support this functionality. The sparkctl command for this will look like the following:

sparkctl create sparkapplication one-time --from=<ScheduledSparkApplication name> 

@spjegan @dharmeshkakadia please let me know WDYT.

[dharmeshkakadia]

LGTM

[spjegan]

Just a quick question. Do we really need to say that this is one-time? Can't this be just
sparkctl create sparkapplication --from=<ScheduledSparkApplication name> as it is anyways a one-time job?

Otherwise, this looks good to me too.

Thanks,
Jegan

[dharmeshkakadia]

@spjegan one-time is not a keywork. one-time is the name of the sparkapplication that will be created from scheduledsparkapplication.

[dharmeshkakadia]

so something like
sparkctl create sparkapplication name_of_my_forced_spark_run --from=<ScheduledSparkApplication name>

[liyinan926]

I'm thinking we can even make the name optional. If a name is not given, it gets derived from the name of the ScheduledSparkApplication, e.g., <SSA-name>-one-time. WDYT?

[dharmeshkakadia]

@liyinan926 then you can't create more than one runs from a given ScheduledSparkApplication. I like specifying the name of a run also because I can give it a meaningful name like forced_catchup_run and stuff. I dont think we should generate names,

[liyinan926]

Yes, that makes sense. So I think sparkctl create sparkapplication <name_of_my_forced_spark_run> --from=<ScheduledSparkApplication name> sounds good.

[spjegan]

@dharmeshkakadia thanks for the clarification.

This looks good to me.
sparkctl create sparkapplication <name_of_my_forced_spark_run> --from=<ScheduledSparkApplication name>.

This is quite close to the kubectl equivalent

kubectl create job job-name --from=cronjobs/my-cronjob