Enhancements: CronJobs (previously ScheduledJobs)

[soltysh]

Enhancement Description

• One-line feature description (can be used as a release note):
  CronJobs (previously ScheduledJobs) are meant for performing all time-related actions, namely backups, report generation and the like. Each of these tasks should be allowed to run repeatedly (once a day/month, etc.) or once at a given point in time.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/19-Graduate-CronJob-to-Stable
• Discussion Link: sig-apps agenda
• Primary contact (assignee): @soltysh
• Responsible SIGs: sig-apps
• Enhancement target (which target equals to which milestone):
  
  
  
  • [x] Alpha release target 1.4 (as ScheduledJobs)
  
  
  • [x] Beta release target 1.8 (as CronJobs)
  
  
  • [ ] Stable release target 1.21/1.22
  
  

[soltysh]

@erictune fyi

[philips]

@soltysh On what SIG can I discuss this feature? I want to have a longer discussion about third-party resources for this feature and why we feel it needs to be built into core.

[erictune]

Let's use SIG-Apps for now. There has not been a lot of discussion of controllers there, that I have seen, but lets try and see how it goes.

[gtaylor]

Is this being considered for exclusion from the core? I had thought the proposal was already accepted.

[soltysh]

@gtaylor nothing has been decided yet. Currently it'll be part of alpha group in batch, what will happen when this migrates to more stable is not yet known.

[davidopp]

My understanding was that this was accepted into core, while job workflow was rejected for core. In fact we had originally planned to have it in 1.3.

[soltysh]

@davidopp my comment was based on the discussion we've previously had with @philips @erictune here. Although, personally I'd prefer SJ stay in the core :sunglasses:

[davidopp]

@soltysh I interpreted the comment as implying that it would be in core (based on the mention of alpha/beta and the statement "If someone were to produce a third-party version of scheduledJob, well before 1.4, and show that was comparably useful, that would be persuasive argument for the latter path" where the latter path was ThirdParty).

[erictune]

@davidopp that comment was made when I thought SJ would be Beta in 1.4. Now it is going to Alpha in 1.4. The philosophy was that we can cancel an alpha feature for whatever reason, but we should have a pretty high bar for cancelling a beta feature.

[erictune]

Also, to all working on SJ: we should continue work at full speed, despite the above conversation. Much of the work will apply either way, and we need to get some kind of feature out to users so they can give feedback.

[soltysh]

All SJ work is at full speed, the only remaining problem (hopefully) is to have https://github.com/kubernetes/kubernetes/pull/29187 in. I'm hoping to discuss this issue with @smarterclayton today or over the weekend and have it merged so next week we should see one after the other SJ PRs being merged.

[eghobo]

@soltysh: it looks like #29187 was merged, does it mean next 1.4 alpha release will have SJ ready to play?

[soltysh]

@eghobo that's the plan.

[erictune]

This needs docs in k8s.io, but it looks like code is in. Awesome!

[davidopp]

+100

[janetkuo]

@soltysh are docs considered done or are you adding more examples / tutorials? If docs are done we can check the docs box

[soltysh]

@janetkuo I usually mark them done once they merge. With that in mind I've checked the one against branch 1.4, the other will wait for merge.

[soltysh]

Since this was renamed to CronJobs I'll update the title to reflect that change as well.

[ConorNevin]

Is this going to be in beta for 1.5?

[soltysh]

@ConorNevin unfortunately not, see the beta requirements in the issue description what we need to address first to promote it to beta. Sorry :disappointed: Help with fixing those is highly encouraged though :smiley:

[stp-ip]

There is still https://github.com/wercker/cronetes, if one needs cronjob like functionality now without the ability to run alpha features.

[vinay-g]

Would run at most once feature be implemented in CronJobs ?
I see that it was not included in the alpha - https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/scheduledjob.md#decision

[soltysh]

@vinay-g eventually, I guess so, but I have no idea when that will. Help is always welcomed though :)

[JeanMertz]

This feature is still attached to milestone 1.4, and the 1.6 milestone spreadsheet doesn't mention anything about Cron/ScheduledJobs.

Is this still on-schedule for a 1.6 beta release? As a GKE customer, I would _love_ to start moving all our out-of-cluster crons onto the cluster itself (without using cronetes).

[evmin]

Indeed. This a much needed feature has been dragging on from 1.3 version if I am not mistaken. Myself I am in the same position - can not wait till it comes across so I can start pulling the on premise jobs onto GKE.

[soltysh]

I've modified the milestone to next, there are still quite a lot of work to be done for CronJobs to stabilize them, I'd love to push it forward more aggressive, but unfortunately the lack of time is the main factor I can't atm.

[avaranovich]

is imagePullSecrets supported in the ChronJob template?

[idvoretskyi]

@soltysh thank you for updating.

[soltysh]

is imagePullSecrets supported in the ChronJob template?

@avaranovich should be, since we're creating a Pod from the template. If it's not please fill in an issue and tag me there.

[ApsOps]

Hey @soltysh . I'd love to see in this beta soon! I want to help but not sure what is required/ the next step here. Can you please refine the checklist a bit (maybe create/point to relevant issues/docs)? 🙂

[soltysh]

@ApsOps before beta we definitely need to implement server-side removal, support google app engine and chronos format, and allow using timezones. Probably it would be nice to sweep the bugs that are on me that are related to CronJobs. I doubt it's doable for 1.6, but for next releases it's doable. Most importantly tag me in every issue/pr you create wrt to this topic.

[erictune]

@soltysh

I think quite a few people use CronJobs as they are now, and I mostly hear "when will it be beta", an not so much issues with alpha.

If we believe that we can add server-side removal, app-engine and chronos, and timezones without breaking the current API, then I don't see any reason not to move beta now, and add those things before GA.

[evmin]

+1 For CronJobs beta. As soon as they are beta and the cluster reset is not required, it would be a killing feature for a number of our workflows here.

[niclashedam]

We are looking forward to this feature going beta and we have had no problems with alpha.

[soltysh]

Personally I'd prefer address at least the removal part, there's someone already working on adding configurable limits for how many successful and failed jobs are left behind. And the removal is a key element, I think. Let's discuss the option of moving this to beta for 1.6 or 1.7 during upcoming sig-apps call. @michelleN mind adding this as a topic?

[soltysh]

@NiclasHedam have your problems been addressed, are there open issues I haven't seen, mind opening/pinging me in those?

[niclashedam]

They have all been addressed in 1.4.7

[oscarbatori]

Is there any plan to add GUI support this? For example, I can currently see the following:

[obatori@obatori:~] >> kubectl get cronjobs
NAME         SCHEDULE      SUSPEND   ACTIVE    LAST-SCHEDULE
cron-hello   */1 * * * *   False     0         Tue, 25 Jul 2017 09:11:00 -0400
hello        0 22 * * *    False     0         <none>

However, in the GUI I am only able to see historical execution, not current jobs and the schedule associated with them? Also, kubectl get does not list cronjob/cronjobs as valid resource types, though they do indeed work.

Naturally, I could be missing something, but exhaustive searching of the various parts of the GUI has yet to show me my jobs!

[luxas]

@oscarbatori Please open an issue in kubernetes/dashboard repository and ask for that enhancement.

[oscarbatori]

@luxas will do, thanks for your prompt response.

[palak2011]

@soltysh Can you please add the k8s.io Docs PR for this feature for 1.8 release here: https://docs.google.com/spreadsheets/d/1AFksRDgAt6BGA3OjRNIiO3IyKmA-GU7CXaxbihy48ns/edit#gid=0

[idvoretskyi]

@soltysh this feature is listed at the features tracking spreadsheet - https://docs.google.com/spreadsheets/d/1AFksRDgAt6BGA3OjRNIiO3IyKmA-GU7CXaxbihy48ns/edit#gid=0, but doesn't have 1.8 milestone assigned.

Does this feature target 1.8?

[soltysh]

@idvoretskyi partially, the promotion to beta was targeted for 1.8 and happened in that timeframe. There's no milestone set for this, b/c there's no clear plan for the future promotion to stable, yet.

[idvoretskyi]

@soltysh understood. So, I'll mark with 1.8 milestone.

[soltysh]

Thanks!

[soltysh]

@idvoretskyi since there's one feature (Ability to start CronJobs manually) we'll be trying to get into 1.9 release related to CronJobs, I'll add 1.9 milstone here, is that ok? I don't want to create another issue just to track that single item.

[soltysh]

I'll probably try to update the initial description so that it reflects changes introduced (planned as well) for CronJob related features.

[idvoretskyi]

@soltysh any progress with the feature description update? :)

Please, use the new template - https://github.com/kubernetes/features/blob/master/ISSUE_TEMPLATE.md

[zacharysarah]

@soltysh :wave: Please indicate in the 1.9 feature tracking board
whether this feature needs documentation. If yes, please open a PR and add a link to the tracking spreadsheet. Thanks in advance!

[janetkuo]

@idvoretskyi since there's one feature (Ability to start CronJobs manually) we'll be trying to get into 1.9 release related to CronJobs, I'll add 1.9 milstone here

This specific feature won't be in 1.9. Shall we move the milestone to 1.10?

[soltysh]

For the 1.10 milestone there are 3 topics:

1. TimeZone support in CronJob (https://github.com/kubernetes/kubernetes/pull/47266) - @iterion see this comment for reasoning why
2. CronJob manual instantiation (https://github.com/kubernetes/kubernetes/pull/53988) - @erhudy
3. (?) Re-write the controller to use shared informers (https://github.com/kubernetes/kubernetes/issues/17130) - @soltysh

[idvoretskyi]

@soltysh and still beta, right?

[soltysh]

Stable requirements:

1. Shared informers in the controller (https://github.com/kubernetes/kubernetes/issues/17130)
2. Support different time formats (ISO 8601, GCE time format).

[Bradamant3]

@soltysh feature tracking spreadsheet indicates doc updates are needed. Is this indeed the case? If so, please get your PR in asap (today is doc deadline, but I haven't been good about notifying folks). If not, could you please update the spreadsheet? Thanks!

[Bradamant3]

@soltysh docs ping -- deadline for merging docs PRs is this Friday, March 9. See previous comment. Thanks! /cc @idvoretskyi

[soltysh]

@Bradamant3 sorry for the delay, no doc update is needed for this feature. I've added a comment in the linked spreadsheet.

[justaugustus]

@soltysh
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[soltysh]

Any plans for this in 1.11?

Controller re-write to satisfy https://github.com/kubernetes/kubernetes/issues/17130 but I'm still struggling with time. So this is more of a wishful thinking than actual plans :wink:

[justaugustus]

Okay cool. I'm going to push the milestone on this.

[spiffxp]

any update on plans to bring this to stable? I'm assuming based on lack of milestone this isn't going to happen for 1.12?

[justaugustus]

any update on plans to bring this to stable? I'm assuming based on lack of milestone this isn't going to happen for 1.12?

@soltysh ^^

[justaugustus]

@spiffxp -- I spoke with @soltysh earlier. Nothing planned for 1.12.

[kacole2]

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[soltysh]

@kacole2 this is not moving anywhere until we fix the biggest issue with cronjob controller, which is shared informers. We'll be discussing this topic during next SIG-Apps call.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[wadadli]

How about a prefix to jobs such as 20190212T2157Z

[mattfarina]

/remove-lifecycle rotten

[mattfarina]

/lifecycle frozen

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[kacole2]

Hello @soltysh , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. As usual, a KEP will need to be merged before this can progress.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[kfox1111]

As a user, I've been successfully using this resource kind for a long time now. I have not seen any major problems with the api. Is it time to ship it as GA?

[kow3ns]

We are working on a KEP for graduation

[kfox1111]

Ok. Thanks for the update.

[kacole2]

Hi @kow3ns @soltysh , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[palnabarun]

Hey there @soltysh @kow3ns , 1.17 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancement-tracking). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[palnabarun]

Hey there @soltysh @kow3ns ,

1.18 Enhancements team member here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.18? Enhancements are going to be frozen on January 28.

If you do, I'll add it to the 1.18 tracking sheet (https://bit.ly/k8s-1-18-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. :+1:

Thanks!

[palnabarun]

The current release schedule is:

• Monday, January 6th - Release Cycle Begins
• Tuesday, January 28th EOD PST - Enhancements Freeze
• Thursday, March 5th, EOD PST - Code Freeze
• Monday, March 16th - Docs must be completed and reviewed
• Tuesday, March 24th - Kubernetes 1.18.0 Released

[soltysh]

Hey @palnabarun @barney-s is working on closing the KEP in time, the implementation will proceed.

[palnabarun]

Thanks, @soltysh for the updates. I'm guessing the enhancement is targeted to be stable for the release. I am updating the same in the tracking sheet. Please let me know if it is otherwise.

[palnabarun]

/stage stable

[palnabarun]

/milestone v1.18

[palnabarun]

@barney-s Just a friendly reminder, we are just 7 days away from the Enhancement Freeze (Tuesday, January 28th).

Do you have any updates on the KEP?

[jeremyrickard]

Per @mattfarina on Slack, this won't be graduating to stable in 1.18. I'm going to remove it from the 1.18 milestone and drop it from the release tracking sheet.

[palnabarun]

KEP PR: https://github.com/kubernetes/enhancements/pull/978

[jeremyrickard]

/milestone clear

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[harshanarayana]

Hey there @soltysh @kow3ns, 1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

1. The KEP PR must be merged in an implementable state
2. The KEP must have test plans
3. The KEP must have graduation criteria.

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

If you do, I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[harshanarayana]

Hey @soltysh / @kow3ns, I'm following up on my previous update on this Enhancement being part of the v1.19 release.

Do you happen to have any update on the possiblity of this being included in the release v1.19?

Thanks again for your time and contributions. 🖖

[harshanarayana]

Hey @soltysh / @kow3ns, I'm following up on my previous update on this Enhancement being part of the v1.19 release.

Do you happen to have any update on the possiblity of this being included in the release v1.19?

Thanks again for your time and contributions. 🖖

[harshanarayana]

Hey @soltysh / @kow3ns, any plans for the Enhancements to be included in v1.19? Please let me know so that I can update the tracking sheet to show the inclusion state.

_Enhancements freeze is on May 19_

Note that recently the KEP format has changed. Additionally, #1620 merged recently, adding production readiness review questions to the KEP template.
Please take this opportunity to reformat your KEP and also answer the questions added to the template in that PR.

Thanks,
🖖

[harshanarayana]

Hey @soltysh / @kow3ns, Unfortunately the deadline for the 1.19 Enhancement freeze has passed and the KEP #978 is still in flight. For now this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[soltysh]

/lifecycle frozen

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[kikisdeliveryservice]

/lifecycle frozen

Hi @soltysh

Enhancements Lead here. Any plans for this in 1.20?

Thanks,
Kirsten

[soltysh]

@kikisdeliveryservice yes, we are planning on moving this slowly, see https://github.com/kubernetes/enhancements/pull/1996 for proposal, so 1.20 is when we'll be introducing the new controller as alpha. I've just updated the initial description with all the proper links and to match the current template.

[soltysh]

/milestone v1.20

[kikisdeliveryservice]

@kikisdeliveryservice yes, we are planning on moving this slowly, see #1996 for proposal, so 1.20 is when we'll be introducing the new controller as alpha. I've just updated the initial description with all the proper links and to match the current template.

OK I read through this a bunch and I'm a bit confused 😄
The KEP is in beta. It will stay in beta?? until 1.21 GA?

# The target maturity stage in the current dev cycle for this KEP.
stage: beta

# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
latest-milestone: "v1.20"

# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
  alpha: "v1.4"
  beta: "v1.9"
  stable: "v1.21"

It sounds like work will be done during 1.20 (new controller, etc...) to get this through to GA but that work may take a release or 2 to finish before GA? Did I get it right? So this wouldn't need to be tracked for the 1.20 release?

(Please correct me if I'm wrong!!)

[soltysh]

It sounds like work will be done during 1.20 (new controller, etc...) to get this through to GA but that work may take a release or 2 to finish before GA? Did I get it right? So this wouldn't need to be tracked for the 1.20 release?

That is correct. We're not targeting 1.20 per se, but important chunk of work (the new controller) will land in 1.20. That's why I think it should be tracked for 1.20, no?

[soltysh]

/stage beta

[kikisdeliveryservice]

@soltysh Makes sense let's roll with it :+1:

[kikisdeliveryservice]

For my own record we're just waiting for the PR (which meets criteria) https://github.com/kubernetes/enhancements/pull/1996 to merge by October 6th

[kikisdeliveryservice]

KEP merged! :partying_face:

[kikisdeliveryservice]

Hey @soltysh !

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

[SomtochiAma]

Hello @soltysh, 1.20 Docs shadow here.
Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th

Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[soltysh]

Roger that :+1:

[SomtochiAma]

Hi @soltysh
The docs placeholder deadline is almost here. Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline

Also, please keep in mind the important upcoming dates:

• Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for Review
  As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[soltysh]

• Controller implementation - https://github.com/kubernetes/kubernetes/pull/93370
• Documentation - https://github.com/kubernetes/website/pull/24885

[kikisdeliveryservice]

Hi @soltysh !

Looks like kubernetes/kubernetes#93370 is still open but being actively reviewed. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

[soltysh]

Yup, I'm on it, if we don't get the PR merged within next few hours we'll fill in exception.

[kikisdeliveryservice]

It merged! Awesome!!