Server: Feature under development: Scheduled commands

[jesperstarkar]

In an ongoing project we're working on adding scheduled commands. @Julusian is the lead programmer. Related to #871 and #850.

Based on a channels timecode, all channel (and layer)-related commands can be buffered and scheduled to be executed by the server in the future, triggered by a timecode.

Proposed API

>>TIME [channel] 
<<2xx [some positive response]
<<[00:00:00:00] //some timecode
returns current timecode of the channel

>>SCHEDULE [timecode] [token] [command] 
<<2xx [some optimistic positive response]
Schedules a command for a given timecode. Multiple commands can be stacked into the same frame/timecode. Token needs to be unique. Scheduling a new command (at any timecode) with an existing token removes the old scheduled command with the same token.

>>UNSCHEDULE  [token]
<<2xx [some positive response] // or
<<xxx [some negative response] // if token is missing or in the past
Removes a scheduled command

>>UNSCHEDULE  ALL
<<2xx [some positive response]
Removes all scheduled commands

>>LIST SCHEDULED
<<2xx [some positive response]
<<[00:00:00:00] [token] //list, multiple lines
Return a list of all scheduled timecodes with their tokens, in chronological order. Only commands of the future, never current or previous ones.

>>LIST SCHEDULED [timecode]
<<2xx [some positive response]
<<[00:00:00:00] [token] //list, multiple lines
Return a list of scheduled timecodes with their tokens for a given timecode, in chronological order. Only commands of the future, never current or previous ones.

Other ideas

• command EXPEDITE [token] to trigger a scheduled command immediately.
• relative timing offset in timecode (+00:00:03:00)

The first prototype implementation is working well and it is insanely cool to be able to batch commands like this.

[premultiply]

I would not recommend to do it by waiting for a specific or absolute timecode to occur. Because you can never trust timcodes to be monotonic and there are issues with wrap around, commands sent to late and played out 24h later etc.

I woud suggest to do it with relative timing.
Other commands like "DELAY" would benefit from it.
Many commands could by scheduled. Most of the time they would have an implied offet of 0 (do it now) but they may be also delayed by any time offset.

[jesperstarkar]

You do have a valid point on the 24h paradox. However, I think we will aim for absolute timecodes, and leave the housekeeping of validating scheduled commands to the client.

@Julusian Perhaps we should warn/handle if a scheduled command didn't fire, and at a given time is scheduled for the past (timecode have moved past the given time, the command is stil scheduled).

[ronag]

Why timecodes? Why not just use system time?

[jesperstarkar]

System time will be one of the three modes a channel can run on: System time, Slaved (external input) and Free running.

External input will be useful for timed liveshows and slaving to backtracks or light cues. External input will also be useful to fire events upon an incoming live source with embedded timecode, regardless of the delays in the signal path upstream.

[MauriceVeraart]

Would be a nice feature!!

[TomKaltz]

I know @ronag favors keeping the core server as simple as possible and there is some merit to that idea but I am 100% excited for this feature!!!

[ronag]

If you want it frame accurate it needs to be in the server.

[Julusian]

Because you can never trust timcodes to be monotonic and there are issues with wrap around
I woud suggest to do it with relative timing.

@premultiply The issue with relative timing is that it will be relative to when the server scheduled it, not when the client sent it. And also makes it a lot harder or potentially impossible to reliably queue commands for the same frame.
If the timecode is non monotonic, then the command will be fired on the first time it is hit. I am thinking to run commands for a short period after their target frame, so that if a frame is dropped then commands will at worst run a few frames late.

Other commands like "DELAY" would benefit from it.

Do you mean the delay exposed in the client? I believe that is currently done client side, but we are considering supporting relative timing, so that could be done with this too, which should make it more reliable

Perhaps we should warn/handle if a scheduled command didn't fire, and at a given time is scheduled for the past (timecode have moved past the given time, the command is still scheduled).

Yes, as I said above I was planning to handle a small overshoot gracefully, but any larger jumps may want some thought as to how they should be handled and that could affect how the scheduled queue is implemented

[premultiply]

@Julusian We do frame accurate timing with CasparCG 2.0.7 and broadcast playout automation system with AMCP over serial connection. This is really not the problem.
We had many many issues before with a "professional" CG system with a timecode interface you like to implement now. One of the major problem was that scheduled contents where not played because CG system internally decided that commands where received to late. Other problems were the source timecode was non-continous etc. All this timecode-based playlist stuff is very complex and is getting worse if you have to sync two playlist-driven systems (Client and CasparCG in this case).
After removing all this timecode nonsense and switched to simple scheduled commands by automation system all the problems and complexity was gone in our case.
And the lesson we have learned is that it is less important that contents will be played 100% frame accurate but they have to be played at all. But CasparCG told us that it is even possible to have very accurate playout under controlled conditions.

With DELAY i mean a new AMCP command which will pause the serial command execution on an AMCP control channel for an ammount of time.

Think of

PLAY 1-1 ...
DELAY 2400ms
MIXER 1-1 ...

All these commands may be sent to server at once (or saved as macro file on server) and it would execute in order without any further interaction.

[jesperstarkar]

@premultiply We too do frame-accurate-ish Client-side rundowns, but we think this will be a good addition to the server side. I appreciate you sharing your experiences and concerns, and I think our combination of client-side logic with this SCHEDULE api will be rock solid.

[ronag]

I guess the reason you want this with timecode is that you want it to happen frame accurately together with things outside of CasparCG. How will the latency be handled?

[MauriceVeraart]

I guess the reason you want this with timecode is that you want it to happen frame accurately together with things outside of CasparCG. How will the latency be handled?

True in broadcast it happens less often I guess but in theatre & staging lights/sound/video and whatever are quite often synchronized with timecode.

scheduling against the system clock would be a nice feature too for sync starting multiple channels.
but I wonder how to implement this in the AMCP protocol in the most convenient way.

[jesperstarkar]

but I wonder how to implement this in the AMCP protocol in the most convenient way.
Have a look at my suggestions above, and please comment with suggestions and ideas!

[ronag]

One thing missing here is that you probably want to be able to schedule multiple commands in a single call. Or you could end up in weird scenarios. Could [command] be a comma-separated list?

[ronag]

Actually, what I think we also need is a way to batch/transact multiple AMCP commands in a way where they are guaranteed to all be performed in a single tick. That way my comment above is unnecessary.

[MauriceVeraart]

hi jesper / Ronald

reading your suggestion one's more it now makes sense to me.

I have 3 scenario's in mind for using this.

1 fully programmed slaved timecode show (where jespers protocol makes sense & looks really good)
2 sync starting multiple channels (in a instant/nonplanned/adhoc way Clips need to start as soon as possible but at the sametime) the beauty of casparcg imo
3 timecodechase (when timecode off the loaded clip matches incoming timecode, clip will follow/chase when incoming timecode stops the clip stops)

I guess scheduling more commands on the same timecode / time is like using te same timecode with a different token right ?

[jesperstarkar]

1 fully programmed slaved timecode show (where jespers protocol makes sense & looks really good)
2 sync starting multiple channels (in a instant/nonplanned/adhoc way Clips need to start as soon as possible but at the sametime) the beauty of casparcg imo

Exactly!

3 timecodechase (when timecode off the loaded clip matches incoming timecode, clip will follow/chase when incoming timecode stops the clip stops)

Will not be supported

I guess scheduling more commands on the same timecode / time is like using te same timecode with a different token right ?

Yes! I have already tested this. It's not on the same tick, as Robert suggests, but it is for sure within a frame. And it is über-cool.

[MauriceVeraart]

Love to help with beta testing

[MauriceVeraart]

Is this code available somewhere just to look into it.
i understand it is not ready yet

[jesperstarkar]

We'll share/open as soon as possible. This is currently our number three priority, after #920 and #776

[MauriceVeraart]

Ok thanks first things first of coarse ;)
sorry for my impatience

whats roughly the status

• proof of concept

• buggy

• nearly ready

LS
Maurice

[jesperstarkar]

PoC. @Julusian want to chime in?

[Julusian]

I would class it as a proof of concept.
I have been busy on other things, so haven't much time to work on it since this discussion was started.

[Julusian]

Over the weekend I have implemented some more of the schedule commands. Here is my revised proposed list of commands. Any feedback is welcome.

The key differences:

• Commands are now SCHEDULE [cmd] to make the names more clear and consistent
• SET command has token as first parameter
• Many of the response codes have been filled in more accurately

>>TIME [channel] 
<<201 TIME OK
<<[00:00:00:00] //some timecode
returns current timecode of the channel

>>SCHEDULE SET [token] [timecode] [command] 
<<202 SCHEDULE SET OK // or
<<xxx [some negative response] // if timecode is invalid
<<xxx [various negative responses] // if command is invalid
Schedules a command for a given timecode. Multiple commands can be stacked into the same frame/timecode. Token needs to be unique. Scheduling a new command (at any timecode) with an existing token removes the old scheduled command with the same token.

This will send a response upon execution to the client who scheduled the command, using the schedule token as the RES token.

>>SCHEDULE REMOVE [token]
<<202 SCHEDULE REMOVE OK // or
<<403 SCHEDULE REMOVE FAILED // if token is missing or in the past
Removes a scheduled command

>>SCHEDULE CLEAR
<<202 SCHEDULE CLEAR OK
Removes all scheduled commands

>>SCHEDULE LIST
<<200 SCHEDULE LIST OK
<<[00:00:00:00] [token] //list, multiple lines
Return a list of all scheduled timecodes with their tokens, in chronological order. Only commands of the future, never current or previous ones.

>>SCHEDULE LIST [timecode]
<<403 SCHEDULE LIST FAILED // if timecode invalid or
<<200 SCHEDULE LIST OK
<<[00:00:00:00] [token] //list, multiple lines
Return a list of scheduled timecodes with their tokens for a given timecode. Only commands of the future, never current or previous ones. 

[MauriceVeraart]

This looks really good to me !
maybe SCHEDULE LIST [token] would be a nice addition to search on which timecode you scheduled a token a bit off the inverse off SCHEDULE LIST [timecode]

[Julusian]

maybe SCHEDULE LIST [token] would be a nice addition

That would be easy to add.
I think that SCHEDULE INFO [token] would be a better name, as you are asking about a certain entry, not multiple. Or perhaps SCHEDULE INSPECT [token]? I'm also thinking that more info can be added to this response in future if there is some data worth returning, but not to SCHEDULE LIST.

[MauriceVeraart]

happy to hear it is easy to implement

i take the syntax off the command the way it comes ;)
but you have good points there.
i like short so my 2cents to SCHEDULE INFO [token]

keep up the good work

ls
Maurice

[Julusian]

I have SCHEDULE INFO defined as the following:

>>SCHEDULE INFO [token]
<<403 SCHEDULE INFO FAILED // if token not found or
<<201 SCHEDULE INFO OK
<<[00:00:00:00]

Some notes about how the scheduler is currently implemented:
When a scheduled command gets executed, it will send a response back to the initiating client for confirmation of execution. It sends the same response as it would have if executed normally (eg 202 PLAY OK), using the RES syntax with the schedule token used as the id:

SCHEDULE SET abcd 00:01:00:00 PLAY 1-10 GREEN
202 SCHEDULE SET OK
...
RES abcd 202 PLAY OK

The timecode paramater is of the form hour:minute:second:frames. In interlaced channels fields are counted rather than frames, as each field output is rendered as a full frame internally. This means it isnt presented as true SMPTE timecode, as that can't do framerates above 30 without specifying flags, but I think this makes most sense to the user

[Julusian]

This also includes command batching, as it uses the same execution method and just required some additional parsing code.
The new names are BEGIN COMMIT and DISCARD

>>REQ abc BEGIN
>>PLAY 1-10 GREEN
>>TIME 1
>>REQ bbb PLAY 1-11 BLUE
>>PLAY
<<xxx PLAY ERROR // parsing errors immediately return an error, but the batch is continued.
>>COMMIT
<<202 PLAY OK
<<201 TIME OK
<<00:00:33:27
<<RES bbb 202 PLAY OK
<<RES abc 202 COMMIT OK

>>BEGIN
>>PLAY 1-10 GREEN
>>DISCARD

Some things to note:

• You can schedule inside a batch (will be useful if relative timecode is added)
• You cannot schedule a batch
• using REQ inside a batch gets respected for any responses, but is optional
• If you want to know what timecode the batch is executed at, then include TIME [channel] as one of the steps

[TomKaltz]

Looking forward to your branch being made public so we can test this new cool feature!

[baltedewit]

>> SCHEDULE CLEAR
<< 202 SCHEDULE CLEAR OK
Removes all scheduled commands

It might be a good idea to broadcast this to all connected clients. e.g. we might leave commands in memory to wait for them to be executed, however, if they get cancelled by another client without our knowledge they will end up in staying memory indefinitely.

[Julusian]

We were originally planning to make the timecode string used be valid SMPTE timecode, but that doesn't cleanly support high framerates. When SMPTE goes above 30p it counts in a confusing manner, eg for 50p:

01:00:00:23
01:00:00:23*
01:00:00:24
01:00:00:24*
01:00:00:00
01:00:00:00*

In some places (eg decklink driver) no suffix is used when in the string format, each number is simply used twice. Adobe software looks to not use SMPTE timecode. I'm not sure what suffix is normally used, so am open to suggestions if we go that route.

I think the suffix approach will be confusing for users who read and write these timecodes. Instead it would be simpler to count up to the framerate. I haven't seen anything on supporting framerates above 60p which will need considering when either #674 is implemented or 24G-SDI is published

Additionally, in 2.2 50i channels render internally as 50p and interlaced on output, so do we want the scheduling timecode to give us field resolution or just whole frames (in this case 25p)?

Any input on this is much appreciated :)

[premultiply]

When talking about "timecode" you are always talking about SMPTE Timecode. So yes, it needs to be implemented the way the SMPTE has specified it. But it is not as bad as you may think at the first moment. Many devices like frame synchronizers, vision mixers, NLE software, Dolby encoder/decoders, VANC data insertion in general etc. still need to be aligned to the "standard framerates" (24, 25, 30) to support proper conversion to/from interlaced images even if they support "double framerates" (50, 60). "Nobody want a mixer to cut sources between fields of a frame"!
Of course you can implement your own timecode but that would be incompatible with all other devices

[Julusian]

@premultiply To clarify, the timecode output via consumers will definitely be valid smpte timecode, I am wondering about the timecode we use in amcp and osc, as that doesn't need to be smpte unless we want it to

[premultiply]

Ok, then I would prefer fractions of seconds "timecode" for internal use with "endless" precision/resolution.
HH:MM:SS.ffffff or SECONDS.fractions with decimal datatype. This would support any framerate.

[MauriceVeraart]

Hi Julusian
What is your input ??
is it like the framenumber off the FFMPEG producer??

i remember the * from the early tape days ;)

hmm normally you would use this with other gear using timecode too,
when notation is going to be different from the other gear this will be confusing for operational staff.

TC shows i am doing with other gear on 30+ framerates are normally using real SMPTE notation were the "*" timecodes are just ignored (lowering the time resolution to half FPS)

[Julusian]

@premultiply that is one option. I think I would end up converting that fraction to a frame number internally, rounded to the nearest frame, and so responses would end up being the ideal number.
so that if you scheduled in 25p on 01:00:00.111 it would report that back under SCHEDULE LIST as 01:00:00:120.

@MauriceVeraart The timecode source will have accuracy down to the field as the timecode comes from either the system clock or decklink producer with some help if the framerate doesnt match, but I can shuffle the numbers around as needed to expose less accuracy to the user if desired. That is already being done on the decklink consumer.

I can understand that a different scheme would be confusing if using other gear with timecode, but equally using smpte would be confusing to those who do not use timecode elsewhere.
Also I would like to avoid lowering the resolution if possible, as that would be an annoying limitation on a feature aimed at frame accurate command execution.

[Julusian]

This is at the point where it is working pretty well (with some limitations), so I think it is time to share a preview build.

Some warnings and known issues:

• The timecode is hard coded to increment correctly for 50i. In any other format it won't behave correctly
• Channel time will start at 00:00:00:00 and will freerun, it can't be set any other way
• AMCP is expecting the timecode to be valid smpte (pending conclusion to the above)
• the scheduler may get a bit confused when the timecode crosses the midnight mark
• This is based on our 2.1 branch which has some other changes thrown in (it will be ported to 2.2 later)
• It is still a work in progress

Let me know if you have any thoughts or find any issues, it would be greatly appreciated.

Code: https://github.com/nrkno/tv-automation-caspar-server
Binary: https://drive.google.com/open?id=1zb2Dau5z3YcRipK23-D96NukU3Aa412H

[MauriceVeraart]

cool you have the timecode on OSC also ;D

[Julusian]

I had a thought. The scheduler commands are currently channel agnostic. This will be fine if all channels are on the same timecode, but if they are different (perhaps off by 1, different fps) then the list would be rather confusing.

One option would be to explicitly specify the channel in each of the commands. The quirk here (or advantage?) is that you could do the following: SCHEDULE 1 SET abc 00:01:00:00 PLAY 2-10 AMB which could be a bit confusing if inspecting the list of scheduled operations, but would also be more flexible and allow for more reliable frame accurate commands across channels.

Alternatively, the list command could include the channel number in the response and optionally take a channel number as a parameter.

[Julusian]

This is now complete and just needs further testing and porting to 2.2
I have a document describing the final functionality including new commands and config as well as known limitations, which will eventually make its way to the wiki
Doc: https://docs.google.com/document/d/1xkksxIdNHMO95RFF8wJ3vYRbXCHwKnjkMwVqwAyjv3A/edit?usp=sharing
Build: https://drive.google.com/open?id=1krPtXGHIxes2MRqzxuagtw7AzZYrIgj7

[PeterAkakpo]

What does it mean to set time code source to FREE. I i have tried CLOCK and it uses the system time.

[PeterAkakpo]

i tried the following and the clip1 and clip2 played concurrently on their respective layers
BEGIN
play 1-0 clip1
play 1-1 clip2
COMMIT

the following only played the last clip2
BEGIN
play 1-0 clip1
play 1-0 clip2
COMMIT

the question is how can we schedule multiple command on the same layer to play one after the other

--using the following commands also just plays the last command (from the documentation, it stated that you can stack multiple commands with same time code provided the token is unique) my understanding of that statement is that both clips will play one after the other.

schedule set 1 13:20:00:00 PLAY 1-0 CLIP1
schedule set 2 13:20:00:00 PLAY 1-0 CLIP2

[baltedewit]

What does it mean to set time code source to FREE. I i have tried CLOCK and it uses the system time.

Free running means it starts at 00:00:00:00 when you start the server and then it keeps running from there on.

you can stack multiple commands with same time code provided the token is unique

my understanding of that statement is that both clips will play one after the other.

I'm afraid you have misunderstood that. What it means is you can execute multiple commands on a single frame, e.g. a MIXER command and a PLAY command. If you schedule two PLAY commands on the same frame then they will both be executed that frame, so two play commands on the same frame.
You can schedule a LOADBG command with an AUTO parameter to make a second clip play after the first. (But note that you can have only 1 clip loaded in the background)

[PeterAkakpo]

Just as a response in the form RES token PLAY OK is sent when a schedule starts playing, I think it will also be good to have a response after it has finished playing or when it is interrupted by another schedule. Some command to also check the state of each schedule wouldn't be bad. Currently the info command does not give much information.

[Julusian]

@PeterAkakpo I can see why you would want that, but that would not be easy to add. After a producer is created, there is no knowledge of where it came from, which makes async responses pretty impossible without some large changes.
That would also be a pretty big change in how the amcp protocol behaves, so I expect is unlikely to happen anytime soon.

If you need that data, you could either schedule another command to run at the expected endpoint of the video, or you can get the current position in the file over OSC and detect when the video finishes from there

[PeterAkakpo]

Thanks Julusian , I think I will go for the first option. Osc is not perfect for such situations because it is not consistent with all media types, using the file/time or file/frame string sometime reports the same increasing value for position and duration from start to finish, duration is never constant, mkv file as one such example, and that really messed by my client application. This is not true with version 2.2 but 2.1, I am making this point because the current implementation of the scheduler is in 2.1.

[PeterAkakpo]

I want to know the difference in terms of execution between these sets of command

1. Schedule set 1 13:00:00:00 play 1-0 blue
   Schedule set 2 13:00:00:00 mixer 1-2 clear

And

1. Begin
   Schedule set 1 13:00:00:00 play 1-0 play blue
   Schedule set 2 13:00:00:00 mixer 1-2 clear
   Commit

If the result is the same, then what's the importance of the begin/commit commands

[Julusian]

There is no difference between those, the batching syntax was added as it internally uses a lot of the same code and is another requested feature.

In your example at 13:00:00:00 it adds a single command to command queue which will perform the play and mixer in the same tick.
The begin/commit syntax allows you to queue this same 'single' command without having to create it through the scheduler.

So they aren't very useful when working with scheduler commands, but have other uses.

[Besedin86]

Hi @Julusian .
When we can expect this feature in one of 2.2 BetaX versions? Could I try it on Linux?

[Julusian]

Hi @Besedin86
I have been very busy on other things lately, so haven't had time to finish porting this to 2.2 yet but hopefully will do very soon. I think it is too late to consider merging it for 2.2 also as it is quite a large change.
I haven't yet tested it on linux, but I will be making sure it works there too.