time and synchronization for dummies · clocks have different rates and precision clocks measure...
TRANSCRIPT
![Page 1: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/1.jpg)
Time and Synchronization for dummies
Edward Hervey
![Page 2: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/2.jpg)
Goals
● Feel comfortable with timing and synchronization in GStreamer
● From reallife examples ...● .. to what GStreamer can do ...● .. and understand why
![Page 3: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/3.jpg)
Time
● Ordering (past, present, future) of events
● Measure of duration of events and intervals between them
● “a certain number of repetitions of one or another standard cyclical event ... constitutes one standard unit of time such as the second”
![Page 4: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/4.jpg)
Chronometer (clocks)
● Sundials, water clocks (clepsydra)● Hourglass (Magellan) and candles● Mechanical clock● Quartz and atomic (caesium)
clocks
![Page 5: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/5.jpg)
Time● Clocks have different rates and
precision● Clocks measure the passing of time
(duration, intervals)● The absolute time is not useful
● Meeting at 232895437843294 !● You need a reference
● 01/10/2013 00:00 UTC is 332895437840000
● Meeting in 10mins from “now”
![Page 6: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/6.jpg)
GstClock
● API and base implementation● Monotonic rate● Get current clock absolute time
● gst_clock_get_time()● Schedule event for absolute time
● gst_clock_id_wait()/wait_async()● Works without a pipeline
![Page 7: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/7.jpg)
GstClock
● Different implementations● Doesn't matter for rest of talk● Assume it's the monotonic POSIX
clock
![Page 8: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/8.jpg)
ClockAbsoluteTime
![Page 9: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/9.jpg)
Buffer timestamps
● videotestsrc ! timeoverlay ! xvimagesink sync=False
● Timestamps on buffers produced by videotestsrc
● 0, 1/30s, 2/30s, 3/30s, ....
![Page 10: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/10.jpg)
Buffer timestamps
● Argh, everything goes too fast !● Without synchronization, it's like unix
shell piping (+/)● But the buffers had timestamps !● Synchronization, it's useful (c) (tm)● How are we going to synchronize
against a clock ?
![Page 11: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/11.jpg)
Running time
● We want buffers to be synchronized N seconds after we start playing
● Moment the pipeline switches to playing (base time)● + N seconds, AKA : Clock running time
![Page 12: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/12.jpg)
Base TimeClockAbsoluteTime
RunningTime
0
![Page 13: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/13.jpg)
Segment
● How do we figure out the running time for buffers ?● Take the absolute value ? What if the
first buffer PTS is not 0 ?● We need a reference (from which
to calculate the running time of each buffer)
● Enter GstSegment !
![Page 14: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/14.jpg)
Segment
● Helps define the various time relationships in a stream
● start,stop: first and last valid buffer timestamp
● For any buffer:● PTS – Segment.start => buffer running
time● (not final formula)
![Page 15: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/15.jpg)
Base TimeClockAbsoluteTime
RunningTime
0
Segment.start Segment.stopBuffer.PTS
BufferTimestamps
![Page 16: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/16.jpg)
Base Time
● What if I pause ?● Running time is the amount of time
“spent” in PLAYING● Need to update base_time● PLAYING=>PAUSED : remember
running_time● PAUSED=>PLAYING : base_time =
current_absolute_time – running_time.
![Page 17: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/17.jpg)
Segment rate
● What if I play faster/slower ?● I want buffers to be synchronized
faster/slower● Segment rate property● running_time gets adjusted
accordingly● (B.PTS – S.start) / ABS(S.rate)
![Page 18: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/18.jpg)
Base TimeClockAbsoluteTime
RunningTime
0
Segment.start Segment.stopBuffer.PTS
BufferTimestamps
Segment.rate > 1.0
![Page 19: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/19.jpg)
Segment rate
● What if I play backwards (in reverse) ?
● Segment.rate < 0.0● Buffer have decreasing timestamps● Running time is calculated using
Segment.stop● (S.stop B.PTS) / ABS(S.rate)
![Page 20: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/20.jpg)
Base TimeClockAbsoluteTime
RunningTime
0
Segment.start Segment.stopBuffer.PTS
BufferTimestamps
Segment.rate < 0.0
![Page 21: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/21.jpg)
Stream time● “Userfacing time”
● Position reporting● Seek values
● Quite confusing since it's quite often the same as buffer time.
● When isn't it the same ?● RTP usecases● DVB usecases● Some formats
![Page 22: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/22.jpg)
Stream Time● You connect to a live presentation
via RTP which started 30mins before● RTP timestamps (i.e. Buffer
timestamps) can be anything● You want to be shown how much
in the presentation you are● => Stream Time● Segment.time (reference for stream
time)
![Page 23: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/23.jpg)
Base TimeClockAbsoluteTime
RunningTime
0
Segment.start Segment.stopBuffer.PTS
BufferTimestamps
Segment.time
StreamTimes
![Page 24: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/24.jpg)
So far....
● Absolute Clock Time● Running Time● Buffer Time● Stream Time● Need base_time and segment
![Page 25: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/25.jpg)
Live sources/pipelines
● Time and Clocks are not just used for synchronizing buffers/events,
● Also used for knowing when an event happened.
● Live sources (webcam, microphone, ...)
![Page 26: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/26.jpg)
Live sources/pipelines
● A “live” event is an event that happens “now”
● If you try to capture too early/late you will miss it
● “now” is the current running time of the clock.
![Page 27: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/27.jpg)
Real World EventClockAbsoluteTime
Audio Capture Delay
Video Capture Delay
RunningTime
0
![Page 28: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/28.jpg)
Live sources/pipelines
● The same event captured over different sources should have the same timestamp
● But we have different capture time (1 audio segment duration vs 1 webcam frame duration)
● So we just subtract that value from the current running time ?
![Page 29: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/29.jpg)
Real World EventClockAbsoluteTime
Audio Capture Delay
Video Capture Delay
BufferTime
0
![Page 30: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/30.jpg)
Live sources/pipelines
● Subtracting capture delay from runningtime helps ...
● But would result in all buffers always arriving late (if you wanted to play them back in the same pipeline)
● Enter latency !
![Page 31: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/31.jpg)
Latency
● Ensure buffers/events will be able to be synchronized downstream (i.e. Not dropped)
● As quickly as possible● Not too early and not too late
(grmbl !!!)● How do we figure that
![Page 32: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/32.jpg)
Latency
● Let every element in the pipeline report what● is the minimum latency it is introducing
(for producing/processing data)● Is the maximum latency it can support
(before dropping/blocking)● GST_QUERY_LATENCY● Pipeline emits and distributes ideal
latency
![Page 33: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/33.jpg)
Real World Event
Audio Capture Delay(min latency)
Video Capture Delay(min latency)
RunningTime
0
Audio ringbuffer size(max latency)
V4l2 queue size(max latency)
MAX (min latency) : Global Latency
MIN (max latency)
![Page 34: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/34.jpg)
Latency
● Rendering time becomes:● Latency + running_time
![Page 35: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/35.jpg)
Real World Event
Audio Capture Delay(min latency)
Video Capture Delay(min latency)
RunningTime
0
Pipeline Latency
RenderTime
![Page 36: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/36.jpg)
Latency
● Other elements can introduce latency● Decoders (frame reordering)● Transformation elements● ....
● Or increase maxlatency● Queue !
![Page 37: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/37.jpg)
No more time !
● Different clocks● Slaving clocks and distributed
synchronization● Advanced techniques● Go see Jan's talk● You have the basics !
![Page 38: Time and Synchronization for dummies · Clocks have different rates and precision Clocks measure the passing of time (duration, intervals) The absolute time is not useful Meeting](https://reader034.vdocument.in/reader034/viewer/2022042807/5f7a3a2868e1981e9368f344/html5/thumbnails/38.jpg)
Time for questions ?
● Or time for lunch ?