Which Clock Should Danmaku Follow? Rewriting Danmaku Sync in BiliKit

The danmaku stack and a few concepts#

BiliKit is a native macOS Bilibili client written with Swift 6, SwiftUI + AppKit, and AVPlayer. Rendering has also changed since the previous post: each comment is now a CALayer whose content is a texture prerendered with Core Text, rather than a CATextLayer. A scrolling comment moves with a CABasicAnimation whose beginTime is the comment’s appearance time. Comments are fetched from Bilibili’s API in 6-minute segments.

The pipeline has three layers. Layout assigns each comment a lane, which is the row it travels along on screen. Presentation decides which comments are on screen at the moment and when their textures are prepared. Rendering manages the CA layers and textures.

A few concepts come up throughout the post.

• timebase: AVPlayerItem.timebase is the player’s internal media clock. You can think of it as a straight line: at some host time (a point on the system clock), media time has some value, and from there it advances at the playback rate. When paused, the rate is 0 and the line goes flat. On a seek, or when the system resets the clock internally, the whole line shifts, and media time jumps.
• snapshot: A periodic observer on the main thread fires every 1/30 of a second and reads the current playback position. Each reading is a snapshot. The original danmaku scheduling advanced by snapshots.
• motion and the on-screen set: Danmaku has to answer two questions. One is where each comment is on screen at the moment, which is determined by its animation and the clock driving that animation. The other is which comments should be on screen at the moment. I call that group of comments the on-screen set.

The original implementation scheduled at runtime. The main thread advanced a cursor by snapshot and added each comment when its appearance time arrived.

Backfilling after seeks kept growing#

The first change replayed a short time window from the landing point after a seek settled, adding the comments that should still be in flight.

After that merged, other problems kept appearing: flashes, small jitters around the landing point, leftover frames while paused, and a flash when starting playback after seeking while paused. I fixed them one by one on a branch called danmaku-seek-catchup, and each fix added another heuristic to the scheduler: a 0.5-second tolerance for backward jumps, a 2.5-second overrun limit, a check for whether the landing point was stable, and a rule that a drag only counted as landed after holding still for 150 ms.

All of these rules inferred from changes in position whether the user had seeked. The backward-jump tolerance, for example, meant that if the position moved back by less than 0.5 seconds, it was not treated as a seek. Every threshold stood for a guess about the user’s intent. The rules kept piling up, and the branch was never merged. Since then, my main test for whether a rule is a patch has been whether it fixes a side effect of another rule.

Then I found that the system provides explicit clock events. The timebase posts TimeJumped and EffectiveRateChanged notifications. The player layer subscribed to both and published two states, “seeking” and “landed at a position.” The danmaku scheduler handled only the landing: clear the screen, then backfill once from the landing point. All of the earlier thresholds were removed.

This approach merged, but starting playback after seeking while paused still flashed once, and I had not found the cause at the time. Another problem was that TimeJumped is a low-level clock event. The system also posts it when it resets the clock internally, so it does not mean the user intended to seek, and the rate alone cannot reliably tell the two apart. The rule I set then was that anything the next step could solve would not get another patch in this approach.

Making danmaku depend only on media time#

The next round changed direction. If which comments are on screen at a given moment, and where each one is, can be computed entirely from the media time t, then the picture depends only on the current t no matter how the user seeks, drags, or changes speed, and there is no need to detect seeks at all. I made three changes along these lines.

• Motion is driven by the media clock. The danmaku layer sits under an AVSynchronizedLayer, which makes the animations of its sublayers follow the player’s time, so pausing, rate changes, and seeks are all handled by the system.
• Lanes use deterministic layout. Comments are sorted offline by (time, id), and each comment’s lane is computed in advance. The same moment therefore always produces the same picture.
• The on-screen set looks at one time window. A comment still on screen at time t cannot have appeared earlier than t − maximum duration. Adding a lookahead lets upcoming comments be prepared in advance. So each update only needs to add and remove comments in the interval [t − maximum duration, t + lookahead] (a reconcile).

Tested on an HLS stream served over local loopback, seeking, dragging, rate changes, buffering, and the end of playback all lined up, and attaching 1,000 layers at once took about 9 ms.

Before starting this round, I looked at how open-source implementations handle it. In the table, “runtime greedy” means finding a free lane only when a comment appears, and “precomputed layout” means placing every comment ahead of time.

Only the precomputed-layout group fills the screen after a seek. Their positions are already a function of time, so jumping to any moment gives the picture for that moment directly. I did not find any open-source danmaku project using AVSynchronizedLayer. I also tested Bilibili’s website and app, and both fill the screen immediately after a seek, so laying comments out in advance matches the official experience.

This round had problems of its own. The first version laid comments out in physical pixels, so scroll duration and lane count both depended on the size of the video area, and any window change meant redoing the entire layout. That led to a set of compensations: layout seeds carried across segments, re-layout for segments that arrived late, full re-layout on fullscreen, zoom, or settings changes, and a crossfade of about 200 ms. The crossfade was removed before merging.

Danmaku slid backward on pause#

After switching to AVSynchronizedLayer, scrolling comments slid backward by 4–10 pt when pausing.

I recorded the moment of pausing with Instruments (Time Profiler + System Trace) and found two blocking periods.

The first came from Live Text (the system feature that recognizes text in a paused frame). pause() triggers a KVO callback for the rate, and inside that callback AVKit’s frame analysis reads currentTime on the main thread. At that moment the player holds a lock for the audio pause handshake, so the main thread waited about 20 ms longer. Turning off allowsVideoFrameAnalysis reduced the blocking on pause from about 55 ms to about 35 ms. The danmaku layer covers the video anyway, so selecting text in a paused frame was never usable there.

The second came from AwaitIOCycle in AudioQueuePause, which waits for the current round of audio I/O to finish. It also holds a lock.

When AVSynchronizedLayer freezes depends on the main thread. While the main thread is blocked, the comment animations keep moving forward. When the layer finally freezes, the comments go back to their positions at the moment of the pause, which looks like sliding backward.

1
beginTime  = now
2
timeOffset = Ta + (now − Ha) · rate
3
speed      = rate

Rethinking the requirements#

At this point I went back to the original requirement. “Keep danmaku accurately in sync with video time” was not really a requirement. It was a means introduced by using AVSynchronizedLayer. Much of the earlier patching was about that means’s side effects, and most of the “requirements” listed above had been inferred backward from specific bugs.

So I wrote down a few invariants instead, properties that must hold however it is implemented. They describe only what the viewer sees.

The next thing to check was whether the player had a perfect time that would make the first invariant, “danmaku time is approximately viewer time,” hold directly. If it did, danmaku could simply follow it.

How the player’s clocks actually behave#

To find out, I wrote a probe program. Its AVPlayerView configuration is identical to BiliKit’s. The test video runs at 60 fps with a barcode encoded in every frame, used to read which frame is currently on screen. The HLS stream has two bitrates, with audio as a separate EXT-X-MEDIA rendition, and the local server can throttle or pause. The program reuses the repository’s timeline adapter unchanged and records the timebase, main-thread snapshots, the on-screen frame number, and main-thread blocking at the same time across a range of scenarios.

AVFoundation’s documentation rarely covers these behaviors. When the timebase jumps and by how much, which thread AVKit calls seek on, and when KVO fires synchronously all have to be measured. Every design decision this time started with data from a separate probe program.

The numbers, split by output device:

The size of the jump in the first row also changes with the output device. In other words, every time playback starts, the timebase first steps back a little, and danmaku that strictly follows the timebase slides back with it.

Other observations:

• On Bluetooth, the timebase silently re-anchors without any notification, producing discrete jumps of +2…+31 ms, with an error of up to ±49 ms. Without a notification, the danmaku side has no timely way to know.
• The timebase stays the same across a bitrate switch. The line does not change, and there is no notification.
• Playing HLS at 4x speed produces a burst of TimeJumped notifications, 6–66 of them every 4 seconds.
• When dragging into an unbuffered region, the snapshot jumps to the target in about 4 ms, while the timebase stays put for 0.2–4 seconds or longer, so the two differ by tens of seconds. If part of the logic reads the snapshot and another part reads the timebase, they see times tens of seconds apart.
• When the main thread stalls, snapshots stop coming. They are neither queued nor coalesced.
• On wired output, the on-screen frame is exactly floor(timebase × fps), so the video and the timebase agree. After a seek, the target frame starts presenting before the timebase restarts and holds for about 93 ms.
• After a pause, the main thread is blocked for about 31 ms; after a rate change, for 24–42 ms.

Two time sources#

While running the spikes, I asked a subagent to do an adversarial review of every danmaku implementation in the codebase’s history: the original runtime scheduling, the catchup branch, the system-event approach, the media-clock approach, the rounds from the viewport through the clock layer, and the clock filter.

The review concluded that every version had two time sources. As described earlier, danmaku has to answer two questions: where each comment is, and which comments should be on screen. The clock answering the first question changed several times, from the root layer to the synchronized layer to the clock layer. The second question, however, was always decided by main-thread snapshots, which arrive every 1/30 of a second, can be coalesced, and can jump backward. The two times were never unified, so whenever they disagreed, a comment’s position and whether it should appear at all would not match.

For example, when playback starts after a seek, the timebase jumps back 57 ms, which puts the snapshot outside the window that has already been laid out. That triggers a recompute, comments that had already finished get added back, and the frame that was being held for a full-screen display gets broken up.

The earlier “first principles” research had only examined the clock and never looked at how the on-screen set was decided. Research should cover the whole system, without assuming in advance that any part does not need to change.

Two practices helped during the review. The reviews did not know about each other, and the design doc required its author to list the three points they thought were most worth attacking, which the reviews then worked through.

The review produced a 14-item checklist. Some of the items:

1. Single time source: a comment’s position and the on-screen set are computed from the same clock at the same moment.
2. No intent detection: do not guess seeks with thresholds, counters, or out-of-range checks.
3. A rule budget: no rule whose only job is fixing a side effect of the previous rule.
4. Tests assert that comments are actually visible, not only that they have been added (installed).

Another review confirmed that the forward-only rule and the single time source had to ship together. Otherwise, under the burst of TimeJumped notifications at 4x speed, 30% of visible comments would be missing.

The current design#

The current design still treats the timebase as the source of truth and no longer tries to change the player’s behavior. It only adds one rule on top, from the viewer’s point of view: during forward playback, time only moves forward.

1
beginTime  = now + (D − T) / r
2
timeOffset = D
3
fillMode   = .backwards

Known limitations#