[kixelated]

Part of the moq-video VAAPI quest, moq-dev/moq#8013, which wires this into moq-video's VAAPI decode backend.

Stacked on moq-dev/vaapi#9: the first commit here is #9's, so review efb73fb, 38dfca1, and d1c7c3a alone until #9 lands.

Problem

The decoder handles H.264 only, so an H.265 broadcast falls back to software decode in moq-video even on hardware that decodes H.265.

Approach

• decode::Config gains codec, H.264 unless set. Decoder dispatches to the existing H.264 decoder or a new H.265 one in src/decode/h265.rs.
• The H.265 decoder is ported from discord/cros-codecs's stateless H.265 decoder and its VA-API backend, with its parser, DPB, and picture modules vendored beside the H.264 ones as codec::h265. It derives each picture's reference picture set (8.3.2) and each slice's reference lists (8.3.4), skips RASL pictures after a CRA that starts the stream over, and handles dependent slice segments.
• Output follows C.5.2: the DPB bumps by the SPS's reorder limit, so a stream coded without B-frames comes out the call it was decoded in. H.264 still trails by its DPB depth, as before.
• Screen content and range extensions are left out. A sequence is accepted when it is Main profile, or Main 10 or Main Still Picture with 8-bit samples, 4:2:0, and cropped only on the right and bottom, the same limits the H.264 side has.
• The two decoders share the VA side. A sequence is keyed on a Format (codec, VA profile, coded and visible size, DPB size), and a picture is submitted through one helper.
• Differences from upstream: reference picture sets and lists are vectors rather than 16-entry arrays, so a malformed count fails instead of indexing past the end; reference lists are built by indexing the RPS modulo its length, which is what 8-8 to 8-11 repeat out to; an inter slice with an empty RPS is an error, where upstream's guard against looping forever only fired for screen content streams; and an end-of-sequence NAL unit lets the DPB out at once.
• Parser fixes over upstream, from review: entry point offsets are a vector sized by the slice's count, so a 4K wavefront slice with more than 32 no longer panics; sublayer profiles and levels are read on their own present flags; tile layouts up to Table A.8's 20 columns and 22 rows parse, and more is an error; the bumping process's latency limit is SpsMaxLatencyPictures (C.5.2.2); and a field-coded sequence (field_seq_flag) is refused, since its pictures would come out as unpaired fields.

Impact

• Breaking: decode::probe takes the codec it checks for. decode::Config has a new public field, so a struct literal without ..Config::new() no longer compiles.
• New: decode::Config::codec, and Codec at the crate root (the same enum moq-dev/vaapi#10 adds for encode; whichever lands second resolves a one-line conflict in codec.rs, the README, and the Cargo description).
• The vendored H.265 parser is about 4,100 lines.

Validation

On Intel Arrow Lake (iHD 26.1.2) and AMD Radeon RX 9070 (radeonsi, Mesa 26.0.8), just ci (fmt, clippy, tests, docs, cargo-deny) and the full test suite. The AMD run pointed the decoder at its render node for the run only.

• decoded_pictures_match_a_software_decoder now covers both codecs, coded by x264 and x265 with two B-frames and three references, at 320x240 and cropped at 852x478. Every picture matches ffmpeg's software decode bit for bit, comes out reordered, and keeps its timestamp. The x265 stream has an IDR and an open-GOP CRA. Half the pictures go through decode_exported and download.
• h265_without_reordering_comes_out_as_it_decodes decodes an x265 stream with no B-frames and checks each call returns its own picture.
• a_new_h265_sequence_decodes_at_its_own_size concatenates two x265 streams at different sizes and checks the first one's tail comes out ahead of the second, at the right size.
• a_4k_h265_wavefront_stream_decodes decodes a 3840x2160 x265 stream with wavefronts and 64-pixel CTBs, whose slices carry more than 32 entry points.
• Unit tests cover the IQ matrix's diagonal scan, the latency limit counting from the reorder limit, and the refusal of a field-coded sequence.
• The H.264 tests pass unchanged.

Alternatives

• Vendoring cros-codecs's generic stateless decoder framework: what the H.264 side already declined, for the same reasons.
• A separate H265Decoder type: two public decoders with identical methods, where moq-video would pick one from the catalog's codec anyway.

Follow-ups

• RASL pictures are not exercised: x265's open GOPs here produced a CRA without leading pictures.
• Main 10 output (P010) would need a second surface format through export and download.

Review

Codex (quest review, gpt-6.1-sol, medium), three rounds against #9's head dba9a75.

• Round 1 on efb73fb, five findings, all fixed in 38dfca1: entry point offsets overflowing a 32-entry array on 4K wavefront input; sublayer profiles gated on the wrong flag; the inferred last tile column or row overflowing its array; the latency limit compared against the wrong threshold; field-coded sequences accepted.
• Round 2 on 38dfca1, one finding, fixed in d1c7c3a: the entry point bound multiplied the tile counts as u8, which overflows at 20x22 tiles.
• Round 3 on d1c7c3a: no findings.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: e859ca9a-15ac-4d76-b7b3-4cf519810117

Commits

Reviewing files that changed from the base of the PR and between af121c7 and d1c7c3a.

Files selected for processing: 12

• Cargo.toml
• README.md
• src/codec.rs
• src/codec/h265.rs
• src/codec/h265/dpb.rs
• src/codec/h265/parser.rs
• src/codec/h265/picture.rs
• src/decode.rs
• src/decode/h265.rs
• src/lib.rs
• src/pool.rs
• src/vpp.rs

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn