Codecs, RTP payloads, and streaming design

Match the codec profile, level and packetisation to the decoder and recording system. Include playback and export when checking compatibility.

Overview#

A codec compresses media. A payload format maps codec units into RTP. A container stores or segments tracks. A session/control protocol describes or controls delivery. A transport moves bytes. Supporting H.264, for example, doesn't imply support for every H.264 profile/level, RTP packetisation mode, container, transport, resolution, or bitrate.

Common physical security media formats#

Codec licensing and patent obligations are jurisdiction, use, distribution, and implementation dependent. This page isn't legal advice.

ONVIF and WebRTC compatibility#

ONVIF Profile T covers advanced video and includes H.264 and H.265 capabilities. ONVIF Profile V was still a release candidate on 25 August 2026 and requires H.264 for its WebRTC oriented video capability. These statements don't mean every Profile T product exposes every encoder combination or that every browser decodes H.265. Query device and client capabilities and negotiate an intersection. ONVIF T ONVIF V

WebRTC's protocol profile and codec requirements are defined across RFCs 8834, 7742, and 7874. Browser implementations evolve, so a deployment must verify the exact client matrix rather than infer support from the WebRTC label. WEBRTC MEDIA WEBRTC VIDEO WEBRTC AUDIO

H.264 payload essentials#

H.264 video is composed of Network Abstraction Layer (NAL) units. RFC 6184 defines transport forms including single NAL units, aggregation packets, and fragmentation units. Practical considerations:

• negotiate packetization-mode; mode 0 and mode 1 have different packetisation capabilities;
• treat profile-level-id as a constraint, not decorative metadata;
• obtain and refresh sequence/picture parameter sets (SPS/PPS) according to the negotiated mechanism;
• request or wait for an independently decodable picture after join, loss, or decoder reset;
• don't assume RTP marker means “this packet is a keyframe”; its meaning is payload format specific and related to access unit boundaries;
• bound reconstructed NAL and access unit size before allocation.

H.265 payload essentials#

HEVC adds Video Parameter Sets (VPS) and different NAL/aggregation/fragmentation rules. RFC 7798 also supports temporal scalability concepts. Record the exact profile, tier, level, bit depth, chroma format, VPS/SPS/PPS strategy, and decoder maximums. A gateway that merely forwards RTP still needs to validate packetisation and size; a transcoder must also handle codec state and timestamps safely. H265 RTP

Audio payload essentials#

Opus uses a 48 kHz RTP timestamp clock even when internal audio bandwidth differs. Packet duration, in band FEC, discontinuous transmission, stereo/channel mapping, and maximum playback rate affect intercom quality and latency. OPUS RTP

AAC/RTP has multiple modes and configuration data. Don't derive decoder setup from the codec name alone. For telephony codecs, distinguish sample rate, RTP clock rate, channels, packetisation interval, and DTMF transport.

Keyframes and group of pictures design#

Longer GOPs reduce bitrate but increase random access and recovery delay. Shorter GOPs increase bitrate/storage and encoder work but improve join, seek, failover, and loss recovery. Document:

• keyframe/IDR or random access interval;
• whether scene change keyframes are enabled;
• B frame/reordering depth and decoder delay;
• what feedback requests a recovery picture;
• rate limit for picture loss/full intra requests;
• recorder segment boundaries and whether each segment begins with a usable random access point.

Don't let unauthenticated or unbounded keyframe requests amplify encoder load or bandwidth.

Bitrate and rate control#

Plan using measured peak bitrate, audio, RTP/transport/container overhead, retransmission/FEC, metadata, TLS/QUIC overhead, concurrency, and failover, not the configured average video number alone.

RTP packetisation, MTU, and loss#

• Keep packet size below the effective path MTU after IP, UDP, RTP, SRTP, and tunnel overhead.
• Avoid IP fragmentation; loss of one fragment discards the complete IP packet.
• One lost fragmented NAL packet can invalidate a large access unit.
• Reordering and jitter buffers must be bounded by time and memory.
• Retransmission, forward error correction, and feedback are profile negotiated; don't assume they exist.
• Apply decoder and parser limits before reassembly and decompression.
• Treat codec dimensions, cropping, stride, color format, and frame rate as untrusted until checked against resource limits.

Streaming families#

RFC 8216 defines the published IETF HLS base. Apple continues work on a second edition Internet Draft; an Internet Draft must not be represented as a final RFC. Pin the exact HLS feature set used by a product. HLS HLS BIS

HLS implementation contract#

• Authenticate both playlists and segments; use short lived, scoped URLs where appropriate.
• Prevent cache sharing across tenants/users by setting correct cache keys and response controls.
• Validate every URI obtained from a playlist before server side retrieval.
• Bound playlist size, variant count, segment duration, discontinuities, and live window history.
• Define behaviour for key rotation, expired authorisation, sequence wrap, discontinuity, missing segment, and clock shift.
• Don't place long lived secrets in playlist URLs or logs.
• Treat encryption key delivery and DRM as separate security designs; TLS alone protects only in transit to its endpoint.

MPEG DASH implementation contract#

ISO/IEC 23009-1:2026 is the sixth edition of MPEG DASH Part 1, covering the Media Presentation Description (MPD) and segment formats. An MPD can organize a presentation into periods, adaptation sets, representations, initialisation information, and media segments. It can describe multiple codecs, bitrates, languages, resolutions, roles, timelines, and URL derivation rules; it doesn't guarantee that a particular player supports the resulting profile or media combination. MPEG DASH

Pin the applicable DASH profile and the exact interpretation of:

• static versus dynamic MPD behaviour and its update cadence;
• Period, AdaptationSet, Representation, and dependency relationships;
• BaseURL, SegmentTemplate, SegmentTimeline, SegmentList, and SegmentBase use;
• initialisation segments, indexes, media segment numbering/time, discontinuities, and end of presentation behaviour;
• UTCTiming, availability windows, suggested delay, clock source, and uncertainty for live content;
• codec strings, MIME types, dimensions, frame/audio rates, bandwidth declarations, and decoder limits;
• EventStream/InbandEventStream or other timed metadata used by the application;
• ContentProtection descriptors and the separately selected key/licence system.

Manifest and segment security#

An authorised MPD response doesn't authorise every URL that it references. Apply authorisation to the MPD, initialisation segments, indexes, media segments, keys/licences, timed metadata, thumbnails, and alternate representations. Bind entitlement to tenant, user/workload, camera/resource, purpose, time window, and representation policy.

• Resolve BaseURL and every enabled external reference under an approved scheme/host/path allowlist.
• For server side retrieval, revalidate DNS results and every permitted redirect; reject loopback, link local, metadata service, management, and unexpected private network destinations.
• Cap MPD bytes, XML depth, Period/AdaptationSet/Representation counts, timeline entries, URL length/expansion, segment duration/count, and update rate before allocation or fan out.
• Use secure XML processing with external entity resolution disabled and no implicit network retrieval.
• Prevent one tenant's MPD or segment from satisfying another tenant's cache request. Include authenticated resource/tenant/entitlement dimensions in cache design or use correctly scoped unguessable URLs with bounded lifetime.
• Keep bearer tokens, signatures, licence challenges, device identifiers, and personal data out of cache keys, analytics URLs, referrers, and ordinary logs where possible.
• Define behaviour when MPD authorisation remains valid but a segment or licence expires, and when revoked viewers retain cached bytes.

ContentProtection and DRM descriptors signal protection system information; they aren't proof that media is encrypted correctly, that a licence was issued to the intended principal, or that screen/output controls meet policy. Review media encryption, key derivation/rotation, licence authorisation, client robustness, offline use, revocation, and failure behaviour as a separate design.

HLS, DASH, and shared media#

HLS and DASH can reuse related fragmented MP4/CMAF media assets in some deployments, but shared segments don't make playlist and MPD semantics interchangeable. Authentication, sequence/timeline interpretation, codec/profile signalling, low latency behaviour, encryption metadata, and player feature support remain manifest family specific.

For recordings or evidence derived from adaptive streaming, preserve the exact manifest snapshot and response metadata, selected representation, initialisation and media segment identities/hashes, gaps and discontinuities, clock evidence, decryption/licence context where policy permits, and every remux/transcode transformation. A mutable live MPD or CDN cache isn't a chain of custody record.

Metadata, privacy, and evidence#

Video/audio can contain faces, voices, locations, access events, and behavioral data. Codec metadata may reveal device models, timestamps, encoder configuration, or topology. Define minimisation, retention, export authorisation, watermark/signature verification, chain of custody, and redaction independently of stream delivery.

For evidentiary export, preserve original bytes where policy requires, record hashes and provenance, use a documented time source, and retain the decoder/player requirements. Transcoding changes the bitstream and can remove metadata; record that transformation explicitly.

Safe negotiation fixture#

The following synthetic SDP fragment is suitable as an offline parser/negotiation fixture. It contains no endpoint or credential:

m=video 0 RTP/AVP 96 97
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42e01f
a=rtpmap:97 JPEG/90000
a=sendonly

A parser must not choose the first payload blindly. Intersect the offered payload, direction, format parameters, decoder limits, security profile, and application policy.

Engineering checklist#

• Codec, profile/tier/level, bit depth, chroma, resolution, frame rate, and payload format explicit
• Parameter set/configuration delivery and changes handled
• Keyframe interval and post loss/join recovery measured in the target environment
• Peak bitrate, overhead, concurrency, and failover capacity budgeted
• MTU, fragmentation, reordering, jitter, feedback, and packet loss behaviour bounded
• Decoder dimensions, allocation, CPU/GPU, queue depth, and decompression limits enforced
• Client/browser/hardware codec matrix verified for the supported deployment baseline
• Transcoding latency, quality loss, failure, and trust boundary documented
• HLS playlist or DASH MPD profile, limits, URL resolution, authorisation, and update behaviour explicit where used
• Manifest, initialisation, segment, key/licence, CDN cache, redirect, SSRF, and tenant boundaries reviewed
• Privacy, retention, evidence export, and redaction requirements mapped

Sources#

• H264 RTP, RFC 6184: RTP Payload Format for H.264 Video, IETF, May 2011.
• H265 RTP, RFC 7798: RTP Payload Format for High Efficiency Video Coding, IETF, March 2016.
• AV1 RTP, RTP Payload Format for AV1 v1.0.0, Alliance for Open Media, final deliverable, 15 December 2024.
• JPEG RTP, RFC 2435: RTP Payload Format for JPEG compressed Video, IETF, October 1998.
• OPUS RTP, RFC 7587: RTP Payload Format for the Opus Codec, IETF, June 2015.
• MPEG4 RTP, RFC 3640: RTP Payload Format for Transport of MPEG 4 Elementary Streams, IETF, November 2003.
• WEBRTC MEDIA, RFC 8834: Media Transport and Use of RTP in WebRTC, IETF, January 2021.
• WEBRTC VIDEO, RFC 7742: WebRTC Video Processing and Codec Requirements, IETF, March 2016.
• WEBRTC AUDIO, RFC 7874: WebRTC Audio Codec and Processing Requirements, IETF, May 2016.
• HLS, RFC 8216: HTTP Live Streaming, IETF, August 2017.
• HLS BIS, HTTP Live Streaming 2nd Edition Internet Draft, IETF Datatracker, work in progress, accessed 25 August 2026.
• MPEG DASH, ISO/IEC 23009-1:2026, sixth edition, ISO/IEC, Dynamic adaptive streaming over HTTP (DASH), Part 1: Media presentation description and segment formats.
• ONVIF T, Profile T, ONVIF, accessed 25 August 2026.
• ONVIF V, Profile V, ONVIF, release candidate dated 9 July 2026.

Section overview · Wiki home