The Anatomy of MPEG-TS

Recently, I built a reader and writer for MPEG-TS, the Moving Picture Experts Group’s transport stream format, in Pion’s format package. I began with the smallest pieces a four-byte header, a packet, and then the layers above it. Working upward forced me to understand why video needs a container at all.

This essay is the mental model I wish I had before I started. It begins with raw video, separates the codec from the container, and examines MPEG-TS one layer at a time. The implementation is in Pion format pull request #3, but you do not need the code to follow the explanation.

The central distinction is between a codec and a container. H.264 and H.265 are codecs. They compress raw frames into a smaller byte stream and later reconstruct pictures from it. MP4 and MPEG-TS are containers. They identify the streams around those bytes and preserve the timing that makes them one program.

Why containers exist

Video starts as pixels. Take a 1920 × 1080 frame in red, green, and blue (RGB) format. Each pixel has one byte for each color, so a single frame is about 6.2 MB. At 30 frames per second:

1920 × 1080 pixels
× 3 bytes per pixel
× 30 frames per second
= 186,624,000 bytes per second

That is about 187 MB every second, or 1.49 gigabits per second. A ten-second clip would take about 1.9 GB. A minute would take about 11 GB, and an hour roughly 672 GB. Those figures include no audio.

Much of that data repeats. In a camera view of someone sitting at a desk, the wall, chair, and most of the person barely change from one frame to the next.

Frame 1: room + person + desk
Frame 2: room + person + desk + hand moved slightly
Frame 3: room + person + desk + hand moved slightly again

Saving each frame as a complete picture stores the same information again and again. A codec removes that repetition.

The codec makes video smaller

H.264 and H.265 look for repetition within a frame and across nearby frames. Instead of describing every pixel again, an encoder can express the change from information it has already encoded:

Start with this picture.

Most of the next picture is the same.

This block moved here.

These pixels changed.

Here is the remaining difference.

The algorithms are more involved, but the principle is plain do not store information you can predict from information you already have. The result is a compressed byte stream instead of a sequence of raw RGB frames.

H.264 and H.265 divide this stream into Network Abstraction Layer (NAL) units. Some NAL units contain part of a picture. Others contain parameter sets and control information that tell the decoder how to interpret it. The NAL units that describe one coded picture form an access unit.

Camera frames
    ↓
H.264 encoder
    ↓
NAL unit
NAL unit
NAL unit
NAL unit
…

Compression makes the video practical to store or send, but a camera may also have a microphone. If its sound uses Advanced Audio Coding (AAC), the result is two independent streams:

H.264 video  ──→ bytes

AAC audio    ──→ bytes

Writing both byte streams into one file does not explain their relationship. A reader still has to determine:

which bytes are video

which bytes are audio

when each video frame should appear

which audio samples should play with it

where each stream begins and ends

H.264 cannot answer these questions because it knows only about video. An audio codec knows only about audio. Compression gives us smaller streams, but it does not turn them into one timed program.

The container adds order

A container supplies that missing structure. It groups encoded streams, identifies them, and records when their contents should play.

                ┌─ H.264 video
                │
Container  ─────┼─ AAC audio
                │
                └─ timing + stream information

The distinction can be reduced to two lines:

Codec:
compressed bytes → picture or sound

Container:
multiple streams → one timed program

H.264 is not tied to a particular container. The same video can appear inside MP4, Matroska, or MPEG-TS. In the other direction, MPEG-TS can contain H.264, H.265, MPEG-2 video, audio, subtitles, and private data.

Codec:     represent video with fewer bytes
Container: identify those bytes and place them in time

MPEG-TS answers the container problem by dividing the program into fixed-size packets. Every transport packet is exactly 188 bytes. The rest of the format follows from that peculiar number.

Why MPEG-TS looks strange

An MPEG-TS stream looks disordered when you first inspect it. Video, audio, timing, and program information are mixed into a sequence of 188-byte packets. Some of that information appears again and again.

The design follows from how the stream is used. A receiver may begin reading after transmission has already started:

… packet packet packet packet packet packet …
                       ↑
                    join here

Everything before that point is gone, yet the receiver must still find packet boundaries, discover the available streams, recover their timing, and notice missing data. MPEG-TS keeps enough structure inside the running stream to rebuild that context. Fixed packets, identifiers, repeated tables, timestamps, counters, and recovery points all follow from this requirement.

The 188-byte packet

The smallest unit is one 188-byte transport packet. Its first four bytes form the header. The remaining bytes contain payload, an adaptation field, or both. Here, payload means the table or media data inside the packet.

byte 0                                           byte 187
┌───────────────┬───────────────────────────────────────┐
│ 4-byte header │ adaptation field, payload, or both    │
└───────────────┴───────────────────────────────────────┘
                       total = 188 bytes

Every packet begins with the sync byte 0x47. If a reader loses alignment, it can search for that value and check for another exactly 188 bytes later. Repeated matches reveal the packet grid again.

0x47 ┄┄┄┄┄ 188 bytes ┄┄┄┄┄ 0x47 ┄┄┄┄┄ 188 bytes ┄┄┄┄┄ 0x47
  ↑                            ↑                            ↑
packet                       packet                       packet

The fixed size solves framing because it reveals where each packet begins and ends. The header solves the next problem: deciding which logical stream owns the packet.

PIDs separate the streams

Knowing a packet’s boundary does not reveal its contents. Video, audio, and program information still appear in the same sequence:

packet | packet | packet | packet | packet | packet
video    audio    video    table    audio    video

A 13-bit Packet Identifier (PID) in the four-byte header separates them.

188-byte transport packet

┌─────────────────────┬────────────────────────────────┐
│ 4-byte header       │ payload / adaptation field     │
│                     │                                │
│ contains PID        │                                │
└─────────────────────┴────────────────────────────────┘

Packets with the same PID belong to the same logical stream. One transport stream might assign its PIDs like this:

PID 0x0100  → video
PID 0x0101  → audio
PID 0x0000  → program information

Those packets remain interleaved on disk or on the wire:

[0000] [0100] [0101] [0100] [0100] [0101] [0100]

Reading the PID turns that mixed sequence back into separate streams:

PID 0x0100  → [video] [video] [video] [video]

PID 0x0101  → [audio] [audio]

PID 0x0000  → [program information]

PIDs let several logical streams share one packet sequence without losing their identities. A few values have fixed meanings. PID 0x0000 always contains the Program Association Table (PAT), while PID 0x1FFF is reserved for null packets. The stream assigns most other values.

A PID groups packets, but its number does not describe them. Nothing inside 0x0100 says:

0x0100 means H.264 video

MPEG-TS therefore needs a directory that assigns meaning to those numbers.

PAT and PMT give PIDs meaning

Grouping packets by PID is only the first half of discovery. Suppose a reader sees this sequence:

PID 0x0100
PID 0x0101
PID 0x0100
PID 0x0100
PID 0x0101

It knows which packets belong together, but not whether 0x0100 contains video, audio, or something else.

Two tables inside the stream supply the answer: the Program Association Table (PAT) and Program Map Table (PMT). PAT always uses PID 0x0000, so the reader has a fixed place to begin. PAT points to the PMT rather than directly to the video.

PID 0x0000

PAT
 │
 └── program 1 → PMT is on PID 0x1000

The reader follows that entry to PID 0x1000, where the PMT describes the streams in the program:

PMT

H.264 video → PID 0x0100
AAC audio   → PID 0x0101

Discovering the video therefore takes two lookups:

PAT
 ↓
find PMT
 ↓
PMT
 ↓
find video PID
 ↓
PID 0x0100

The two levels exist because one transport stream can contain several programs, such as several television channels in one broadcast. PAT lists those programs and the PID of each PMT. Each PMT then lists the video, audio, and other tracks in its program.

For a stream with one video track, the model is smaller:

PID groups packets.

PAT tells us where to find the description.

PMT tells us what each PID contains.

The reader now knows that PID 0x0100 contains H.264 video. It still has to rebuild pictures that are much larger than one transport packet.

A picture spans many packets

To the decoder, one encoded H.264 picture can look like a single block of bytes:

┌──────────────────────────────────────────────┐
│              encoded picture                 │
└──────────────────────────────────────────────┘

MPEG-TS cannot fit that block into one packet. After the four-byte header, at most 184 bytes remain for payload, so the encoded data must be split:

encoded video

┌──────────────────────────────────────────────┐
│                                              │
└──────────────────────────────────────────────┘

                    ↓ split

┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
│ TS packet │ │ TS packet │ │ TS packet │ │ TS packet │
│ PID 0100  │ │ PID 0100  │ │ PID 0100  │ │ PID 0100  │
└───────────┘ └───────────┘ └───────────┘ └───────────┘

The shared PID tells the reader that these packets belong to the video stream. It does not mark where one larger media unit ends and the next begins.

For that boundary, MPEG-TS wraps encoded video in a Packetized Elementary Stream (PES) packet. A PES header sits in front of the video bytes:

PES packet

┌──────────────┬───────────────────────────────┐
│ PES header   │ encoded video                 │
└──────────────┴───────────────────────────────┘

MPEG-TS then divides the complete PES packet across transport packets:

one PES packet

┌──────────────────────────────────────────────┐
│ header + encoded video                       │
└──────────────────────────────────────────────┘

                    ↓

┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ TS packet 1  │ │ TS packet 2  │ │ TS packet 3  │
│ PID 0x0100   │ │ PID 0x0100   │ │ PID 0x0100   │
│ start = 1    │ │ start = 0    │ │ start = 0    │
└──────────────┘ └──────────────┘ └──────────────┘

The first transport packet sets the payload-unit start indicator in its header. Packets that follow on the same PID continue that PES payload until another start indicator appears. The reader can therefore rebuild the larger unit in order:

find video PID
      ↓
see start of PES
      ↓
collect payload
      ↓
collect payload
      ↓
collect payload
      ↓
next PES starts
      ↓
previous PES is complete

A writer performs the reverse operation. It creates a PES packet, divides it into fragments, and writes those fragments into transport packets with the same video PID. Reassembling the bytes restores the encoded picture, but not its place in time.

PES adds time

A sequence of decoded pictures still needs a schedule. Video is not merely:

picture
picture
picture
picture

It is a sequence of pictures placed on a timeline:

picture at time 0
picture at time 33 ms
picture at time 66 ms
picture at time 100 ms

The PES header can contain a Presentation Time Stamp (PTS) and a Decoding Time Stamp (DTS):

┌─────────────────────────────────────────┐
│ PES header                              │
│                                         │
│ PTS → picture presentation time         │
│ DTS → picture decoding time             │
├─────────────────────────────────────────┤
│ encoded H.264 or H.265 video            │
└─────────────────────────────────────────┘

PTS tells the player when the picture should appear. DTS tells the decoder when to process it. Both values are needed because decoding order can differ from presentation order.

Consider this presentation order:

presentation order:

I0 → B1 → B2 → P3

Pictures B1 and B2 depend on information from P3, so the decoder needs P3 before either of them. The decoding order becomes:

I0 → P3 → B1 → B2

The same picture can therefore occupy two positions:

PTS → where it belongs on screen
DTS → where it belongs in the decoder

When decoding and presentation order match, PTS and DTS can match as well. Once the reader has these timestamps, only the codec bytes remain.

Inside the PES packet

Removing the PES header reveals the encoded H.264 or H.265 stream:

transport packets
       ↓
join payloads
       ↓
PES packet
       ↓
remove PES header
       ↓
H.264 / H.265 encoded video

As described earlier, these codecs divide their output into NAL units. Several NAL units can describe one coded picture:

NAL unit
NAL unit
NAL unit
    ↓
one coded picture

That group is the access unit passed to the decoder. The path from container bytes to a picture now looks like this:

188-byte transport packets
        ↓
read PID
        ↓
PAT
        ↓
PMT
        ↓
find video PID
        ↓
collect its packets
        ↓
rebuild PES
        ↓
read PTS / DTS
        ↓
remove PES header
        ↓
H.264 / H.265 access unit
        ↓
decoder
        ↓
picture

Each part has one job:

PID         → groups packets from one logical stream

PAT / PMT   → identify the contents of each PID

PES         → joins fragments into a larger media unit

PTS / DTS   → schedule decoding and presentation

Access unit → supplies codec bytes to the decoder

The remaining fields keep the receiver on the sender’s clock, detect missing packets, fill unused space, and restore context after a late join. They support the media path without changing its order.

The adaptation field

A transport packet sometimes needs information that does not belong to the encoded video. Clock samples, random-access flags, discontinuity state, and padding all describe the packet rather than its media. MPEG-TS places them in an adaptation field between the header and payload.

188-byte transport packet

┌───────────────┬──────────────────┬─────────────────┐
│ 4-byte header │ adaptation field │ payload         │
└───────────────┴──────────────────┴─────────────────┘

Its most visible use is stuffing. A transport packet must remain 188 bytes even when its payload does not fill the available space. If the final PES fragment needs only 80 bytes, the packet cannot shrink to 84 bytes. The adaptation field occupies the remainder with stuffing bytes.

┌────────┬───────────────┬─────────────┐
│ header │ actual data  │ adaptation + stuffing │
└────────┴──────────────┴───────────────────────┘
                  188 bytes

The same field can contain a Program Clock Reference, random-access indicator, or discontinuity flag. These values tell the receiver how to handle the packet. They do not describe the codec payload.

The program clock

PTS and DTS place pictures on a timeline, but the receiver also needs to reproduce the pace of that timeline. Program Clock Reference (PCR) samples in adaptation fields keep its local playback clock aligned with the sender.

PCR → where the program clock is now

DTS → when this picture should be decoded

PTS → when this picture should appear

Packet arrival time cannot serve as playback time. A file can be read faster than real time, while a network may deliver several packets in a burst. PTS and DTS place media on the timeline. PCR tells the receiver how that timeline advances.

The PMT identifies the PID that contains PCR. Its exact representation combines a 90 kHz base with a higher-resolution extension. The useful distinction is that PCR advances the clock, while PTS and DTS place pictures on it.

Detecting missing packets

A PES packet divided across several transport packets becomes incomplete when any fragment disappears. MPEG-TS cannot recover the bytes, but a four-bit continuity counter in every transport header makes the gap visible.

Payload-carrying packets on one PID increment their counter from 0 to 15, then wrap to 0. Each PID has its own sequence, so interleaved audio packets do not affect the video counter:

video PID 0x0100:  7 → 8 → 9  → 10
audio PID 0x0101:  3 → 4 → 5 → 6

video PID 0x0100:  7 → 8 → 10
                                ↑
                         packet 9 is missing

The jump from 8 to 10 reveals the missing packet. The counter cannot repair it. Pion drops the incomplete media unit after this discontinuity rather than joining bytes from opposite sides of the gap.

Tables can span packets

PAT and PMT live inside the same 188-byte packet system as media. MPEG-TS encodes them as Program Specific Information (PSI) sections so a table can extend beyond one packet.

transport packet 1         transport packet 2
┌──────────────────┐      ┌──────────────────┐
│ start of section │      │ rest of section  │
└──────────────────┘      └──────────────────┘
          └──────── one PSI section ────────┘

Each PSI section declares its length. A pointer field locates the next section boundary, and a cyclic redundancy check (CRC) lets the reader reject a damaged table. Pion usually writes PAT and PMT in one packet, but its reader also handles sections that span several.

Joining halfway through

A file reader normally begins at byte zero. A live receiver can join after the stream has started and first encounter PID 0x0100 without knowing what that PID means:

… video · audio · video · PAT · PMT · video · audio …
                              ↑
                           join here

MPEG-TS repeats PAT and PMT throughout the stream. The receiver waits for the next copies, follows PAT to the PMT, and learns the media PIDs:

video · audio · video · PAT · PMT · video · audio
            ↑             │     │
         join here         │     └─ learn the media PIDs
                           └─ find the PMT

The same pattern appears at every layer. A sync byte restores packet alignment. PAT and PMT restore the directory. A new PES start restores a media boundary, and a random-access picture gives the decoder a place to resume.

MPEG-TS neither retransmits missing packets nor guarantees delivery. It exposes damage and repeats enough structure for a receiver to recover its understanding without returning to the beginning.

From packet to picture

The full read path now follows directly from the pieces above. The reader finds packet boundaries, reads each PID, uses PAT and PMT to discover the video PID, rebuilds PES packets, reads their timestamps, and passes the encoded access unit to the decoder:

MPEG-TS file or live stream
        ↓
188-byte transport packet
        ↓
read PID
        ↓
PAT → PMT → video PID
        ↓
collect video packets
        ↓
rebuild PES
        ↓
read PTS / DTS
        ↓
extract H.264 / H.265 access unit
        ↓
decode
        ↓
picture

Each layer solves one problem. The packet supplies a fixed frame. The PID groups related packets. PAT and PMT explain those groups. PES restores larger timed units, and the codec turns those units into pictures.

The supporting machinery protects that path. Adaptation fields hold packet-level information, PCR maintains the program clock, continuity counters expose loss, PSI lets tables span packets, and repeated tables restore context after a late join.

MPEG-TS on a network

The word “transport” is misleading. MPEG-TS is not a network transport protocol like Transmission Control Protocol (TCP) or User Datagram Protocol (UDP). It is a media systems layer that can be written to a file or carried by Real-time Transport Protocol (RTP) over the Internet Protocol (IP).

H.264 or H.265 access unit
        ↓
PES packet
        ↓
188-byte MPEG-TS packets
        ↓
RTP payload
        ↓
UDP datagram
        ↓
IP packet
        ↓
Ethernet, Wi-Fi, or another link

RFC 2250 requires an RTP payload containing MPEG-TS to hold a whole number of 188-byte packets. RTP adds a sequence number and a 90 kHz timestamp for network delivery and jitter handling. These fields do not replace the PIDs, continuity counters, PCR, PTS, or DTS inside MPEG-TS.

Why seven packets fit

Seven transport packets occupy 1,316 bytes. Add a 12-byte RTP header, an 8-byte UDP header, and a 20-byte IPv4 header, and the Internet Protocol (IP) packet reaches 1,356 bytes. It remains below a 1,500-byte Ethernet maximum transmission unit. Eight transport packets already occupy 1,504 bytes before any of those headers.

7 × 188 = 1,316 bytes of MPEG-TS
+ 12 bytes RTP
+  8 bytes UDP
+ 20 bytes IPv4
= 1,356 bytes

Seven is a practical packing choice, not a rule of MPEG-TS. RFC 2250 requires a whole number of transport packets. The path, outer protocols, and maximum transmission unit determine the useful count.

What packet loss looks like

If a UDP datagram carrying seven transport packets disappears, RTP shows a sequence-number gap. Each affected PID may also show a continuity-counter gap inside MPEG-TS. The demuxer can then discard a partial PES packet or access unit instead of sending damaged bytes to the decoder.

Recovery happens at several boundaries. The next sync byte restores packet framing. The next payload-unit start restores a PES boundary. The next Instantaneous Decoder Refresh (IDR) or Intra Random Access Point (IRAP) picture restores codec random access. Repeated PAT and PMT restore the program map.

None of these mechanisms replaces another. A sync byte cannot repair an incomplete picture. A random-access picture cannot identify its PID without a program map. A continuity counter exposes a gap but cannot reconstruct the missing bytes.

The same stream on disk and wire

MPEG-TS treats a file and a live connection as two destinations for the same packet sequence. A recorder can write each packet as it arrives. It does not need to seek backward and finalize a central index before the earlier bytes become meaningful.

The cost is packet overhead and limited native indexing. A random seek may land between table repetitions or inside a predictive video sequence. The player must scan forward for PAT and PMT, identify the video PID, and find a random-access picture. Applications that need precise arbitrary seeking can build an external index or use a container designed around that job.

Null packets reveal the format’s broadcast origins. PID 0x1FFF fills unused capacity when a channel must maintain a constant transport bitrate. File writers can omit these packets, but every demuxer must recognize and ignore them.

Building from the bottom up

The implementation order in the Pion pull request became the best learning order because each layer depends on the one before it:

1. Packet constants and headers: establish 188-byte framing, sync, PID, flags, and continuity state.
2. Adaptation fields: add PCR, random-access signaling, discontinuity state, and stuffing.
3. CRC and PSI sections: build the envelope shared by program tables.
4. PAT and PMT: make programs and elementary streams discoverable.
5. PES headers: attach stream identity, PTS, and DTS to codec data.
6. H.264 and H.265 helpers: find Annex-B NAL units, Access Unit Delimiter (AUD) boundaries, and random-access pictures.
7. Writer: repeat tables, create PES, split payload, add PCR, increment counters, and write packets.
8. Reader: discover tracks, route PIDs, detect loss, reassemble sections and PES, and unwrap timestamps.
9. Tests: verify known bytes, round trips, malformed input, fuzz seeds, and FFmpeg-generated fixtures.

Building only the writer would have hidden its assumptions. Round-trip tests made each boundary observable. If the reader could not recover the same codec, timestamps, random-access state, and access-unit bytes, the fault had to lie in one of those boundaries.

What MPEG-TS does

MPEG-TS takes already compressed video and audio and stores or sends them as one continuous sequence of 188-byte packets. It does not compress the media or guarantee network delivery. Its job is to preserve identity and timing as a writer records or transmits the packets.

sync byte     → find each 188-byte packet
PAT           → find the PMT
PMT           → find the video and audio PIDs
video PID     → collect packets from the video stream
PES           → rebuild the timed video unit
PTS and DTS   → decide when to decode and show it
access unit   → send the encoded picture to the decoder

PCR keeps playback moving at the sender’s pace, while the continuity counter reveals missing packets. Repeated PAT and PMT tables let a receiver join halfway through and still discover the program. If data disappears, the next packet boundary, PES start, and random-access picture give the receiver places to resume.

This was the idea I was missing before I built the format. MPEG-TS looks repetitive because it assumes the reader may start late or lose part of the stream. Once that constraint is clear, each field answers a concrete question: where a packet begins, what it contains, when its media should play, and whether anything went missing.