Binary Codec Library: Pattern Coverage

How the combinator library handles each binary parsing pattern, with Elixir comparisons showing where the library is simpler, comparable, or more complex.

Context

This report is a response to RFC #241: Elixir Style Binary Pattern Matching. The RFC observes that binary parsing in Pony today requires manual bitmasks and byte-level operations — tedious, error-prone, and hard to read compared to Elixir's <<>> pattern syntax. The examples in the RFC (IPv4 header parsing and a badge-reader protocol) make the pain concrete.

Rather than adding new language syntax, this report explores a pure library approach: a combinator library modeled on scodec (Scala) and Kennedy's pickler combinators. You define a Codec[A] value that handles both parsing (bytes → value) and building (value → bytes) from a single definition. Codecs compose — small codecs combine into larger ones, the same way functions compose.

We identified 13 binary parsing patterns from real-world protocols. For each pattern, this report shows:

The Pony code shown here is an API sketch, not a finished implementation. The goal is to evaluate whether this approach addresses the RFC's concerns well enough to proceed without language changes.

Elixir vs. Pony Combinator Library

Elixir's <<>> syntax is built into the language. The combinator library is pure Pony — no compiler changes. That difference shapes the tradeoffs.

Where Elixir is simpler

Elixir wins on the patterns that map directly to its pattern syntax: fixed-layout headers and literal matching. A UDP header is one line of pattern matching in Elixir; in Pony it's a class definition, a Seq4 combinator, and an Mapped with two lambdas. This is the base cost of a library approach — the codec is a value you construct, not syntax you write.

Where the Pony library is simpler

Elixir's pattern syntax can't express repetition, nesting, or "parse N of these" — you write recursive helper functions for each. The combinator library handles all three with single combinator calls: Many(codec), Count(U16BE, codec), or nesting codecs inside each other. Elixir also has no bidirectional story — you write decode and encode separately, and they must stay in sync manually. Every combinator in this library handles both directions from a single definition.

Pattern-by-pattern breakdown

PatternVerdictKey difference
Fixed-layout headersElixir simplerOne-line pattern vs. Seq + Mapped ceremony
Sub-byte bit fieldsComparableBoth handle sub-byte naturally; Elixir terser, Pony typed
Length-prefixed dataElixir simplerInline dependent size vs. VariableSize/FlatZip combinator
Discriminated messagesElixir simplerFunction clause matching vs. class defs + Mapped + widen + builder
Multi-message parsingPony simplerMany(codec) vs. manual recursive extract
TCP stream framingComparableSame structure; Pony's FrameDecoder is reusable
Conditional fieldsComparableBoth need branching; Pony is bidirectional
Literal matchingElixir simplerInline bytes vs. Literal combinator
Nested structuresPony simplerCodec composition vs. Elixir's no-nesting limitation
Repeated elementsPony simplerCount(n, codec) vs. manual recursive counting
ChecksumsComparableBoth need validation logic beyond pattern matching
Endianness detectionComparableBoth branch; Elixir can't parameterize endianness
Alignment / paddingComparableSame structure, different syntax

The short version: Elixir's syntax is hard to beat for simple, flat, one-shot parsing. The combinator library seems to earn its keep on composition — nesting, repetition, and multi-message — and on bidirectionality, where Elixir has no answer at all. The RFC's two examples (IPv4 header, badge commands) sit in Elixir's sweet spot; real-world protocols tend to need the patterns where the library appears stronger.

Why not add Elixir's syntax to Pony?

Elixir's binary pattern matching gets much of its power from function clause dispatch — multiple function heads with different patterns, tried top-to-bottom. The discriminated message example is three function clauses, each matching a different tag byte. This is method overloading on argument shape, and it's central to how Elixir expresses protocol parsing.

Pony doesn't have method overloading, and there's a reason for that. Method overloading interacts poorly with Pony's generics and iftype specialization — the combination introduces ambiguity in dispatch that creates real footguns. This was explored and discussed in the ponyc repo; the conclusion was that the interaction with other language features makes it undesirable.

Pony can express the same dispatch with a single method and a match statement. It works, but it lacks the structural elegance of Elixir's clause-per-variant style. Adding binary pattern syntax to Pony would give you the field extraction (<<version::4, ihl::4>>), but without method overloading to dispatch on, you'd still land in a match block. The result would likely be a hybrid that's neither as clean as Elixir's nor as composable as a combinator library.

This is part of why I think the library approach is a good fit for Pony: it works with the language as it is, builds on Pony's strengths (type safety, composition, reference capabilities), and doesn't require features that conflict with the rest of the type system.

01

Fixed-Layout Headers

IPv4 UDP Ethernet MPEG-TS

Multiple fields of known widths and endianness, laid out consecutively. The canonical binary parsing problem. Every protocol starts here.

Real-world example

A UDP header: source port (16-bit BE), destination port (16-bit BE), length (16-bit BE), checksum (16-bit BE). Four fields, 8 bytes total, all big-endian.

How it works

Fixed-arity Seq combinators sequence the field codecs. Mapped maps the resulting tuple to a named domain type.

class val UdpHeader
  let src_port: U16
  let dst_port: U16
  let length: U16
  let checksum: U16

  new val create(s: U16, d: U16, l: U16, c: U16) =>
    src_port = s; dst_port = d; length = l; checksum = c

let udp_codec: Codec[UdpHeader] =
  Mapped[(U16, U16, U16, U16), UdpHeader](
    Seq4(U16BE, U16BE, U16BE, U16BE),
    {(t) => UdpHeader(t._1, t._2, t._3, t._4)},
    {(h) => (h.src_port, h.dst_port, h.length, h.checksum)})
In Elixir
<<src_port::16-big, dst_port::16-big,
  length::16-big, checksum::16-big>> = data

Pony is more complex here. Elixir destructures in one pattern — four fields, one line. The Pony version needs a class definition, a Seq4 combinator, and an Mapped with two lambdas to map between the tuple and the domain type. This is the base cost of a library approach vs. language syntax: you're building codecs compositionally, which pays off in later patterns but is overhead on simple fixed layouts.

Larger headers (10+ fields) compose from smaller Seq groups. An IPv4 header with 12 fields becomes two Seq6 groups mapped via Mapped, then combined with Seq2.

02

Sub-Byte Bit Fields

IP version/IHL DNS flags TCP flags MPEG-TS PID

Fields narrower than a byte: 4-bit nibbles, 1-bit flags, 3-bit opcodes, 13-bit fragment offsets. The IPv4 header alone has 4-bit, 6-bit, 2-bit, 13-bit, and 3-bit fields.

Real-world example

The first byte of an IPv4 header packs two 4-bit fields: version (always 4) and IHL (header length in 32-bit words). These are not two U8 values — together they make one byte.

How it works

The library is bit-native. BitCursor tracks position at bit granularity, so sub-byte fields compose identically to byte-aligned ones. BitsU8(4) reads 4 bits and returns a U8. BoolBit reads one bit as a Bool.

// First byte of IPv4: version (4 bits) + IHL (4 bits)
let first_byte = Seq2(BitsU8(4), BitsU8(4))
// Codec[(U8, U8)] — two 4-bit values, together consuming one byte

// DNS flags: 16 bits broken into 1+4+1+1+1+1+3+4 = 16
let dns_flags = Seq4(
  BoolBit,                // QR: query or response
  BitsU8(4),              // opcode
  Seq4(BoolBit, BoolBit, BoolBit, BoolBit), // AA, TC, RD, RA
  Seq2(BitsU8(3), BitsU8(4)))              // Z, RCODE
In Elixir
<<version::4, ihl::4>> = <<0x45>>
# version => 4, ihl => 5

# DNS flags: 1+4+1+1+1+1+3+4 = 16 bits
<<qr::1, opcode::4, aa::1, tc::1, rd::1,
  ra::1, z::3, rcode::4>> = flags_bytes

Roughly comparable. Both handle sub-byte fields naturally. Elixir's syntax is terser — ::4 vs. BitsU8(4) — but the structure is the same: list the fields with their bit widths. The Pony version carries type information (BitsU8 vs. BitsU16) that Elixir doesn't — all Elixir integers are untyped at runtime.

Bit-level codecs and byte-level codecs mix freely in a Seq. No mode switching is needed — a BitsU8(4) followed by a U16BE reads 4 bits then 16 bits. Byte alignment is not required between fields (though Bytes and Literal do require alignment at their own boundary).

03

Length-Prefixed Variable Data

TLV HTTP/2 frames MessagePack MQTT

A length field determines the size of the payload that follows. The core of TLV formats, HTTP/2 frames, MessagePack strings, and virtually every application protocol. The length field is structural — it's not part of the domain model.

Real-world example

An HTTP/2 frame: 24-bit length, 8-bit type, 8-bit flags, 31-bit stream ID (1 reserved bit + 31 data bits), then length bytes of payload.

How it works

VariableSize encapsulates the pattern. On decode: read the size field, then decode the body from exactly that many bytes. On encode: encode the body, measure it, write the size, then the body. The length field never appears in the decoded type.

// Simple: U16 length prefix + byte payload
let message = VariableSize[Array[U8] val](U16BE, BytesRest)
// Codec[Array[U8] val]
// Decode [0x00, 0x05, 0x48, 0x65, 0x6C, 0x6C, 0x6F]
//        ↑ length=5   ↑ "Hello"
// → Array[U8] val containing "Hello"
// Encode "Hi" → [0x00, 0x02, 0x48, 0x69]

// When the length field needs a non-standard width or encoding:
let http2_payload = VariableSize[Array[U8] val](
  BitsU32(24),   // 24-bit length
  BytesRest)

For cases where the length field itself is part of the domain model (the caller needs both the length and the data), FlatZip exposes it:

let with_length = FlatZip[U16, Array[U8] val](
  U16BE,
  {(len) => Bytes(len.usize())})
// Codec[(U16, Array[U8] val)] — length stays in the result
In Elixir
<<len::16-big, payload::binary-size(len)>> = data
# len => 5, payload => "Hello"

Pony is more complex here. Elixir handles dependent lengths in a single pattern — a later field's size references an earlier field's value. The Pony version needs either VariableSize (when the length isn't part of the domain model) or FlatZip (when it is). Straightforward once you know the combinators, but more machinery than Elixir's syntax. However, the Pony version is bidirectional — VariableSize automatically computes and writes the length on encode, while Elixir requires a separate construction expression.

04

Discriminated Message Types

MessagePack MQTT badge reader RPC

A tag value selects which variant to decode. The tag might be a byte, a string literal, or a bit pattern. Each variant has its own body layout.

Real-world example

Sean's badge reader: a 10-byte badge ID followed by a 2-byte ASCII command. "UD" = up-key-down, "DD" = down-key-down, "LD" = left-key-down. The command literal acts as the discriminator.

How it works

A single Discriminated class handles any number of variants. The user defines the union type externally and registers each variant with its tag, its codec, and a narrowing function for the encode direction.

class val UDMessage
  let badge_id: Array[U8] val
class val DDMessage
  let badge_id: Array[U8] val
class val LDMessage
  let badge_id: Array[U8] val

type BadgeMessage is (UDMessage | DDMessage | LDMessage)

// Each variant: 10-byte badge ID + 2-byte literal command
let ud = Mapped[Array[U8] val, UDMessage](
  ThenConst(Bytes(10), Literal([as U8: 'U'; 'D'])),
  {(b) => UDMessage(b)}, {(m) => m.badge_id})

let dd = Mapped[Array[U8] val, DDMessage](
  ThenConst(Bytes(10), Literal([as U8: 'D'; 'D'])),
  {(b) => DDMessage(b)}, {(m) => m.badge_id})

let ld = Mapped[Array[U8] val, LDMessage](
  ThenConst(Bytes(10), Literal([as U8: 'L'; 'D'])),
  {(b) => LDMessage(b)}, {(m) => m.badge_id})

// Widen each to Codec[BadgeMessage], combine with Alt
let badge_codec: Codec[BadgeMessage] = Alt3(
  ud.widen[BadgeMessage]({(m) => m as UDMessage}),
  dd.widen[BadgeMessage]({(m) => m as DDMessage}),
  ld.widen[BadgeMessage]({(m) => m as LDMessage}))

For byte-tag dispatch (the more common case), a DiscriminatedBuilder registers each variant with its tag, codec, and narrowing function. The builder enforces consistency: no duplicate tags, and the tag/codec pairing can't get out of sync because they're registered together.

type ProtocolMsg is (Ping | Pong | DataMsg)

let proto = DiscriminatedBuilder[U8, ProtocolMsg](U8Codec)
  .>case_of[Ping](0x01, ping_codec,
    {(m) => m as Ping})
  .>case_of[Pong](0x02, pong_codec,
    {(m) => m as Pong})
  .>case_of[DataMsg](0x03, data_codec,
    {(m) => m as DataMsg})
  .build()
// Codec[ProtocolMsg]
// Decode: reads tag byte, dispatches to matching codec
// Encode: tests each case's narrowing fn, writes matching tag + body
In Elixir
def handle_packet(<<0x01, payload::binary>>), do: handle_ping(payload)
def handle_packet(<<0x02, len::16, data::binary-size(len)>>), do: handle_data(data)
def handle_packet(<<0x03, code::8>>), do: handle_error(code)

Pony is more complex here. Elixir dispatches via function clause matching — three lines, each a tag pattern + body parse. The Pony version needs class definitions for each variant, Mapped wrappers with lambdas, widen calls, and either Alt or a DiscriminatedBuilder. That's substantially more code. The upside: the Pony version is bidirectional (encode is derived from the same definition), and the builder enforces no duplicate tags. But the setup cost is real.

05

Multi-Message Parsing

batch decode log files packet captures

A buffer containing multiple messages. Decode all of them. This is independent of how the buffer was obtained — it could be a file read, a UDP datagram, or a pre-assembled byte array.

Real-world example

A pcap file body: a sequence of packet records, each with a header (timestamp, lengths) followed by packet data. Read the whole file into a buffer, decode every record.

How it works

Many applies a codec repeatedly until the input is exhausted. It stops on InsufficientData (not enough bytes for another complete element) and errors on MalformedData.

// Parse all messages from a buffer
let all_messages = Many[MyMessage](my_message_codec)
// Codec[Array[MyMessage] val]

let data: Array[U8] val = FileReader.read("capture.bin")
match all_messages.decode(data, 0)
| (let msgs: Array[MyMessage] val, let consumed: USize) =>
  for msg in msgs.values() do
    process(msg)
  end
| let m: MalformedData =>
  env.out.print("Bad data: " + m.string())
end

Many is bidirectional: on encode, it encodes each element in sequence.

In Elixir
# Recursive extraction — the standard Erlang pattern
defp extract(<<ts::32, len::16, payload::binary-size(len),
              rest::binary>>, acc) do
  extract(rest, [{ts, payload} | acc])
end
defp extract(partial, acc), do: {Enum.reverse(acc), partial}

Pony is simpler here. The Elixir recursive extract looks clean, but the catch-all clause (defp extract(partial, acc)) silently treats all match failures as "incomplete data." A malformed message also fails to match — so a corrupt record is indistinguishable from a truncated one. In a streaming context, this means a single corrupt message causes the buffer to grow forever, waiting for bytes that would "complete" something that was never a valid message. Each protocol writes its own extract function from scratch, and each one has this same silent-failure mode. Many(codec) handles the loop once, and the codec's error model (InsufficientData vs. MalformedData) propagates correctly — a corrupt record stops the parse with an error instead of silently accumulating. It also encodes, which the recursive function doesn't.

05b

TCP Stream Framing

TCP chunked delivery framing

Data arrives in chunks over TCP. A message might span chunks. Accumulate bytes, extract complete messages, carry the remainder. This is orthogonal to the codec itself — any codec can be wrapped for streaming.

How it works

FrameDecoder wraps any Codec[A] for use inside a TCPConnectionNotify. It holds a ref buffer, appends incoming iso chunks (consuming them), and runs the codec in a loop. The buffer is ref, which satisfies decode's box parameter since ref <: box.

class ref FrameDecoder[A: Any val]
  let _codec: Codec[A]
  var _buffer: Array[U8] ref
  var _offset: USize

  fun ref received(data: Array[U8] iso,
    notify: FrameNotify[A] ref)
  =>
    _buffer.append(consume data)
    var continue = true
    while continue do
      // _buffer is ref, decode takes box, ref <: box
      match _codec.decode(_buffer, _offset)
      | (let value: A, let consumed: USize) =>
        notify.frame_received(value)
        _offset = _offset + (consumed / 8)
      | InsufficientData => continue = false
      | let m: MalformedData =>
        notify.frame_error(m)
        continue = false
      end
    end
    _trim_consumed()

The error model matters here: InsufficientData means "wait for more bytes" (normal in streaming), while MalformedData means "this data is wrong" (a real error). A partial function (?) can't make that distinction.

In Elixir
# The classic Erlang buffer-accumulation loop
def loop(buffer, parent) do
  receive do
    {:tcp, _sock, chunk} ->
      data = buffer <> chunk
      rest = extract(data, parent)
      loop(rest, parent)
    {:tcp_closed, _sock} -> :ok
  end
end

Roughly comparable. Both require the same fundamental structure: accumulate chunks, extract complete messages in a loop, carry the remainder. Elixir writes the buffer management inline; Pony wraps it in FrameDecoder. The FrameDecoder is reusable across protocols — plug in any codec. In Elixir, each protocol writes its own extract loop. Elixir also offers :gen_tcp's packet: N option for simple length-prefixed framing, which has no Pony equivalent.

The buffer is ref, not iso. It never leaves the actor, so isolation isn't needed — ref is the right cap. Incoming iso chunks are consumed into it via append. decode takes box, and ref <: box, so no copy or capability conversion is needed. Integer extraction is by value. Bytes(n) copies its N bytes — O(field size), not O(buffer size).

06

Conditional / Optional Fields

MPEG-TS TLS DNS extensions

A field that is present or absent depending on a flag decoded earlier. The field's existence is data-dependent, not statically known.

Real-world example

MPEG-TS adaptation field: a PCR (program clock reference, 48 bits) is present only if the pcr_flag bit is set. Five other optional fields follow, each gated by its own flag bit.

How it works

FlatZip threads the flag value to a Conditional codec. The result type includes None for the absent case.

// Flag bit controls whether 48-bit PCR follows
let pcr_field = BoolBit.flat_zip[(U64 | None)](
  {(flag) =>
    Conditional[U64](flag, BitsU64(48))})
// Codec[(Bool, (U64 | None))]
// flag=true  → (true, 0x1234ABCD0000)
// flag=false → (false, None)

// Multiple conditional fields chained:
let adaptation = Seq2(
  BoolBit.flat_zip[(U64 | None)](
    {(f) => Conditional[U64](f, BitsU64(48))}),  // PCR
  BoolBit.flat_zip[(U64 | None)](
    {(f) => Conditional[U64](f, BitsU64(48))})   // OPCR
)
In Elixir
# Can't express in a single pattern — needs two clauses or imperative logic
case pcr_flag do
  1 -> <<pcr::48, rest::binary>> = remaining
       {pcr, rest}
  0 -> {nil, remaining}
end

Roughly comparable. Neither language handles this in one expression. Elixir needs a case or multiple function clauses; Pony uses FlatZip + Conditional. The Pony version is more mechanical but still bidirectional — on encode, the flag is set based on whether the value is None. Elixir requires separate encode logic.

07

Literal / Constant Matching

magic numbers sync bytes protocol IDs

Fixed byte sequences that identify a protocol, message type, or frame boundary. On decode, the bytes must match exactly or the parse fails. On encode, the bytes are emitted verbatim.

Real-world example

MPEG-TS sync byte (0x47) at the start of every 188-byte packet. PNG magic bytes (89 50 4E 47 0D 0A 1A 0A). The "UD"/"DD" strings in the badge protocol.

How it works

Literal matches fixed bytes without binding a value (its type is Codec[None]). ConstThen and ThenConst pair a literal with a data-carrying codec, keeping only the data.

// MPEG-TS: sync byte, then 3 bytes of header data
let ts_packet = ConstThen[U32](
  Literal([as U8: 0x47]),
  BitsU32(24))
// Decode: checks 0x47, reads 24 bits → Codec[U32]
// Encode: writes 0x47, then 24 bits of the U32 value

// PNG magic: just validate, bind nothing
let png_magic = Literal(
  [as U8: 0x89; 0x50; 0x4E; 0x47; 0x0D; 0x0A; 0x1A; 0x0A])
// Codec[None] — fails on mismatch, advances cursor on match

// Badge: data first, then the command literal
let ud_badge = ThenConst[Array[U8] val](
  Bytes(10),
  Literal([as U8: 'U'; 'D']))
// Codec[Array[U8] val] — 10 bytes of badge ID
In Elixir
# Literal bytes as guards in a pattern
<<0x47, header::binary-size(3), _rest::binary>> = data
# or string literals
<<"UD", badge_id::binary-size(10)>> = data

Elixir is more concise. Literal bytes are inline in the pattern — no separate combinator needed. The Pony Literal / ConstThen / ThenConst naming is more verbose, but both do the same work. The Pony version emits the literal bytes on encode automatically; Elixir's construction syntax is separate.

08

Nested Structures

Ethernet → IP → TCP TLV trees file formats

One structure contains another: an Ethernet frame wraps an IP packet which wraps a TCP segment which wraps application data. Each layer has its own header and its own codec.

How it works

Codecs compose directly. A field in an outer codec can itself be a codec for a complex inner type. VariableSize bounds the inner codec to the right number of bytes.

// Inner: a simple key-value TLV entry
class val TlvEntry
  let tag: U8
  let value: Array[U8] val

let tlv_entry = Mapped[(U8, Array[U8] val), TlvEntry](
  Seq2(U8Codec, VariableSize[Array[U8] val](U8Codec, BytesRest)),
  {(t) => TlvEntry(t._1, t._2)},
  {(e) => (e.tag, e.value)})

// Outer: a message containing nested TLV entries
class val TlvMessage
  let version: U8
  let entries: Array[TlvEntry] val

let tlv_message = Mapped[(U8, Array[TlvEntry] val), TlvMessage](
  Seq2(U8Codec, Many[TlvEntry](tlv_entry)),
  {(t) => TlvMessage(t._1, t._2)},
  {(m) => (m.version, m.entries)})

Each layer is an independent codec. The nesting comes from using one codec as a building block in another — no special nesting syntax or combinator needed.

In Elixir
# Can't nest binary patterns — each layer is a separate function
def parse_outer(<<version::8, rest::binary>>) do
  entries = parse_entries(rest, [])
  %TlvMessage{version: version, entries: entries}
end

defp parse_entries(<<tag::8, len::8,
                    value::binary-size(len),
                    rest::binary>>, acc) do
  parse_entries(rest, [%TlvEntry{tag: tag, value: value} | acc])
end
defp parse_entries(<<>>, acc), do: Enum.reverse(acc)

Pony is simpler here. Elixir can't nest <<>> patterns — each layer needs its own function and manual recursive iteration. The Pony version composes codecs directly: Many[TlvEntry](tlv_entry) inside a Seq2. The nesting is the composition — no separate functions, no manual recursion, and bidirectional.

09

Repeated Elements

DNS records TLV sequences array payloads

N instances of a sub-structure, either counted by a preceding field or repeated until the input is exhausted.

Real-world example

DNS response: the header says ancount = 3, meaning 3 answer resource records follow. Each record has its own variable-length layout.

How it works

Count reads a count field, then decodes that many instances. Many repeats until the input is consumed (or InsufficientData for a partial element).

// Counted: U16 count, then that many TlvEntry values
let counted_entries = Count[TlvEntry](U16BE, tlv_entry)
// Codec[Array[TlvEntry] val]
// Decode: reads U16 = 3, then decodes 3 TlvEntry instances
// Encode: writes 3 as U16, then encodes each entry

// Greedy: decode TlvEntry values until input exhausted
let all_entries = Many[TlvEntry](tlv_entry)
// Codec[Array[TlvEntry] val]
// Stops on InsufficientData (not enough for another entry)
// Errors on MalformedData (data present but invalid)

Count is bidirectional: on encode, it counts the array, writes the count field, then encodes each element. The count field value is never part of the domain type.

In Elixir
# "Parse N records" requires a recursive function
defp parse_n(data, 0, acc), do: {Enum.reverse(acc), data}
defp parse_n(<<tag::8, len::8,
              value::binary-size(len),
              rest::binary>>, n, acc) do
  parse_n(rest, n - 1, [%{tag: tag, value: value} | acc])
end

# Called after reading the count:
<<count::16-big, rest::binary>> = data
{entries, _remaining} = parse_n(rest, count, [])

Pony is simpler here. Elixir can't express "parse N of these" in a pattern — it needs a recursive helper that counts down. Count(U16BE, tlv_entry) does it in one line, and encodes the count automatically on the write side.

10

Checksums and Computed Fields

IP checksum CRC length fields

Fields whose values are derived from other fields during encoding (checksum = function of data, length = payload size) and validated during decoding.

How it works

This pattern uses Mapped with a validating forward function and a computing reverse function. The checksum field is decoded as a plain integer; the Mapped's forward function validates it against the data. The reverse function computes it from the domain value.

// A frame: U32 CRC followed by payload bytes
// Domain type holds just the payload; CRC is computed
class val CheckedFrame
  let payload: Array[U8] val

let checked_frame = Mapped[(U32, Array[U8] val), CheckedFrame](
  Seq2(U32BE, BytesRest),
  {(t) =>
    // Validate: compute CRC of payload, compare to field
    let expected = CRC32(t._2)
    if t._1 != expected then error end
    CheckedFrame(t._2)},
  {(f) =>
    // Compute: CRC is derived from payload on encode
    (CRC32(f.payload), f.payload)})
In Elixir
# Decode: extract CRC and payload, validate manually
<<crc::32-big, payload::binary>> = data
^crc = :erlang.crc32(payload)  # match fails if wrong

# Encode: compute CRC, build binary
crc = :erlang.crc32(payload)
<<crc::32-big, payload::binary>>

Roughly comparable. Both require validation logic beyond simple pattern matching. Elixir uses pinned variables (^crc) or guard clauses; Pony uses a partial Mapped forward function. The Pony version bundles the validation and computation into one codec definition; Elixir writes decode and encode as separate expressions that must stay consistent.

The forward function is partial (?) — a checksum mismatch produces a decode error. The Mapped combinator would need to handle partial forward functions, surfacing the error as MalformedData. This is an extension to the basic Mapped sketch.

11

Endianness Detection

pcap ELF TIFF

The file or stream declares its byte ordering in a header field. All subsequent multi-byte integers use that ordering. The decoder must read the header, determine endianness, then use the right codec for every field that follows.

How it works

This is a FlatZip: decode the endianness marker, use its value to select between BE and LE codec sets for the rest of the structure. The endianness choice flows through the dependency chain.

// pcap: magic number determines endianness
// 0xA1B2C3D4 = native (big-endian file)
// 0xD4C3B2A1 = swapped (little-endian file)

primitive PcapEndian
  fun codec(): Codec[(Bool, PcapHeader)] =>
    U32BE.flat_zip[PcapHeader](
      {(magic) =>
        let be = magic == 0xA1B2C3D4
        if be then
          pcap_header_be()  // all fields big-endian
        else
          pcap_header_le()  // all fields little-endian
        end})

  fun pcap_header_be(): Codec[PcapHeader] =>
    Mapped[(U16, U16, U32, U32), PcapHeader](
      Seq4(U16BE, U16BE, U32BE, U32BE), ...)

  fun pcap_header_le(): Codec[PcapHeader] =>
    Mapped[(U16, U16, U32, U32), PcapHeader](
      Seq4(U16LE, U16LE, U32LE, U32LE), ...)

The endianness choice is explicit at each level — no global state or implicit threading. Each codec function picks BE or LE for its fields. This is verbose for all-same-endianness formats, but a factory function (like pcap_header_be() vs pcap_header_le()) keeps the repetition in one place.

In Elixir
# Read magic number, branch on endianness
case data do
  <<0xA1B2C3D4::32, rest::binary>> ->
    parse_pcap_header(rest, :big)
  <<0xD4C3B2A1::32, rest::binary>> ->
    parse_pcap_header(rest, :little)
end

# But endianness can't be a variable in a pattern —
# you need two full sets of patterns or use :binary.decode_unsigned/2

Roughly comparable. Both need to branch on the endianness marker and use different decoders for each path. Elixir can't parameterize endianness in a pattern — ::big and ::little are compile-time modifiers, not runtime values — so you need two separate pattern sets or fall back to :binary.decode_unsigned/2. The Pony version has the same branching structure, but each branch is a reusable codec value.

12

Alignment and Padding

MPEG-TS IP options file formats

Fields that must start on byte or word boundaries. Padding bytes fill the gap between the end of one field and the required alignment of the next.

How it works

SkipBits consumes bits without binding a value. For alignment, the skip count is calculated from the current position — which requires a dependent codec.

// Skip 3 reserved bits between fields
let with_padding = Seq3(
  BitsU8(4),           // 4-bit field
  SkipBits(3),         // 3 reserved bits
  BoolBit)             // 1-bit flag — now byte-aligned
// Codec[(U8, None, Bool)] — None from SkipBits

// Fixed padding to word boundary
let padded = Seq2(
  Bytes(3),            // 3 bytes of data
  SkipBits(8))         // 1 byte padding to 4-byte alignment
In Elixir
# Skip reserved bits explicitly
<<field::4, _reserved::3, flag::1>> = byte

# Padding to alignment
<<data::binary-size(3), _pad::8, rest::binary>> = input

Roughly comparable. Both handle static padding by naming the skip width explicitly. Neither has built-in dynamic alignment (skip to the next N-bit boundary). Elixir uses underscore-prefixed variables to discard padding; Pony uses SkipBits. The same structure, different syntax.

Dynamic alignment (skip N bits to reach the next byte boundary, where N depends on how many bits have been consumed) would need access to the cursor's current bit offset inside a FlatZip. The current design doesn't expose cursor position to codec constructors. For now, only static skip counts are supported. Dynamic alignment is a possible future extension.

Appendix

Streaming: Elixir vs. Pony

The patterns above cover parsing a complete buffer. Streaming is the harder problem: data arrives in chunks over TCP, and a message can span multiple chunks. A frame protocol says "expect 50 bytes of payload" but you might have the 2-byte header and only 2 bytes of payload so far. The system must recognize "not enough data yet," hold onto what it has, and resume when more arrives.

The concrete problem

A length-prefixed frame protocol: 2-byte big-endian length, then that many bytes of payload. Three TCP chunks arrive:

Chunk 1:  [00 32 AA BB]         — header says 50 bytes, but only 2 payload bytes
Chunk 2:  [CC DD ... 48 more]   — now we have all 50
Chunk 3:  [00 03 EE FF]         — next header says 3 bytes, only 2 here

After chunk 1, the parser has the length field (50) but only 2 of the 50 payload bytes. It must:

After chunk 2, it has enough to deliver the first message. After chunk 3, it's partway through the second message and must wait again.

In Elixir

The simple path: Erlang's :gen_tcp has a packet: N option that handles this automatically. Set packet: 2 on the socket (2-byte length prefix) and the runtime buffers internally — each {:tcp, socket, data} message delivers exactly one complete payload with the length header stripped. This is hard to beat when both sides can agree on the framing format.

Custom protocols don't get this for free. The standard pattern:

def loop(buffer, parent) do
  receive do
    {:tcp, _sock, chunk} ->
      data = buffer <> chunk
      {messages, rest} = extract(data, [])
      Enum.each(messages, &process/1)
      loop(rest, parent)
    {:tcp_closed, _sock} -> :ok
  end
end

defp extract(<<len::16, payload::binary-size(len),
              rest::binary>>, acc) do
  extract(rest, [payload | acc])
end
defp extract(partial, acc), do: {Enum.reverse(acc), partial}

Walking through our scenario:

This works correctly for the incomplete-data case. But consider what happens if byte 3 of chunk 1 were corrupted — say the length field read as 0xFFFF (65535). The pattern would try to match 65535 bytes, fail, and fall through to the same catch-all. The system would now buffer 65KB of data waiting for a "message" that was never actually sent. The catch-all clause can't distinguish "haven't received enough yet" from "the data is wrong." Both are just a failed match.

In Pony

The combinator library separates framing from parsing. The codec handles the protocol. FrameDecoder handles the streaming:

// The protocol codec — same one used for one-shot parsing
let frame_codec = VariableSize[Array[U8] val](U16BE, BytesRest)

// Wrap it for streaming
let framer = FrameDecoder[Array[U8] val](frame_codec)

// Inside TCPConnectionNotify.received():
framer.received(consume data, my_notify)

Walking through the same scenario:

Now the corruption scenario: the length field reads as 65535. VariableSize reads the length via U16BE (succeeds — it's a valid U16). Then it tries to read 65535 bytes via BytesRest. Not enough data — InsufficientData.

This is the same outcome as Elixir: the system waits. The length field is a valid integer; there's no way to know 65535 is "wrong" without application-level knowledge (e.g., a maximum message size). Both systems need an upper bound check for this case. The difference shows up when the corruption is in the payload, not the length — if the inner codec encounters data that is structurally invalid (a tag byte that matches no known variant, a nested structure that doesn't parse), it returns MalformedData. FrameDecoder calls notify.frame_error() instead of silently buffering. Elixir's catch-all treats this the same as "not enough data."

Key differences

ElixirPony combinator library
Simple framingpacket: N on the socket — automatic, zero codeNo equivalent; always explicit FrameDecoder
Custom framingManual extract function, written per protocolReusable FrameDecoder wrapping any codec
Partial messageMatch fails, catch-all keeps bufferInsufficientData, FrameDecoder keeps buffer
Corrupt payloadMatch fails — indistinguishable from incompleteMalformedData — reported as an error
Corrupt lengthBuffers forever (same problem in both)Buffers forever (needs app-level max-size check in both)
Buffer managementBinary concatenation in process loopFrameDecoder owns a ref buffer, trims consumed bytes
Codec reuseSame patterns work one-shot and streamingSame codec works one-shot and in FrameDecoder

The honest summary: for simple length-prefixed protocols, Elixir's packet: N is unbeatable — zero code. For custom protocols, both approaches handle partial delivery correctly. The Pony library's potential advantage is distinguishing corrupt data from incomplete data, and making the framing logic reusable rather than hand-written per protocol.