How the combinator library handles each binary parsing pattern, with Elixir comparisons showing where the library is simpler, comparable, or more complex.
This report is a response to RFC #241: Elixir Style Binary Pattern Matching. The RFC observes that binary parsing in Pony today requires manual bitmasks and byte-level operations — tedious, error-prone, and hard to read compared to Elixir's <<>> pattern syntax. The examples in the RFC (IPv4 header parsing and a badge-reader protocol) make the pain concrete.
Rather than adding new language syntax, this report explores a pure library approach: a combinator library modeled on scodec (Scala) and Kennedy's pickler combinators. You define a Codec[A] value that handles both parsing (bytes → value) and building (value → bytes) from a single definition. Codecs compose — small codecs combine into larger ones, the same way functions compose.
We identified 13 binary parsing patterns from real-world protocols. For each pattern, this report shows:
The Pony code shown here is an API sketch, not a finished implementation. The goal is to evaluate whether this approach addresses the RFC's concerns well enough to proceed without language changes.
Elixir's <<>> syntax is built into the language. The combinator library is pure Pony — no compiler changes. That difference shapes the tradeoffs.
Elixir wins on the patterns that map directly to its pattern syntax: fixed-layout headers and literal matching. A UDP header is one line of pattern matching in Elixir; in Pony it's a class definition, a Seq4 combinator, and an Mapped with two lambdas. This is the base cost of a library approach — the codec is a value you construct, not syntax you write.
Elixir's pattern syntax can't express repetition, nesting, or "parse N of these" — you write recursive helper functions for each. The combinator library handles all three with single combinator calls: Many(codec), Count(U16BE, codec), or nesting codecs inside each other. Elixir also has no bidirectional story — you write decode and encode separately, and they must stay in sync manually. Every combinator in this library handles both directions from a single definition.
| Pattern | Verdict | Key difference |
|---|---|---|
| Fixed-layout headers | Elixir simpler | One-line pattern vs. Seq + Mapped ceremony |
| Sub-byte bit fields | Comparable | Both handle sub-byte naturally; Elixir terser, Pony typed |
| Length-prefixed data | Elixir simpler | Inline dependent size vs. VariableSize/FlatZip combinator |
| Discriminated messages | Elixir simpler | Function clause matching vs. class defs + Mapped + widen + builder |
| Multi-message parsing | Pony simpler | Many(codec) vs. manual recursive extract |
| TCP stream framing | Comparable | Same structure; Pony's FrameDecoder is reusable |
| Conditional fields | Comparable | Both need branching; Pony is bidirectional |
| Literal matching | Elixir simpler | Inline bytes vs. Literal combinator |
| Nested structures | Pony simpler | Codec composition vs. Elixir's no-nesting limitation |
| Repeated elements | Pony simpler | Count(n, codec) vs. manual recursive counting |
| Checksums | Comparable | Both need validation logic beyond pattern matching |
| Endianness detection | Comparable | Both branch; Elixir can't parameterize endianness |
| Alignment / padding | Comparable | Same structure, different syntax |
The short version: Elixir's syntax is hard to beat for simple, flat, one-shot parsing. The combinator library seems to earn its keep on composition — nesting, repetition, and multi-message — and on bidirectionality, where Elixir has no answer at all. The RFC's two examples (IPv4 header, badge commands) sit in Elixir's sweet spot; real-world protocols tend to need the patterns where the library appears stronger.
Elixir's binary pattern matching gets much of its power from function clause dispatch — multiple function heads with different patterns, tried top-to-bottom. The discriminated message example is three function clauses, each matching a different tag byte. This is method overloading on argument shape, and it's central to how Elixir expresses protocol parsing.
Pony doesn't have method overloading, and there's a reason for that. Method overloading interacts poorly with Pony's generics and iftype specialization — the combination introduces ambiguity in dispatch that creates real footguns. This was explored and discussed in the ponyc repo; the conclusion was that the interaction with other language features makes it undesirable.
Pony can express the same dispatch with a single method and a match statement. It works, but it lacks the structural elegance of Elixir's clause-per-variant style. Adding binary pattern syntax to Pony would give you the field extraction (<<version::4, ihl::4>>), but without method overloading to dispatch on, you'd still land in a match block. The result would likely be a hybrid that's neither as clean as Elixir's nor as composable as a combinator library.
This is part of why I think the library approach is a good fit for Pony: it works with the language as it is, builds on Pony's strengths (type safety, composition, reference capabilities), and doesn't require features that conflict with the rest of the type system.
Multiple fields of known widths and endianness, laid out consecutively. The canonical binary parsing problem. Every protocol starts here.
A UDP header: source port (16-bit BE), destination port (16-bit BE), length (16-bit BE), checksum (16-bit BE). Four fields, 8 bytes total, all big-endian.
Fixed-arity Seq combinators sequence the field codecs. Mapped maps the resulting tuple to a named domain type.
class val UdpHeader let src_port: U16 let dst_port: U16 let length: U16 let checksum: U16 new val create(s: U16, d: U16, l: U16, c: U16) => src_port = s; dst_port = d; length = l; checksum = c let udp_codec: Codec[UdpHeader] = Mapped[(U16, U16, U16, U16), UdpHeader]( Seq4(U16BE, U16BE, U16BE, U16BE), {(t) => UdpHeader(t._1, t._2, t._3, t._4)}, {(h) => (h.src_port, h.dst_port, h.length, h.checksum)})
<<src_port::16-big, dst_port::16-big, length::16-big, checksum::16-big>> = data
Pony is more complex here. Elixir destructures in one pattern — four fields, one line. The Pony version needs a class definition, a Seq4 combinator, and an Mapped with two lambdas to map between the tuple and the domain type. This is the base cost of a library approach vs. language syntax: you're building codecs compositionally, which pays off in later patterns but is overhead on simple fixed layouts.
Larger headers (10+ fields) compose from smaller Seq groups. An IPv4 header with 12 fields becomes two Seq6 groups mapped via Mapped, then combined with Seq2.
Fields narrower than a byte: 4-bit nibbles, 1-bit flags, 3-bit opcodes, 13-bit fragment offsets. The IPv4 header alone has 4-bit, 6-bit, 2-bit, 13-bit, and 3-bit fields.
The first byte of an IPv4 header packs two 4-bit fields: version (always 4) and IHL (header length in 32-bit words). These are not two U8 values — together they make one byte.
The library is bit-native. BitCursor tracks position at bit granularity, so sub-byte fields compose identically to byte-aligned ones. BitsU8(4) reads 4 bits and returns a U8. BoolBit reads one bit as a Bool.
// First byte of IPv4: version (4 bits) + IHL (4 bits) let first_byte = Seq2(BitsU8(4), BitsU8(4)) // Codec[(U8, U8)] — two 4-bit values, together consuming one byte // DNS flags: 16 bits broken into 1+4+1+1+1+1+3+4 = 16 let dns_flags = Seq4( BoolBit, // QR: query or response BitsU8(4), // opcode Seq4(BoolBit, BoolBit, BoolBit, BoolBit), // AA, TC, RD, RA Seq2(BitsU8(3), BitsU8(4))) // Z, RCODE
<<version::4, ihl::4>> = <<0x45>> # version => 4, ihl => 5 # DNS flags: 1+4+1+1+1+1+3+4 = 16 bits <<qr::1, opcode::4, aa::1, tc::1, rd::1, ra::1, z::3, rcode::4>> = flags_bytes
Roughly comparable. Both handle sub-byte fields naturally. Elixir's syntax is terser — ::4 vs. BitsU8(4) — but the structure is the same: list the fields with their bit widths. The Pony version carries type information (BitsU8 vs. BitsU16) that Elixir doesn't — all Elixir integers are untyped at runtime.
Bit-level codecs and byte-level codecs mix freely in a Seq. No mode switching is needed — a BitsU8(4) followed by a U16BE reads 4 bits then 16 bits. Byte alignment is not required between fields (though Bytes and Literal do require alignment at their own boundary).
A length field determines the size of the payload that follows. The core of TLV formats, HTTP/2 frames, MessagePack strings, and virtually every application protocol. The length field is structural — it's not part of the domain model.
An HTTP/2 frame: 24-bit length, 8-bit type, 8-bit flags, 31-bit stream ID (1 reserved bit + 31 data bits), then length bytes of payload.
VariableSize encapsulates the pattern. On decode: read the size field, then decode the body from exactly that many bytes. On encode: encode the body, measure it, write the size, then the body. The length field never appears in the decoded type.
// Simple: U16 length prefix + byte payload let message = VariableSize[Array[U8] val](U16BE, BytesRest) // Codec[Array[U8] val] // Decode [0x00, 0x05, 0x48, 0x65, 0x6C, 0x6C, 0x6F] // ↑ length=5 ↑ "Hello" // → Array[U8] val containing "Hello" // Encode "Hi" → [0x00, 0x02, 0x48, 0x69] // When the length field needs a non-standard width or encoding: let http2_payload = VariableSize[Array[U8] val]( BitsU32(24), // 24-bit length BytesRest)
For cases where the length field itself is part of the domain model (the caller needs both the length and the data), FlatZip exposes it:
let with_length = FlatZip[U16, Array[U8] val]( U16BE, {(len) => Bytes(len.usize())}) // Codec[(U16, Array[U8] val)] — length stays in the result
<<len::16-big, payload::binary-size(len)>> = data
# len => 5, payload => "Hello"
Pony is more complex here. Elixir handles dependent lengths in a single pattern — a later field's size references an earlier field's value. The Pony version needs either VariableSize (when the length isn't part of the domain model) or FlatZip (when it is). Straightforward once you know the combinators, but more machinery than Elixir's syntax. However, the Pony version is bidirectional — VariableSize automatically computes and writes the length on encode, while Elixir requires a separate construction expression.
A tag value selects which variant to decode. The tag might be a byte, a string literal, or a bit pattern. Each variant has its own body layout.
Sean's badge reader: a 10-byte badge ID followed by a 2-byte ASCII command. "UD" = up-key-down, "DD" = down-key-down, "LD" = left-key-down. The command literal acts as the discriminator.
A single Discriminated class handles any number of variants. The user defines the union type externally and registers each variant with its tag, its codec, and a narrowing function for the encode direction.
class val UDMessage let badge_id: Array[U8] val class val DDMessage let badge_id: Array[U8] val class val LDMessage let badge_id: Array[U8] val type BadgeMessage is (UDMessage | DDMessage | LDMessage) // Each variant: 10-byte badge ID + 2-byte literal command let ud = Mapped[Array[U8] val, UDMessage]( ThenConst(Bytes(10), Literal([as U8: 'U'; 'D'])), {(b) => UDMessage(b)}, {(m) => m.badge_id}) let dd = Mapped[Array[U8] val, DDMessage]( ThenConst(Bytes(10), Literal([as U8: 'D'; 'D'])), {(b) => DDMessage(b)}, {(m) => m.badge_id}) let ld = Mapped[Array[U8] val, LDMessage]( ThenConst(Bytes(10), Literal([as U8: 'L'; 'D'])), {(b) => LDMessage(b)}, {(m) => m.badge_id}) // Widen each to Codec[BadgeMessage], combine with Alt let badge_codec: Codec[BadgeMessage] = Alt3( ud.widen[BadgeMessage]({(m) => m as UDMessage}), dd.widen[BadgeMessage]({(m) => m as DDMessage}), ld.widen[BadgeMessage]({(m) => m as LDMessage}))
For byte-tag dispatch (the more common case), a DiscriminatedBuilder registers each variant with its tag, codec, and narrowing function. The builder enforces consistency: no duplicate tags, and the tag/codec pairing can't get out of sync because they're registered together.
type ProtocolMsg is (Ping | Pong | DataMsg) let proto = DiscriminatedBuilder[U8, ProtocolMsg](U8Codec) .>case_of[Ping](0x01, ping_codec, {(m) => m as Ping}) .>case_of[Pong](0x02, pong_codec, {(m) => m as Pong}) .>case_of[DataMsg](0x03, data_codec, {(m) => m as DataMsg}) .build() // Codec[ProtocolMsg] // Decode: reads tag byte, dispatches to matching codec // Encode: tests each case's narrowing fn, writes matching tag + body
def handle_packet(<<0x01, payload::binary>>), do: handle_ping(payload) def handle_packet(<<0x02, len::16, data::binary-size(len)>>), do: handle_data(data) def handle_packet(<<0x03, code::8>>), do: handle_error(code)
Pony is more complex here. Elixir dispatches via function clause matching — three lines, each a tag pattern + body parse. The Pony version needs class definitions for each variant, Mapped wrappers with lambdas, widen calls, and either Alt or a DiscriminatedBuilder. That's substantially more code. The upside: the Pony version is bidirectional (encode is derived from the same definition), and the builder enforces no duplicate tags. But the setup cost is real.
A buffer containing multiple messages. Decode all of them. This is independent of how the buffer was obtained — it could be a file read, a UDP datagram, or a pre-assembled byte array.
A pcap file body: a sequence of packet records, each with a header (timestamp, lengths) followed by packet data. Read the whole file into a buffer, decode every record.
Many applies a codec repeatedly until the input is exhausted. It stops on InsufficientData (not enough bytes for another complete element) and errors on MalformedData.
// Parse all messages from a buffer let all_messages = Many[MyMessage](my_message_codec) // Codec[Array[MyMessage] val] let data: Array[U8] val = FileReader.read("capture.bin") match all_messages.decode(data, 0) | (let msgs: Array[MyMessage] val, let consumed: USize) => for msg in msgs.values() do process(msg) end | let m: MalformedData => env.out.print("Bad data: " + m.string()) end
Many is bidirectional: on encode, it encodes each element in sequence.
# Recursive extraction — the standard Erlang pattern defp extract(<<ts::32, len::16, payload::binary-size(len), rest::binary>>, acc) do extract(rest, [{ts, payload} | acc]) end defp extract(partial, acc), do: {Enum.reverse(acc), partial}
Pony is simpler here. The Elixir recursive extract looks clean, but the catch-all clause (defp extract(partial, acc)) silently treats all match failures as "incomplete data." A malformed message also fails to match — so a corrupt record is indistinguishable from a truncated one. In a streaming context, this means a single corrupt message causes the buffer to grow forever, waiting for bytes that would "complete" something that was never a valid message. Each protocol writes its own extract function from scratch, and each one has this same silent-failure mode. Many(codec) handles the loop once, and the codec's error model (InsufficientData vs. MalformedData) propagates correctly — a corrupt record stops the parse with an error instead of silently accumulating. It also encodes, which the recursive function doesn't.
Data arrives in chunks over TCP. A message might span chunks. Accumulate bytes, extract complete messages, carry the remainder. This is orthogonal to the codec itself — any codec can be wrapped for streaming.
FrameDecoder wraps any Codec[A] for use inside a TCPConnectionNotify. It holds a ref buffer, appends incoming iso chunks (consuming them), and runs the codec in a loop. The buffer is ref, which satisfies decode's box parameter since ref <: box.
class ref FrameDecoder[A: Any val] let _codec: Codec[A] var _buffer: Array[U8] ref var _offset: USize fun ref received(data: Array[U8] iso, notify: FrameNotify[A] ref) => _buffer.append(consume data) var continue = true while continue do // _buffer is ref, decode takes box, ref <: box match _codec.decode(_buffer, _offset) | (let value: A, let consumed: USize) => notify.frame_received(value) _offset = _offset + (consumed / 8) | InsufficientData => continue = false | let m: MalformedData => notify.frame_error(m) continue = false end end _trim_consumed()
The error model matters here: InsufficientData means "wait for more bytes" (normal in streaming), while MalformedData means "this data is wrong" (a real error). A partial function (?) can't make that distinction.
# The classic Erlang buffer-accumulation loop def loop(buffer, parent) do receive do {:tcp, _sock, chunk} -> data = buffer <> chunk rest = extract(data, parent) loop(rest, parent) {:tcp_closed, _sock} -> :ok end end
Roughly comparable. Both require the same fundamental structure: accumulate chunks, extract complete messages in a loop, carry the remainder. Elixir writes the buffer management inline; Pony wraps it in FrameDecoder. The FrameDecoder is reusable across protocols — plug in any codec. In Elixir, each protocol writes its own extract loop. Elixir also offers :gen_tcp's packet: N option for simple length-prefixed framing, which has no Pony equivalent.
The buffer is ref, not iso. It never leaves the actor, so isolation isn't needed — ref is the right cap. Incoming iso chunks are consumed into it via append. decode takes box, and ref <: box, so no copy or capability conversion is needed. Integer extraction is by value. Bytes(n) copies its N bytes — O(field size), not O(buffer size).
A field that is present or absent depending on a flag decoded earlier. The field's existence is data-dependent, not statically known.
MPEG-TS adaptation field: a PCR (program clock reference, 48 bits) is present only if the pcr_flag bit is set. Five other optional fields follow, each gated by its own flag bit.
FlatZip threads the flag value to a Conditional codec. The result type includes None for the absent case.
// Flag bit controls whether 48-bit PCR follows let pcr_field = BoolBit.flat_zip[(U64 | None)]( {(flag) => Conditional[U64](flag, BitsU64(48))}) // Codec[(Bool, (U64 | None))] // flag=true → (true, 0x1234ABCD0000) // flag=false → (false, None) // Multiple conditional fields chained: let adaptation = Seq2( BoolBit.flat_zip[(U64 | None)]( {(f) => Conditional[U64](f, BitsU64(48))}), // PCR BoolBit.flat_zip[(U64 | None)]( {(f) => Conditional[U64](f, BitsU64(48))}) // OPCR )
# Can't express in a single pattern — needs two clauses or imperative logic case pcr_flag do 1 -> <<pcr::48, rest::binary>> = remaining {pcr, rest} 0 -> {nil, remaining} end
Roughly comparable. Neither language handles this in one expression. Elixir needs a case or multiple function clauses; Pony uses FlatZip + Conditional. The Pony version is more mechanical but still bidirectional — on encode, the flag is set based on whether the value is None. Elixir requires separate encode logic.
Fixed byte sequences that identify a protocol, message type, or frame boundary. On decode, the bytes must match exactly or the parse fails. On encode, the bytes are emitted verbatim.
MPEG-TS sync byte (0x47) at the start of every 188-byte packet. PNG magic bytes (89 50 4E 47 0D 0A 1A 0A). The "UD"/"DD" strings in the badge protocol.
Literal matches fixed bytes without binding a value (its type is Codec[None]). ConstThen and ThenConst pair a literal with a data-carrying codec, keeping only the data.
// MPEG-TS: sync byte, then 3 bytes of header data let ts_packet = ConstThen[U32]( Literal([as U8: 0x47]), BitsU32(24)) // Decode: checks 0x47, reads 24 bits → Codec[U32] // Encode: writes 0x47, then 24 bits of the U32 value // PNG magic: just validate, bind nothing let png_magic = Literal( [as U8: 0x89; 0x50; 0x4E; 0x47; 0x0D; 0x0A; 0x1A; 0x0A]) // Codec[None] — fails on mismatch, advances cursor on match // Badge: data first, then the command literal let ud_badge = ThenConst[Array[U8] val]( Bytes(10), Literal([as U8: 'U'; 'D'])) // Codec[Array[U8] val] — 10 bytes of badge ID
# Literal bytes as guards in a pattern <<0x47, header::binary-size(3), _rest::binary>> = data # or string literals <<"UD", badge_id::binary-size(10)>> = data
Elixir is more concise. Literal bytes are inline in the pattern — no separate combinator needed. The Pony Literal / ConstThen / ThenConst naming is more verbose, but both do the same work. The Pony version emits the literal bytes on encode automatically; Elixir's construction syntax is separate.
One structure contains another: an Ethernet frame wraps an IP packet which wraps a TCP segment which wraps application data. Each layer has its own header and its own codec.
Codecs compose directly. A field in an outer codec can itself be a codec for a complex inner type. VariableSize bounds the inner codec to the right number of bytes.
// Inner: a simple key-value TLV entry class val TlvEntry let tag: U8 let value: Array[U8] val let tlv_entry = Mapped[(U8, Array[U8] val), TlvEntry]( Seq2(U8Codec, VariableSize[Array[U8] val](U8Codec, BytesRest)), {(t) => TlvEntry(t._1, t._2)}, {(e) => (e.tag, e.value)}) // Outer: a message containing nested TLV entries class val TlvMessage let version: U8 let entries: Array[TlvEntry] val let tlv_message = Mapped[(U8, Array[TlvEntry] val), TlvMessage]( Seq2(U8Codec, Many[TlvEntry](tlv_entry)), {(t) => TlvMessage(t._1, t._2)}, {(m) => (m.version, m.entries)})
Each layer is an independent codec. The nesting comes from using one codec as a building block in another — no special nesting syntax or combinator needed.
# Can't nest binary patterns — each layer is a separate function def parse_outer(<<version::8, rest::binary>>) do entries = parse_entries(rest, []) %TlvMessage{version: version, entries: entries} end defp parse_entries(<<tag::8, len::8, value::binary-size(len), rest::binary>>, acc) do parse_entries(rest, [%TlvEntry{tag: tag, value: value} | acc]) end defp parse_entries(<<>>, acc), do: Enum.reverse(acc)
Pony is simpler here. Elixir can't nest <<>> patterns — each layer needs its own function and manual recursive iteration. The Pony version composes codecs directly: Many[TlvEntry](tlv_entry) inside a Seq2. The nesting is the composition — no separate functions, no manual recursion, and bidirectional.
N instances of a sub-structure, either counted by a preceding field or repeated until the input is exhausted.
DNS response: the header says ancount = 3, meaning 3 answer resource records follow. Each record has its own variable-length layout.
Count reads a count field, then decodes that many instances. Many repeats until the input is consumed (or InsufficientData for a partial element).
// Counted: U16 count, then that many TlvEntry values let counted_entries = Count[TlvEntry](U16BE, tlv_entry) // Codec[Array[TlvEntry] val] // Decode: reads U16 = 3, then decodes 3 TlvEntry instances // Encode: writes 3 as U16, then encodes each entry // Greedy: decode TlvEntry values until input exhausted let all_entries = Many[TlvEntry](tlv_entry) // Codec[Array[TlvEntry] val] // Stops on InsufficientData (not enough for another entry) // Errors on MalformedData (data present but invalid)
Count is bidirectional: on encode, it counts the array, writes the count field, then encodes each element. The count field value is never part of the domain type.
# "Parse N records" requires a recursive function defp parse_n(data, 0, acc), do: {Enum.reverse(acc), data} defp parse_n(<<tag::8, len::8, value::binary-size(len), rest::binary>>, n, acc) do parse_n(rest, n - 1, [%{tag: tag, value: value} | acc]) end # Called after reading the count: <<count::16-big, rest::binary>> = data {entries, _remaining} = parse_n(rest, count, [])
Pony is simpler here. Elixir can't express "parse N of these" in a pattern — it needs a recursive helper that counts down. Count(U16BE, tlv_entry) does it in one line, and encodes the count automatically on the write side.
Fields whose values are derived from other fields during encoding (checksum = function of data, length = payload size) and validated during decoding.
This pattern uses Mapped with a validating forward function and a computing reverse function. The checksum field is decoded as a plain integer; the Mapped's forward function validates it against the data. The reverse function computes it from the domain value.
// A frame: U32 CRC followed by payload bytes // Domain type holds just the payload; CRC is computed class val CheckedFrame let payload: Array[U8] val let checked_frame = Mapped[(U32, Array[U8] val), CheckedFrame]( Seq2(U32BE, BytesRest), {(t) => // Validate: compute CRC of payload, compare to field let expected = CRC32(t._2) if t._1 != expected then error end CheckedFrame(t._2)}, {(f) => // Compute: CRC is derived from payload on encode (CRC32(f.payload), f.payload)})
# Decode: extract CRC and payload, validate manually <<crc::32-big, payload::binary>> = data ^crc = :erlang.crc32(payload) # match fails if wrong # Encode: compute CRC, build binary crc = :erlang.crc32(payload) <<crc::32-big, payload::binary>>
Roughly comparable. Both require validation logic beyond simple pattern matching. Elixir uses pinned variables (^crc) or guard clauses; Pony uses a partial Mapped forward function. The Pony version bundles the validation and computation into one codec definition; Elixir writes decode and encode as separate expressions that must stay consistent.
The forward function is partial (?) — a checksum mismatch produces a decode error. The Mapped combinator would need to handle partial forward functions, surfacing the error as MalformedData. This is an extension to the basic Mapped sketch.
The file or stream declares its byte ordering in a header field. All subsequent multi-byte integers use that ordering. The decoder must read the header, determine endianness, then use the right codec for every field that follows.
This is a FlatZip: decode the endianness marker, use its value to select between BE and LE codec sets for the rest of the structure. The endianness choice flows through the dependency chain.
// pcap: magic number determines endianness // 0xA1B2C3D4 = native (big-endian file) // 0xD4C3B2A1 = swapped (little-endian file) primitive PcapEndian fun codec(): Codec[(Bool, PcapHeader)] => U32BE.flat_zip[PcapHeader]( {(magic) => let be = magic == 0xA1B2C3D4 if be then pcap_header_be() // all fields big-endian else pcap_header_le() // all fields little-endian end}) fun pcap_header_be(): Codec[PcapHeader] => Mapped[(U16, U16, U32, U32), PcapHeader]( Seq4(U16BE, U16BE, U32BE, U32BE), ...) fun pcap_header_le(): Codec[PcapHeader] => Mapped[(U16, U16, U32, U32), PcapHeader]( Seq4(U16LE, U16LE, U32LE, U32LE), ...)
The endianness choice is explicit at each level — no global state or implicit threading. Each codec function picks BE or LE for its fields. This is verbose for all-same-endianness formats, but a factory function (like pcap_header_be() vs pcap_header_le()) keeps the repetition in one place.
# Read magic number, branch on endianness case data do <<0xA1B2C3D4::32, rest::binary>> -> parse_pcap_header(rest, :big) <<0xD4C3B2A1::32, rest::binary>> -> parse_pcap_header(rest, :little) end # But endianness can't be a variable in a pattern — # you need two full sets of patterns or use :binary.decode_unsigned/2
Roughly comparable. Both need to branch on the endianness marker and use different decoders for each path. Elixir can't parameterize endianness in a pattern — ::big and ::little are compile-time modifiers, not runtime values — so you need two separate pattern sets or fall back to :binary.decode_unsigned/2. The Pony version has the same branching structure, but each branch is a reusable codec value.
Fields that must start on byte or word boundaries. Padding bytes fill the gap between the end of one field and the required alignment of the next.
SkipBits consumes bits without binding a value. For alignment, the skip count is calculated from the current position — which requires a dependent codec.
// Skip 3 reserved bits between fields let with_padding = Seq3( BitsU8(4), // 4-bit field SkipBits(3), // 3 reserved bits BoolBit) // 1-bit flag — now byte-aligned // Codec[(U8, None, Bool)] — None from SkipBits // Fixed padding to word boundary let padded = Seq2( Bytes(3), // 3 bytes of data SkipBits(8)) // 1 byte padding to 4-byte alignment
# Skip reserved bits explicitly <<field::4, _reserved::3, flag::1>> = byte # Padding to alignment <<data::binary-size(3), _pad::8, rest::binary>> = input
Roughly comparable. Both handle static padding by naming the skip width explicitly. Neither has built-in dynamic alignment (skip to the next N-bit boundary). Elixir uses underscore-prefixed variables to discard padding; Pony uses SkipBits. The same structure, different syntax.
Dynamic alignment (skip N bits to reach the next byte boundary, where N depends on how many bits have been consumed) would need access to the cursor's current bit offset inside a FlatZip. The current design doesn't expose cursor position to codec constructors. For now, only static skip counts are supported. Dynamic alignment is a possible future extension.
The patterns above cover parsing a complete buffer. Streaming is the harder problem: data arrives in chunks over TCP, and a message can span multiple chunks. A frame protocol says "expect 50 bytes of payload" but you might have the 2-byte header and only 2 bytes of payload so far. The system must recognize "not enough data yet," hold onto what it has, and resume when more arrives.
A length-prefixed frame protocol: 2-byte big-endian length, then that many bytes of payload. Three TCP chunks arrive:
Chunk 1: [00 32 AA BB] — header says 50 bytes, but only 2 payload bytes Chunk 2: [CC DD ... 48 more] — now we have all 50 Chunk 3: [00 03 EE FF] — next header says 3 bytes, only 2 here
After chunk 1, the parser has the length field (50) but only 2 of the 50 payload bytes. It must:
After chunk 2, it has enough to deliver the first message. After chunk 3, it's partway through the second message and must wait again.
The simple path: Erlang's :gen_tcp has a packet: N option that handles this automatically. Set packet: 2 on the socket (2-byte length prefix) and the runtime buffers internally — each {:tcp, socket, data} message delivers exactly one complete payload with the length header stripped. This is hard to beat when both sides can agree on the framing format.
Custom protocols don't get this for free. The standard pattern:
def loop(buffer, parent) do receive do {:tcp, _sock, chunk} -> data = buffer <> chunk {messages, rest} = extract(data, []) Enum.each(messages, &process/1) loop(rest, parent) {:tcp_closed, _sock} -> :ok end end defp extract(<<len::16, payload::binary-size(len), rest::binary>>, acc) do extract(rest, [payload | acc]) end defp extract(partial, acc), do: {Enum.reverse(acc), partial}
Walking through our scenario:
[00 32 AA BB]): The pattern tries to match <<len::16, payload::binary-size(len), rest::binary>>. It reads len = 50, but binary-size(50) needs 50 bytes and only 2 are available. The match fails, falls through to the catch-all, and returns {[], <<0, 50, 0xAA, 0xBB>>}. The 4 bytes are kept as the buffer for next time.<<0, 50, ...52 bytes total>>. The pattern matches: len = 50, 50 payload bytes, rest = <<>>. Message delivered.[00 03 EE FF]): len = 3, but only 2 payload bytes. Match fails, buffer carried.This works correctly for the incomplete-data case. But consider what happens if byte 3 of chunk 1 were corrupted — say the length field read as 0xFFFF (65535). The pattern would try to match 65535 bytes, fail, and fall through to the same catch-all. The system would now buffer 65KB of data waiting for a "message" that was never actually sent. The catch-all clause can't distinguish "haven't received enough yet" from "the data is wrong." Both are just a failed match.
The combinator library separates framing from parsing. The codec handles the protocol. FrameDecoder handles the streaming:
// The protocol codec — same one used for one-shot parsing let frame_codec = VariableSize[Array[U8] val](U16BE, BytesRest) // Wrap it for streaming let framer = FrameDecoder[Array[U8] val](frame_codec) // Inside TCPConnectionNotify.received(): framer.received(consume data, my_notify)
Walking through the same scenario:
FrameDecoder appends the chunk to its buffer, calls frame_codec.decode(buffer, 0). The codec reads len = 50 via U16BE, then BytesRest tries to read 50 bytes but only 2 are available. It returns InsufficientData. FrameDecoder stops the loop and keeps the buffer intact.FrameDecoder calls notify.frame_received(payload), advances the offset, and tries to decode again. InsufficientData (nothing left). Trims consumed bytes.InsufficientData, wait for more.Now the corruption scenario: the length field reads as 65535. VariableSize reads the length via U16BE (succeeds — it's a valid U16). Then it tries to read 65535 bytes via BytesRest. Not enough data — InsufficientData.
This is the same outcome as Elixir: the system waits. The length field is a valid integer; there's no way to know 65535 is "wrong" without application-level knowledge (e.g., a maximum message size). Both systems need an upper bound check for this case. The difference shows up when the corruption is in the payload, not the length — if the inner codec encounters data that is structurally invalid (a tag byte that matches no known variant, a nested structure that doesn't parse), it returns MalformedData. FrameDecoder calls notify.frame_error() instead of silently buffering. Elixir's catch-all treats this the same as "not enough data."
| Elixir | Pony combinator library | |
|---|---|---|
| Simple framing | packet: N on the socket — automatic, zero code | No equivalent; always explicit FrameDecoder |
| Custom framing | Manual extract function, written per protocol | Reusable FrameDecoder wrapping any codec |
| Partial message | Match fails, catch-all keeps buffer | InsufficientData, FrameDecoder keeps buffer |
| Corrupt payload | Match fails — indistinguishable from incomplete | MalformedData — reported as an error |
| Corrupt length | Buffers forever (same problem in both) | Buffers forever (needs app-level max-size check in both) |
| Buffer management | Binary concatenation in process loop | FrameDecoder owns a ref buffer, trims consumed bytes |
| Codec reuse | Same patterns work one-shot and streaming | Same codec works one-shot and in FrameDecoder |
The honest summary: for simple length-prefixed protocols, Elixir's packet: N is unbeatable — zero code. For custom protocols, both approaches handle partial delivery correctly. The Pony library's potential advantage is distinguishing corrupt data from incomplete data, and making the framing logic reusable rather than hand-written per protocol.