Add binary_codec package

A combinator library for bidirectional binary parsing and building. Define a Codec[A] once and it handles both decoding bytes into A and encoding A back to bytes.
17 files changed +2,998

Motivation

Pony programs that speak binary protocols (TCP, UDP, file formats) need to decode incoming bytes and encode outgoing values. Without a library, every project writes its own ad-hoc parsing with manual offset arithmetic, byte-shifting, and error checking. This is repetitive, error-prone, and impossible to test in isolation.

The binary_codec package provides a composable, type-safe approach: small primitive codecs (U16BE, BitsU8(4), Bytes(16)) compose into larger structures via combinators (Seq4, Mapped, FlatZip, Discriminated). A single codec definition drives both directions, so parsing and serialization can't diverge.

The library is bit-native. Sub-byte fields (4-bit nibbles, 1-bit flags, 13-bit offsets) work the same way as byte-aligned fields. Errors are data: InsufficientData means "wait for more bytes," MalformedData means "this data is wrong." FrameDecoder uses this distinction to implement zero-copy TCP stream framing.

File Map

▶ Core types 3 files
▶ Primitive codecs 3 files
▶ Combinators 4 files
▶ Repetition & framing 2 files
▶ Internal & tests 3 files

Public API Surface

Core trait and error types

New trait val Codec[A: Any val]
A bidirectional codec that decodes bytes into a value and encodes a value into bytes. Codecs are val — immutable and shareable across actors. The bit offset lets sub-byte fields compose naturally.
  • fun decode(data: Array[U8] val, offset: USize): ((A, USize) | DecodeError)
  • fun encode(value: A): (Array[U8] val | EncodeError)
New primitive InsufficientData
Returned when decode needs more bytes than are available. In streaming, this means "wait for more data."
  • fun string(): String iso^
New class val MalformedData
Returned when decode encounters structurally invalid data — a literal mismatch, an unknown tag, a constraint violation. No additional data will fix it.
  • new val create(message': String, bit_offset': USize = 0)
  • fun message(): String
  • fun bit_offset(): USize
  • fun string(): String iso^
New type DecodeError is (InsufficientData | MalformedData)
The error union returned by decode.
New class val EncodeViolation
Returned when encode fails — a value doesn't fit the codec's constraints or the output isn't byte-aligned.
  • new val create(message': String)
  • fun message(): String
  • fun string(): String iso^
New type EncodeError is EncodeViolation
The error type returned by encode.

Byte-aligned primitives

New primitive U8Codec is Codec[U8]
Read or write a single unsigned byte (8 bits).
New primitive I8Codec is Codec[I8]
Read or write a signed byte (8 bits).
New primitive BoolBit is Codec[Bool]
Read or write a single bit as a Bool.
New primitive U16BE is Codec[U16]
16-bit unsigned integer, big-endian.
New primitive U16LE is Codec[U16]
16-bit unsigned integer, little-endian.
New primitive I16BE is Codec[I16]
16-bit signed integer, big-endian.
New primitive I16LE is Codec[I16]
16-bit signed integer, little-endian.
New primitive U32BE is Codec[U32]
32-bit unsigned integer, big-endian.
New primitive U32LE is Codec[U32]
32-bit unsigned integer, little-endian.
New primitive I32BE is Codec[I32]
32-bit signed integer, big-endian.
New primitive I32LE is Codec[I32]
32-bit signed integer, little-endian.
New primitive U64BE is Codec[U64]
64-bit unsigned integer, big-endian.
New primitive U64LE is Codec[U64]
64-bit unsigned integer, little-endian.
New primitive I64BE is Codec[I64]
64-bit signed integer, big-endian.
New primitive I64LE is Codec[I64]
64-bit signed integer, little-endian.
New primitive U128BE is Codec[U128]
128-bit unsigned integer, big-endian.
New primitive U128LE is Codec[U128]
128-bit unsigned integer, little-endian.
New primitive I128BE is Codec[I128]
128-bit signed integer, big-endian.
New primitive I128LE is Codec[I128]
128-bit signed integer, little-endian.
New primitive F32BE is Codec[F32]
32-bit IEEE 754 float, big-endian.
New primitive F32LE is Codec[F32]
32-bit IEEE 754 float, little-endian.
New primitive F64BE is Codec[F64]
64-bit IEEE 754 float, big-endian.
New primitive F64LE is Codec[F64]
64-bit IEEE 754 float, little-endian.

Bit-width codecs

New class val BitsU8 is Codec[U8]
Read or write an unsigned integer of 1–8 bits. Sub-byte fields: BitsU8(4) reads a nibble.
  • new val create(count: U8)
New class val BitsU16 is Codec[U16]
Read or write an unsigned integer of 1–16 bits, big-endian.
  • new val create(count: U8)
New class val BitsU32 is Codec[U32]
Read or write an unsigned integer of 1–32 bits, big-endian.
  • new val create(count: U8)
New class val BitsU64 is Codec[U64]
Read or write an unsigned integer of 1–64 bits, big-endian.
  • new val create(count: U8)

Byte and literal codecs

New class val Bytes is Codec[Array[U8] val]
Read or write exactly count bytes. Requires byte alignment. Returns a zero-copy trim view of the input array.
  • new val create(count: USize)
New primitive BytesRest is Codec[Array[U8] val]
Read all remaining bytes, or write a byte array with no length constraint. Requires byte alignment.
New class val Literal is Codec[None]
Match or emit a fixed byte sequence. On decode, returns MalformedData if the bytes don't match. The decoded value is None.
  • new val create(expected: Array[U8] val)

Sequence combinators

New class val Seq2[A, B] is Codec[(A, B)]
Sequence two codecs, producing a tuple of their decoded values.
  • new val create(a: Codec[A], b: Codec[B])
New class val Seq3[A, B, C] is Codec[(A, B, C)]
Sequence three codecs, producing a 3-tuple.
  • new val create(a: Codec[A], b: Codec[B], c: Codec[C])
New class val Seq4[A, B, C, D] is Codec[(A, B, C, D)]
Sequence four codecs, producing a 4-tuple.
  • new val create(a: Codec[A], b: Codec[B], c: Codec[C], d: Codec[D])
New class val Seq5[A, B, C, D, E] is Codec[(A, B, C, D, E)]
Sequence five codecs, producing a 5-tuple.
  • new val create(a: Codec[A], b: Codec[B], c: Codec[C], d: Codec[D], e: Codec[E])
New class val Seq6[A, B, C, D, E, F] is Codec[(A, B, C, D, E, F)]
Sequence six codecs, producing a 6-tuple. Larger structures compose from smaller Seq groups via Mapped.
  • new val create(a: Codec[A], b: Codec[B], c: Codec[C], d: Codec[D], e: Codec[E], f: Codec[F])

Structural combinators

New class val Mapped[A, B] is Codec[B]
Bidirectional type mapping between a wire type A and a domain type B. The forward function may fail (partial) to signal malformed data.
  • new val create(c: Codec[A], f: {(A): B ?} val, r: {(B): A} val)
New class val Widen[V, Msg] is Codec[Msg]
Widen a codec's type from a specific variant V to a broader union Msg. Used internally by DiscriminatedBuilder.
  • new val create(codec: Codec[V], narrow: {(Msg): V ?} val)
New class val ThenConst[A] is Codec[A]
Decode a value followed by a literal, keeping only the value. On encode, write both.
  • new val create(codec: Codec[A], literal: Literal)
New class val ConstThen[A] is Codec[A]
Decode a literal followed by a value, keeping only the value. On encode, write both.
  • new val create(literal: Literal, codec: Codec[A])
New class val FlatZip[A, B] is Codec[(A, B)]
Dependent sequencing: the second codec is determined by the first decoded value. On encode, A is available from the tuple so the same dispatch works in reverse.
  • new val create(ca: Codec[A], f: {(A): Codec[B]} val)
New class val VariableSize[A] is Codec[A]
Length-prefixed data: a size codec determines how many bytes the body reads from. On encode, the body is encoded first, measured, then the size is written before it.
  • new val create(size_codec: Codec[U16], body_codec: Codec[A])
New class val Conditional[A] is Codec[(A | None)]
Conditionally present field: if the flag is true, decode/encode using the inner codec; if false, the value is None.
  • new val create(flag: Bool, codec: Codec[A])
New class val SkipBits is Codec[None]
Skip a fixed number of bits on decode, write zero bits on encode. For reserved or padding fields.
  • new val create(count: USize)

Repetition

New class val Count[A] is Codec[Array[A] val]
Read a count field, then decode that many instances of the body codec. On encode, write the array length as the count, then each element.
  • new val create(count_codec: Codec[U16], body_codec: Codec[A])
New class val Many[A] is Codec[Array[A] val]
Decode elements repeatedly until the input is exhausted. Stops on InsufficientData; propagates MalformedData as an error.
  • new val create(codec: Codec[A])

Tagged union dispatch

New class val Discriminated[D, Msg] is Codec[Msg]
Tagged union dispatch: a discriminator codec selects which variant codec to use. On decode, reads the tag and dispatches. On encode, tests each variant's narrowing function. Built via DiscriminatedBuilder.
New class ref DiscriminatedBuilder[D, Msg]
Builder for constructing a Discriminated codec. Register each variant with case_of, then call build().
  • new ref create(disc_codec: Codec[D])
  • fun ref case_of[V: Msg](disc: D, codec: Codec[V], narrow: {(Msg): V ?} val): DiscriminatedBuilder[D, Msg] ref
  • fun ref build(): Discriminated[D, Msg]

TCP stream framing

New class ref FrameDecoder[A]
Wraps any Codec[A] for TCP stream framing. Incoming iso chunks are concatenated with unconsumed remainder into a val buffer for zero-copy decoding. InsufficientData means "wait for more bytes"; MalformedData is reported via FrameNotify.
  • new ref create(codec: Codec[A])
  • fun ref received(data: Array[U8] iso, notify: FrameNotify[A] ref)
New interface ref FrameNotify[A]
Callback interface for FrameDecoder. Receives complete decoded values and error notifications.
  • fun ref frame_received(value: A)
  • fun ref frame_error(err: MalformedData)

Architecture

CORE TRAIT Codec[A] PRIMITIVES U8..U64 BE/LE BitsU8..U64 Bytes Literal BoolBit COMBINATORS Seq2..6 Mapped FlatZip Many Count Discriminated ThenConst ConstThen VariableSize Conditional SkipBits FRAMING FrameDecoder FrameNotify DecodeError (InsufficientData | MalformedData) • EncodeError (EncodeViolation)

Primitive codecs (U16BE, BitsU8, Bytes, etc.) implement the Codec trait directly. Combinators compose codecs: Seq for tuples, Mapped for type mapping, FlatZip for dependent parsing, Discriminated for tagged unions, Many/Count for repetition. All combinators are themselves codecs, so they compose recursively.

FrameDecoder wraps any composed codec for TCP streaming, using the InsufficientData/MalformedData distinction to separate "wait for more" from "corrupt data." It notifies the application through the FrameNotify interface.

Release Notes

Add binary_codec package

A new standard library package for bidirectional binary parsing. Define a Codec[A] once — the same definition decodes bytes into a value and encodes the value back to bytes.

Sub-byte fields (4-bit nibbles, 1-bit flags, 13-bit offsets) work the same as byte-aligned fields. decode takes Array[U8] val so byte-slice codecs return zero-copy views via Array.trim.

Codecs are val and compose via combinators: Seq2–Seq6 sequence fields into tuples, Mapped maps between wire tuples and domain types, FlatZip threads a decoded value into a dependent codec, Many and Count handle repetition, and Discriminated dispatches on a tag value.

Decode errors are data: InsufficientData means the input is incomplete, MalformedData means the data is structurally wrong. FrameDecoder wraps any codec for TCP stream framing and uses this distinction to separate “not enough data yet” from “corrupt message.”

use "binary_codec"

class val UdpHeader
  let src_port: U16
  let dst_port: U16
  let length: U16
  let checksum: U16

  new val create(s: U16, d: U16, l: U16, c: U16) =>
    src_port = s; dst_port = d; length = l; checksum = c

actor Main
  new create(env: Env) =>
    let codec: Codec[UdpHeader] =
      Mapped[(U16, U16, U16, U16), UdpHeader](
        Seq4[U16, U16, U16, U16](U16BE, U16BE, U16BE, U16BE),
        {(t) => UdpHeader(t._1, t._2, t._3, t._4)},
        {(h) => (h.src_port, h.dst_port, h.length, h.checksum)})

    let data: Array[U8] val =
      [0x00; 0x50; 0x00; 0x51; 0x00; 0x08; 0x12; 0x34]

    match codec.decode(data, 0)
    | (let h: UdpHeader, let bits: USize) =>
      env.out.print("src=" + h.src_port.string())
      env.out.print("dst=" + h.dst_port.string())
    | InsufficientData =>
      env.out.print("need more data")
    | let e: MalformedData =>
      env.out.print("bad data: " + e.string())
    end