ponyc runtime · PR #6285 · October 2026
Pony's current backpressure mechanism uses binary muting: when an actor sends to an overloaded receiver, the sender is immediately removed from scheduling. Under fan-in topologies, this cascades — muting propagates backwards through the actor graph, freezing the entire system for seconds at a time.
This report documents a graduated alternative: per-actor batch sizes that adjust dynamically under backpressure. Overloaded actors process more messages per run; senders to overloaded receivers process fewer. The mechanism eliminates the fan-in collapse and triples sustained-overload throughput.
Sustained overload
3.4×
22M msg/s vs 6.5M baseline
Fan-in collapse
Eliminated
40M msg/s vs 130 msg/s
Code change
+38 lines
3 files, 0 bytes added to actor struct
All benchmarks run on a 16-core / 32 GB Linux box with --ponymaxthreads 16. Baseline is ponyc 0.75.0.
| Benchmark | Configuration | Baseline | Graduated |
|---|---|---|---|
| Sustained overload | 10K senders, 1 receiver | ~6.5M msg/s | ~22M msg/s |
| Fan-in | 100s → 2K analyzers, 10ms workload | 39M → 119 msg/s | 40M msg/s stable |
| Fan-in | 100s → 2K analyzers, 5ms workload | 39M ↔ 99 msg/s | 41M msg/s stable |
| Fan-in | 100s → 2K analyzers, 1ms workload | 38M msg/s stable | 42M msg/s stable |
| Fan-in (high load) | 500s → 2K analyzers, 10ms workload | 40M → 181 msg/s | 44M msg/s stable |
| Fan-in (low fan-in) | 100s → 200 analyzers, 10ms workload | — | 51M msg/s stable |
The fan-in collapse — throughput dropping from 39M to <200 msg/s with a 20-second freeze — is completely eliminated across all configurations tested. The 1ms workload case, where the baseline was already stable, shows a ~10% throughput improvement from the receiver's batch growth.
Two new fields on pony_actor_t: a batch_size (uint16_t, starts at PONY_SCHED_BATCH, default 100) and a sent_to_overloaded flag (bool) cleared each run. Both pack into existing alignment padding — sizeof(pony_actor_t) is unchanged. Two compile-time constants: PONY_SCHED_BATCH (default batch, 100) and PONY_SCHED_BATCH_STEP (adjustment step, 25).
The fan-in benchmark: 100 senders → 2000 analyzers → 1 receiver.
Every 10 seconds, all 2000 analyzers send one message to the receiver. The receiver takes 10ms per message, so draining 2000 messages takes 20 seconds — longer than the 10-second tick interval. The receiver stays permanently overloaded.
With binary muting, every analyzer that sends to the overloaded receiver is immediately and fully muted. Muted analyzers stop processing messages from senders. Senders see their target analyzers as muted and are muted themselves. Within one tick, most of the system is frozen.
This is a limit cycle: the system oscillates between full throughput (39M msg/s) and near-total freeze (~130 msg/s). The feedback gain is greater than 1 — each muted actor causes further muting — producing an unstable oscillation.
The 5ms workload case is the boundary: drain time (10s) equals the tick interval, so the system alternates between stable and collapsed intervals. The 1ms case (drain time 2s) stays stable because the receiver always catches up.
When 2000 analyzers each send one message to the overloaded receiver, each analyzer's batch drops by 25 (100 → 75). All 2000 analyzers are still scheduled at batch 75. They continue processing messages from senders. No muting cascades backward.
Meanwhile, the receiver's batch grows: 100 → 125 → 150 → … It processes more messages per scheduling run, increasing its effective service rate. The receiver's batch stabilizes when its throughput matches the arrival rate.
On the next tick, if the receiver is no longer overloaded, each analyzer recovers: 75 → 100 in one clean run. If the receiver is still overloaded, analyzers drop to 50, then 25 — but they never reach 0 because each analyzer sends to the receiver only once per tick (one step decrement per tick).
The mechanism replaces a bang-bang controller (binary mute/unmute) with a proportional controller (graduated throttling). In queuing theory terms:
Each actor is a single-server queue. Messages arrive at rate λ. The actor processes up to batch_size messages per scheduling run, giving effective service rate μ(b). An actor is overloaded when λ > μ(b) for the current interval.
The system converges because the receiver's batch growth is monotonic during overload and persistent after overload clears. This creates a ratchet: the batch stabilizes at whatever value lets the receiver keep up with arrivals. Since batch growth is unbounded, convergence is guaranteed for any finite per-message processing time.
From the Pollaczek-Khinchine formula, as utilization ρ approaches 1, mean queue length diverges. The only remedies are: decrease λ (throttle senders) or increase μ (process faster). The graduated mechanism does both. Capping the batch would cap μ, potentially leaving ρ ≥ 1 for large enough bursts — guaranteeing unbounded queue growth.
In practice, a very large batch means an actor monopolizes its scheduler thread longer. This is acceptable: the actor genuinely has that many messages waiting, other scheduler threads serve other actors, and the actor yields after its batch regardless.
| Property | TCP (AIMD) | Graduated batch | Binary muting |
|---|---|---|---|
| Decrease | Multiplicative (halve) | Additive (−25) | Total (→ 0) |
| Increase | Additive (+1/RTT) | Additive (+25/run) | Binary (0 → 100) |
| Convergence | Proven stable | Stable (monotonic ratchet) | Limit cycle |
| Bottleneck adaptation | No (link capacity fixed) | Yes (batch grows) | No |
The mechanism's most distinctive property is bottleneck adaptation. TCP has no analogue because network link capacity is physical. In the actor model, an actor's "capacity" is its batch size — a scheduling parameter, not a physical constraint. This extra degree of freedom makes the system more adaptive than network congestion control.
An actor with a batch of 10,000 monopolizes its scheduler thread for 10,000 messages. With P scheduler threads and one large-batch actor, P−1 threads are unaffected. Large batches result from sustained overload, which is typically concentrated in a small number of receivers.
The ratio B₀/Δ = 100/25 = 4 means an actor gets 4 "strikes" before muting. This is aggressive enough to respond quickly to congestion, but gradual enough to avoid cascading. Both values are compile-time constants (PONY_SCHED_BATCH and PONY_SCHED_BATCH_STEP).
Under sustained overload where arrival rate permanently exceeds service rate, queues grow regardless of mechanism. The graduated approach doesn't make this worse — the receiver processes its queue faster (larger batch), so memory growth is actually slower than baseline for the same arrival rate.
The system could oscillate if senders recover and resume full-rate sending, re-triggering overload. This is damped because the receiver's elevated batch persists after overload clears — it can handle a higher arrival rate before becoming overloaded again. Each cycle leaves the receiver with a higher batch, so the system converges.
Three files changed (PR #6285):
#define PONY_SCHED_BATCH 100
#define PONY_SCHED_BATCH_STEP 25
typedef struct pony_actor_t
{
...
uint8_t internal_flags;
bool sent_to_overloaded;
uint16_t batch_size;
uint32_t live_asio_events;
...
}
if(triggers_muting(to) && ctx->current != to)
{
ctx->current->sent_to_overloaded = true;
if(has_sync_flag_any(ctx->current,
ACTOR_SYNC_FLAG_OVERLOADED |
ACTOR_SYNC_FLAG_UNDER_PRESSURE))
{
if(ctx->current->batch_size > PONY_SCHED_BATCH_STEP)
ctx->current->batch_size -= PONY_SCHED_BATCH_STEP;
}
else if(ctx->current->batch_size <= PONY_SCHED_BATCH_STEP)
{
ctx->current->batch_size = 0;
ponyint_sched_mute(ctx, ctx->current, to);
}
else
{
ctx->current->batch_size -= PONY_SCHED_BATCH_STEP;
}
}
actor->batch_size += PONY_SCHED_BATCH_STEP;
if(!actor->sent_to_overloaded
&& actor->batch_size < PONY_SCHED_BATCH)
{
actor->batch_size += PONY_SCHED_BATCH_STEP;
if(actor->batch_size > PONY_SCHED_BATCH)
actor->batch_size = PONY_SCHED_BATCH;
}
actor->batch_size = PONY_SCHED_BATCH_STEP;