Fix backpressure muting cascade

Replace binary muting with graduated per-actor batch sizes that adjust dynamically under backpressure, eliminating the fan-in muting cascade.
9 files changed +459 −9

Motivation

Under fan-in topologies, the runtime's binary muting cascaded backward through the entire actor graph. When a receiver became overloaded, every actor that sent to it was immediately muted. Their senders then saw muted targets and were muted in turn. Within one scheduling interval, most of the system was frozen — throughput dropped from tens of millions of messages per second to near zero.

This replaces binary muting with proportional throttling. Senders to overloaded receivers have their batch size decremented rather than being removed from scheduling entirely. Overloaded receivers grow their batch to process more messages per run. The cascade is broken because throttled actors remain scheduled — their senders never see them as muted.

File Map

▶ Release notes 1 file

Behavioral Changes

Proportional throttling replaces binary muting

Senders to overloaded receivers have their per-actor batch_size decremented by PONY_SCHED_BATCH_STEP (25) per send, rather than being immediately muted. An actor is only fully muted when its batch reaches 0 — after 4 sends to overloaded receivers in one run at the default batch of 100.

Overloaded actors process larger batches

When an overloaded actor processes a full batch without emptying its queue, its batch_size grows by 25 with no upper bound. This lets bottleneck actors increase their effective service rate to match the arrival rate.

Overloaded actors are never muted

An actor that is itself overloaded (or under pressure) and sends to another overloaded actor has its batch decremented, but floored at PONY_SCHED_BATCH_STEP (25). It is never muted — the actor graph stays connected.

Gradual recovery

Unmuted actors restart at batch 25 and ramp up over 4 clean runs (25 → 50 → 75 → 100), avoiding a thundering-herd restart. Actors that weren't muted but were throttled recover the same way: +25 per run where they don't send to any overloaded receiver, capped at the default.

Persistent equilibrium

Clearing overload does not reset the batch. An actor that found its required service rate at batch 350 keeps that batch, avoiding re-triggering overload on the next burst.

Zero struct growth

The two new fields (batch_size as uint16_t, sent_to_overloaded as bool) pack into existing alignment padding between internal_flags and live_asio_events. sizeof(pony_actor_t) is unchanged.

API Changes

▶ Changed maybe_mark_should_mute() — graduated throttling

The core backpressure decision point. Previously muted the sender unconditionally; now decrements batch size proportionally, with special handling for overloaded senders.

if(triggers_muting(to) &&
  !has_sync_flag_any(ctx->current,
    ACTOR_SYNC_FLAG_OVERLOADED |
    ACTOR_SYNC_FLAG_UNDER_PRESSURE) &&
  ctx->current != to)
{
  ponyint_sched_mute(ctx, ctx->current, to);
}
if(triggers_muting(to) && ctx->current != to)
{
  ctx->current->sent_to_overloaded = true;
  if(has_sync_flag_any(ctx->current,
    ACTOR_SYNC_FLAG_OVERLOADED |
    ACTOR_SYNC_FLAG_UNDER_PRESSURE))
  {
    if(ctx->current->batch_size > PONY_SCHED_BATCH_STEP)
      ctx->current->batch_size -= PONY_SCHED_BATCH_STEP;
  }
  else if(ctx->current->batch_size <= PONY_SCHED_BATCH_STEP)
  {
    ctx->current->batch_size = 0;
    ponyint_sched_mute(ctx, ctx->current, to);
  }
  else
  {
    ctx->current->batch_size -= PONY_SCHED_BATCH_STEP;
  }
}
▶ Changed pony_actor_t — two new fields in padding

Two fields added between internal_flags and live_asio_events, fitting in the existing 3-byte alignment gap. No change to struct size.

uint8_t internal_flags;
uint32_t live_asio_events;
uint8_t internal_flags;
bool sent_to_overloaded;
uint16_t batch_size;
uint32_t live_asio_events;
▶ Changed ponyint_actor_run() — per-actor batch size

The batch limit is now read from the actor's own field instead of the compile-time constant.

size_t batch = PONY_SCHED_BATCH;
actor->sent_to_overloaded = false;
size_t batch = actor->batch_size;
▶ Added PONY_SCHED_BATCH_STEP — compile-time constant

PONY_SCHED_BATCH moved from actor.c to actor.h (so scheduler.c can reference it). PONY_SCHED_BATCH_STEP added alongside it.

#define PONY_SCHED_BATCH 100
#define PONY_SCHED_BATCH_STEP 25
▶ Added batch recovery — end of ponyint_actor_run()

After the overload check, if the actor didn't send to any overloaded receiver this run, its batch is incremented toward the default.

if(!actor->sent_to_overloaded
  && actor->batch_size < PONY_SCHED_BATCH)
{
  actor->batch_size += PONY_SCHED_BATCH_STEP;
  if(actor->batch_size > PONY_SCHED_BATCH)
    actor->batch_size = PONY_SCHED_BATCH;
}
▶ Added sched_reschedule_unmuted() — gradual restart

Unmuted actors start at one step instead of the full default.

actor->batch_size = PONY_SCHED_BATCH_STEP;

Architecture

Before: Binary Muting Sender A (batch=100) Sender B (batch=100) Analyzer (MUTED) Receiver (OVERLOADED) ✖ also muted (cascade) ✖ muted immediately After: Graduated Batch Sizes Sender A (batch=100) Sender B (batch=100) Analyzer (batch=75) Receiver (batch=250) ✔ still scheduled ✔ throttled, still scheduled ✔ batch grows to catch up No cascade: throttled actors remain scheduled, senders unaffected

Release Notes

Fix backpressure muting cascade

Under fan-in topologies — many actors sending to a shared slow receiver — the runtime's binary muting cascaded backward through the entire actor graph. When the receiver became overloaded, every actor that sent to it was immediately muted. Their senders then saw muted targets and were muted in turn. Within one scheduling interval, most of the system was frozen, dropping throughput from tens of millions of messages per second to near zero.

The runtime now uses graduated per-actor batch sizes instead of binary muting. Each actor's batch size adjusts dynamically: senders to overloaded receivers are throttled proportionally (one step per send) rather than muted outright, and overloaded receivers process larger batches to catch up. Actors are only fully muted after repeated sends to overloaded receivers within a single scheduling run. This eliminates the cascade — throttled actors remain scheduled, so their senders never see them as muted.