Congestion control

​

Overview

1. The sender’s NIC sends its data packets as ECN-capable.
2. A switch with a congested egress queue marks those packets Congestion Experienced (CE). The NIC itself never marks a packet; marking is the switch’s job (ECN marking explains how the Scala Switch decides).
3. The receiver’s NIC, on a CE-marked packet, sends a congestion notification packet (CNP) to the sending queue pair. The sender’s NIC reacts to CNPs by lowering that queue pair’s rate, and raises it again in steps once CNPs stop.

​

Key concepts

​

ECN on the wire

​

Congestion notification

​

The rate control loop

1. CNP reception starts an episode. When a CNP arrives for a queue pair that is not congested, the loop runs at once, starting with a reduction. The queue pair is congested from then on, and its target rate is set to the rate it had when the episode began.
2. The control interval. The loop then runs every half ControlIntervalDurationNs, alternating a reduction half and an increase half, so each control interval of 45 µs by default holds one of each. The CNP and packet counts are cleared after each reduction half, so each interval counts afresh.
3. The congestion estimate. Every run of the loop updates the queue pair’s congestion estimate, a moving average of how strongly the queue pair’s traffic is being marked, taken from the CNPs it received and the packets it sent in the current interval. CongestionProbabilityWeight is the weight each new interval carries in the average: a higher weight makes the estimate rise faster and forget earlier congestion sooner.
4. Reduction. In a reduction half that saw at least one CNP, the queue pair makes a reduction. On the reduction that begins an episode, and during fast recovery, the current rate is cut by a fraction set by the congestion estimate, at most half, and the recovery count goes back to 0. Once the queue pair has moved on to additive increase, a reduction instead first lowers the target rate by one additive step, then moves the current rate halfway toward the new target rate, and takes one off the recovery count. When the count falls below FastRecoveryAttempt, the queue pair is back in fast recovery: its next increase leaves the target rate alone, and a reduction that comes before increases bring the count back to FastRecoveryAttempt is a fast recovery reduction. A reduction half with no CNP leaves the rate alone.
5. Increase. An increase half that has seen no CNP since the last reduction half makes an increase. An increase half that did see CNPs makes one only on every (NumOfControlLoopsPerIncrease + 1)-th such half, every sixth at the default 5.
6. Fast recovery. While the recovery count is below FastRecoveryAttempt, each increase moves the current rate halfway toward the target rate, without changing the target, and adds one to the count. Each halves the gap between them.
7. Additive increase. Once the recovery count reaches FastRecoveryAttempt, each further increase first raises the target rate by the additive step, up to line rate, and then moves the current rate halfway toward it. The step is fixed when the episode begins:
   additive step = target rate at the start of the episode × (1 − InitialTargetRatePercentage) ÷ NumberActiveIncreaseSteps
   
8. The floor. While a queue pair is congested, its rate never goes below MinimumRatePercentage × DataRate, whatever the reductions would give.
9. The return to line rate. When an increase brings the current rate within 0.1 Gbps of line rate, the episode ends: the queue pair runs at line rate again, the loop stops, and the next CNP starts a new episode.

​

How the rate is applied

​

How each attribute shapes the loop

​

Worked example: the loop’s timing and steps at the defaults

control interval               ControlIntervalDurationNs            = 45 µs
loop period                    45 µs ÷ 2                            = 22.5 µs, alternating reduction and increase
target rate at the start       the rate when the episode begins     = 400 Gbps
additive step                  400 Gbps × (1 − 0.90) ÷ 10           = 4 Gbps
floor                          400 Gbps × 0.10                      = 40 Gbps
increase while CNPs continue   every (5 + 1)th increase half        = at most once every 6 × 45 µs = 270 µs
end of the episode             current rate within 0.1 Gbps of 400 Gbps

​

What congestion control records