Release 0.1.2544

​

At a glance

• Logs during a run. Simulation logs can be read while a simulation on the optimized execution path is still running.
• Clearer failure reasons. They now name out-of-memory failures, the simulator’s own fatal error, and exhausted cloud capacity.
• Topology report. It names each component’s POD and rack, and each link’s plane and scale-up group.
• Corrections. Several reports and multi-process timing are corrected; see the note below.
• API specification. workloadMetadata is fully described, and a public OpenAPI file is published with the documentation.
• Security. The release includes security updates.

Some results change in this release. Check these before comparing runs with 0.1.2376:

• Multi-process simulations are now reproducible. Messages between simulator processes that arrive at the same simulated time are now handled in a fixed order, so repeated runs of a configuration give the same results. Runs spread across several simulator processes can differ slightly from earlier releases.
• Scale-up timing across simulator processes is corrected. A scale-up packet whose receiver ran in another simulator process arrived late, by the topology’s largest link delay. Two concurrent transfers between the same pair of ranks could also corrupt each other. Runs whose scale-up groups span simulator processes now give different, correct results.
• The application-report DSCP column is corrected. It reported the low six bits of the QoS byte instead of the DSCP value.
• global-sim-report tier totals are corrected. ECN and PFC totals for components outside the named tiers are now counted under other.
• Chakra traces with an unroutable send or receive now fail. Previously those operations were dropped silently, and the run could complete without that traffic.
• topology-report gains eight columns at the end of each row. Existing columns do not move.

No defaults change. The new Chakra workload 4.1.0 settings apply only to optical scale-up groups.

​

Customer-reported issues resolved

​

WorkloadMetadata missing from the OpenAPI specification

• shape.rankCount;
• the parallelism degrees dp, tp, sp, ep and pp, through WorkloadShape and WorkloadParallelism.

• each parallelism degree, including cp and fsdp, which are not factors of the rank count;
• the rank-count rules;
• the descriptive members you can add.

​

Typed scale-up fabrics and optical circuit switching

ScaleUpGroupId, ScaleUpBandwidthGbps, ScaleUpPerHopSwitchLatencyNs, ScaleUpNpuCableLatencyNs, Type, NumOfPlanes

• Type is EPS or OCS-XBar, an optical circuit switch crossbar.
• NumOfPlanes is from 1 to 64. The group’s bandwidth is divided across its planes, and it must divide by NumOfPlanes with no remainder.
• Planes are set in the trace. Every scale-up send and receive of a typed group carries a scale_up_plane attribute that selects its plane, from 1 to NumOfPlanes. It is written as a JSON integer under uint64Val. The guide shows the exact form.
• On an OCS-XBar plane, a transfer first obtains a circuit to its receiver from the fabric’s optical circuit controller. A plane carries traffic to one receiver at a time. Later transfers to the same receiver use the circuit while it is held. ScaleUpPerHopSwitchLatencyNs is the latency of one pass through the crossbar.
• Rows with four fields are unchanged. An untyped row is an EPS group with one plane, and it needs no scale_up_plane attribute.

​

New in this release

​

Chakra mapping file generator

• Generate from a short config. chakra_mapping_generator/generate_mapping.py writes a mapping file from a short JSON config. It needs Python 3.10 or later.
• Placement and scale-up groups. It places rank i on server i. It can place ranks in scale-up groups stamped from templates you define.
• Examples. An examples folder holds three configs, each with the file it generates.
• Set rankCount to your traceset’s rank count. A file whose rank count differs from the traceset’s is rejected when you attach it.
• Typed groups are added by hand. The generator writes untyped scale-up groups; to use a typed fabric, add Type and NumOfPlanes to its Section II rows.

​

Simulation logs while a simulation runs

​

Clearer failure reasons

• Out of memory. On the optimized execution path, a simulation stopped for running out of memory now reports The simulation ran out of memory. Reduce its size, for example the number of ranks, or contact support. instead of simulator-runtime-error.
• Simulator fatal errors. On the optimized execution path, failureReason now carries the simulator’s own error message, instead of only simulator-runtime-error.
• Cloud capacity. When no availability zone has capacity for a launch, the simulation now fails as capacity-exhausted instead of provisioning-error.

​

Topology report: POD, rack, plane and scale-up group

• pod, rack and pod_config_name for each end of a link;
• the same three with a peer_ prefix, for the other end;
• plane;
• scaleup_group.

​

Stricter validation of traces and mapping files

• Unroutable sends and receives. A Chakra send or receive fails the simulation while it is being prepared if:
  • it names a process group the trace’s process-group metadata does not define; or
  • it has no process group and no peer.
• Section II rows with five fields, or more than six, are refused.
• A scale-up group with both latencies at 0 is refused.

​

API specification changes

• SimulationResults (GET /api/v1/simulations/{sim_id}/results) no longer declares summary or dashboardUrl. The endpoint never returned either, so responses are unchanged. Clients generated from the new specification no longer have the two fields.
• Transport summary. The API reference now documents mode=summary for roce-transport-stats and uet-transport-stats, which returns the whole-run transport view. It previously described this mode as not implemented.

​

Model catalog changes

​

Platform reliability and stability

• Terminate on the standard execution path. A terminate request now reaches the right simulation. Previously, if the simulation’s name had been used by an earlier, deleted simulation, the request could report success without stopping the run.
• Chakra simulations end sooner. They no longer wait a fixed 20 seconds while they gather collective statistics at the end of a run.
• No hang at the end of multi-process Chakra runs. A run in which some simulator process hosts no Chakra application no longer risks hanging there.

​

Security updates

​

Documentation

• Workload metadata: a new page, and a published OpenAPI file. See Customer-reported issues resolved above.
• Chakra mapping files: the generator, typed scale-up fabrics, and the scale_up_plane trace attribute.
• Model reference: new pages for the Scala MRC NIC, the Scala RDMA Application and the Scala Traffic Generator.
• Simulation output files: audited against the simulator. The reference now lists every record written by default, including the Chakra records, uet-transport-stats, mrc-transport-stats, scala-switch-load-balancing-stats and rdma-perf. It also covers the new topology-report columns.
• Authentication: how to keep a bearer token valid past its one-hour lifetime.
• Examples and API text: the quickstart and run-simulation examples, and descriptions that no longer matched what the platform serves, are corrected.

​

Not included in this release

• Chakra workloads on the Scala MRC NIC. A Chakra simulation on MRC NICs does not yet run to completion.
• Mapping files above 4,096 ranks. Mapping-file support covers simulations of up to 4,096 ranks. A larger simulation with a mapping file attached fails with Mapping file <name> (<id>) could not be delivered.
• Mapping files with the platform’s own Chakra tracesets. A simulation from such a configuration fails before the simulator starts. Mapping files work on tracesets you upload.
• Summary metrics for the largest result sets. Summaries for the largest runs, such as 8,192 ranks, can still be refused as too large. Raw result files remain available through the results download operations.
• Pre-simulation run-time estimation. There is still no supported way to tell in advance whether a trace will exceed the configured simulation time limit.
• ECN-marked packet counts received at the NIC. NIC-side receive counters for ECN-marked packets are not yet reported. Switch-side ECN information remains available.
• Mixed NIC types in a Chakra simulation. All servers in a Chakra simulation must use the same NIC type.

​

Upgrade notes

• Automatic updates. The updated simulator build and the Chakra workload 4.1.0 model are applied with the platform upgrade. Existing configurations are unaffected unless they use the features above.
• Optical scale-up groups. To set the optical switching delay or control latency, use the Chakra workload 4.1.0.
• Check results comparability against the list in At a glance before comparing runs with 0.1.2376.
• Update column-count checks. Readers that check the column count of topology-report must allow for the eight appended columns.
• Regenerate API clients to pick up the SimulationResults change and the fully described workloadMetadata.
• Fix unroutable sends and receives in your traces. Traces whose sends or receives name an undefined process group, or have neither a process group nor a peer, now fail. Fix the trace’s process-group metadata or the operations’ peers before rerunning.