At a glance
This release adds the Scala UET NIC, a model of a NIC running Ultra Ethernet Transport. Scala Switch 3.7.0 adds a UET buffer policy, packet trimming and dynamic load balancing. Simulator software moves to 4.8.1, and the RoCE NIC’s retransmission timer is reworked. Other changes:- Chakra simulations of 8,192 ranks or more no longer fail at start.
- On the optimized execution path, a simulation can be terminated at any point before it finishes, and it keeps the results it has produced.
- Traceset uploads now declare their rank count.
- On tracesets you upload, simulations can now run with a custom Chakra mapping file, which sets rank placement and scale-up groups.
- Validation of configurations and traces is stricter.
- New results query modes return transport, utilization, fairness and completion-time data over time.
- The release includes security updates.
Some results change in this release. Check these before comparing runs with 0.1.1828:The new Scala Switch attributes default to off, so switch behaviour is unchanged for existing configurations.
- RoCE NIC loss recovery. The RoCE NIC’s retransmission timing and NAK handling are reworked; see Scala RoCE NIC 4.6.0. Runs in which NAKs, retransmission timeouts or duplicate packets occur produce different results.
- Chakra computation time is corrected. The computation time in
chakra-completion-stats(total_chakra_computation_time_us) and in the results summary was over-counted for any rank that completed more than one computation. Reported values are now lower; simulated timing is unchanged.- The
cdfsummary now sums ranks. A multi-rank progress curve now reaches 1.0. Previously ranks were averaged and the curve levelled off at 1/N.- Chakra traces that name an undefined process group now fail. Previously their collectives were dropped silently, and the run could complete without that traffic.
- Two reports gain columns at the end of each row.
roce-transport-statsandscala-switch-load-balancing-statseach gain four columns. Readers that check the column count must be updated. Existing columns do not move.- Remote trace logging is off by default for new configurations. See Remote trace logging is off by default.
Customer-reported issues resolved
8K-rank simulation failed
Before the simulator reads a Chakra trace, the platform annotates each operation with its position and size. The size field held six digits, so any operation of 1 MB or more misaligned every offset in the annotated trace. The simulator then stopped on the first operation with a parse error (rapidjson parse error). Traces of 8,192 ranks or more commonly contain operations that large.
The size field now holds nine digits. If an operation is still too large for it, the run fails while it is being prepared, and the message names the operation.
No change to your tracesets is needed. Annotation runs as part of every simulation, so rerunning picks up the fix.
Large multi-rank runs could also fill the compute disk with a debug log. That log is now off by default; see Remote trace logging is off by default. If an existing configuration has it set to true, set it to false before rerunning a large simulation.
Terminate API works only in running state
On the optimized execution path, a simulation can now be terminated at any point before it finishes:- while it is queued, including while its compute is still being acquired;
- while it is being prepared;
- while it is running.
- The request returns the simulation as it is at that moment.
- A run that has started records
terminationRequestedAtand settles asterminated, typically within 15 minutes. - A run that had not started settles as
terminatedin about a minute, with no results.
- A run that has started records
- Results produced before the stop are kept. They can be listed and downloaded through the existing results operations. Data the simulator still held in memory is not written out, and a file that was being written at the moment of the stop may be incomplete.
- Finished or finishing simulations are unaffected. A request for a simulation that has already finished, or is delivering its results, is accepted and has no effect.
List models API reported Rack and Pod as switches
GET /api/v1/models now reports type: "rack" for ScalaRack and type: "pod" for POD. This matches the model detail and components operations. type=rack and type=pod are also accepted as filters.
Clients generated from the previous API specification may reject the new values. See Upgrade notes.
Scala UET NIC and UET support in the Scala Switch
The ScalaUETNIC400G model (4.6.0) is now available in the model catalog. It models a 400 Gb/s NIC running Ultra Ethernet Transport (UET). It offers:- packet spraying across equal-cost paths, with a choice of spraying methods;
- UET congestion control, including Quick Adapt;
- failure detection;
- optional per-packet monitoring.
- UET buffer policy (
SharedBufferManager.UETPolicyEnabled). When it is on, every NIC in the topology must be a UET NIC or an MRC NIC. Otherwise the simulation stops before it starts. Turning the policy on while PFC is enabled logs a warning. - Packet trimming (
PacketTrimmer.TrimmingEnabled, withTrimmingStatsEnabledfor its statistics). Trimming requires the UET buffer policy, and PFC turned off on every traffic class. With the switch’s default PFC settings, a configuration that turns trimming on stops before it starts.
uet-transport-stats by default. The per-packet uet-monitor record is written when EnableUETMonitor or one of its triggers is on. A time series of uet-transport-stats is available through the results API; see Results, logs and uploads.
Chakra workloads run on UET NICs. All servers in a Chakra simulation must use the same NIC type.
New in this release
Simulator software 4.8.1
Simulator software 4.8.1 carries the model updates below and the new UET and MRC NIC models.Scala RoCE NIC 4.6.0
Retransmission timing per queue pair. The RoCE NIC now times outstanding requests with one InfiniBand-style transport timer per queue pair, replacing the per-packet timers. In addition:- a NAK now acknowledges the packets before it;
- duplicate packets are acknowledged again;
- acknowledgments for packets never sent, and stale transmit payload, are discarded.
RetransmitTimeout keeps its default of 64 ms.
roce-transport-stats gains four columns, appended at the end of each row:
Scala Switch 3.7.0: dynamic load balancing
The switch can now move flows between the members of an equal-cost set. The default,none, keeps the previous behaviour, so 3.6.0 results are unchanged.
scala-switch-load-balancing-stats gains paused_skips and paused_selections, with their interval columns, at the end of each row. They count decisions that skipped a member because PFC had paused it, and decisions that selected a member while it was paused.
Uploaded tracesets declare their rank count
- New field on upload requests.
POST /api/v1/tracesets/upload-urlnow takesworkloadMetadata, which declares the traceset’s rank count in one of two ways:shape.rankCount;- parallelism degrees
dp,tp,sp,epandpp, multiplied together. An absent degree counts as 1.
- The declaration is required for any upload that includes trace files. A missing or invalid declaration returns
400VALIDATION_ERRORnaming the field. - Client support. The Python client takes
workload_metadata, and the command-line tool takes--workload-metadata. - Uploaded tracesets can be used right away. An upload completes as
ready, and the traceset can be attached to a configuration immediately. Previously, attaching an uploaded traceset could be refused with a conflict that asked you to retry, and retrying did not help. - The count is checked at launch. Before trace processing begins, the declared count is compared with the per-rank traces in the traceset. A mismatch fails the simulation with
Declared rank count N does not match the M per-rank traces found in the traceset. - A traceset with no rank count cannot be attached. Attaching it now returns
422(previously409). Upload it again with a declaration.
Chakra mapping files
A Chakra mapping file sets which server runs each Chakra rank, replacing the default in-order placement. It can also declare scale-up groups. Traffic inside a group uses the scale-up fabric instead of the scale-out network.- Upload once, attach to a configuration. Every simulation created from that configuration is launched with the file, and records it in
appliedMappingFile. - Manage files through the API. Mapping files can be listed, downloaded byte for byte, and deleted. Detaching a file restores in-order placement.
- Mapped simulations run to completion on tracesets you upload. The traceset must have a rank count (see above). The file must place every rank on its own Chakra-capable server in the topology.
- A run never falls back to in-order placement. A file that cannot be delivered to the simulator fails the simulation, with the reason.
- The platform owns the mapping parameters while a file is attached. It sets
UseMappingandMappingFileName, and a change to either returns422MAPPING_PARAMS_LOCKED. - Files can be up to 64,000,000 bytes.
- You provide the file. The platform does not generate mapping files. Write one, or generate it with your own tooling.
Stricter validation of configurations and traces
These checks can reject configurations or traces that earlier releases accepted. In each case the configuration or trace could not have run correctly. The problem is now reported before or at the start of the run, with the reason.- Parameters are checked against the model catalog. Configurations and templates are rejected with
400, listing every issue, for:- undeclared parameters;
- wrong types or enum values;
- values out of range;
- malformed units or lists.
- Range checks are unit-aware for time, data-rate, byte and queue-size values. Previously only time values were checked.
- Chakra traces whose collectives name an undefined process group fail while the simulation is being prepared.
- A corrupt zip archive anywhere in a traceset fails the simulation on the standard execution path. Previously a corrupt archive could be skipped silently.
- Refused launches return a client error. A launch refused because of its configuration now returns
400LAUNCH_CONFIG_INVALIDwith the reason, or422LAUNCH_REJECTED, instead of503. - Compute requests above 1,024 instances are rejected with
400“Invalid compute configuration”.
Results, logs and uploads
New query modes onGET /api/v1/simulations/{sim_id}/results/data:
A metric and mode that do not combine return
400 UNSUPPORTED_COMBINATION. Switch bandwidth by tier is available from GET /api/v1/configurations/{config_id}/versions/{version}/switch-bandwidth.
Changes to existing results behaviour:
- Large result sets. If a summary takes longer than the request allows, computing it now continues, and the result is cached for later requests. Large result files are downloaded in parallel parts.
- Empty metrics. A run that finished at least 60 minutes ago and has no data for a metric now returns
200with an empty summary, instead of404. - Tier filter on the performance summary. Every completion is recorded at a host. So for
perfandchakra-perf,tier=ssw,fsworrswnow returns an empty summary (count 0), instead of repeating the whole-network values.tier=allandtier=nicare unchanged. - Logs. For log files over 8 MiB on the optimized execution path, the logs operation now returns the last 8 MiB, where failures appear, with a notice that the start is not shown. A pagination cursor obtained before the upgrade gets one
400; start the walk again. - Sorting the simulation list.
GET /api/v1/simulationsaccepts a single sort field. A cursor replayed under a different sort returns400. - Uploads. Upload part URLs are valid for one hour, up from 15 minutes. Responses include
sizeandpartUrlsExpireAt. Completing an upload of more than about 1 GB no longer stays stuck incompleting.
Remote trace logging is off by default
EnableRemoteTraceLogging, a global simulation parameter, now defaults to false in the shipped templates and in new configurations. The debug file it enables, {simulation}-scala-remote-trace-log-{shard}.csv, is therefore no longer produced by default. On large multi-rank runs that file can reach hundreds of gigabytes, enough to fill the run’s disk and stop the simulation.
Saved configurations keep their stored value. See Upgrade notes.
Scala MRC NIC (in testing)
The ScalaMrcNic model (4.0.5) is now in the model catalog, marked Beta. It models a NIC with Multi-path Reliable Connection (MRC) transport. The model is still in testing, but it can be used now, including with Chakra workloads and the RDMA applications.Model catalog changes
The shipped default templates now reference simulator software 4.8.1 and set remote trace logging off.
Platform reliability and stability
- Fewer provisioning failures. When the cloud provider lacks capacity in one availability zone, provisioning now retries in the others before failing.
- Concurrent requests. Concurrent simple-simulation requests for the same workload no longer fail each other with a server error.
- Long-running Python scripts. The Python client now refreshes its access token, so scripts that run for over an hour no longer fail with
401. - Standard execution path:
- A finished simulation no longer shows
runningfor hours when a storage sync stalls. - A launch that is denied access to its storage now fails promptly, instead of after about 15 minutes of retries.
- A finished simulation no longer shows
- Terminate after a platform restart. Terminate requests no longer return
503after a platform restart in which the messaging service started late.
Security updates
This release contains security updates. We recommend upgrading. In keeping with our disclosure practice, we do not publish details of security fixes. Customers with a specific compliance question should contact their Scala representative.Documentation
- Model reference pages for Scala Switch 3.7.0, Scala RoCE NIC 4.6.0, the Chakra workload 4.0.3 and Scala UET NIC 4.6.0. The model pages now sit in the Documentation tab, below the API Reference.
- A new Chakra mapping files guide.
- Output-file reference: the new
roce-transport-statscolumns, and the remote trace logging default with its disk-use warning. - API reference and changelog: updated for this release.
Not included in this release
The following remain open and are not addressed by 0.1.2376:- Mapping files with the platform’s own Chakra tracesets. A mapping file can be attached to a configuration that uses one of the platform’s Chakra tracesets. A simulation from that configuration currently fails before the simulator starts, with
Mapping file <name> (<id>) could not be delivered. Mapping files work on tracesets you upload. - Summary metrics for the largest result sets. Computing a summary now continues after a request times out (see above), so a later request returns the metrics. Summaries for the largest runs, such as 8,192 ranks, can still be refused as too large. Raw result files remain available through the results download operations.
- Pre-simulation run-time estimation. There is still no supported way to tell in advance whether a trace will exceed the configured simulation time limit.
- ECN-marked packet counts received at the NIC. NIC-side receive counters for ECN-marked packets are not yet reported. Switch-side ECN information remains available.
- Mixed NIC types in a Chakra simulation. All servers in a Chakra simulation must use the same NIC type.
Upgrade notes
- Automatic upgrades. Simulator software 4.8.1 and the updated models are applied with the platform upgrade.
- Declare a rank count on traceset uploads. API callers add
workloadMetadata. Python client users passworkload_metadata, and command-line users pass--workload-metadata. Tracesets uploaded earlier that report no rank count must be uploaded again with a declaration;rankCountDerivedis absent onGET /api/v1/tracesets/{id}for those tracesets. - Regenerate API clients, or update hand-written ones, for these changes:
- model types
rackandpod; - component-schema
min/maxvalues are now objects withvalueandunit; - single-field sort on the simulation list;
- the launch error codes
LAUNCH_CONFIG_INVALID(400) andLAUNCH_REJECTED(422).
- model types
- Turn off remote trace logging in saved configurations. Set
EnableRemoteTraceLoggingtofalsewithPATCH /api/v1/configurations/{config_id}/parameters, especially before large multi-rank runs. - Expect stricter validation. Configurations with undeclared parameters or out-of-range values, and Chakra traces with undefined process groups, now fail before the run instead of during or after it.
- Check results comparability against the list in At a glance before comparing runs with 0.1.1828.
- Update column-count checks. Readers that check the column count of
roce-transport-statsorscala-switch-load-balancing-statsmust allow for the four appended columns. - Restart log pagination that was in progress before the upgrade.