Current limitations
A simulation created from a configuration with a mapping file attached runs to completion on a traceset you upload, and records the file inappliedMappingFile. These limits remain:
- Generating a mapping file. The platform does not generate a mapping file for a configuration. Write one yourself, or generate one with the mapping file toolkit Scala provides (see Generate a mapping file), and upload it.
- Tracesets you upload need a rank count. Attaching a mapping file requires the configuration’s traceset to have a rank count. A traceset you upload reports its rank count as
rankCountDerivedonGET /api/v1/tracesets/{id}; when that field is present, the traceset has a rank count. An upload of trace files declares its rank count inworkloadMetadata, asshape.rankCountor as parallelism degrees (dp,tp,sp,ep,pp, multiplied together, with an absent degree counting as 1), andPOST /api/v1/tracesets/upload-urlrefuses one that declares neither. A traceset uploaded before rank counts were declared may report no rank count. Behind an uploaded traceset with no rank count, the mapping-file attach answers422UNPROCESSABLE_ENTITY, and retrying does not change the answer: upload the traces again with a declaration. The traceset attach refuses the same traceset with the same422, although a configuration create or update can still name it asactiveTraceset. - The platform’s own Chakra tracesets (coming soon). These have a rank count, so a mapping file attaches to a configuration that uses one. A simulation created from that configuration currently fails before the simulator starts, with the
failureReasonMapping file <name> (<id>) could not be delivered, until an update to the platform’s simulation hosts is in service.
The file format
A mapping file is plain text with one required section and one optional section.- Everything from a
#to the end of its line is a comment. Blank lines are ignored. - Fields are separated by spaces, commas, or any run of both. Tabs are not separators, and the file must use Unix (LF) line endings, not Windows (CRLF).
- A line may carry at most 127 bytes before its first comment, and at most 4,096 bytes in total.
- A comment block at the top of the file must use
#. A line starting with@is reserved for Section II.
Section I: rank placement
Section I lists one rank per line, with exactly two or exactly four fields:
All fields are non-negative base-10 integers that fit in 32 bits, and
4294967295 is not accepted as a serverId. The ids must cover the contiguous range 0 through rankCount - 1, each exactly once, and no two ranks may name the same serverId. A rank with no scale-up fields belongs to no group, and all of its traffic uses the scale-out network.
Section II: scale-up group parameters (optional)
Section II begins at the first line that starts with@, after at least one Section I entry. Every line starting with @ is a comment, so the marker line can carry a heading, and inside Section II both # and @ start a comment anywhere in a line. Each remaining line declares one group, with four fields or six fields:
A line with four fields is an untyped group. An untyped group is an
EPS group with one plane. A line with six fields is a typed group. A line with five fields, or with more than six fields, is refused.
The bandwidth in bit/s is ScaleUpBandwidthGbps × 1e9, truncated to a whole number. For a typed group it must divide by NumOfPlanes with no remainder: 100 Gbit/s on 3 planes is refused, and 99 Gbit/s on 3 planes is accepted. On an OCS-XBar group, ScaleUpPerHopSwitchLatencyNs is the latency of one pass through the crossbar.
A typed group (EPS or OCS-XBar, with any NumOfPlanes, including 1) needs a scale_up_plane attribute on every scale-up SEND and RECV node of the workload. A scale-up node is a node whose partner is in the same scale-up group. The attribute selects the plane of the transfer, and its value is an unsigned integer from 1 to NumOfPlanes. In the Chakra trace, scale_up_plane is an entry in the node’s attr list. The entry has name set to scale_up_plane, and its value is a JSON integer under the key uint64Val. The value is not a string. A string under uint64Val (the form comm_size uses) stops the run, and so does a value under any other key. For plane 2, the entry is:
- has no
scale_up_planeattribute, - has a value that is not a JSON integer under
uint64Val, or - has a value outside 1 to
NumOfPlanes.
scale_up_plane attribute, and it ignores one.
The two latency fields are also bounded as a pair, using the group’s switch latency s and cable latency c, both in picoseconds:
- The smallest crossing,
1 × s + 2 × c(one switch hop and two cable segments, the crossing between two ranks of the same subgroup), is at least 1 ps. A group with both latencies at 0 is refused. A group with one latency at 0 and the other positive is accepted. - The largest round trip,
2 × (3 × s + 4 × c), is not more than 9223372036854775807 ps, the largest time the simulator represents. A switch latency of about 1.5e15 ns, or a cable latency of about 1.15e15 ns, reaches this limit on its own, well below the per-field ceiling in the table.
Examples
An 8-rank file with no scale-up groups, placing the ranks in reverse order. The comment on each row is free-form; here it records where the server sits in the topology:OCS-XBar group of 8 planes. Its switch latency is 0, which the pair rule accepts because its cable latency is positive:
When each rule is checked
Every rule above is checked when the upload completes. A file that breaks one is refused with422 MAPPING_FILE_INVALID, the message names the first offending line (line <n>: <reason>) or the first missing rank, and nothing is stored.
Two things depend on a configuration, so they are checked later:
- Rank count. Attach requires the file’s
rankCountto equal the rank count of the configuration’s traceset. - Chakra-capable servers. Whether each
serverIdnames a Chakra-capable server is a fact about the topology. A file that names another server uploads and attaches, and the simulation fails when the simulator starts, with the simulator’s error naming theserverId.
Generate a mapping file
The platform does not generate a mapping file for a configuration. Scala provides a mapping file toolkit for writing one: a generator script,chakra_mapping_generator/generate_mapping.py, that writes a mapping file from a short JSON config; sample configs to start from, in chakra_mapping_generator/examples; and this documentation. Request the generator and its sample configs from Scala.
The generator places rank i on serverId i, and can place the ranks in scale-up groups stamped from templates you define. Every file it writes follows the rules in The file format. It needs Python 3.10 or later and nothing outside the standard library.
Run the generator
--output, the file is written to standard output. When the config breaks a rule, the generator writes nothing, prints error: <field>: <reason>, and exits with status 2.
The generator does not read your topology. It assumes servers 0 through rankCount - 1 are Chakra-capable servers in it. As with any mapping file, a serverId that is not Chakra-capable is reported when the simulation starts.
Start from an example
Thechakra_mapping_generator/examples directory holds three configs, each with the file it generates beside it:
Copy the one closest to what you need, set
rankCount to your traceset’s rank count, and run the generator on your copy:
rankCount that differs from the traceset’s rank count makes the attach fail with 422 MAPPING_RANK_COUNT_MISMATCH.
The config file
Template fields, in
defaults or in an override:
The three numeric fields accept the values Section II accepts (see Section II). Each template must also meet the bound on the two latencies as a pair, so a template with both latencies at 0 is refused. After overrides are applied, every template must have
size, bandwidthGbps, perHopSwitchLatencyNs and npuCableLatencyNs. The generator has no built-in bandwidth or latency. Unknown and repeated keys are refused, so a misspelled field fails instead of being ignored.
How the config expands
- Build the templates.
groupCountcreates templates numbered0togroupCount - 1. Each starts as a copy ofdefaults, and its entry inoverrides, if it has one, replaces the fields that entry names. - Stamp groups in turn. Ranks are taken in order. Template 0 stamps the first group from the next
sizeranks, template 1 the next group from the ranks after that, and so on. After the last template the cycle returns to template 0. Every block is a new group, numbered0, 1, 2, …in the order it is created, so groupguses templateg mod groupCount. - Number the subgroups. Inside each group, ranks are split in order into subgroups of
subgroupSize. Subgroup ids restart at 0 in every group, and the last subgroup is smaller whensubgroupSizedoes not divide the group. - Stop at the last rank. When the ranks run out partway through a block, that last group has fewer ranks than its template’s
size. It is still a group. - Write the file. Section I has one row per rank:
rank,serverId,ScaleUpGroupId,ScaleUpSubgroupId, withserverIdequal to the rank. Section II has one row per group created, carrying its template’s bandwidth and latencies. Without templates, each row isrank,serverIdand there is no Section II.
Worked example
The config above, with 18 ranks and two templates. Step 1, the templates. Template 0 has no override, so it is the defaults. Template 1 overrides three fields and inherits both latencies.
Steps 2 to 4, the groups.
Step 5, the file.
More cases
- With
rankCount4and the samescaleUp, only group 0 is created, with ranks 0 to 3: ranks 0 to 2 in subgroup 0 and rank 3 in subgroup 1. Section II has one row. - With
rankCount20, group 3 is full (ranks 16 to 19). - With no
scaleUp, orgroupCount0, every row isrank,rankand the file has no Section II. - A whole number in a numeric field is written as a decimal:
800is written800.0, and0is written0.0. - A file larger than 64,000,000 bytes, the upload limit, is refused before anything is written.
Upload, attach, and simulate
1. Set up your environment
2. Request upload URLs
Send the file’s size in bytes and the workspace it belongs to.name is optional and defaults to chakra-mapping.txt. It must be 1 to 255 bytes of A-Z, a-z, 0-9, ., _ and -, must not be ., and must not contain .. anywhere. sizeBytes is an upper bound of at most 64,000,000 bytes (the 64 MB per-file limit), and description holds at most 2,048 characters. The name is display metadata only: inside the simulator the file is always called chakra-mapping.txt.
3. Upload the bytes
PUT each part to its presigned URL and keep the ETag response header of each one.
4. Complete the upload
Send the upload id and each part’s number and ETag. The platform validates the file, derivesrankCount and scaleUpGroupCount, stores the file, and answers 201 with the new mapping file. sizeBytes in the response is the stored file’s actual length.
200 with that existing file and duplicateOfExisting set to true. Uploading the bytes of a deleted file creates a new mapping file with a new id; a deleted file is never revived.
5. Attach it to a configuration
- a configuration outside the template workspace (
400otherwise); - at least one application with
trafficTypechakra(422MAPPING_REQUIRES_CHAKRA_APP); - an attached traceset (
409MAPPING_REQUIRES_TRACESET) that has a rank count: a traceset without one is answered422UNPROCESSABLE_ENTITY, attach changes nothing, and retrying does not change the answer; - a mapping file in the configuration’s own workspace that has not been deleted (
404otherwise); - a mapping file whose
rankCountequals the traceset’s rank count (422MAPPING_RANK_COUNT_MISMATCH, naming both counts and the missing or extra ranks).
If-Match is optional. The configuration’s ETag is its version number in quotes, and a stale one answers 412. Attaching writes a new configuration version whose activeMappingFile names the file (excerpt):
UseMapping to true and MappingFileName to chakra-mapping.txt in every Chakra application’s ChakraConfiguration parameters. While the file is attached, the platform owns those two parameters: a configuration update, an application-parameter update, or a new Chakra application that would change them is refused with 422 MAPPING_PARAMS_LOCKED. A traceset with a different rank count is refused with 409 TRACESET_CONFLICTS_WITH_MAPPING. A configuration update whose body carries activeMappingFile is refused with 400: attach and detach are the only ways to set it. To make any of these changes, detach the mapping file, make the change, and attach it again.
One mapping file can be attached to any number of configurations in its workspace, and each configuration holds its own reference.
6. Create a simulation
Every simulation created from the configuration is launched with the attached file (see Current limitations).appliedMappingFile from the moment it is created, and keeps naming it after the file is deleted, with deleted reporting that state. A simulation just created (excerpt):
appliedMappingFile, and its ranks are placed in order.
Creating a simulation is refused, and no simulation is created, when:
- the attached mapping file has been deleted:
409MAPPING_FILE_DELETED_REFERENCE, naming the file. Detach it, or attach another, and create again; - the attached mapping file’s name contains
..or a shell metacharacter:400VALIDATION_ERROR, naming the field. Upload rejects such names, but a file stored before that rule keeps its name; upload it again under another name and attach it in place of the old one.
failureReason reads Mapping file <name> (<id>) could not be delivered. On some runs a long <name> is shortened and ends in …; the id is always given in full. A failure the platform does not attribute to the mapping file, such as a storage or infrastructure fault while staging it, reports a general failure reason instead.
7. Check the placement the run used
When a mapped run completes, the simulator has written the placement it applied to one CSV result file whose name contains-chakra-mapping-. List the run’s CSV results, find that file by name (page through with next if needed), and download it through its downloadUrl or the download-url endpoint:
ChakraId,Hostname,NicNodeID,ServerNodeID,ServerID,Core,IPAddress, plus ScaleUpGroupId,ScaleUpSubgroupId when the run uses scale-up groups. The first column is the rank and the fifth is the serverId it ran on, so each row can be compared with the Section I entry you uploaded. Rows are not in rank order. See Simulation Output Files for downloading result files.
Browse, download, and delete
List the live mapping files in a workspace, newest first.workspaceId is required, and limit is at least 1:
hasMore is true, pass the returned nextCursor as the next query parameter to fetch the following page.
Fetch one mapping file’s details, or a presigned URL (url, valid until expiresAt) that serves its exact uploaded bytes:
204 whether or not a configuration refers to the file; deleting it again answers 404:
- Simulations already created are unaffected. They keep the file they were created with, and their
appliedMappingFilereadsdeleted: true. - A configuration that refers to the file keeps its
activeMappingFile, which readsdeleted: true, and creating a simulation from it is refused until you detach the file or attach another. - Fetching the file or its download URL answers
410MAPPING_FILE_DELETED.
Errors
All errors use the shared error shape: