Validator Contract

A validator is an executable - an OCI image in production, any command during development - that implements five subcommands. The service talks to it through a job directory (passed via --job) and JSON documents; nothing else crosses the boundary. Validators are stateless: every invocation is a pure function of its command line and the job directory contents.

Validators reach the server as published container images: see Challenges in the Client API for the publish, revise, and retire endpoints. There is no in-repository registry file.

This contract is a strict, closed schema. contract_version must be "3"; unknown fields are rejected. Contract v2 manifests are not accepted by this server.

Problem, task, and view

The validator author declares one or more problems P. Each problem has a human-readable title, one output requirement, and one sweep axis. The server and web UI then reason about concrete tasks P+s, where s is one element of the problem's sweep vector. A view is only an input mode for generating and scoring that same concrete task; it does not change the identity of the problem or the task.

The public structure is:

• validator -> problems
• problem -> concrete tasks
• task -> s plus the set of allowed views

The five commands

describe

Print the validator manifest as JSON on stdout. The manifest declares the validator's name and version, its views, its problems, the output file expected for each problem, and the metrics used for ranking and display.

generate

generate --task <task_id> --view <view_id> --seed <hex> --job <dir>

Create a fresh dataset. The validator writes the participant-visible challenge into <dir>/public/ and everything needed for scoring into <dir>/private/. The service pre-creates both directories (and <dir>/submission/) before the call; the validator only writes files into them.

Two hard rules:

• generate must be byte-deterministic: the same (task, view, seed) triple must produce byte-identical files.
• public/ must never contain the seed string, answer keys, or any other material needed to score the submission - it is served to the client verbatim as a tar archive.

score

score --job <dir>

Read the task and view metadata from the files generate stored in <dir>/private/, then read <dir>/private/ and <dir>/submission/, compare them, and write the verdict to <dir>/result.json - or to <dir>/error.json when the submission is invalid.

self-test

self-test --job <dir>

Run one Validator-owned, known-valid scoring example and write the same result.json that score writes. The fixture and any answer material stay inside the Validator image; the command must not print or place them in public/. Publication requires self_test: true in the manifest and a successful self-test whose metrics and details satisfy the declared contract. This prevents a Validator that rejects every submission from passing merely because it rejects the empty conformance fixture correctly.

artifact

artifact --task <task_id> --artifact <artifact_id> --job <dir>

Materialize one public artifact declared by that task. Write the requested content to <dir>/public/artifact and a JSON provenance document to <dir>/public/provenance.json. Both must be regular files no larger than 16 MiB; all artifacts in one validator are capped at 64 MiB total. The publishing worker generates and stores them before the revision goes live, so the API never runs validator containers while serving a download.

The job directory

The service owns the job directory; the validator only ever reads and writes inside it.

Exit codes

Before finishing with exit code 2, score must write error.json explaining what was wrong with the submission.

Packaging as an image

In production the validator is an OCI image, and the image's ENTRYPOINT is the contract CLI: the service appends only the subcommand and its flags, so docker run <image> describe must print the manifest.

The hosted service at bench.qudeleap.com runs linux/amd64 images. Multi-platform images must include that platform; an arm64-only image is not sufficient. Self-hosted deployments must document their own worker platform.

• Job directory. The service bind-mounts the job directory at /job and passes --job /job. Do not assume any other path, and write nothing outside it.
• `describe` runs with no job mount. It is invoked before any job exists - during publishing, to read the manifest - so it must not touch /job.
• No network. Every command runs with --network=none. Data files, circuits, and reference tables must be baked into the image; nothing is downloaded at generate or score time.
• Reduced privileges. Containers run with --cap-drop=ALL and --security-opt=no-new-privileges, are removed after each command, and share nothing between invocations.
• Resource limits. The publisher may request cpus, memory, and pids_limit; each is clamped to the operator's configured maximum and applied with --cpus, --memory, and --pids-limit. Design for the clamped values, not the requested ones.
• Process output. describe stdout may contain at most 1 MiB of JSON. Every invocation has an 8 MiB combined stdout/stderr budget. The service reads both streams incrementally; exceeding either applicable limit fails the command and stops/removes its container. For diagnostics, only the last 64 KiB of each stream is retained (except describe stdout, which is kept complete). These limits apply to terminal output, not dataset, artifact or result files. Keep progress logging concise and write contract data to the designated files.
• Pullable and pinned. The image must be pullable by the server, with pull credentials when the registry is private. The publish resolves the reference to repository@sha256:<64 hex> once and runs that digest from then on, so moving a tag never changes a live benchmark.
• Affordable to check. Publish-time conformance runs the whole kit - three generate calls and one score - against every task/view pair by default. A validator with expensive tasks may declare a non-empty publish_conformance_pairs list of representative task/view pairs instead. Each declared view mode should be represented, and the validator's own CI must still run the full, unfiltered conformance kit. The whole publish shares one MIRRORQ_PUBLISH_TIMEOUT budget (default 14400 seconds). Scratch is not the constraint: only one pair's job directories exist at a time, so peak disk is the cost of a single task, not of the manifest.

Manifest format

describe prints one JSON object:

{
  "contract_version": "3",
  "name": "example-validator",
  "version": "1.0.0",
  "description": "Measure answer quality on blinded tasks.",
  "self_test": true,
  "views": [
    {"view_id": "view_a", "label": "View A"},
    {"view_id": "view_b", "label": "View B"}
  ],
  "problems": [
    {
      "problem_id": "example_problem",
      "title": "Example Problem (P)",
      "display": {
        "family": "example_family",
        "setting": "blinded",
        "p": [0.001, 0.002, 0.003]
      },
      "sweep_key": "p",
      "submission": {
        "filename": "answer.bin",
        "media_type": "application/octet-stream"
      },
      "tasks": [
        {"task_id": "example_problem_p001", "benchmark_version": "1.0.0", "program_timeout_seconds": 1920, "s": 0.001, "views": ["view_a", "view_b"]},
        {"task_id": "example_problem_p002", "benchmark_version": "1.0.0", "s": 0.002, "views": ["view_a", "view_b"]}
      ]
    }
  ],
  "primary_metric": {
    "name": "score_value",
    "label": "Score value",
    "direction": "lower_is_better"
  },
  "metrics": [
    {"name": "score_ci_low", "label": "Score CI low", "direction": "lower_is_better"},
    {"name": "score_ci_high", "label": "Score CI high", "direction": "lower_is_better"}
  ],
  "leaderboard": {
    "ranking": [
      {"source": {"type": "metric", "name": "score_value"}, "direction": "ascending"},
      {"source": {"type": "platform", "field": "runtime_seconds"}, "direction": "ascending"}
    ],
    "columns": [
      {
        "column_id": "rank",
        "label": "",
        "sources": [{"type": "platform", "field": "rank"}],
        "format": {"type": "integer"}
      },
      {
        "column_id": "solver",
        "label": "Solver",
        "sources": [{"type": "platform", "field": "registered_name"}],
        "format": {"type": "text"}
      },
      {
        "column_id": "score",
        "label": "Score",
        "sources": [{"type": "metric", "name": "score_value"}],
        "format": {"type": "number", "notation": "decimal"}
      }
    ],
    "details": [
      {
        "detail_id": "username",
        "label": "Username",
        "sources": [{"type": "platform", "field": "username"}],
        "format": {"type": "text"}
      },
      {
        "detail_id": "device",
        "label": "Device",
        "sources": [{"type": "platform", "field": "device_name"}],
        "format": {"type": "text"}
      },
      {
        "detail_id": "accepted_items",
        "label": "Accepted items",
        "sources": [{"type": "detail", "name": "accepted_items"}],
        "format": {"type": "integer"}
      }
    ]
  }
}

Top-level fields:

Each entry of views:

Each problem in problems:

Each task in tasks:

Each task artifact declares artifact_id, label, a safe ASCII filename, media_type, and optional description, role (download, guide, sample, or checker), and absolute HTTPS source_url without embedded credentials. Artifact IDs are unique within a task. When any artifact command fails validation, the whole challenge revision is rejected before it can go live.

preflight.sample_artifact_id must reference an artifact whose role is sample; preflight.checker_artifact_id must reference one whose role is checker. The public sample must be a tar or tar+zstd dataset bundle that uses the same task-program interface as a real run. The checker receives <sample-directory> <output-file> after its own path. MirrorQ CLI downloads both files from the platform but never runs the checker unless the participant explicitly supplies a checker runtime with mirrorq preflight --checker.

Each methodology stage declares a nonblank title and description, plus an optional short notation line. A methodology section declares a title, one to six paragraphs, and up to eight optional bullet items. Reference links declare label, an absolute HTTPS url without credentials, and an optional description. The platform renders these fields as plain text and safe links; validators do not supply HTML.

Display and sweep rules:

• display preserves author order and may contain scalars, arrays, nested arrays, and objects.
• sweep_key must name a field in display.
• display[sweep_key] must be a non-empty, finite, strictly increasing numeric vector with no duplicates.
• The s values in tasks must correspond one-to-one with that vector, in the same order.
• Every views entry must exist in the top-level views array.
• The output requirement is declared once on the problem, not repeated per task.

The primary_metric object:

Each entry of metrics:

Complete public profile

New public Validators should set public_contract_version to "1". This activates publication-time checks for the information and fixtures a participant needs before spending server capacity. The manifest must then provide:

• at least one resource and usage_terms; citations are optional when no publication represents the benchmark as a whole;
• a description for every view, problem input, metric, and artifact;
• a unit for every metric;
• an immutable benchmark_version, a format guide, and preflight sample and checker references for every task.

Existing Contract v3 Validators remain loadable when the field is omitted. They can migrate without changing their scientific definitions, then opt into the strict profile once all participant-facing declarations and public fixtures are ready.

The leaderboard object:

Each entry of ranking declares a source plus direction (ascending or descending). A source is either:

• {"type":"platform","field":"runtime_seconds"|"published_at"}
• {"type":"metric","name":"<declared metric name>"}

published_at is the time the owner most recently published the result. If a result is withdrawn and later republished, the new publication time is used.

Each column declares column_id, label, sources, and format. A source is either:

• {"type":"platform","field":"rank"|"registered_name"|"username"|"device_name"|"runtime_seconds"}
• {"type":"metric","name":"<declared metric name>"}

Each detail entry declares detail_id, a non-empty label, sources, and format. Its source may be:

• {"type":"platform","field":"registered_name"|"username"|"device_name"|"notes"|"runtime_seconds"|"attempts"|"reference_url"} (published_at is ranking-only)
• {"type":"metric","name":"<declared metric name>"}
• {"type":"detail","name":"<key written to result.details>"}

The platform renders these fields in declaration order in a generic key/value panel. It controls the layout; validators provide data declarations, not HTML. On desktop, short values are paired two per row while long, multiline, and structured values span the full row; mobile uses one field per row. The web page shows a fixed chevron outside the 1..6 main columns only when details is non-empty, and keeps at most one score expanded. Interactive details are web-only; mirrorq leaderboard still uses the declared main columns.

format is a strict object:

• {"type":"text"} - exactly one text platform source; detail entries may instead use one Validator result detail
• {"type":"integer"} - exactly one rank platform source or one metric source
• {"type":"number","notation":"decimal"|"scientific","precision":{...}} - exactly one metric source
• {"type":"percentage","decimal_places":0..12} - exactly one metric source
• {"type":"duration","decimal_places":0..3} - exactly one runtime_seconds platform source
• {"type":"range","notation":"decimal"|"scientific","precision":{...}} - 2..6 metric sources
• {"type":"fraction"} - exactly two metric sources, rendered as numerator/denominator
• {"type":"link"} - exactly one reference_url platform source in an expanded detail

When present, precision is {"mode":"decimal_places"|"significant_digits","digits":...}. number and range default notation to decimal. Only the column whose single source is platform rank may use an empty label.

For detail entries, integer may use platform attempts, numeric formats may also read a Validator result detail; range may combine 2..6 metric or result-detail sources, and fraction may combine exactly two such sources. Platform rank is never valid in the expandable panel. Missing optional platform values render as —.

Platform text fields have deliberately narrow meanings:

• registered_name is the solver/decoder name chosen at registration.
• username is the account's current chosen display name. Accounts that have not completed profile setup temporarily fall back to a GitHub login when one exists; email addresses are never substituted or published. A later display name change is reflected on old scores.
• device_name is the optional, self-reported mirrorq --device-name value. It is not verified by the platform.
• notes is optional public free text attached to one submission/run. It may include implementation details, links, hardware context, or other useful information and does not propagate to another run.
• attempts counts completed submissions for the same owner, solver, and task.
• reference_url is the solver's optional public HTTPS source-code, paper, or project link. Supplying a new link on a later submission updates old and new public rows for that solver.

Validation rules enforced by the server:

• views, problems, and every problem's tasks must all be non-empty.
• view_id, problem_id, and task_id must each be unique and non-empty.
• title is required and must be unique within the validator.
• Unknown fields at any level reject the manifest.
• A task may only use declared view_ids.
• Every task in a problem must use the same view set.
• Every leaderboard metric source must match either primary_metric.name or a declared secondary metric name.
• Ranking sources must be unique; registered_name and rank are never valid ranking sources. Neither username, device_name, nor Validator result details may influence ranking.
• column_id and detail_id values are unique within their respective lists.
• score must not require --task; it reads the task and view metadata from the job directory.
• The validator should treat the manifest as the source of truth; the server does not infer domain meaning from field names.

Score result

A successful score writes result.json:

{
  "metrics": {"score_value": 0.0035},
  "details": {"accepted_items": 350}
}

Error document

An invalid submission writes error.json before exiting with code 2:

{"message": "answer.bin has 12 rows, expected 20000"}

message (string, required) is the only field; it is shown to the client in the HTTP 400 response.

Versions and provenance

The manifest's top-level version identifies the validator software/image release. Each task's benchmark_version identifies the benchmark semantics of that concrete task. A validator image may therefore be released without retiring results for unchanged tasks. Bump benchmark_version only for every task affected by a change to its generation behavior, input/output protocol, allowed view IDs, task parameters, metric names or directions, or scoring behavior. Labels, descriptions, public artifacts, preflight helpers, and leaderboard presentation may change without invalidating existing scores. A new task starts with its own version. At load time, the service hashes a canonical semantic definition containing the v3 contract version, validator name, concrete task parameters, problem protocol, allowed view IDs, and metric names and directions. It records that digest for each (benchmark, version) and refuses to load a validator that presents a previously seen version with a different definition. A publish that presents one fails at the version-hash step and tells the publisher to bump the affected task's benchmark_version. For backward compatibility, omitting benchmark_version makes that task follow the top-level version, which is the pre-task-version behavior. Explanatory methodology copy is also excluded from this digest, so documentation can improve without creating a scientifically new benchmark version.

Each validator ships a CHANGELOG.md. Its first version heading must be ## <version> (an optional v prefix is accepted) and must match the manifest. The conformance kit reports this as the changelog-current check, which runs only when --changelog <path> is given. The server's publish checks have no access to the file, so keeping the changelog current is the publisher's own discipline - run the kit with --changelog in your CI.

The service always runs a published validator by immutable OCI digest, for example repository/name@sha256:<64 hexadecimal characters>, and records that digest with every score. A tag is resolved to its digest once, when the revision is pulled. For development local_command validators it records a digest of the validator source tree instead.

Validators must remain byte-deterministic for a given task, view, and seed. The service stores the seed privately with the dataset so an authorized audit can replay the instance recipe; the seed is never returned by the participant API.

Conformance

The repository ships an executable check of the mechanical contract: manifest validity and stability, a known-valid scoring self-test, deterministic generate, seed sensitivity, private-data hygiene, and invalid-submission handling. Run it in your CI:

python tools/validator_conformance.py \
  --command "<argv>" \
  --task <task_id> \
  --view <view_id> \
  --changelog path/to/CHANGELOG.md

Pass --image <reference> instead of --command to check the container you are about to publish; the two are mutually exclusive and one is required. The checks are describe-valid, describe-stable, score-accepts-known-valid, changelog-current (only with --changelog), generate-succeeds, generate-deterministic, generate-seed-sensitive, no-private-leak, and score-rejects-empty.

no-private-leak rejects the literal seed, complete private files, and sampled 12-byte fragments from the beginning, middle, and end of private files when they appear in public/. This catches common packaging mistakes but cannot prove semantic secrecy (for example, an encoded or transformed answer leak). Validator review and Validator-owned adversarial tests remain required.

Without --task and --view the standalone kit checks every task and view the manifest declares, regardless of publish_conformance_pairs, and reports a per-pair failure as <check>: <task_id>/<view_id>: <detail>; --task and --view narrow that to the single pair you name. During the conformance step of a publish, the server uses publish_conformance_pairs when declared and otherwise checks every pair. The declaration controls publish cost only; it does not remove tasks or views from the public manifest and does not change normal benchmark execution.

Changes from v2

Contract v3 is a breaking change from v2:

1. contract_version must be "3" and unknown fields are rejected.
2. views and problems are top-level. Tasks live under a problem, not at the top level.
3. display is problem metadata only; it is ordered and generic, and the server does not invent titles, curve groups, or presentation hierarchy.
4. submission is declared once per problem.
5. curves is gone. primary_metric and metrics are still present, but curves are now derived by the server from problems, tasks, and scores.
6. generate now requires --view.