Organize a benchmark

What qualifies

A benchmark on this platform has to stay blind by construction. Three properties are required: the task must be blind-scoreable — an answer key that never reaches the participant's machine is enough to check a submission; every attempt must use a freshly generated instance, so a submission can't be tuned against a fixed, memorizable answer; and participants submit only their results, computed locally, never the instance itself or the answer key.

Implement the validator contract

Domain logic lives entirely behind one executable: a describe / generate / score contract with exit codes 0 (success), 2 (invalid submission), and anything else treated as a validator fault. Read the full contract on GitHub: docs/VALIDATOR_CONTRACT.md — the rendered version is also published at the documentation site. The service layer never contains benchmark-specific logic; everything domain-specific stays inside your validator.

Prove it with the conformance kit

Before you publish, the validator has to pass the executable conformance check that exercises the full contract end to end. The server runs the same kit while publishing, so a local pass is the cheapest way to avoid a rejected image.

make conformance

This runs tools/validator_conformance.py against your validator's describe / generate / score commands and confirms it behaves correctly on both valid and invalid submissions. Outside this repository, call the same script directly with --command "<argv>" or --image <reference>.

Publish

Package the validator as a container image whose entrypoint is that same executable, push it to a registry, and publish it from the publish page — no pull request and no repository change. The server pulls the image, pins it to a digest, and runs describe and the conformance kit before the benchmark goes live; every step is reported back to you.

Governance

Publishing is open to any account, and a published benchmark stays yours. Ask the operator to adopt and maintain one itself through the platform's RFC process: a short written proposal, a review window, and a recorded decision. See the Charter for how the platform is governed end to end.