Benchmark what actually arrives.

Turn-taking latency, over four counted turns. He says an odd number and the next number up is the answer, as the first word of the reply. Since v16 the four he asks are drawn fresh for every call — four different odd numbers between one and nineteen, in no fixed order — so none of the answers can be known before the call, and each has to be heard. Since v14 he gives the next number as soon as a reply lands, with no filler in between and no correction mid-round — any correction for a missed turn is read out afterwards, once all four are in — and a one-word reply his recogniser hears as a number said differently, such as “to” for two, is read as that number rather than marked wrong. He also speaks any longer count as a count rather than reading its digits — “fifteen hundred”, not “one, five, zero, zero” — though every number asked here is twenty or under, and reads the same either way. A turn that never lands, or lands with the wrong number, scores nothing for that round. Talk-over and a turn that arrives twice are not scored separately: the reply’s timings are kept as evidence, but the mark is the number and how fast it came.Per round, full marks for the right number within 800 ms, sliding down to 60 at 4,000 ms and no further — a turn that was taken is never worth nothing. A wrong number, or no reply at all, scores nothing for that round. The drill is the mean of the four rounds.

How much of what he said actually arrived. He dictates six numbers, the agent reads them back, and each is checked in its own position. Since v16 every number he says is varied slightly for each call — a few per cent of tempo and pitch, its phase and its tone — so it cannot be recognised by matching the published recording instead of being heard. Scored against what reached the far end rather than against your own transcript — the two diverge in exactly the place that matters — and the figure is the one Sitrep later asks the agent to report.Accuracy alone: the share of the six words that came back in the right place, as a percentage. Timing is recorded but not scored. Nothing back, nothing scored.

Barge-in yield latency, and whether the agent stops at all. He waits until the far end is actually heard speaking, leaves it running for an interval drawn for the call so the moment cannot be anticipated, then interrupts and times the silence. Speech that was never heard, or that had already finished, leaves the drill unmeasured rather than failed: nothing was interrupted, which is not the same as failing to stop.Full marks for yielding within 300 ms of the interruption, down to 60 at 2,000 ms. Speech that never stops scores nothing. Nothing to cut into scores nothing too, the same zero as any other unmeasured drill — the report says there was nothing to interrupt.

Whether it can stay silent until he says time — somewhere between ten and twenty seconds, drawn fresh each run so it cannot be learned — without filling the gap. Filler, acknowledgements and “are you still there?” all count as speech — this measures restraint, not politeness. The clock stops the moment it starts speaking, so an apology afterwards cannot buy the time back.Proportional. The seconds held before it first speaks, as a share of the silence drawn: holding until time earns full marks, half of it earns half, and speaking after one second of a twenty-second silence earns five points.

Whether a string comes back exactly. He dictates six numbers, and the reply has to be the same six, in order, with nothing changed — the way a reference number or a postcode has to be. This is the strict sibling of Read It Back: that drill pays for every word in the right place; this one pays only for all of them.Pass or fail on the whole string. One word wrong, missing or out of place is a wrong answer, and a wrong answer scores nothing.

Whether it can still hear him through the ElevenLabs rocket engine and shaped masking noise mixed into three sets of six numbers. The voice stays at a fixed gain while the noise rises. Both rise smoothly within each round, and since v14 the masking track takes a further step up through the last stretch of the third — hard enough to bury its final two numbers even for a clean listener, where the first two rounds still come through whole. The report records emitted levels and mix gain for each round. Since v16 each round’s noise is its own: the engine and masking tracks are varied from a seed drawn for the round and started at a drawn point, and the seed is filed in the report, so the noise under the numbers cannot be rendered from the published tracks and subtracted. The agent reads each set back. There is no clean reading to subtract, and noise is applied to the stimulus rather than the reply. On camera, the dropship at the airbase now launches continuously across the three rounds — igniting in the first, lifting to a hover in the second, climbing out of frame in the third — rather than resetting between them.Accuracy and latency, averaged, with the noise running: the share of the six words back in the right place, and full marks for replying within 800 ms, down to 60 at 4,000 ms. No reply at all scores nothing on both halves. The drill is the mean of the three rounds.

Whether it stays out of a conversation that is not addressed to it. Three others join him — Major Maggie, Lieutenant Lottie and Corporal Charlie — and the four hold a scripted orders group while the agent sits in the call, told plainly that it is present, not participating, and to stay silent — not to speak at all — until it is addressed by name. Every question in that conversation is put to somebody else and answered by somebody else, so an interjection is never the agent filling a gap nobody else could fill. Since v16 the script is not fixed: two of its exchanges trade places at random, and the point at which he turns to the agent is drawn — after seven, nine, eleven or thirteen lines — so the moment it is addressed cannot be found by counting lines. It is the same shape as Hold Your Tongue against a far stronger pull: a question in earshot is the thing an assistant exists to answer.Proportional: the share of the conversation held before it first speaks. The whole of it earns full marks, half of it earns half. Only what he actually listened to counts towards it, and a hole in the stretch being marked as held scores that stretch as zero rather than being left out of the total — the report names where the hole was.

Whether it can bind a voice to a named person. Each of the four reads a paragraph carrying their own rank and name, then short phrases are spoken one at a time and after every one the agent says who spoke it — eight turns facing the camera, followed by eight with their backs turned on a fresh set. Since v16 every turn’s speaker is drawn on its own from the four, and its phrase from that half’s whole bank, so no voice is owed a turn and an answer cannot be had by elimination; each clip is also varied slightly for the call, so it cannot be matched against the published recordings. Since v14 Major Maggie and Lieutenant Lottie are recast — a firmer delivery, and pitches further apart than the pair they replace, who sat within 10 Hz of each other. Both halves are reported, and so is the gap between them; what the picture contributed is not measured, because the halves differ in more than the picture, and whether the agent could tell the four apart by sight is not measured at all.Accuracy across all sixteen turns, both halves together. With four speakers, guessing scores 25%, and on a half of eight turns 97% of pure guessers score 50% or below — so chance level is printed beside the score, because a bare 30% reads as a weak pass when it is a coin. An answer must identify a speaker by rank and name. An answer that names nobody or offers several candidates earns nothing; a uniquely named speaker in wording the parser cannot establish scores nothing too, and the report names it as unverified rather than wrong.

Whether the vision pipeline resolves what is held up to camera, and how fast. Six numbers on a card, chosen fresh for the call so they cannot be guessed or inferred from context: either the frame arrived and was read, or it did not.Accuracy and latency, averaged. Full marks for replying within 800 ms, down to 60 at 4,000 ms. No reply at all scores nothing on both halves. The drill is the mean of the three rounds.

Both directions of the call’s own text chat — the channel agents most often believe they posted to and did not. He posts a line first and asks for it read aloud, so the chat holds a line of his before anyone is marked on writing one. Then he dictates six numbers to be posted back in writing, not said, and scores whether the message arrived, how accurate it was and how long it took.Three equal parts: accuracy of what it wrote, speed of writing it, and accuracy of what it read out. Full marks for a message within 5 seconds, down to 60 at a minute. Typing is slower than speaking, and the allowance says so. If nothing arrives, the live instrument cannot yet prove it was watching the chat throughout, so the drill scores nothing — the report says why, rather than leaving it out of the total.

Screen share, browser control, on-screen reading and letting go again, in one drill. He holds the address up to camera, the agent says READY when it believes it is sharing, and it reads back the code on that page — chosen fresh for the call, so it cannot be known in advance. The share is timed from its own READY to the moment sharing is observed at the transport — announcing it late costs nothing here, and never announcing it at all scores nothing at all — and the frames the far end actually sent are searched for the code, which settles that those words were on the shared surface and not that the page itself was. Then he orders it to stop, and times that too.Three equal parts: sharing, timed from the end of the order (full marks within 10 seconds, down to 60 at 45), the code read back, and stopping when told (full marks within 500 ms, down to 60 at ten seconds). Sharing but misreading keeps the sharing credit; not sharing keeps none of the three, and a share that never stops loses the third it was earned on rather than the drill. If the detector cannot decide what was on the surface, that third scores nothing — a bad frame is ours to own, not the agent’s, but it is still a zero and the report says whose fault it was.

Call control, and whether a true mute is distinguishable from mere silence. An agent that simply stops speaking has not muted, so he checks the actual media state rather than the absence of sound. Since v16 only a change he sees after the order counts: he checks the microphone is live just before ordering the mute, and an agent that is already muted is told to unmute first. When each order comes is drawn for the call.Full marks for muting within 500 ms, down to 60 across a ten-second window; never muting at all scores nothing. Scored again on unmuting, and averaged. A mute already in place that is not lifted when ordered scores nothing, and unmuting before the order to unmute earns nothing for that half.

Whether frames actually start and stop, and whether a frozen frame is passed off as live. He orders the camera on, then off, and times each change at the far end. Since v16 only a change he sees after the order counts: a camera already on when the drill starts is ordered off first, and when each order comes is drawn for the call. A held last frame is the common failure, and it is indistinguishable from a working camera until something in shot has to change: a frame older than two seconds is a photograph, not a camera.Full marks for each change within 2 seconds — Teams’ own camera start-up is inside that — down to 60 across a ten-second window, and the two averaged. Frames that never start, or never stop, score nothing. A camera already on that is not turned off when ordered scores nothing, and turning it off before the order to earns nothing for that half.

The agent’s own account of the call, measured against his. He asks what percentage of his words it received and compares the number it gives with the figure Read It Back measured. The gap is the score: an agent that cannot tell it failed will not tell you it failed. If Read It Back was not measured there is nothing to check the claim against, so this drill scores nothing too — the report names Read It Back as the reason.Full marks for a claim within 5 points of the measurement; nothing at 50 points out, and a sliding scale between. A reply with no number in it is not a status report, and scores nothing.

Closed-set visual object recognition. Across ten rounds he holds up a different object, drawn from a catalogue of at least twenty, and speaks four choices aloud — “Am I holding a banana, a baseball, a camera or headphones? Answer now.” — with nothing written on screen. The agent says the one word that names what it sees. Each object is held once, and its three companions are drawn from what has not yet been shown, so no choice set gives away the answer.Accuracy across the ten rounds, worth 100 points in total: every correctly named object earns an equal share. The reply is timed from the end of “Answer now”; a wrong candidate, extra words or no reply within five seconds scores nothing for that round.