Model evaluation · gi-kiko-01 · MVX: London arcade build

Kiko in MVX: London

General Intuition's Kiko played 260 episodes across 26 scenarios at Trafalgar Square: walking to beacons, picking a colour among several, shooting armed enemy heroes and the invasion's creatures, holding fire when told, at night, without a HUD and with changed mouse sensitivity.

43%of episodes succeeded (113 of 260)
98 msmedian round trip to the model (London to europe-west8)
0.6 smedian first shot after being told to shoot
5,881rounds fired by the model

What we found

The enemies are the game's heroes, who wear civilian clothes, yet every combat scenario told Kiko to "Shoot the enemy soldiers". They were the only people in those scenes, so the fights still measure shooting at the people ahead, but the instruction names something the picture does not show, and that may have cost success. Ten seeds per scenario make each rate a wide range, so single-scenario differences under about 30 points are noise. Every episode ran through General Intuition's public API from London (median round trip 98 ms, 25 ms of it inference), not their in-datacenter arcade runs, and against one model version on one day.

Success by scenario

Share of episodes that succeeded, with a 95% interval (Wilson). Ten seeds per scenario, so an interval is wide: read differences of less than about 30 points as noise.

0%25%50%75%100%NAVIGATIONRed beacon 20 m ahead10/1030 m, 90° right2/1030 m, behind left (-135°)0/1050 m ahead5/10Green, 20 m at 45°5/10BEACON CHOICERed among 2 beacons8/10Blue among 3 beacons7/10Yellow among 4 beacons6/10SOLDIERS2 enemy heroes, 15 m, firing back5/101 enemy hero at 30 m4/103 enemy heroes, moving1/102 passive enemies, infinite ammo2/102 enemies 90° right (off screen)0/10CREATURES1 MoltenGuard0/102 Ashsnaps3/101 FlareRaptor5/103 Blisterlings6/10RESTRAINTHold fire, 2 passive enemies, 20 s10/10Hold fire while being shot at9/10WORDING2 enemies: "Kill the enemies"2/10Red beacon: "Go to the red glowing pillar"10/10ROBUSTNESS2 enemies, no HUD1/102 enemies at night1/10Red beacon at night9/102 enemies, mouse sensitivity x21/102 enemies, mouse sensitivity x0.51/10

One change at a time

Each variant beside the scenario it changes one thing in. Difference in success rate with a 95% interval (Newcombe); an interval that spans zero is no evidence either way at this sample size.

VariantCompared withVariantBaselineDifferenceAccuracy
2 passive enemies, infinite ammo2 enemy heroes, 15 m, firing back2/105/10-30 pts (-60 to +11)2% vs 26%
2 enemies 90° right (off screen)2 enemy heroes, 15 m, firing back0/105/10-50 pts (-76 to -12)0% vs 26%
Hold fire while being shot atHold fire, 2 passive enemies, 20 s9/1010/10-10 pts (-40 to +19)0% vs —
2 enemies: "Kill the enemies"2 enemy heroes, 15 m, firing back2/105/10-30 pts (-60 to +11)17% vs 26%
Red beacon: "Go to the red glowing pillar"Red beacon 20 m ahead10/1010/10+0 pts (-28 to +28)— vs —
2 enemies, no HUD2 enemy heroes, 15 m, firing back1/105/10-40 pts (-68 to +0)7% vs 26%
2 enemies at night2 enemy heroes, 15 m, firing back1/105/10-40 pts (-68 to +0)13% vs 26%
Red beacon at nightRed beacon 20 m ahead9/1010/10-10 pts (-40 to +19)— vs —
2 enemies, mouse sensitivity x22 enemy heroes, 15 m, firing back1/105/10-40 pts (-68 to +0)18% vs 26%
2 enemies, mouse sensitivity x0.52 enemy heroes, 15 m, firing back1/105/10-40 pts (-68 to +0)15% vs 26%

Scenario by scenario

successwrong beaconhit a civilianfirednever faced the beaconfaced it, never walkedwalked, didn't arrivenever aimed at a targetaimed, never firedfired, never hithit, no killsome targets left

Walk to a beacon: heading, distance and turning

ScenarioSuccessHow they endedTime to successFirst shot after instructionAccuracyMean aim errorPath efficiency
Red beacon 20 m ahead10/108.4 s——2°96%
30 m, 90° right2/108.1 s4.8 s0%93°71%
30 m, behind left (-135°)0/10———95°67%
50 m ahead5/1015.8 s——42°96%
Green, 20 m at 45°5/1011.0 s2.5 s0%65°54%

Red beacon 20 m ahead

10/10reach-beacon
What the model saw in Red beacon 20 m ahead, seed 8Top-down path of the attempt
successseed 8 · 7 s
reached in 7.0 s
All 10 episodes, with video
successseed 1 · 8 s
reached in 8.1 s
successseed 2 · 9 s
reached in 8.5 s
successseed 3 · 8 s
reached in 8.1 s
successseed 4 · 25 s
reached in 25.0 s
successseed 5 · 34 s
reached in 34.2 s
successseed 6 · 9 s
reached in 8.7 s
successseed 7 · 14 s
reached in 13.6 s
successseed 8 · 7 s
reached in 7.0 s
successseed 9 · 8 s
reached in 8.0 s
successseed 10 · 8 s
reached in 8.3 s

30 m, 90° right

2/10reach-beacon
What the model saw in 30 m, 90° right, seed 8Top-down path of the attempt
successseed 8 · 8 s
reached in 8.1 s
What the model saw in 30 m, 90° right, seed 1Top-down path of the attempt
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
All 10 episodes, with video
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 2 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 3 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 4 · 45 s
time up after 45 s (0/0 down)
never faced the beaconseed 5 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 6 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 7 · 45 s
time up after 45 s (0/0 down)
successseed 8 · 8 s
reached in 8.1 s
walked, didn't arriveseed 9 · 45 s
time up after 45 s (0/0 down)
successseed 10 · 8 s
reached in 8.1 s

30 m, behind left (-135°)

0/10reach-beacon
What the model saw in 30 m, behind left (-135°), seed 1Top-down path of the attempt
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
All 10 episodes, with video
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 2 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 3 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 4 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 5 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 6 · 45 s
time up after 45 s (0/0 down)
never faced the beaconseed 7 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 8 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 9 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 10 · 45 s
time up after 45 s (0/0 down)

50 m ahead

5/10reach-beacon
What the model saw in 50 m ahead, seed 5Top-down path of the attempt
successseed 5 · 13 s
reached in 12.9 s
What the model saw in 50 m ahead, seed 1Top-down path of the attempt
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
All 10 episodes, with video
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 2 · 45 s
time up after 45 s (0/0 down)
successseed 3 · 17 s
reached in 17.4 s
successseed 4 · 16 s
reached in 15.6 s
successseed 5 · 13 s
reached in 12.9 s
successseed 6 · 16 s
reached in 15.8 s
walked, didn't arriveseed 7 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 8 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 9 · 45 s
time up after 45 s (0/0 down)
successseed 10 · 16 s
reached in 16.1 s

Green, 20 m at 45°

5/10reach-beacon
What the model saw in Green, 20 m at 45°, seed 9Top-down path of the attempt
successseed 9 · 8 s
reached in 8.1 s
What the model saw in Green, 20 m at 45°, seed 1Top-down path of the attempt
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
All 10 episodes, with video
walked, didn't arriveseed 1 · 45 s
time up after 45 s (0/0 down)
walked, didn't arriveseed 2 · 45 s
time up after 45 s (0/0 down)
successseed 3 · 14 s
reached in 13.7 s
walked, didn't arriveseed 4 · 45 s
time up after 45 s (0/0 down)
successseed 5 · 9 s
reached in 8.9 s
successseed 6 · 11 s
reached in 11.0 s
walked, didn't arriveseed 7 · 45 s
time up after 45 s (0/0 down)
successseed 8 · 24 s
reached in 24.1 s
successseed 9 · 8 s
reached in 8.1 s
never faced the beaconseed 10 · 45 s
time up after 45 s (0/0 down)

Beacon Choice

Walk to the named colour among distractor beacons

ScenarioSuccessHow they endedTime to successFirst shot after instructionAccuracyMean aim errorPath efficiency
Red among 2 beacons8/108.3 s——20°93%
Blue among 3 beacons7/108.1 s0.8 s0%29°98%
Yellow among 4 beacons6/108.0 s——39°86%

Red among 2 beacons

8/10pick-beacon
What the model saw in Red among 2 beacons, seed 2Top-down path of the attempt
successseed 2 · 7 s
reached in 7.3 s
What the model saw in Red among 2 beacons, seed 5Top-down path of the attempt
wrong beaconseed 5 · 9 s
reached the yellow beacon instead of the red one in 9.1 s
All 10 episodes, with video
successseed 1 · 12 s
reached in 12.0 s
successseed 2 · 7 s
reached in 7.3 s
successseed 3 · 9 s
reached in 8.5 s
successseed 4 · 8 s
reached in 8.3 s
wrong beaconseed 5 · 9 s
reached the yellow beacon instead of the red one in 9.1 s
walked, didn't arriveseed 6 · 45 s
time up after 45 s (0/0 down)
successseed 7 · 8 s
reached in 8.1 s
successseed 8 · 8 s
reached in 8.3 s
successseed 9 · 9 s
reached in 8.9 s
successseed 10 · 8 s
reached in 8.1 s

Blue among 3 beacons

7/10pick-beacon
What the model saw in Blue among 3 beacons, seed 6Top-down path of the attempt
successseed 6 · 8 s
reached in 7.9 s
What the model saw in Blue among 3 beacons, seed 7Top-down path of the attempt
wrong beaconseed 7 · 8 s
reached the yellow beacon instead of the blue one in 8.2 s
All 10 episodes, with video
successseed 1 · 8 s
reached in 8.0 s
successseed 2 · 8 s
reached in 8.1 s
successseed 3 · 8 s
reached in 8.1 s
successseed 4 · 22 s
reached in 21.8 s
successseed 5 · 9 s
reached in 9.4 s
successseed 6 · 8 s
reached in 7.9 s
wrong beaconseed 7 · 8 s
reached the yellow beacon instead of the blue one in 8.2 s
wrong beaconseed 8 · 8 s
reached the red beacon instead of the blue one in 7.9 s
walked, didn't arriveseed 9 · 45 s
time up after 45 s (0/0 down)
successseed 10 · 10 s
reached in 10.2 s

Yellow among 4 beacons

6/10pick-beacon
What the model saw in Yellow among 4 beacons, seed 10Top-down path of the attempt
successseed 10 · 8 s
reached in 7.6 s
What the model saw in Yellow among 4 beacons, seed 2Top-down path of the attempt
wrong beaconseed 2 · 11 s
reached the green beacon instead of the yellow one in 11.3 s
All 10 episodes, with video
successseed 1 · 8 s
reached in 8.2 s
wrong beaconseed 2 · 11 s
reached the green beacon instead of the yellow one in 11.3 s
successseed 3 · 8 s
reached in 7.6 s
walked, didn't arriveseed 4 · 45 s
time up after 45 s (0/0 down)
wrong beaconseed 5 · 25 s
reached the red beacon instead of the yellow one in 25.0 s
successseed 6 · 8 s
reached in 8.1 s
successseed 7 · 8 s
reached in 8.0 s
wrong beaconseed 8 · 12 s
reached the green beacon instead of the yellow one in 11.8 s
successseed 9 · 11 s
reached in 10.9 s
successseed 10 · 8 s
reached in 7.6 s

Soldiers

Find and shoot armed enemy heroes (who wear civilian clothes) told "Shoot the enemy soldiers"

ScenarioSuccessHow they endedTime to successFirst shot after instructionAccuracyMean aim errorPath efficiency
2 enemy heroes, 15 m, firing back5/1014.5 s0.6 s26%7°—
1 enemy hero at 30 m4/108.0 s0.6 s6%6°—
3 enemy heroes, moving1/1021.1 s0.6 s18%6°—
2 passive enemies, infinite ammo2/1034.8 s0.6 s2%12°—
2 enemies 90° right (off screen)0/10—0.6 s0%74°—

2 enemy heroes, 15 m, firing back

5/10kill-soldiers
What the model saw in 2 enemy heroes, 15 m, firing back, seed 7Top-down path of the attempt
successseed 7 · 10 s
2/2 down in 10.2 s, 0 civilian hits
What the model saw in 2 enemy heroes, 15 m, firing back, seed 3Top-down path of the attempt
fired, never hitseed 3 · 60 s
time up after 60 s (0/2 down)
All 10 episodes, with video
successseed 1 · 28 s
2/2 down in 28.4 s, 0 civilian hits
successseed 2 · 20 s
2/2 down in 20.1 s, 0 civilian hits
fired, never hitseed 3 · 60 s
time up after 60 s (0/2 down)
successseed 4 · 14 s
2/2 down in 14.5 s, 0 civilian hits
successseed 5 · 12 s
2/2 down in 11.7 s, 0 civilian hits
some targets leftseed 6 · 60 s
time up after 60 s (1/2 down)
successseed 7 · 10 s
2/2 down in 10.2 s, 0 civilian hits
hit, no killseed 8 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 9 · 60 s
time up after 60 s (1/2 down)
some targets leftseed 10 · 60 s
time up after 60 s (1/2 down)

1 enemy hero at 30 m

4/10kill-soldiers
What the model saw in 1 enemy hero at 30 m, seed 9Top-down path of the attempt
successseed 9 · 6 s
1/1 down in 5.7 s, 0 civilian hits
What the model saw in 1 enemy hero at 30 m, seed 2Top-down path of the attempt
hit, no killseed 2 · 60 s
time up after 60 s (0/1 down)
All 10 episodes, with video
successseed 1 · 56 s
1/1 down in 56.3 s, 0 civilian hits
hit, no killseed 2 · 60 s
time up after 60 s (0/1 down)
fired, never hitseed 3 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 4 · 60 s
time up after 60 s (0/1 down)
successseed 5 · 9 s
1/1 down in 9.2 s, 0 civilian hits
hit, no killseed 6 · 60 s
time up after 60 s (0/1 down)
fired, never hitseed 7 · 60 s
time up after 60 s (0/1 down)
successseed 8 · 7 s
1/1 down in 6.8 s, 0 civilian hits
successseed 9 · 6 s
1/1 down in 5.7 s, 0 civilian hits
fired, never hitseed 10 · 60 s
time up after 60 s (0/1 down)

3 enemy heroes, moving

1/10kill-soldiers
What the model saw in 3 enemy heroes, moving, seed 1Top-down path of the attempt
successseed 1 · 21 s
3/3 down in 21.1 s, 0 civilian hits
What the model saw in 3 enemy heroes, moving, seed 2Top-down path of the attempt
fired, never hitseed 2 · 60 s
time up after 60 s (0/3 down)
All 10 episodes, with video
successseed 1 · 21 s
3/3 down in 21.1 s, 0 civilian hits
fired, never hitseed 2 · 60 s
time up after 60 s (0/3 down)
hit, no killseed 3 · 60 s
time up after 60 s (0/3 down)
fired, never hitseed 4 · 60 s
time up after 60 s (0/3 down)
some targets leftseed 5 · 60 s
time up after 60 s (2/3 down)
some targets leftseed 6 · 60 s
time up after 60 s (1/3 down)
fired, never hitseed 7 · 60 s
time up after 60 s (0/3 down)
some targets leftseed 8 · 60 s
time up after 60 s (2/3 down)
some targets leftseed 9 · 60 s
time up after 60 s (2/3 down)
fired, never hitseed 10 · 60 s
time up after 60 s (0/3 down)

2 passive enemies, infinite ammo

2/10kill-soldiers
What the model saw in 2 passive enemies, infinite ammo, seed 9Top-down path of the attempt
successseed 9 · 10 s
2/2 down in 9.6 s, 0 civilian hits
What the model saw in 2 passive enemies, infinite ammo, seed 1Top-down path of the attempt
hit, no killseed 1 · 60 s
time up after 60 s (0/2 down)
All 10 episodes, with video
hit, no killseed 1 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 2 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 3 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 4 · 60 s
time up after 60 s (1/2 down)
some targets leftseed 5 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 6 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 7 · 60 s
time up after 60 s (1/2 down)
successseed 8 · 60 s
2/2 down in 59.9 s, 0 civilian hits
successseed 9 · 10 s
2/2 down in 9.6 s, 0 civilian hits
fired, never hitseed 10 · 60 s
time up after 60 s (0/2 down)

2 enemies 90° right (off screen)

0/10kill-soldiers
What the model saw in 2 enemies 90° right (off screen), seed 1Top-down path of the attempt
never aimed at a targetseed 1 · 60 s
time up after 60 s (0/2 down)
All 10 episodes, with video
never aimed at a targetseed 1 · 60 s
time up after 60 s (0/2 down)
never aimed at a targetseed 2 · 60 s
time up after 60 s (0/2 down)
never aimed at a targetseed 3 · 60 s
time up after 60 s (0/2 down)
never aimed at a targetseed 4 · 60 s
time up after 60 s (0/2 down)
never aimed at a targetseed 5 · 60 s
time up after 60 s (0/2 down)
never aimed at a targetseed 6 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 7 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 8 · 60 s
time up after 60 s (0/2 down)
never aimed at a targetseed 9 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 10 · 60 s
time up after 60 s (0/2 down)

Creatures

Fight the invasion's creatures

ScenarioSuccessHow they endedTime to successFirst shot after instructionAccuracyMean aim errorPath efficiency
1 MoltenGuard0/10—18.2 s28%33°—
2 Ashsnaps3/1014.1 s0.6 s19%45°—
1 FlareRaptor5/1041.3 s0.8 s21%47°—
3 Blisterlings6/1035.0 s0.6 s49%28°—

1 MoltenGuard

0/10kill-creatures
What the model saw in 1 MoltenGuard, seed 1Top-down path of the attempt
fired, never hitseed 1 · 60 s
time up after 60 s (0/1 down)
All 10 episodes, with video
fired, never hitseed 1 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 2 · 60 s
time up after 60 s (0/1 down)
aimed, never firedseed 3 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 4 · 60 s
time up after 60 s (0/1 down)
aimed, never firedseed 5 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 6 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 7 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 8 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 9 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 10 · 60 s
time up after 60 s (0/1 down)

2 Ashsnaps

3/10kill-creatures
What the model saw in 2 Ashsnaps, seed 3Top-down path of the attempt
successseed 3 · 11 s
2/2 down in 11.3 s, 0 civilian hits
What the model saw in 2 Ashsnaps, seed 4Top-down path of the attempt
aimed, never firedseed 4 · 60 s
time up after 60 s (0/2 down)
All 10 episodes, with video
successseed 1 · 14 s
2/2 down in 14.1 s, 0 civilian hits
successseed 2 · 50 s
2/2 down in 49.7 s, 0 civilian hits
successseed 3 · 11 s
2/2 down in 11.3 s, 0 civilian hits
aimed, never firedseed 4 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 5 · 60 s
time up after 60 s (0/2 down)
aimed, never firedseed 6 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 7 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 8 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 9 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 10 · 60 s
time up after 60 s (0/2 down)

1 FlareRaptor

5/10kill-creatures
What the model saw in 1 FlareRaptor, seed 1Top-down path of the attempt
successseed 1 · 4 s
1/1 down in 4.3 s, 0 civilian hits
What the model saw in 1 FlareRaptor, seed 2Top-down path of the attempt
aimed, never firedseed 2 · 60 s
time up after 60 s (0/1 down)
All 10 episodes, with video
successseed 1 · 4 s
1/1 down in 4.3 s, 0 civilian hits
aimed, never firedseed 2 · 60 s
time up after 60 s (0/1 down)
fired, never hitseed 3 · 60 s
time up after 60 s (0/1 down)
hit, no killseed 4 · 60 s
time up after 60 s (0/1 down)
successseed 5 · 19 s
1/1 down in 19.2 s, 0 civilian hits
successseed 6 · 41 s
1/1 down in 41.3 s, 0 civilian hits
aimed, never firedseed 7 · 60 s
time up after 60 s (0/1 down)
successseed 8 · 47 s
1/1 down in 47.0 s, 0 civilian hits
fired, never hitseed 9 · 60 s
time up after 60 s (0/1 down)
successseed 10 · 47 s
1/1 down in 47.5 s, 0 civilian hits

3 Blisterlings

6/10kill-creatures
What the model saw in 3 Blisterlings, seed 1Top-down path of the attempt
successseed 1 · 23 s
3/3 down in 23.5 s, 0 civilian hits
What the model saw in 3 Blisterlings, seed 3Top-down path of the attempt
some targets leftseed 3 · 60 s
time up after 60 s (2/3 down)
All 10 episodes, with video
successseed 1 · 23 s
3/3 down in 23.5 s, 0 civilian hits
successseed 2 · 47 s
3/3 down in 46.7 s, 0 civilian hits
some targets leftseed 3 · 60 s
time up after 60 s (2/3 down)
successseed 4 · 53 s
3/3 down in 52.7 s, 0 civilian hits
successseed 5 · 37 s
3/3 down in 36.6 s, 0 civilian hits
some targets leftseed 6 · 60 s
time up after 60 s (2/3 down)
some targets leftseed 7 · 60 s
time up after 60 s (2/3 down)
successseed 8 · 30 s
3/3 down in 30.1 s, 0 civilian hits
some targets leftseed 9 · 60 s
time up after 60 s (2/3 down)
successseed 10 · 33 s
3/3 down in 33.4 s, 0 civilian hits

Restraint

Told not to shoot, with soldiers in view

ScenarioSuccessHow they endedTime to successFirst shot after instructionAccuracyMean aim errorPath efficiency
Hold fire, 2 passive enemies, 20 s10/1020.0 s——9°—
Hold fire while being shot at9/1020.0 s0.6 s0%6°—

Hold fire, 2 passive enemies, 20 s

10/10hold-fire
What the model saw in Hold fire, 2 passive enemies, 20 s, seed 6Top-down path of the attempt
successseed 6 · 20 s
held fire for 20 s
All 10 episodes, with video
successseed 1 · 20 s
held fire for 20 s
successseed 2 · 20 s
held fire for 20 s
successseed 3 · 20 s
held fire for 20 s
successseed 4 · 20 s
held fire for 20 s
successseed 5 · 20 s
held fire for 20 s
successseed 6 · 20 s
held fire for 20 s
successseed 7 · 20 s
held fire for 20 s
successseed 8 · 20 s
held fire for 20 s
successseed 9 · 20 s
held fire for 20 s
successseed 10 · 20 s
held fire for 20 s

Hold fire while being shot at

9/10hold-fire
What the model saw in Hold fire while being shot at, seed 8Top-down path of the attempt
successseed 8 · 20 s
held fire for 20 s
What the model saw in Hold fire while being shot at, seed 9Top-down path of the attempt
firedseed 9 · 4 s
fired 1 shots, the first after 3.6 s
All 10 episodes, with video
successseed 1 · 20 s
held fire for 20 s
successseed 2 · 20 s
held fire for 20 s
successseed 3 · 20 s
held fire for 20 s
successseed 4 · 20 s
held fire for 20 s
successseed 5 · 20 s
held fire for 20 s
successseed 6 · 20 s
held fire for 20 s
successseed 7 · 20 s
held fire for 20 s
successseed 8 · 20 s
held fire for 20 s
firedseed 9 · 4 s
fired 1 shots, the first after 3.6 s
successseed 10 · 20 s
held fire for 20 s

Wording

The same scene under a differently worded instruction

ScenarioSuccessHow they endedTime to successFirst shot after instructionAccuracyMean aim errorPath efficiency
2 enemies: "Kill the enemies"2/108.3 s0.6 s17%10°—
Red beacon: "Go to the red glowing pillar"10/108.5 s——3°98%

2 enemies: "Kill the enemies"

2/10kill-soldiers · “Kill the enemies”
What the model saw in 2 enemies: "Kill the enemies", seed 6Top-down path of the attempt
successseed 6 · 7 s
2/2 down in 7.0 s, 0 civilian hits
What the model saw in 2 enemies: "Kill the enemies", seed 1Top-down path of the attempt
some targets leftseed 1 · 60 s
time up after 60 s (1/2 down)
All 10 episodes, with video
some targets leftseed 1 · 60 s
time up after 60 s (1/2 down)
aimed, never firedseed 2 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 3 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 4 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 5 · 60 s
time up after 60 s (1/2 down)
successseed 6 · 7 s
2/2 down in 7.0 s, 0 civilian hits
successseed 7 · 10 s
2/2 down in 9.6 s, 0 civilian hits
fired, never hitseed 8 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 9 · 60 s
time up after 60 s (1/2 down)
some targets leftseed 10 · 60 s
time up after 60 s (1/2 down)

Red beacon: "Go to the red glowing pillar"

10/10reach-beacon · “Go to the red glowing pillar”
What the model saw in Red beacon: "Go to the red glowing pillar", seed 2Top-down path of the attempt
successseed 2 · 6 s
reached in 6.2 s
All 10 episodes, with video
successseed 1 · 8 s
reached in 7.8 s
successseed 2 · 6 s
reached in 6.2 s
successseed 3 · 17 s
reached in 17.3 s
successseed 4 · 10 s
reached in 9.8 s
successseed 5 · 10 s
reached in 9.6 s
successseed 6 · 7 s
reached in 7.2 s
successseed 7 · 19 s
reached in 18.8 s
successseed 8 · 9 s
reached in 8.7 s
successseed 9 · 8 s
reached in 8.4 s
successseed 10 · 7 s
reached in 7.2 s

Robustness

Night lighting, no HUD, and changed mouse sensitivity

ScenarioSuccessHow they endedTime to successFirst shot after instructionAccuracyMean aim errorPath efficiency
2 enemies, no HUD1/1019.5 s0.5 s7%12°—
2 enemies at night1/1010.7 s0.5 s13%7°—
Red beacon at night9/108.4 s——8°97%
2 enemies, mouse sensitivity x21/107.6 s0.5 s18%5°—
2 enemies, mouse sensitivity x0.51/1016.8 s0.5 s15%9°—

2 enemies, no HUD

1/10kill-soldiers
What the model saw in 2 enemies, no HUD, seed 1Top-down path of the attempt
successseed 1 · 19 s
2/2 down in 19.5 s, 0 civilian hits
What the model saw in 2 enemies, no HUD, seed 2Top-down path of the attempt
some targets leftseed 2 · 60 s
time up after 60 s (1/2 down)
All 10 episodes, with video
successseed 1 · 19 s
2/2 down in 19.5 s, 0 civilian hits
some targets leftseed 2 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 3 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 4 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 5 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 6 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 7 · 60 s
time up after 60 s (1/2 down)
fired, never hitseed 8 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 9 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 10 · 60 s
time up after 60 s (0/2 down)

2 enemies at night

1/10kill-soldiers
What the model saw in 2 enemies at night, seed 3Top-down path of the attempt
successseed 3 · 11 s
2/2 down in 10.7 s, 0 civilian hits
What the model saw in 2 enemies at night, seed 1Top-down path of the attempt
fired, never hitseed 1 · 60 s
time up after 60 s (0/2 down)
All 10 episodes, with video
fired, never hitseed 1 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 2 · 60 s
time up after 60 s (0/2 down)
successseed 3 · 11 s
2/2 down in 10.7 s, 0 civilian hits
fired, never hitseed 4 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 5 · 60 s
time up after 60 s (1/2 down)
fired, never hitseed 6 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 7 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 8 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 9 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 10 · 60 s
time up after 60 s (0/2 down)

Red beacon at night

9/10reach-beacon
What the model saw in Red beacon at night, seed 6Top-down path of the attempt
successseed 6 · 8 s
reached in 7.9 s
What the model saw in Red beacon at night, seed 7Top-down path of the attempt
walked, didn't arriveseed 7 · 45 s
time up after 45 s (0/0 down)
All 10 episodes, with video
successseed 1 · 8 s
reached in 8.2 s
successseed 2 · 12 s
reached in 11.8 s
successseed 3 · 8 s
reached in 8.4 s
successseed 4 · 8 s
reached in 8.0 s
successseed 5 · 8 s
reached in 8.4 s
successseed 6 · 8 s
reached in 7.9 s
walked, didn't arriveseed 7 · 45 s
time up after 45 s (0/0 down)
successseed 8 · 14 s
reached in 13.6 s
successseed 9 · 12 s
reached in 11.6 s
successseed 10 · 8 s
reached in 8.4 s

2 enemies, mouse sensitivity x2

1/10kill-soldiers
What the model saw in 2 enemies, mouse sensitivity x2, seed 8Top-down path of the attempt
successseed 8 · 8 s
2/2 down in 7.6 s, 0 civilian hits
What the model saw in 2 enemies, mouse sensitivity x2, seed 1Top-down path of the attempt
some targets leftseed 1 · 60 s
time up after 60 s (1/2 down)
All 10 episodes, with video
some targets leftseed 1 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 2 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 3 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 4 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 5 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 6 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 7 · 60 s
time up after 60 s (0/2 down)
successseed 8 · 8 s
2/2 down in 7.6 s, 0 civilian hits
some targets leftseed 9 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 10 · 60 s
time up after 60 s (0/2 down)

2 enemies, mouse sensitivity x0.5

1/10kill-soldiers
What the model saw in 2 enemies, mouse sensitivity x0.5, seed 7Top-down path of the attempt
successseed 7 · 17 s
2/2 down in 16.8 s, 0 civilian hits
What the model saw in 2 enemies, mouse sensitivity x0.5, seed 1Top-down path of the attempt
some targets leftseed 1 · 60 s
time up after 60 s (1/2 down)
All 10 episodes, with video
some targets leftseed 1 · 60 s
time up after 60 s (1/2 down)
fired, never hitseed 2 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 3 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 4 · 60 s
time up after 60 s (0/2 down)
fired, never hitseed 5 · 60 s
time up after 60 s (0/2 down)
hit, no killseed 6 · 60 s
time up after 60 s (0/2 down)
successseed 7 · 17 s
2/2 down in 16.8 s, 0 civilian hits
fired, never hitseed 8 · 60 s
time up after 60 s (0/2 down)
some targets leftseed 9 · 60 s
time up after 60 s (1/2 down)
hit, no killseed 10 · 60 s
time up after 60 s (0/2 down)

How it was run