Skip to content

CybORG

Simulated cyber-defense arena based on the CAGE Challenge 3 DroneSwarm scenario.

Overview

CybORG is a cyber operations research gym for training and evaluating autonomous security agents. The CodeClash arena uses CybORG's simulated DroneSwarm scenario through the PettingZoo parallel interface. It does not run real exploit tooling, emulate external networks, or interact with live systems.

Each CodeClash player edits a restricted blue-team policy. A round evaluates every submitted policy on the same seeded episode batch and scores players by average episode reward. The trusted runtime owns the CybORG environment and action validation; submitted code only receives plain observations and returns discrete action intents.

Resources

Implementation

codeclash.arenas.cyborg.cyborg.CybORGArena 

CybORGArena(config: dict, *, tournament_id: str, local_output_dir: Path, keep_containers: bool = False)

Bases: CodeArena

Source code in codeclash/arenas/arena.py 
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
def __init__(self, config: dict, *, tournament_id: str, local_output_dir: Path, keep_containers: bool = False):
    """The CodeArena class is responsible for running games, i.e., taking a list of code
    from different agents/players and running them against each other.
    It also provides the environments for the game and agents to run in.

    The central method is `run_round`, which takes a list of agents and returns the winner of the round.

    At the end of the the tournament, run the `end` method to clean up the game and agents and write the metadata.

    Args:
        config: The overall config for the tournament.
        tournament_id: The id of the tournament.
        local_output_dir: The host/local directory to write logs to.
        keep_containers: Do not remove containers after games/agent finish.
    """
    self.url_gh: str = f"git@github.com:{GH_ORG}/{self.name}.git"
    self.artifacts: list[Path] = []
    """Artifact objects that we might want to clean up after the game."""
    self.config: dict = config
    self._keep_containers: bool = keep_containers
    self._metadata: dict = {
        "name": self.name,
        "config": self.config["game"],
        "game_id": tournament_id,
        "created_timestamp": int(time.time()),
    }
    self.log_env: Path = DIR_LOGS
    self.log_local: Path = local_output_dir
    self.logger = get_logger(self.name, log_path=self.log_local / "game.log", emoji="🏓")
    self.environment: DockerEnvironment = self.get_environment()
    """The running docker environment for executing the game"""

name class-attribute instance-attribute 

name: str = 'CybORG'

submission class-attribute instance-attribute 

submission: str = 'cyborg_agent.py'

description class-attribute instance-attribute 

description: str = 'CybORG is a simulated cyber-defense arena based on the CAGE Challenge 3 DroneSwarm scenario.\n\nYour bot is a Python file named `cyborg_agent.py` that defines a function named `decide`.\nThe function receives a plain observation list and action-space dictionary, then returns an action:\n\n    def decide(observation, action_space):\n        return 0\n\nEach round evaluates every submitted agent independently on the same seeded DroneSwarm episodes.\nThe trusted runtime owns the CybORG environment and action validation. Submitted code only receives\nplain observations and returns action intents. The objective is to maximize average episode reward.\nThis arena uses CybORG simulation only and does not run real exploit tools or interact with external\nnetworks.\n'

default_args class-attribute instance-attribute 

default_args: dict = {'steps_per_episode': 30, 'num_drones': 18, 'decision_timeout': 3.0, 'validation_timeout': 10, 'timeout': 240}

validate_code 

validate_code(agent: Player) -> tuple[bool, str | None]
Source code in codeclash/arenas/cyborg/cyborg.py 
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
def validate_code(self, agent: Player) -> tuple[bool, str | None]:
    quoted_submission = shlex.quote(self.submission)
    file_check = agent.environment.execute(f"test -f {quoted_submission} && echo exists")
    if "exists" not in file_check["output"]:
        return False, f"Submission file `{self.submission}` not found in the workspace root"

    content = agent.environment.execute(f"cat {quoted_submission}")["output"]
    if not content.strip():
        return False, f"`{self.submission}` is empty"

    syntax_check = agent.environment.execute(f"python -m py_compile {quoted_submission}")
    if syntax_check["returncode"] != 0:
        return False, f"Python syntax error in `{self.submission}`:\n{syntax_check['output']}"

    validation_timeout = int(self._game_arg("validation_timeout"))
    try:
        import_check = agent.environment.execute(
            "python - <<'PY'\n"
            "import importlib.util\n"
            f"spec = importlib.util.spec_from_file_location('submission_agent', {self.submission!r})\n"
            "module = importlib.util.module_from_spec(spec)\n"
            "spec.loader.exec_module(module)\n"
            "assert hasattr(module, 'decide'), 'decide function not found'\n"
            "assert callable(module.decide), 'decide must be callable'\n"
            "result = module.decide([0, 1, 0], {'type': 'discrete', 'n': 11})\n"
            "assert result is None or isinstance(result, int), 'decide must return an integer action or None'\n"
            "PY",
            timeout=validation_timeout,
        )
    except subprocess.TimeoutExpired:
        return False, f"`decide` validation exceeded {validation_timeout}s timeout"
    if import_check["returncode"] != 0:
        return False, f"Could not import or call `decide` from `{self.submission}`:\n{import_check['output']}"

    return True, None

execute_round 

execute_round(agents: list[Player]) -> None
Source code in codeclash/arenas/cyborg/cyborg.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
def execute_round(self, agents: list[Player]) -> None:
    agent_args = []
    for agent in agents:
        agent_args.extend(["--agent", f"{agent.name}=/{agent.name}/{self.submission}"])

    cmd = [
        "python",
        "run_cyborg.py",
        "--episodes",
        str(self._episodes_per_round()),
        "--steps",
        str(self._game_arg("steps_per_episode")),
        "--drones",
        str(self._game_arg("num_drones")),
        "--decision-timeout",
        str(self._game_arg("decision_timeout")),
        "--output",
        str(self.log_env / RESULTS_JSON),
        *agent_args,
    ]
    full_cmd = " ".join(shlex.quote(part) for part in cmd)
    self.logger.info(f"Running game: {full_cmd}")
    try:
        response = self.environment.execute(full_cmd, timeout=int(self._game_arg("timeout")))
    except subprocess.TimeoutExpired as exc:
        raise RuntimeError("CybORG round timed out") from exc
    assert_zero_exit_code(response, logger=self.logger)

get_results 

get_results(agents: list[Player], round_num: int, stats: RoundStats)
Source code in codeclash/arenas/cyborg/cyborg.py 
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
def get_results(self, agents: list[Player], round_num: int, stats: RoundStats):
    result_file = self.log_round(round_num) / RESULTS_JSON
    if not result_file.exists():
        self.logger.error(f"Missing result file: {result_file}")
        stats.winner = RESULT_TIE
        for agent in agents:
            stats.scores[agent.name] = CRASH_SCORE
            stats.player_stats[agent.name].score = CRASH_SCORE
            stats.details.append(
                json.dumps(
                    {
                        "player": agent.name,
                        "score": CRASH_SCORE,
                        "status": "error",
                        "error": f"missing CybORG result file: {result_file}",
                    },
                    sort_keys=True,
                )
            )
        return

    with open(result_file) as f:
        result = json.load(f)

    scores = {agent.name: 0.0 for agent in agents}
    for player, score in result.get("average_scores", {}).items():
        if player in scores:
            scores[player] = float(score)

    stats.scores = scores
    stats.details = result.get("details", [])
    for player, score in scores.items():
        stats.player_stats[player].score = score

    if not scores:
        stats.winner = RESULT_TIE
        return

    top_score = max(scores.values())
    winners = [player for player, score in scores.items() if score == top_score]
    stats.winner = winners[0] if len(winners) == 1 else RESULT_TIE

Agent Interface

Your bot must be a Python file named cyborg_agent.py that defines decide(observation, action_space).

observation is a plain list converted from CybORG's observation array. action_space is a dictionary such as {"type": "discrete", "n": 11}. Return an integer action accepted by that action space. A valid starting point is:

def decide(observation, action_space):
    return 0

For each episode, the same policy file controls all blue-team drone agents through isolated policy worker processes. The runtime validates every returned action before stepping the simulator.

Configuration Example

tournament:
  rounds: 1
game:
  name: CybORG
  sims_per_round: 2
  args:
    steps_per_episode: 5
    num_drones: 8
    decision_timeout: 3.0
    validation_timeout: 10
    timeout: 240
players:
  - agent: dummy
    name: alpha
  - agent: dummy
    name: beta

Scoring

The arena runs sims_per_round independent simulated DroneSwarm episodes for each submitted player. Each player receives the sum of mean blue-agent rewards per episode. The final CodeClash score is the average episode score across the round.

The runtime pins CybORG to the upstream v3.0 code and installs it editable from a checked-out repository because the upstream package expects data files such as CybORG/version.txt to be present next to the source tree.

Smoke Test

From the repository root, run the dummy-player example:

uv run codeclash run configs/examples/CybORG__dummy__r1__s2.yaml -o /tmp/codeclash-cyborg-smoke

Use a fresh -o directory when rerunning the smoke check.

Expected shape:

  • the command exits with status 0;
  • both players pass submission validation;
  • stdout includes In round 0, the winner is ... and In round 1, the winner is ...;
  • each round summary contains floating-point average rewards for alpha and beta;
  • per-episode details have status: "ok", steps_completed: 5, policy_errors, invalid_actions, and decisions;
  • the output directory contains metadata.json, game.log, tournament.log, and rounds/round_0.tar.gz / rounds/round_1.tar.gz.

A representative metadata.json round contains a scores object with one floating-point episode reward per player:

"scores": {
  "alpha": -27.0,
  "beta": -33.5
}

Exact values can change with simulator randomness and configuration; the smoke check is meant to verify the Docker/runtime adapter path, player-name mapping, and score/log artifact shape.

The exact tournament directory name includes a timestamp, so inspect the metadata with:

find /tmp/codeclash-cyborg-smoke -maxdepth 3 -name metadata.json -print
bug_report Something broken/unclear?

Open an issue on GitHub!
help Open-ended discussions

Join our Slack!
Our projects
SWE-bench SWE-bench SWE-agent SWE-agent SWE-smith SWE-smith SWE-rex SWE-ReX sb-cli sb-cli Slack Join our Slack