CodeArena (Abstract Base)

The CodeArena class is the abstract base class for all game arenas in CodeClash.

Overview

Every game in CodeClash extends CodeArena and implements three key methods:

1. validate_code(): Verify that player submissions are valid
2. execute_round(): Run the actual game
3. get_results(): Determine winners and scores

Key Concepts

Game Lifecycle

1. Initialization: Game container is created with Docker
2. Round Execution: For each round:
3. Pre-round setup (copy player code)
4. Validation (check all submissions)
5. Execution (run the game)
6. Results (determine winner)
7. Post-round (save logs)
8. Cleanup: Remove containers and artifacts

Docker Integration

Each game runs in its own Docker container with:

• Game engine installed
• Git repository initialized
• Player code copied in

Class Reference

codeclash.arenas.arena.CodeArena

CodeArena(config: dict, *, tournament_id: str, local_output_dir: Path, keep_containers: bool = False)

Bases: ABC

The CodeArena class is responsible for running games, i.e., taking a list of code from different agents/players and running them against each other. It also provides the environments for the game and agents to run in.

The central method is run_round, which takes a list of agents and returns the winner of the round.

At the end of the the tournament, run the end method to clean up the game and agents and write the metadata.

Parameters:

Name	Type	Description	Default
config	dict	The overall config for the tournament.	required
tournament_id	str	The id of the tournament.	required
local_output_dir	Path	The host/local directory to write logs to.	required
keep_containers	bool	Do not remove containers after games/agent finish.	False

Source code in codeclash/arenas/arena.py

def __init__(self, config: dict, *, tournament_id: str, local_output_dir: Path, keep_containers: bool = False):
    """The CodeArena class is responsible for running games, i.e., taking a list of code
    from different agents/players and running them against each other.
    It also provides the environments for the game and agents to run in.

    The central method is `run_round`, which takes a list of agents and returns the winner of the round.

    At the end of the the tournament, run the `end` method to clean up the game and agents and write the metadata.

    Args:
        config: The overall config for the tournament.
        tournament_id: The id of the tournament.
        local_output_dir: The host/local directory to write logs to.
        keep_containers: Do not remove containers after games/agent finish.
    """
    self.url_gh: str = f"git@github.com:{GH_ORG}/{self.name}.git"
    self.artifacts: list[Path] = []
    """Artifact objects that we might want to clean up after the game."""
    self.config: dict = config
    self._keep_containers: bool = keep_containers
    self._metadata: dict = {
        "name": self.name,
        "config": self.config["game"],
        "game_id": tournament_id,
        "created_timestamp": int(time.time()),
    }
    self.log_env: Path = DIR_LOGS
    self.log_local: Path = local_output_dir
    self.logger = get_logger(self.name, log_path=self.log_local / "game.log", emoji="🏓")
    self.environment: DockerEnvironment = self.get_environment()
    """The running docker environment for executing the game"""

name instance-attribute

name: str

description instance-attribute

description: str

default_args class-attribute instance-attribute

default_args: dict = {}

submission instance-attribute

submission: str

url_gh instance-attribute

url_gh: str = f'git@github.com:{GH_ORG}/{self.name}.git'

artifacts instance-attribute

artifacts: list[Path] = []

Artifact objects that we might want to clean up after the game.

config instance-attribute

config: dict = config

log_env instance-attribute

log_env: Path = DIR_LOGS

log_local instance-attribute

log_local: Path = local_output_dir

logger instance-attribute

logger = get_logger(self.name, log_path=self.log_local / 'game.log', emoji='🏓')

environment instance-attribute

environment: DockerEnvironment = self.get_environment()

The running docker environment for executing the game

game_config property

game_config: dict

game_id property

game_id: str

image_name property

image_name: str

build_image

build_image()

Build a Docker image for the game using the Dockerfile in the codebase. If running in AWS, pull the image from the AWS Docker registry instead.

Source code in codeclash/arenas/arena.py

def build_image(self):
    """
    Build a Docker image for the game using the Dockerfile in the codebase.
    If running in AWS, pull the image from the AWS Docker registry instead.
    """
    if is_running_in_aws_batch():
        pull_game_container_aws_ecr(game_name=self.name, image_name=self.image_name, logger=self.logger)

    # Hold the lock across check-and-build so concurrent pairs don't race: the first thread
    # builds while the rest wait, then find the image already present and skip.
    with CodeArena._build_lock:
        self.logger.debug(f"Checking if container {self.image_name} exists")
        result = subprocess.run(
            f"docker images -q {self.image_name}",
            shell=True,
            capture_output=True,
            text=True,
        )
        if result.stdout.strip():
            self.logger.debug(f"Container {self.image_name} exists")
            return

        self.logger.info(
            f"Building Docker image {self.image_name}. This may take 1-5 minutes and only work on Linux for some games."
        )

        # NOTE: Assuming Dockerfile is declared in same directory as the arena.
        arena_file = Path(inspect.getfile(self.__class__))
        folder_path = arena_file.parent
        result = subprocess.run(
            f"docker build --no-cache -t {self.image_name} -f {folder_path}/{self.name}.Dockerfile .",
            shell=True,
            capture_output=True,
            text=True,
        )
        if result.returncode == 0:
            self.logger.info(f"✅ Built Docker image {self.image_name}")
        else:
            self.logger.error(f"❌ Failed to build Docker image: {result.stderr}\n{result.stdout}{result.stderr}")
            raise RuntimeError(f"Failed to build Docker image: {result.stderr}")

copy_logs_from_env

copy_logs_from_env(round_num: int) -> None

Copy logs from the game's environment to the local machine.

Source code in codeclash/arenas/arena.py

def copy_logs_from_env(self, round_num: int) -> None:
    """Copy logs from the game's environment to the local machine."""
    (self.log_local / "rounds" / str(round_num)).mkdir(parents=True, exist_ok=True)
    copy_from_container(
        container=self.environment,
        src_path=str(self.log_env) + "/.",
        dest_path=self.log_round(round_num),
    )

    # Remove logs from container to save space
    assert_zero_exit_code(
        self.environment.execute(f"rm -rf {self.log_env}"),
        logger=self.logger,
    )

end

end(cleanup: bool = False)

Source code in codeclash/arenas/arena.py

def end(self, cleanup: bool = False):
    if cleanup:
        for artifact in self.artifacts:
            if artifact.exists():
                subprocess.run(f"rm -rf {artifact}", shell=True)
        self.logger.info(f"🧼 Cleaned up {self.name} game")

log_round

log_round(round_num: int) -> Path

Source code in codeclash/arenas/arena.py

def log_round(self, round_num: int) -> Path:
    return self.log_local / "rounds" / str(round_num)

get_environment

get_environment(branch_name: str | None = None) -> DockerEnvironment

Get docker container ID with the game code installed.

Source code in codeclash/arenas/arena.py

def get_environment(self, branch_name: str | None = None) -> DockerEnvironment:
    """Get docker container ID with the game code installed."""
    self.build_image()
    if not self._keep_containers:
        run_args = ["--rm"]
    else:
        run_args = []
    environment = ClashDockerEnvironment(
        image=self.image_name,
        cwd=str(DIR_WORK),
        env={
            "GITHUB_TOKEN": os.getenv("GITHUB_TOKEN", ""),
            "PAGER": "cat",
            "MANPAGER": "cat",
            "LESS": "-R",
            "PIP_PROGRESS_BAR": "off",
            "TQDM_DISABLE": "1",
        },
        container_timeout="10h",
        logger=self.logger,
        run_args=run_args,
    )

    branch_name = self.game_id if branch_name is None else branch_name

    # Logger setting will likely not take effect for initial container creation logs
    environment.logger = get_logger("environment", emoji="🪴")
    for cmd in [
        f"git branch {branch_name}",
        f"git checkout {branch_name}",
        'git config --global user.email "player@codeclash.com"',
        'git config --global user.name "Player"',
        "git config --global commit.gpgsign false",
    ]:
        assert_zero_exit_code(environment.execute(cmd), logger=self.logger)
    return environment

get_metadata

get_metadata() -> dict

This is what we write to metadata.json. You can subclass extend this to add more details for specific games.

Source code in codeclash/arenas/arena.py

def get_metadata(self) -> dict:
    """This is what we write to metadata.json.
    You can subclass extend this to add more details for specific games.
    """
    return self._metadata

run_round

run_round(agents: list[Player], round_num: int, *, copy_logs: bool = True) -> RoundStats

Run a single round of the game with the given agents.

Returns the log output, result output, and winner name. All bookkeeping should be handled by the tournament class.

Source code in codeclash/arenas/arena.py

def run_round(self, agents: list[Player], round_num: int, *, copy_logs: bool = True) -> RoundStats:
    """
    Run a single round of the game with the given agents.

    Returns the log output, result output, and winner name. All bookkeeping should be
    handled by the tournament class.
    """
    all_names = {agent.name for agent in agents}
    assert len(all_names) == len(agents), "All agents must have unique names"

    random.shuffle(agents)  # Shuffle to ensure fairness in case of positional advantages
    stats = RoundStats(round_num, agents)
    validated: list[Player] = []
    for a in agents:
        is_valid, error = self.validate_code(a)
        if not is_valid:
            self.logger.warning(f"Agent {a.name} failed submission validation: {error}")
            stats.player_stats[a.name].invalid_reason = error
            continue
        self.logger.info(f"Agent {a.name} passed submission validation")
        stats.player_stats[a.name].valid_submit = True
        validated.append(a)

    sims = self.config["game"]["sims_per_round"]
    if len(validated) > 1:
        self._pre_round_setup(validated)
        self.execute_round(validated)
        if copy_logs:
            self.copy_logs_from_env(round_num)
        self.get_results(validated, round_num, stats)
    elif len(validated) == 1:
        self.logger.info(f"Only one valid agent ({validated[0].name}), automatic win")
        stats.winner = validated[0].name
        stats.scores[validated[0].name] = sims
        stats.player_stats[validated[0].name].score = sims
    else:
        self.logger.info("No valid agents, no winner this round (Default tie)")
        stats.winner = RESULT_TIE
        # Split points evenly
        points = sims * 1.0 / len(agents)
        for a in agents:
            stats.scores[a.name] = points
            stats.player_stats[a.name].score = points
    return stats

get_results abstractmethod

get_results(agents: list[Player], round_num: int, stats: RoundStats)

Determine the winner of the game based on the result output. Modifies the stats object in place.

Parameters:

Name	Type	Description	Default
agents	list[Player]	List of agents participating in the round	required

Source code in codeclash/arenas/arena.py

@abstractmethod
def get_results(self, agents: list[Player], round_num: int, stats: RoundStats):
    """Determine the winner of the game based on the result output.
    Modifies the stats object in place.

    Args:
        agents: List of agents participating in the round
    """
    pass

execute_round abstractmethod

execute_round(agents: list[Player])

Subclasses implement their game-specific logic here. This is the low level implementation, you probably want to use run_round instead, which includes the pre-round setup, post-round setup, and winner determination.

Source code in codeclash/arenas/arena.py

@abstractmethod
def execute_round(self, agents: list[Player]):
    """Subclasses implement their game-specific logic here.
    This is the low level implementation, you probably want to use run_round instead, which
    includes the pre-round setup, post-round setup, and winner determination.
    """
    pass

validate_code abstractmethod

validate_code(agent: Player) -> tuple[bool, str | None]

Verify that the given agent can be run by the game.

Parameters:

Name	Type	Description	Default
agent	Player	The agent to verify	required

Returns:

Type	Description
bool	Boolean indicating whether the agent passed verification
str | None	Optional string indicating reason for failure

Source code in codeclash/arenas/arena.py

@abstractmethod
def validate_code(self, agent: Player) -> tuple[bool, str | None]:
    """Verify that the given agent can be run by the game.

    Args:
        agent: The agent to verify

    Returns:
        Boolean indicating whether the agent passed verification
        Optional string indicating reason for failure
    """
    pass

Supporting Classes

PlayerStats

codeclash.arenas.arena.PlayerStats

PlayerStats(name: str)

Source code in codeclash/arenas/arena.py

def __init__(self, name: str):
    self.name = name
    self.invalid_reason: str = ""
    self.score: float = 0.0
    self.valid_submit = False

name instance-attribute

name = name

invalid_reason instance-attribute

invalid_reason: str = ''

score instance-attribute

score: float = 0.0

valid_submit instance-attribute

valid_submit = False

to_dict

to_dict() -> dict[str, Any]

Source code in codeclash/arenas/arena.py

def to_dict(self) -> dict[str, Any]:
    return {
        "name": self.name,
        "invalid_reason": self.invalid_reason,
        "score": self.score,
        "valid_submit": self.valid_submit,
    }

RoundStats

codeclash.arenas.arena.RoundStats

RoundStats(round_num: int, agents: list[Player])

Source code in codeclash/arenas/arena.py

def __init__(self, round_num: int, agents: list[Player]):
    self.winner = None
    self.round_num = round_num
    # Map of player to game metric (e.g. # of wins, assets accumulated)
    self.scores: dict[str, float] = {a.name: 0.0 for a in agents}
    self.player_stats: dict[str, PlayerStats] = {agent.name: PlayerStats(name=agent.name) for agent in agents}
    self.details: list[str] = []

winner instance-attribute

winner = None

round_num instance-attribute

round_num = round_num

scores instance-attribute

scores: dict[str, float] = {(a.name): 0.0 for a in agents}

player_stats instance-attribute

player_stats: dict[str, PlayerStats] = {(agent.name): (PlayerStats(name=agent.name)) for agent in agents}

details instance-attribute

details: list[str] = []

to_dict

to_dict() -> dict[str, Any]

Source code in codeclash/arenas/arena.py

def to_dict(self) -> dict[str, Any]:
    # Going through some pain to ensure that the scores dict is always complete
    player_names = set(self.player_stats.keys()) | set(self.scores.keys())
    return {
        "round_num": self.round_num,
        "winner": self.winner,
        "details": self.details,
        "scores": {name: self.scores.get(name, 0.0) for name in player_names},
        "player_stats": {name: stats.to_dict() for name, stats in self.player_stats.items()},
    }

bug_report Something broken/unclear?

Open an issue on GitHub!

help Open-ended discussions

Join our Slack!

Our projects

SWE-bench SWE-agent SWE-smith SWE-ReX sb-cli Join our Slack