> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openclaas.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval Harness Overview

> Automated evaluation for SDPO continual learning

# Eval Harness

The CLaaS eval harness runs automated feedback loops against a live CLaaS stack and measures whether training shifts the model toward preferred behaviours without collapsing.

## What it does

1. **Initializes a fresh LoRA** adapter for each preference being tested
2. **Runs N feedback steps**, each sending a preference-targeted prompt and feedback to the CLaaS API
3. **Measures metrics** at each step: logprob margins, compliance rates, capability retention, and collapse signals
4. **Generates plots** (optional) showing learning curves over training steps
5. **Produces a summary** with per-preference pass/fail verdicts

## Preferences

The eval harness ships with three built-in preferences:

| Preference | What it trains              | Example feedback                            |
| ---------- | --------------------------- | ------------------------------------------- |
| `no_emoji` | Suppress emoji in responses | "Don't use any emoji in your responses"     |
| `concise`  | Keep responses short        | "Be more concise, use 3 sentences or fewer" |
| `identity` | Adopt a persona             | "Your name is Kuro"                         |

Preferences are configured via the `preferences` list in the Hydra config. You can run any subset:

```bash theme={null}
uv run python -m claas.eval 'preferences=[concise]' num_steps=10
```

## Output structure

Each eval run produces a directory under `data/evals/<run-id>/`:

```text theme={null}
data/evals/<run-id>/
├── summary.json              # Per-preference pass/fail verdicts
└── <preference>/
    ├── metadata.json          # Run config + LoRA ID
    ├── baseline.json          # Pre-training metric snapshot
    └── steps.jsonl            # One JSON object per feedback step
```

Each line in `steps.jsonl` contains: step number, timestamp, feedback given, SDPO training metrics, eval metrics (logprob margin, compliance, general capability, collapse), and rollout transcripts.

## Viewing results

Results can be viewed in the browser via the built-in eval dashboard served by the CLaaS API:

```
GET http://localhost:8080/v1/eval?results_dir=./data/evals
```