Run evaluations locally using promptfoo to assess model quality across accuracy, relevance, and coherence dimensions.
# Install promptfoo npm install -g promptfoo # Navigate to eval directory cd eval/ # Run evaluation promptfoo eval # View results in browser promptfoo view
Below is a sample comparison from a previous evaluation run. Run promptfoo eval to generate fresh results.
| Model | Accuracy | Relevance | Coherence | Avg Latency | Avg Cost |
|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.92 | 0.94 | 0.91 | 1.8s | $0.0042 |
| Claude Opus 4.6 | 0.96 | 0.95 | 0.94 | 3.2s | $0.0210 |
| Amazon Nova Pro | 0.85 | 0.88 | 0.84 | 1.2s | $0.0009 |
| Amazon Nova Lite | 0.78 | 0.82 | 0.80 | 0.6s | $0.0001 |
| Dimension | Description | Scoring |
|---|---|---|
| Accuracy | Factual correctness of the response | LLM-as-judge (0-1) |
| Relevance | How well the response addresses the prompt | LLM-as-judge (0-1) |
| Coherence | Logical flow and clarity of the response | LLM-as-judge (0-1) |