Issue 10 | Performance Benchmarks and Best Practices — Integrating Caveman into Your Development Workflow

🎯 Learning Objectives

After completing this issue, you will master:

1. How to run Caveman's official Benchmark and Eval suites
2. The Three-Arm Evaluation methodology: Why Caveman is better than "please answer briefly"
3. A complete daily development workflow: The full chain from startup to commit
4. Optimal mode selection strategies for different scenarios

📖 Core Content

10.1 Official Benchmark Data

Caveman's token compression effect isn't self-proclaimed—it's backed by real Claude API token count data.

Statistical Summary:

• Range: 22% — 87%
• Average: ~71-75%
• Median: ~75%

💡 Important: Caveman only affects output tokens. Thinking/reasoning tokens are completely unaffected. Caveman doesn't shrink the brain, it just shrinks the mouth.

10.2 Running Official Benchmarks

You can reproduce this data yourself:

# Clone the repository
git clone https://github.com/JuliusBrussee/caveman.git
cd caveman

# Run LLM evaluation (requires Claude CLI and a valid API Key)
uv run python evals/llm_run.py

# Analyze results offline (no API Key needed)
uv run --with tiktoken python evals/measure.py

Three-Arm Evaluation Design (Three-Arm Eval)

Caveman's Eval doesn't simply compare "Normal vs Caveman"—that would conflate Caveman's effect with "generic brief instructions."

graph TD
    A["Three-Arm Evaluation Design"]
    
    A --> B["Arm 1: Verbose
(No Constraints)
Claude Normal Response"]
    A --> C["Arm 2: Terse
(Only 'be brief')
General Brief Instruction"]
    A --> D["Arm 3: Caveman
(Full Skill Rules)
Structured Compression"]
    
    B --> E["Baseline Comparison"]
    C --> F["Proves Caveman ≠ Simply 'be brief'"]
    D --> G["Actual Compression Effect"]
    
    F -.->|"Comparison"| G

Why a Three-Arm Design?

If you only compare Verbose vs Caveman, you cannot distinguish whether the compression effect comes from:

• Caveman's structured rules ([thing] [action] [reason] pattern)
• Or simply because you told the Agent "please answer briefly"

In the three-arm design, Arm 2 (Terse) is the control group—it only says "be brief." If Caveman saves more tokens than Terse and maintains higher accuracy, it proves that Caveman's rule design itself has value, and is not just "asking for brevity."

Actual results: Caveman saves an additional 15-25% tokens compared to Terse mode, with higher technical accuracy.

10.3 Academic Background: Brevity ≠ Coarseness

A March 2026 paper, "Brevity Constraints Reverse Performance Hierarchies in Language Models", found:

graph LR
    A["Traditional Assumption
More Tokens = Better Answer"] -->|"❌ Paper Disproved"| B["Experimental Results
Brevity Constraint Improves Accuracy by 26%"]
    
    C["Large Models (Verbose)"] -->|"Add Brevity Constraint"| D["Accuracy Improvement"]
    E["Small Models (Concise)"] -->|"No Constraint"| F["Accuracy Even Higher"]
    
    D --> G["Conclusion: Verbosity is Noise
Not Signal"]
    F --> G

Key Findings:

1. Brevity constraints improve accuracy by 26 percentage points (on specific benchmarks)
2. Reverses model rankings: Smaller models that originally performed worse surpassed larger models under brevity constraints
3. Verbosity is noise: The computational power models spend on rhetoric could be used for reasoning

This academically validates Caveman's core hypothesis: Remove the fluff, and reasoning becomes more accurate.

10.4 Complete Caveman Daily Workflow

graph TD
    A["🚀 Start Agent Session"] --> B["Hook Automatically Activates Caveman
[CAVEMAN] Badge Lights Up"]
    
    B --> C{"Development Phase"}
    
    C -->|"🔨 Coding"| D["🪨 /caveman full
Concise Technical Answers
Troubleshooting, Writing Code"]
    
    C -->|"🐛 Debugging"| E["🔥 /caveman ultra
Rapid Troubleshooting
Minimal Text to Core Issue"]
    
    C -->|"📖 Learning"| F["🪶 /caveman lite
Retains Full Sentences
Easier Concept Understanding"]
    
    C -->|"🇨🇳 Chinese Projects"| G["📜 /caveman wenyan
Classical Chinese Mode
Most Token-Efficient for Chinese"]
    
    D --> H["✅ Code Modification Complete"]
    E --> H
    F --> H
    G --> H
    
    H --> I["🔍 /caveman-review
One-Line Code Review
L42: 🔴 bug: ..."]
    
    I --> J{"Review Passed?"}
    J -->|"❌ Issues Found"| K["Fix Issues"]
    K --> I
    
    J -->|"✅ Passed"| L["📝 /caveman-commit
Refined Commit Message
fix(auth): token <= not <"]
    
    L --> M["📦 git push"]
    
    M --> N["🗜️ /caveman:compress
Compress CLAUDE.md
Saves Tokens for Next Session"]
    
    N --> O["🎉 Done!"]
    
    style B fill:#FFD700
    style I fill:#87CEEB
    style L fill:#90EE90
    style N fill:#DDA0DD

10.5 Scenario × Mode Selection Matrix

10.6 Full Workflow Comparison Across Platforms

10.7 Advanced Best Practices

Practice 1: CLAUDE.md Layered Strategy

~/.claude/CLAUDE.md          ← Global Caveman always-on (applies to all projects)
<project>/CLAUDE.md          ← Project-specific rules (already compressed)
<project>/CLAUDE.original.md ← Human-readable original (edit this)

Practice 2: Team-wide Unified Configuration

# Commit Caveman configuration in the project root
echo 'Terse like caveman. Technical substance exact...' >> CLAUDE.md
echo 'Terse like caveman. Technical substance exact...' >> GEMINI.md

# Ensure all team members use the same Caveman behavior
git add CLAUDE.md GEMINI.md
git commit -m "chore: add caveman always-on for team"

Practice 3: CI/CD Integration

# .github/workflows/pr-review.yml
- name: Caveman Code Review
  run: |
    # Use Claude Code Action + caveman-review rules
    # Each PR automatically gets a one-line review

Practice 4: Combining with cavemem

# Install cavemem (memory compression)
# Combine with caveman (output compression) for dual optimization
npm install -g cavemem

# caveman compresses output → saves output tokens
# cavemem compresses memory → saves input tokens  
# Combined → total token consumption reduced by 60%+

Practice 5: Customizing Caveman Rules

If you need domain-specific Caveman rules, you can create custom Skills:

<!-- .claude/skills/my-caveman/SKILL.md -->
## My Custom Caveman Rules

Base: Terse like caveman. Technical substance exact.

Additional rules for this project:
- Always mention file paths in full
- Include line numbers when discussing bugs
- Use Chinese for variable name explanations
- Keep API endpoint paths in backticks

📊 Return on Investment Summary

graph LR
    subgraph Investment["💰 Investment"]
        A1["Installation: 1 min"]
        A2["Configuration: 5 min"]
        A3["Learning: This 10-part tutorial"]
    end
    
    subgraph Return["📈 Return"]
        B1["Output Tokens: -75%"]
        B2["Input Tokens: -46%"]
        B3["Response Speed: +3x"]
        B4["Monthly Cost: -$46"]
        B5["Readability: ↑"]
    end
    
    Investment --> Return

📝 Full Series Review

🎓 Graduation Tasks

Complete the following tasks to become a qualified Caveman user:

• Install Caveman on your primary Agent
• Complete a full feature development using full mode
• Review your own code using /caveman-review
• Generate a commit message using /caveman-commit
• Compress your CLAUDE.md using /caveman:compress
• Configure Always-On to ensure automatic activation in the next session
• (Bonus) Commit the configuration to Git so your team can also use Caveman

🔗 References

• Caveman Official Repository (42.9k ⭐)
• Caveman Benchmarks
• Caveman Evals
• Paper: Brevity Constraints Reverse Performance Hierarchies
• cavemem - Memory Compression
• cavekit - Build Optimization
• npx skills tool
• Claude Code Official Documentation