Skip to content

RPG-API DM Quality Improvements

Status: Approved (Post Peer Review) Date: 2025-01-01 Author: Claude Code Reviewer: Human

1. Current Architecture

High-Level System

┌──────────────────────────────────────────────────────────────────┐
│                     rpg-web (Next.js)                             │
│  Chat Interface (assistant-ui) → typewriter animation             │
│  Campaign selector, session persistence (localStorage)           │
└─────────────────────────┬────────────────────────────────────────┘
                          │ HTTP (K8s internal)
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                     palimpsest (FastAPI)                             │
│                                                                   │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │              DM Agent (Pydantic AI + DeepSeek)              │  │
│  │  - System prompt with persona, rules, response format       │  │
│  │  - Multi-tool calls per turn (native Pydantic AI)           │  │
│  │  - Tools: roll_dice, update_pc, get_entity, add_thread...   │  │
│  └────────────────────────────────────────────────────────────┘  │
│                          │                                        │
│  ┌───────────────────────┴────────────────────────────────────┐  │
│  │              Context Injection Layer                        │  │
│  │  ContextBuilder → builds from vault:                        │  │
│  │  - PC state (HP, location, conditions, gold)                │  │
│  │  - NPCs at current location (disposition, role)             │  │
│  │  - Active plot threads (high/medium priority)               │  │
│  │  - Current location description                             │  │
│  │  - Recent timeline events                                   │  │
│  └────────────────────────────────────────────────────────────┘  │
│                          │                                        │
│  ┌───────────────────────┴────────────────────────────────────┐  │
│  │                RAG Layer (pgvector)                         │  │
│  │  - Semantic search on vault documents                       │  │
│  │  - OpenAI text-embedding-3-small (1536 dims)               │  │
│  │  - Threshold-based retrieval (default 0.6)                 │  │
│  │  - Hybrid: deterministic context + RAG on follow-ups       │  │
│  └────────────────────────────────────────────────────────────┘  │
└──────────────────────────┬───────────────────────────────────────┘
          ┌────────────────┴────────────────┐
          ▼                                 ▼
┌─────────────────────┐         ┌────────────────────────┐
│   Campaign Vault    │         │   Supabase (PostgreSQL) │
│   (PVC Mount)       │         │                         │
│   /vault/{campaign} │         │   rpg.campaigns         │
│   ├── canon/        │         │   rpg.sessions          │
│   │   ├── pcs/      │         │   rpg.chat_messages     │
│   │   ├── npcs/     │         │     - injected_context  │
│   │   ├── locations/│         │     - rag_evidence      │
│   │   └── ...       │         │     - tool_calls        │
│   └── sessions/     │         └────────────────────────┘
└─────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                      pgvector (PostgreSQL)                       │
│   rpg_embeddings.embeddings                                      │
│     - campaign_id (UUID5 isolation per campaign)                 │
│     - source_type (npc, location, item, etc.)                   │
│     - source_id, chunk_index                                     │
│     - content, embedding vector (1536 dims)                      │
│     - metadata (JSONB)                                           │
└─────────────────────────────────────────────────────────────────┘

Key Code Paths

File Purpose
api/agent/dm.py DM system prompt + tool definitions
api/context/builder.py SessionContext construction
api/routes/chat.py Chat endpoint + context injection
api/rag/service.py RAG retrieval logic
api/chat_history.py Supabase message persistence
lib/vault.py Vault layer for entity access

2. Gaps Identified

Gap 1: Companions/Familiars Not Included

Discovery: Pip the pseudodragon exists at canon/pcs/pip.md with proper frontmatter but is filtered out.

Root Cause: ContextBuilder._load_pc_state() explicitly filters for type: pc: 

actual_pcs = [p for p in pcs if p.frontmatter.get("type") == "pc"]

Impact: Pip is never mentioned in context injection. DM forgets Pip exists entirely.

Gap 2: Dice Roll Results Not Visible

Problem: DM calls roll_dice tool but results are only in tool response, not visible to players.

Impact: Players can't see dice math. Reduces transparency and D&D feel.

Gap 3: Relationship Texture Missing

Problem: disposition: intimate-ally is shown as a label but doesn't convey behavioral implications.

Impact: Model treats it as abstract. Doesn't produce warmth, physical affection, or genuine concern.

Gap 4: No /context Debug Command

Problem: No way for players to see what context the DM is receiving.

Gap 5: No RAG Source Display

Problem: RAG retrieves evidence but doesn't show sources to players.

3. AnythingLLM Comparison

Aspect AnythingLLM RPG-API Winner
Pip Mentions Pip perches, reacts, nuzzles Pip never mentioned AnythingLLM
Dice Visibility Inline roll display Hidden in tool call AnythingLLM
Marlena Warmth Physical touch, concern Businesslike AnythingLLM
Multi-tool Calls Workaround macros Native support RPG-API
Token Cost ~16k/request ~4k/request RPG-API
State Persistence File-only Supabase + files RPG-API

4. Proposed Changes (Post Peer Review)

4.1 Vault Layer: Add list_party_members()

File: lib/vault.py

Rationale: Don't overload list_pcs(). Create a proper method that returns all party members including companions.

def list_party_members(self, pc_id: str) -> list[VaultEntity]:
    """Get all active party members for a PC.

    Returns:
        - The PC itself
        - Familiars/companions bonded to the PC
        - NPCs with in_party: true
    """
    entities = self.list_pcs()  # All entities in pcs/ folder
    return [
        e for e in entities
        if e.frontmatter.get("status") == "active"
        and (
            e.frontmatter.get("type") == "pc"
            or e.frontmatter.get("bonded_to") == pc_id
            or e.frontmatter.get("in_party") == True
        )
    ]

Key changes from original plan: - Match by pc_id not pc_name (stable identifier) - Filter by status == "active" (exclude dead/dismissed) - Support in_party: true for non-bonded party NPCs (hirelings)

4.2 Companion Frontmatter Schema

File: Update canon/pcs/pip.md (and future companions)

Rationale: Store personality in frontmatter, not markdown sections. Avoids brittle regex parsing.

---
type: familiar
status: active
bonded_to: jake  # Use PC id, not display name
species: Pseudodragon
personality_traits:
  - Mischievous and curious
  - Fiercely loyal to Jake
  - Easily bribed with food
presence: with_pc  # with_pc | outside | stabled
---
# Pip

Pip is Jake's pseudodragon familiar...

New fields: - bonded_to: PC id (lowercase, stable) - personality_traits: List in frontmatter (no regex needed) - presence: Where companion is relative to PC (mounts stay outside)

4.3 Context Builder Updates

File: api/context/builder.py

Add pc_id to SessionContext: 

@dataclass
class SessionContext:
    # ... existing fields ...
    pc_id: str = ""  # Stable identifier for matching
    companions: list[CompanionContext] = field(default_factory=list)

Add CompanionContext: 

@dataclass
class CompanionContext:
    """Companion/familiar for context injection."""
    id: str
    name: str
    type: str  # familiar, companion, mount
    bonded_to: str
    personality_traits: list[str] = field(default_factory=list)
    presence: str = "with_pc"  # with_pc, outside, stabled

Update _load_pc_state() to set pc_id: 

def _load_pc_state(self, ctx: SessionContext) -> None:
    # ... existing code ...
    ctx.pc_id = pc.id  # Store the stable ID
    ctx.pc_name = fm.get("name") or pc.id

Add _load_companions(): 

def _load_companions(self, ctx: SessionContext) -> None:
    """Load companions bonded to active PC."""
    if not ctx.pc_id:
        return

    try:
        party = self.vault.list_party_members(ctx.pc_id)
        for entity in party:
            fm = entity.frontmatter
            entity_type = fm.get("type", "pc")

            # Skip the PC itself
            if entity_type == "pc":
                continue

            # Only include companions that are present
            presence = fm.get("presence", "with_pc")
            if presence == "stabled":
                continue  # Skip stabled mounts

            ctx.companions.append(CompanionContext(
                id=entity.id,
                name=fm.get("name", entity.id),
                type=entity_type,
                bonded_to=fm.get("bonded_to", ""),
                personality_traits=fm.get("personality_traits", []),
                presence=presence,
            ))
    except Exception as e:
        logger.warning(f"Could not load companions: {e}")

Update format_full() for Active Party: 

# Active Party section
lines.extend([
    "### Active Party",
    f"**{ctx.pc_name}**" + (f" ({ctx.pc_class})" if ctx.pc_class else ""),
    f"- HP: {ctx.hp_current}/{ctx.hp_max}",
])

for companion in ctx.companions:
    traits = "; ".join(companion.personality_traits[:2]) if companion.personality_traits else ""
    presence_note = " (outside)" if companion.presence == "outside" else ""
    comp_line = f"  - **{companion.name}** ({companion.type}){presence_note}"
    if traits:
        comp_line += f" - {traits}"
    lines.append(comp_line)

4.4 Server-Side Dice Transcript Injection

File: api/routes/chat.py

Rationale: Don't trust the model to format dice correctly. Inject a deterministic transcript line after each roll.

Add dice formatting helper: 

def format_dice_result(result: DiceResult) -> str:
    """Format dice result for transcript injection."""
    dice_str = ", ".join(str(d) for d in result.dice)
    mod_str = f" + {result.modifier}" if result.modifier > 0 else (
        f" - {abs(result.modifier)}" if result.modifier < 0 else ""
    )
    reason = result.reason or "Roll"
    return f"DICE [{reason}]: [{dice_str}]{mod_str} = {result.total}"

Inject after tool calls in response processing: 

# After agent run, collect dice results from tool calls
dice_lines = []
for tool_call in result.tool_calls:
    if tool_call.name == "roll_dice" and tool_call.result:
        dice_lines.append(format_dice_result(tool_call.result))

# Prepend dice transcript to response
if dice_lines:
    dice_block = "\n".join(dice_lines)
    response_text = f"{dice_block}\n\n{response_text}"

Benefits: - 100% reliable dice display - Consistent formatting - No model "math drift" - Players always see the roll

4.5 RP Cues Mapping for Dispositions

File: api/context/builder.py

Rationale: Don't shout uppercase labels. Provide actionable roleplay cues.

Add disposition-to-cues mapping: 

DISPOSITION_RP_CUES = {
    "intimate-ally": "warm touch welcome; protective; speaks with genuine concern",
    "ally": "cooperative; trusts without question; professional warmth",
    "neutral": "transactional; guarded; requires persuasion",
    "hostile": "suspicious; may deceive or obstruct; hidden agenda",
    "enemy": "actively opposed; will attack or betray",
}

Update NPC formatting in _load_npcs_at_location(): 

npc_line = f"- **{npc.name}**"
if npc.role:
    npc_line += f" ({npc.role})"
if npc.status:
    npc_line += f" [{npc.status}]"

# Add disposition with RP cues
if npc.disposition:
    cues = DISPOSITION_RP_CUES.get(npc.disposition, "")
    npc_line += f" — {npc.disposition}"
    if cues:
        lines.append(npc_line)
        lines.append(f"  *RP: {cues}*")
        continue  # Skip default append

lines.append(npc_line)

Example output: 

### NPCs Present
- **Marlena** (Component dealer) [active] — intimate-ally
  *RP: warm touch welcome; protective; speaks with genuine concern*

4.6 /context Slash Command

File: api/routes/chat.py

@router.post("/{campaign_id}/chat/sync")
async def chat_sync(campaign_id: str, request: ChatRequest) -> ChatResponse:
    message = request.message.strip()

    # Handle slash commands (no LLM call)
    if message.lower() == "/context":
        vault = Vault(CAMPAIGNS_PATH / campaign_id)
        builder = ContextBuilder(vault)
        ctx = builder.build()
        context_dump = builder.format_full(ctx)

        return ChatResponse(
            response=f"**Current Session Context:**\n\n{context_dump}",
            session_id=request.session_id or str(uuid4()),
            tool_calls=[],
        )

    # ... rest of chat handling

Note: Currently shows full context. If secrets are added later, gate with /context --gm or filter.

4.7 RAG Evidence Schema (Structured)

File: api/routes/chat.py

Store RAG evidence as structured JSON, not raw text: 

rag_evidence = [
    {
        "source_type": match.source_type,
        "source_id": match.source_id,
        "chunk_id": match.chunk_id,
        "score": match.score,
        "title": match.metadata.get("title", match.source_id),
    }
    for match in rag_matches
]

# Store as JSON string in DB
await history.add_message(
    ...,
    rag_evidence=json.dumps(rag_evidence),
)

5. Execution Order (Revised)

Step Change File Risk
1 Add list_party_members() to vault lib/vault.py Low
2 Update Pip frontmatter with new schema canon/pcs/pip.md Low
3 Add pc_id, CompanionContext to builder builder.py Low
4 Add _load_companions() method builder.py Low
5 Update format_full() for party display builder.py Low
6 Implement dice transcript injection chat.py Low
7 Add /context command handler chat.py Low
8 Add RP cues mapping for dispositions builder.py Low
9 Update system prompt for party awareness dm.py Low
10 Store RAG evidence as structured JSON chat.py Low
11 Deploy and test deploy-palimpsest.sh Medium

Key reorder: Dice transcript injection moved up to step 6 (highest leverage fix).

6. Test Cases

Party System

curl -X POST "http://localhost:8000/campaigns/seagate/chat/sync" \
  -H "Content-Type: application/json" \
  -d '{"message": "continue"}'

# Response context should include:
# ### Active Party
# **Jake** (Wizard)
# - HP: 13/13
#   - **Pip** (familiar) - Mischievous and curious; Fiercely loyal

Dice Visibility

curl -X POST "http://localhost:8000/campaigns/seagate/chat/sync" \
  -H "Content-Type: application/json" \
  -d '{"message": "I attack the goblin"}'

# Response should START with:
# DICE [Attack Roll]: [14] + 5 = 19
#
# Then narrative follows...

/context Command

curl -X POST "http://localhost:8000/campaigns/seagate/chat/sync" \
  -H "Content-Type: application/json" \
  -d '{"message": "/context"}'

# Should return formatted context dump immediately (no LLM call)

RP Cues Display

# When Marlena present, context should show:
# - **Marlena** (Component dealer) [active] — intimate-ally
#   *RP: warm touch welcome; protective; speaks with genuine concern*

7. Success Metrics

Metric Current Target
Pip mentioned in responses 0% >80%
Dice rolls shown inline 0% 100%
/context command works N/A 100%
Intimate NPCs show warmth ~20% >70%

8. Peer Review Notes

Addressed Issues

Original Issue Resolution
list_pcs() overloaded for companions Added list_party_members() to vault
Match by name (fragile) Match by pc_id (stable)
Regex parsing markdown sections Use personality_traits in frontmatter
Prompt-only dice formatting Server-side transcript injection
Uppercase disposition labels RP cues mapping with actionable hints
Missing status filter Added status == "active" check
Missing mount presence handling Added presence field

Edge Cases Now Handled

  • Multiple companions (familiar + mount)
  • Inactive/dead companions filtered out
  • Mounts marked as outside or stabled
  • Non-bonded party NPCs via in_party: true
  • Dice math guaranteed accurate via server injection

9. Files Modified

lib/vault.py                    # Add list_party_members()
api/context/builder.py          # CompanionContext, _load_companions(), RP cues
api/routes/chat.py              # Dice injection, /context command, RAG schema
api/agent/dm.py                 # System prompt for party awareness
vault/seagate/canon/pcs/pip.md  # Updated frontmatter schema