NEON-CITY/CosySim and the NEXUS project
### **Game mechanics**
Persistent player state lives in `engine/world/player_state.py` (a thread-safe singleton persisted to `data/player_state.json`, broadcasting `hud_update` over Socket.IO so the Neon HUD stays live across every scene):
* **Vitals** — credits (₵), reputation, **heat / wanted level** (0–100), health, hunger, energy
* **Skills & XP** — 8 skills (hacking, combat, stealth, social, tech, driving, medicine, trading) on a use-based XP curve (`skill_progression.py`), with d20-style checks: `success = roll(1–20) + skill_level*4 + modifier ≥ difficulty`, scaling from Trivial(5) to Legendary(25), and a global player level 1–50
* **Factions** — six powers (OmniCorp, NeoTech, BlackMarket, Ghost_Net, SynthSec, DeepState) each with its own personality and a standing scale of −100 (sworn enemy) → 0 → +100 (trusted ally)
* **Territory** — those factions contest 16 districts; control flows from missions, crew ops, and world events, and a >10% swing in one tick triggers a **faction war** that can cascade to adjacent districts
* **Economy** — six good categories (weapons, tech, consumables, contraband, intel, luxury) priced as `base · (1 + (demand − supply)/100)` with territory multipliers layered on top
* **Inventory & equipment** — items carry rarity/condition; equipping cyberware/weapons grants real skill and stat bonuses; consumables resolve effects by category
* **Crew ops** — recruit NPCs you’ve built relationships with into role-based crews; operations resolve via probabilistic skill checks (SUCCESS / PARTIAL / FAILURE) that shift loyalty and pay out scaled rewards
* **Missions & chains** — four branching storylines (heist escalation, faction war, deep-state defection, street-to-syndicate) where outcome and standing route you down divergent paths
See docs/GAME_SYSTEMS.md and docs/ECONOMY_GUIDE.md for the full mechanics.
### **Local-agent simulations: NPCs that perceive, decide, act**
The defining trick of NEON CITY is that its inhabitants are **local LLM agents running a real agent loop** , not scripted dialogue trees. `engine/agents/agent_loop.py` runs a tick-based cycle for every character in a scene:
1. **Perceive** — observe location, nearby characters, and recent events (including world-sim digests)
2. **Decide** — `VirtualAgentManager` produces a _structured_ JSON action against a fixed schema (`speak`, `move`, `interact`, `idle`, `flirt`, …) — batched across agents for parallel inference
3. **Execute** — the action is applied to the scene, broadcast over Socket.IO, and logged to the `EventChain`
DECISION_SCHEMA = {
"type": "object",
"properties": {
"action": {"type": "string",
"enum": ["speak", "move", "interact", "idle",
"flirt", "touch", "kiss", "cuddle", "intimate"]},
"target": {"type": "string"},
"message": {"type": "string"},
},
"required": ["action"],
}
Every reply an agent emits flows through the **MCP interceptor pipeline** (36 interceptors, priority-ordered), which is what wires dialogue into the world: `NexusPrompt` hydrates context from the knowledge base, `FactionContextInterceptor` (pri 40) injects the speaker’s standing toward you, `HeatAwarenessInterceptor` (pri 75) makes NPCs react to your wanted level, `StatSyncInterceptor` (pri 91) applies stat changes, and `SpectatorBroadcastInterceptor` (pri 92) pushes danmaku to onlookers. NPCs even drift through `NaturalMoodDrift` neurochemistry tagging between turns. Agent decisions are also fed into the `DataCollector` for the self-improvement training loop and auto-registered into Nexus’s agent registry. The architecture of that pipeline is documented in docs/MCP_FRAMEWORK.md and docs/ARCHITECTURE.md.
The result is a city where the bartender remembers the slight, the rival faction lieutenant prices you out, and a stranger across the lounge is — genuinely — _deciding_ what to do next, locally, on your machine.
* * *
## **Engine internals: how agents are steered**
_The Oracle scene — a neural-consciousness terminal in NeonCity that doubles as the project’s All-Seeing Eye observability dashboard (real-time error feed, service-health grid, trace links)._
Most “AI character” demos are a system prompt and a `while` loop. CosySim is the opposite: every agent reply passes through a **governed pipeline** of ~38 interceptors, the model’s own output is parsed for **inline control tags** that mutate game state, and inference itself is steered by a **custom LMStudio client/server** that does model affinity, federation, speculative decoding, and ephemeral tool servers — all running on local hardware. This section is the deep dive. Everything below is grounded in real modules you can open and read.
> **Why read this?** It’s a working reference implementation of agent governance, structured-output steering, and observability that you can borrow wholesale. The patterns are deliberately small and composable — an interceptor is ~40 lines; a control tag is a regex plus a state write.
### **The shape of one reply**
When a scene asks a character to respond, it doesn’t call the LLM directly. It calls an `AgentGovernor` (engine/mcp/comms_framework.py) which orchestrates the whole flow:
user_message
│
▼
AgentGovernor.reply()
├─ 1. Load SceneManifest (which skills this scene exposes)
├─ 2. Run AUTO skills ── cooldown + prerequisite gated ──▶ ctx["auto_results"]
├─ 3. pipeline.run_pre(ctx) ◀── ~38 interceptors, priority-ordered
│ (mutate system_prompt + messages: mood, memory, scene, rules…)
├─ 4. LLM call (custom LMStudio client) ──▶ ctx["reply"], response_id, tool_calls
├─ 5. ContentRouter.parse_full(reply) ──▶ ctx["parsed"] (single pass)
└─ 6. pipeline.run_post(ctx) ◀── same interceptors, post phase
(apply [STAT], sync mood, broadcast danmaku, log, shape)
▼
final reply (tags stripped, state mutated, telemetry emitted)
The carrier is a single mutable `ResponseContext` (a `dict` subclass). Every interceptor reads and writes well-known keys (`system_prompt`, `messages`, `reply`, `parsed`, `mood_tags`, `abort`, `skip_llm`…). Any interceptor can short-circuit the chain by setting `ctx["abort"] = True`, or skip the LLM entirely (`ctx["skip_llm"] = True`) to provide a canned reply. The pipeline never lets one bad interceptor crash a reply — each hook is wrapped, and failures are logged through the Oracle, not swallowed.
### **1. The interceptor pipeline (~38 hooks, by priority)**
Interceptors subclass `InterceptorBase` and override `pre_call(ctx)` and/or `post_call(ctx)`. They’re registered in engine/agents/interceptors/__init__.py and sorted by an integer `priority` (lower runs first). Each can declare `applicable_scenes` to limit itself to specific scenes. The registry logs its count at import time, so the live number is always visible in the logs.
The pipeline is the embodiment of the project’s design philosophy: **behaviour is layered, not monolithic.** Context flows _in_ (pre, low→high priority) and gets _applied_ on the way _out_ (post). Pre-call tiers hydrate the prompt; post-call tiers turn the model’s words into consequences.
The full pipeline by priority (pre-call hydration → LLM → post-call application) (click for more details)
Writing a new one is intentionally trivial — and you can register it from anywhere with a decorator:
from engine.agents.interceptors import register_interceptor
from engine.mcp.comms_framework import InterceptorBase, ResponseContext
@register_interceptor
class WeatherMoodInterceptor(InterceptorBase):
name = "weather_mood"
priority = 18 # runs after world state (15), before skills (30)
applicable_scenes = {"neoncity", "penthouse"}
def pre_call(self, ctx: ResponseContext) -> None:
ctx["system_prompt"] += "\n[It is raining outside; the mood is contemplative.]"
### **2. Stream tags — the model steers the world**
CosySim treats the LLM’s output as a **control channel** , not just text. Characters emit inline tags that the engine parses and _applies_ :
**Tag** | **Example** | **Applied by** | **Effect**
---|---|---|---
`[MOOD:x]` | `[MOOD:playful intensity=0.8]` | `MoodSyncInterceptor` (92) | Sets mood, fires threshold rules
`[STAT:x±n]` | `[STAT:arousal+10]` `[STAT:trust=70]` | `StatSyncInterceptor` (91) | Mutates character game state
`[ACTION:x]` | `[ACTION:pour a drink]` | post-call / spectator | Drives animation / narration
`[IMAGE:x]` | `[IMAGE:a selfie in the penthouse]` | scene image pipeline | Triggers ComfyUI generation
`[VOICE:x]` | `[VOICE:whisper]` | `TTSStyleInterceptor` (85) | Selects TTS delivery style
There’s a single canonical parser — `ContentRouter.parse_full()` in engine/agents/content_router.py — that runs **once** per reply (step 5 above) and produces a `ParsedResponse`. Every downstream interceptor reads `ctx["parsed"]` instead of re-scanning with its own regex. For streaming, the mirror is `StreamProcessor` (`engine/agents/stream_processor.py`), which accumulates tags _incrementally_ off the v1 SSE event stream and fires callbacks (`on_mood`, `on_image_request`, `on_stat_delta`) in real time — so a `[MOOD:...]` lights up the UI before the sentence finishes.
The keystone is `StatSyncInterceptor` (priority 91). Before v1.59 these tags were parsed and **discarded** — a character could say `[STAT:trust+10]` and nothing happened. Now the loop is closed: stat tags route through the `CharacterStateCoordinator` (only known stats, with LLM-alias normalization like `desire→horniness`), and because StatSync runs _just before_ `MoodSyncInterceptor` (92), the freshly-updated stats are visible to the threshold-rule auto-evaluation MoodSync performs. A character’s words have mechanical consequences, and those consequences cascade into rule-driven behaviour — all in one reply.
reply: "I lean closer, heart racing. [MOOD:flirtatious] [STAT:arousal+15] [ACTION:lean in]"
│
▼ ContentRouter.parse_full() → ParsedResponse(mood=flirtatious, stat_updates=[arousal+15], actions=[lean in])
▼ StatSync(91): coordinator.update("aria", arousal=+15) → state mutated
▼ MoodSync(92): set mood; arousal now > threshold → rule fires → directive injected next turn
▼ SpectatorBroadcast(92): danmaku "Aria: I lean closer…" in mood color
▼ TTSStyle(85)/clean text: tags stripped → "I lean closer, heart racing."
### **3. The custom LMStudio client/server**
All inference is **local** , through a hand-written native-v1 client — no OpenAI-compat shim. engine/lmstudio/ is a full control plane over LMStudio:
* **`LMSClient`** (`lms_client.py`) — implements every endpoint of the LMStudio v1 REST API (`/api/v1/chat`, model load/unload/download). It exposes the steering knobs that matter: **stateful chats** via `previous_response_id`/`response_id` (conversation branching by reusing any historical id), **structured output** (JSON-schema enforcement at the logit level), full sampling control (`top_k`, `min_p`, `repeat_penalty`, reasoning mode, per-request `context_length`), image input for VLMs, and typed SSE streaming across all 19 event types.
* **`ServerController`** (`server_controller.py`) — CosySim is **both client and server** to LMStudio. The controller does server-side lifecycle: load/unload models, configure inference, **per-agent model instances** (`create_agent_instance("aria", ...)`), TTL-based auto-unload of idle instances, and per-model health (VRAM, request counts, idle time) that feeds the Oracle dashboard.
* **`LMLinkManager`** (`lmlink_manager.py`) — **federation**. Connects multiple LMStudio instances (local + remote over Tailscale) and routes each request to the best peer by **model affinity** , capability, load, and failover. Peers track latency (EMA), error rate, and consecutive failures; transient health blips retry with exponential backoff + jitter rather than flipping a peer unhealthy.
* **`TaskQueue`** (`task_queue.py`) — a priority queue with **model-affinity routing** : `CODE` tasks go to `*coder*` models, `VISION` to `*vl*`/`*llava*`, `ROUTER` to tiny `*0.6b*` models, etc. Workers auto-start on first `submit()`.
* **Ephemeral MCP tool servers** — tools are offered to the model per-request via the v1 `integrations` field. `MCP.ephemeral("http://localhost:8600/mcp/sse")` references a server by URL (no pre-registration), with `allowed_tools` and auth headers; `MCP.plugin("mcp/cosysim")` references a registered one. This is how a character gains tool access _for one call_ without standing infrastructure.
* **Speculative decoding** — `client.enable_speculative(main_model, draft_model)` loads a main+draft pair; LMStudio then activates spec decoding automatically and CosySim passes `draft_model` through the chat payload. Real throughput gains, fully local.
Per-agent affinity, federation, and the task queue together mean a single rig (or a small fleet) can run a tiny router model, a chat model, a coder model, and a vision model concurrently — each agent steered onto the right one.
### **4. The Oracle — one name, two entities**
The Oracle is deliberately dual, and that duality is the project’s signature flourish.
**The telemetry backbone** (engine/observability/oracle.py) is the project-wide observability facade. One import — `from engine.observability.oracle import get_logger` — and on first use it wires the entire stack: a `StructuredLogger` root handler (→ SQLite + JSONL, queryable and traceable), the `CosyLogger` ring buffer (→ the in-game Phone feed), and an `_OracleHandler` that fires only on `ERROR+` (~0.2ms cost). Errors flow into the `ErrorAggregator`, which **fingerprints** them — stripping IDs, numbers, and paths to a stable hash — so 500 log lines collapse into _“LMStudio auth failed: 47× in 5min, affecting phone + lounge + tavern, started 14:32.”_ It’s hardened: a bounded-LRU flood guard caps memory under a storm of unique fingerprints, a throttled rate-alert hook emits one CRITICAL line instead of silence, and a post-install **self-check** confirms the handlers actually attached (a silent no-op install is exactly the failure mode it guards against). `diagnose()` and `scripts/oracle.py` print health, top errors, LLM p95, Nexus KB stats, per-model VRAM, and Gemini service status in one ASCII-safe report.
**The in-game scene** (`content/scenes/oracle/oracle_scene.py`) is a neural-consciousness terminal in NeonCity’s core — meditation, LLM-driven fortune readings, city-pulse displays — _and_ it surfaces the very same telemetry through an “All-Seeing Eye” dashboard: a real-time error feed, a service-health grid, and trace links, all over Socket.IO. The thing watching the city is the same thing watching the code. That’s not a gimmick — it means the project’s observability has a _face_ , and debugging is a first-class, in-world experience.
### **5. Neurochemistry + mood drift**
Underneath the mood tags is a genuine affect model. engine/characters/neurochemistry.py gives every character **6 neurotransmitters** — dopamine, serotonin, oxytocin, cortisol, adrenaline, endorphins — each with a baseline, a half-life decay curve, and a stimulus catalog (`kiss`, `rejection`, `crew_victory`, `level_up`…) that applies clamped deltas. Emotions are **computed** , not hardcoded: high dopamine + low cortisol → _Confident_ ; high cortisol + high adrenaline → _Panicked_. The `NeurochemistryInterceptor` (priority 4) injects this derived state into the system prompt at the very front of the pipeline, and `StimulusDetectInterceptor` (88) closes the loop by detecting stimuli in the conversation post-call and feeding them back.
`NaturalMoodDriftInterceptor` (priority 5) makes the world feel _alive between turns_ : arousal cools, tiredness accumulates, anger fades, happiness regresses toward a personality mean — deliberately slow, so emotions shift gradually rather than snapping. It piggybacks buff-expiry and tag-decay sweeps onto every call and slips the agent a one-line “inner feeling” cue. So a character isn’t a static persona answering questions — it’s a drifting emotional state that your words (and `[STAT:]`/`[MOOD:]` tags, and the threshold rules they trigger) continuously nudge.
* * *
## **NLM + Nexus — frontier-grade AI from local models**
_The Oracle’s All-Seeing Eye surfaces query-router provenance — which tier answered each query, with confidence and tokens-saved logged in Oracle format._
Local models are cheap, private, and fast — but a 0.6B–8B model running in LMStudio is not GPT-class on its own. CosySim closes that gap not by making the model bigger, but by making the model _ask less and remember more_. Two subsystems do the heavy lifting:
* **Nexus KMS** — a persistent SQLite + FTS5 + vector knowledge backbone (`:8700`) that every agent, scene, and dev session reads from and writes back to.
* **NotebookLM (NLM)** — Google’s Gemini, driven headlessly through a reverse-engineered private RPC stack, used as a _free_ distillation and grounding layer.
The thesis is simple and provable in the code: **the first time a question is asked it costs compute; every subsequent time it is served from Nexus for free.** Expensive frontier-grade reasoning happens once, gets distilled into the knowledge base, and is thereafter answered locally — instantly. The local model becomes the _last_ resort, not the first.
> This is the part of CosySim most worth borrowing. The whole pipeline is open and grounded in real modules — read along.
### **The 7-tier query router**
`engine/nexus/query_router.py` (`NexusQueryRouter`) is the heart of the system. Every information-retrieval request — agent context hydration, a player question, a dev lookup — passes through a **confidence-gated cascade, cheapest tier first**. Each tier either clears the `min_confidence` bar and returns, or falls through to the next.
**#** | **Tier** | **Mechanism** | **Cost** | **Confidence**
---|---|---|---|---
0 | Local session cache | In-process MD5-keyed dict, TTL `local_cache_ttl` (300s) | ~0 | inherited
1 | **Q &A cache** | `client.find_qa` exact/fuzzy match, scored by word-overlap relevance (≥0.4 to count) | ~0, instant | up to 0.90
2 | **Vector search** | Gemini Embedding 2 → ChromaDB cosine over `knowledge/qa/code/news` | fast | up to 0.92
2.5 | **File Search** | Google managed RAG with **grounded citations** over uploaded docs | API call | 0.85
3 | **FTS knowledge** | SQLite FTS5 across Nexus entries, title-overlap + length scored | fast | up to 0.85
4 | **Nexus smart-ask** | Server-side hybrid pipeline (FTS + NLM) via `client.ask(depth=…)` | medium | variable
5 | **Direct NLM** | `nlm_unified_ask` — free, Gemini-grounded answer with citations | slow | ~0.8
6 | **LLM fallback** | Local LMStudio inference (`engine.lmstudio.chat`) | local GPU | 0.6
The thresholds are real, tuned constants and every one is config-overridable (`nexus.query_router.*`):
CACHE_CONFIDENCE = 0.90 # Q&A cache hit
VECTOR_CONFIDENCE = 0.82 # strong vector match
FILE_SEARCH_CONFIDENCE = 0.85 # grounded in uploaded docs
SEARCH_HIGH = 0.75 / SEARCH_MEDIUM = 0.50 / SEARCH_LOW = 0.30
MIN_ANSWER_LENGTH = 20
Two details that make it robust rather than naive:
* **Relevance gating, not first-result-wins.** Tier 1 doesn’t trust the top Q&A row blindly — `_question_relevance` computes a stop-word-filtered Jaccard overlap and _scales confidence by it_ (0.4 overlap → 0.72 conf, 1.0 → 0.90). A weak match falls through instead of returning a confidently-wrong answer.
* **Provenance logging.** Every resolution logs `tier=…, confidence=…, tokens_saved=…` in Oracle format, and per-agent hit counts are tracked (`agent_queries` / `agent_hits`) — so you can see exactly which tier answered, for whom, and how much GPU it saved.
### **The self-improving flywheel**
This is what makes local models punch above their weight. Look at tiers 3–6 in `query()`: **every answer that required real work is written back as a Nexus Q &A pair**, which promotes it to tier 1 for all future queries.
# Tier 6: LLM Fallback — store the answer back in Nexus for future reuse
if use_llm:
result = self._llm_fallback(question, ...)
if result.answer and len(result.answer) >= self.MIN_ANSWER_LENGTH:
self._store_qa(client, question, result.answer, ...) # → promotes to tier 1
self._stats.answers_stored += 1
And `_store_qa` doesn’t just cache — it **also feeds the training flywheel** (`_feed_training_flywheel` → `collect_from_qa`), so every fallback simultaneously becomes a future cache hit _and_ a fine-tuning example. The loop is closed:
expensive answer (NLM / LLM)
│ store_qa
▼
Nexus Q&A pair ──────────► future query hits tier 1 (free, instant)
│ collect_from_qa
▼
TrainingFlywheel example ─► fine-tune local model
│
▼
better local fallback ────► cheaper tier 6, more cache hits next cycle
`RouterStats.hit_rate()` measures the payoff directly: hits ÷ total queries. As the cache fills, the rate climbs and `llm_fallbacks` falls. The `nlm_router.py` variant adds an explicit `savings_report()` breaking out `answered_without_gpu = cache_hits + fts_hits + nlm_hits` and `estimated_tokens_saved` — the system reports its own compounding ROI.
### **NLM chain-prompting: where frontier reasoning enters**
NLM is the system’s gateway to Gemini — for free, at NotebookLM rate limits. `engine/nexus/nlm_chain.py` (`NLMChainEngine`) turns a single question into **multi-step distillation** and routes the results straight back into Nexus.
Chains are declarative (defined in `config/nlm_notebooks.yaml`), each step’s output piped into the next via a `{previous_output}` template variable:
engine = NLMChainEngine()
# progressive research: overview → details → examples → gaps
engine.execute_chain("architecture-review", notebook_id,
variables={"task_description": "..."})
# reverse-generate a whole Q&A set from one notebook
engine.distill_notebook("coding", questions=[...])
# weekly fleet sweep across all notebooks
engine.run_batch("weekly-review")
Crucially, `execute_chain` **persists as it goes** : the final synthesis is stored as a Nexus entry, and _every_ substantive step is stored as a Q&A pair (`_store_qa_in_nexus`). So a single chain run — one burst of Gemini-grade reasoning — seeds dozens of tier-1 cache entries that the local stack serves forever after. `generate_action_manifest` even uses the `task_decompose` chain to turn a fuzzy task description into a JSON, agent-executable plan.
Behind it, `nlm_direct_client.py` (`NLMDirectClient`) speaks the raw `batchexecute` / `GenerateFreeFormStreamed` RPC protocol with browser-attached auth (SAPISIDHASH), a 302-operation rpcid registry, and full multimodality — text, URL, YouTube, image, audio, video, PDF in; reports, podcasts, mind-maps, flashcards out. **Every output can become the next call’s input** — recursive self-improvement is the architecture, not an afterthought.
The cache pipeline — Gemini as both generator and evaluator (click for more details)
### **The knowledge pipeline: one funnel, consistent quality**
Every knowledge source — sessions, URL crawls, agent submissions, NLM distillation, manual notes — routes through a single funnel, `engine/nexus/knowledge_pipeline.py` (`KnowledgePipeline.ingest`):
ingest → validate → dedup → store → embed → Q&A → notify → train
Each stage is deliberate: content-hash dedup (SHA-256 of title + first 500 chars) blocks near-duplicates; a quality heuristic gates Q&A generation (`quality ≥ 0.5`); successful entries auto-embed into ChromaDB and auto-generate rule-based Q&A pairs; and everything feeds the `DataCollector` as a `knowledge_synthesizer` training example. The result: anything that enters Nexus is immediately discoverable by **all** retrieval tiers — FTS, vector, _and_ Q&A cache — with no manual bookkeeping.
### **Why this punches above local weight**
* **Frontier reasoning is amortized to zero.** Gemini-grade answers (via NLM) are computed once and distilled into a free, instant local cache. The marginal cost of the 1000th identical query is a dict lookup.
* **Confidence gating prevents quality collapse.** Cheap tiers only answer when they’re actually confident; otherwise the question escalates toward grounded Gemini. You get cache speed _without_ cache staleness lies.
* **Grounded citations on demand.** Tiers 2.5 and 5 return answers with source citations (File Search + NLM), so even “frontier” answers are verifiable, not hallucinated.
* **The system trains the system.** Every fallback is both a cache write and a fine-tuning datum — the local model that handles tier 6 next month was taught by the Gemini that handled tier 5 this month.
* **It’s all observable.** `router.stats`, `savings_report()`, and Oracle provenance logs make the flywheel measurable — you can watch the hit rate climb and the GPU calls fall.
from engine.nexus.query_router import get_query_router
router = get_query_router()
res = router.query("How does the interceptor pipeline work?")
print(res.source, res.confidence, res.tokens_saved) # e.g. "cache" 0.90 450
print(router.stats.to_dict()) # cache/vector/file_search/nlm/llm breakdown + hit rate
The whole stack is open, local-first, and self-documenting — a working example of how to give a small local model a memory that compounds and a tutor that’s free.
