Operations

Configuration

Every constant, threshold and model name lives in one file, and each one carries a comment explaining how its value was chosen. That rule exists because a number that looks tuned and never was is worse than an obvious placeholder.

The rule#

No constant appears inline in code. Environment variables map to field names case-insensitively, so DATABASE_URL becomes database_url, and .env.example documents every one.

Model names are configuration rather than code for a specific reason: free-tier lineups change monthly, and providers retire models with little notice. A 404 from a provider is a one-line fix in .env rather than a code change and a deploy.

Infrastructure#

Variable	Default	Notes
`DATABASE_URL`	postgresql://fishack:fishack@localhost:5432/fishack	Needs the pgvector extension enabled
`REDIS_URL`	redis://localhost:6379/0	Use rediss:// for hosted Redis — Upstash requires TLS
`ADMIN_TOKEN`	(empty)	Empty means the statistics endpoint is OPEN. Set it before anything is public
`API_ORIGIN`	http://localhost:8000	Frontend only. Read on the Next server, never shipped to the browser

LLM providers#

Variable	Default	Notes
`LLM_PROVIDER_ORDER`	groq,gemini,openrouter,ollama	First is primary, the rest are failover targets. Groq leads on the fastest inference and the most generous free request quota; Ollama is last because it is optional and slowest
`GROQ_API_KEY / GROQ_MODEL`	llama-3.1-8b-instant	The only required key
`GOOGLE_API_KEY / GEMINI_MODEL`	gemini-2.5-flash	Optional fallback
`OPENROUTER_API_KEY / OPENROUTER_MODEL`	meta-llama/llama-3.3-70b-instruct:free	Optional fallback
`OLLAMA_ENABLED / OLLAMA_BASE_URL / OLLAMA_MODEL`	false · localhost:11434 · llama3.1:8b	Fully offline, needs roughly 8 GB of RAM

Retries and failover

Setting	Value	Why
`retry_max_attempts`	3	Per provider, before failing over
`retry_base_delay`	1.0 s	Exponential backoff with jitter
`retry_max_delay`	20.0 s	Doubles as the quota-exhaustion signal: a Retry-After LARGER than this means the daily quota is spent, so the client fails over immediately instead of sleeping. The line is drawn here because this value is already defined as the longest we are ever willing to wait on one provider
`llm_timeout_seconds`	60.0	Per request

Free tiers signal two very different situations with the same 429. Retry-After: 2 is a burst limit and retrying the same provider is right; Retry-After: 3600 means that provider is dead for the window and no amount of waiting fixes it.

Retrieval and ranking#

Setting	Value	Provenance
`retrieval_candidates_per_leg`	20	design
`retrieval_fusion_top_k`	20	design
`rerank_input_top_k`	8	measured
`rerank_top_k`	5	design
`rrf_k`	60	literature
`rrf_weight_bm25 / _vector`	1.0 / 1.0	open question
`hnsw_ef_search`	100	design
`embedding_model_name`	BAAI/bge-small-en-v1.5	design
`embedding_dim`changing it needs a column migration, a reindex AND a full re-ingest	384	hard constraint
`RERANKER_ENABLED`	true	deployment lever
`reranker_model_name`	BAAI/bge-reranker-base	design
`reranker_batch_size`	16	design
`reranker_max_length`	512	model limit
`conditional_rerank_enabled`	false	deliberate
`rerank_ambiguity_window`	5	design
`rerank_margin_threshold`	0.10	measured

Generation and validation#

Setting	Value	Provenance
`confidence_threshold_rerank`	0.45	guess
`confidence_threshold_fused`	0.015	guess
`llm_temperature`	0.1	design
`llm_max_tokens`	1024	design
`query_rewrite_enabled`	true	design
`query_rewrite_history_turns`	6	guess
`query_rewrite_max_tokens`	120	design
`citation_validation_enabled`	true	design
`citation_similarity_threshold`	0.50	deliberate
`abstention_message`the prompt, the detector and the eval assertions must all agree on it	fixed string	contract

Caching#

Setting	Value	Why
`CACHE_ENABLED`	true	Turn off entirely on a Redis-free deployment. Every call is wrapped anyway, so a failure degrades to a miss
`cache_ttl_seconds`	3600	Shorter than the freshness requirement. It is a backstop for bugs in active invalidation, not the primary mechanism
`semantic_cache_enabled`	true	The fuzzy path
`semantic_cache_threshold`	0.95	Survivable only because identifier queries skip this path entirely and abstentions are never cached
`semantic_cache_max_candidates`	200	Bounds the similarity scan

What the provenance labels mean#

Label	Means
design	Comes from the system design, and has a stated rationale
literature	From a published result, cited in the code comment
measured	An experiment produced this number, and the experiment is reproducible
guess	Provisional, and labelled so in the config file itself. These are the ones to distrust first
deliberate	Chosen against the obvious value, for a reason recorded in an ADR
open question	Evidence exists that this is wrong, and the experiment has not been run

The virtual price table#

Real spend is zero, so cost is modelled: for each model in use, the paid-tier price of the same model, or the closest comparable hosted model. A conservative mid-range fallback covers anything not in the table, so cost is never silently zero for an unknown model.

app/config.py

VIRTUAL_PRICES = {                        # USD per 1M tokens · snapshot: July 2026
    "llama-3.1-8b-instant":      (0.05, 0.08),
    "llama-3.3-70b-versatile":   (0.59, 0.79),
    "qwen/qwen3-32b":            (0.29, 0.59),
    "gemini-2.5-flash":          (0.30, 2.50),
    "…:free":                    (0.59, 0.79),   # priced as the paid route
    "llama3.1:8b":               (0.05, 0.08),   # local — priced as a comparable hosted 8B
}
DEFAULT_VIRTUAL_PRICE = (0.50, 1.50)

With paid APIs this table disappears — cost comes from the provider invoice and the tracking code stays identical.

The decision record →