Your hybrid system is now too slow for production
You built a beautiful neuro-symbolic system: LLM → knowledge graph → constraint solver → LLM explanation. Works great in testing.
Deploy it and watch what happens:
- LLM calls are 1 second (slow)
- Graph queries are 100ms (okay)
- Constraint solving is 5 seconds (oh no)
- LLM explanation is 1 second
- Total: 7+ seconds per query
Your users expect 500ms. You’re off by 14x.
This is the production problem neuro-symbolic systems have. They’re powerful but slow. This post is about making them fast.
The production architecture
Request → Router → Decision
├─ If simple: Fast path (LLM only, 500ms)
├─ If medium: Hybrid path (LLM + graph, 2s)
└─ If complex: Full path (all layers, batch)
↓
Cache/queue if slow
↓
Response
Layer 1: The router (detect problem complexity)
Not all queries need the full stack.
class QueryRouter:
def route(self, query):
"""Classify query by complexity"""
# Simple queries: "What is product X?"
if is_lookup_query(query):
return "fast_path" # Just LLM + cached answer
# Medium queries: "Which customers use feature X?"
if is_graph_query(query):
return "hybrid_path" # LLM + graph
# Complex queries: "Find the optimal vendor assignment"
if is_optimization_query(query):
return "batch_path" # Queue for batch processing
# Unknown: Route to LLM for guidance
return "guidance_path"
router = QueryRouter()
path = router.route(user_query)
Layer 2: Fast path (for simple queries)
def fast_path(query):
"""Simple retrieval, no reasoning needed"""
# Check cache first
if query in cache:
return cache[query]
# If not cached, call LLM
result = call_claude(query, model="claude-3.5-haiku") # Smaller, faster
# Cache for 1 hour
cache[query] = result
return result
# Example: "What's the price of Product X?"
# Cache hit rate: 60% (queries repeat)
# Latency: 50ms (cached) or 300ms (LLM)
Layer 3: Hybrid path (for medium queries)
def hybrid_path(query):
"""LLM reasons over graph"""
# Step 1: LLM parses the query (200ms)
parsed = call_claude(f"Parse this for a graph query: {query}")
# Step 2: Execute graph queries (100ms total)
graph_results = execute_queries(parsed)
# Step 3: LLM synthesizes answer (200ms)
answer = call_claude(
f"Given these graph results: {graph_results}, answer: {query}"
)
return answer
# Total: 500ms (acceptable)
Layer 4: Batch path (for complex queries)
def batch_path(query):
"""Complex optimization, async processing"""
# Queue the query
job_id = queue.enqueue(
solver_task,
query,
timeout=300 # 5 minutes max
)
# Return immediately to user
return {
"status": "processing",
"job_id": job_id,
"check_back_in": "5 minutes"
}
@queue.job
def solver_task(query):
"""Run the expensive computation"""
# Full stack: LLM → graph → solver → LLM
# Takes 5-30 seconds
result = full_neuro_symbolic_pipeline(query)
# Store result
store_result(job_id, result)
# Notify user (email, webhook, etc.)
notify_user(job_id, result)
Caching strategies
Caching is your biggest speed boost. Cache aggressively.
Cache layer 1: Query results
# Cache exact queries and their results
cache_key = hash(query)
if cache_key in redis:
return redis[cache_key]
result = expensive_computation(query)
redis[cache_key] = result
redis.expire(cache_key, 3600) # 1 hour TTL
return result
Cache hit rate: 30-60% (repeated queries are common)
Cache layer 2: Graph subgraphs
# Cache commonly accessed subgraphs
# "Customers in region X" is queried 1000x/day
subgraph = find_subgraph("region_x_customers")
if subgraph in cache:
return cache[subgraph]
subgraph = query_graph(...)
cache[subgraph] = subgraph
return subgraph
Cache hit rate: 80%+ (few unique subgraphs)
Cache layer 3: LLM outputs
# LLM explanations are deterministic for same input
key = hash(facts + explanation_template)
if key in cache:
return cache[key]
explanation = call_claude(facts + template)
cache[key] = explanation
return explanation
Cache hit rate: 90%+ (explanations repeat)
Monitoring and observability
You can’t optimize what you don’t measure.
Key metrics to track
# Latency by path
metrics.histogram("query_latency_fast_path", latency_ms)
metrics.histogram("query_latency_hybrid_path", latency_ms)
metrics.histogram("query_latency_batch_path", latency_ms)
# Cache effectiveness
metrics.counter("cache_hits_by_layer", layer)
metrics.counter("cache_misses_by_layer", layer)
metrics.gauge("cache_hit_rate", hit_rate_pct)
# Model usage
metrics.counter("llm_calls_by_model", model)
metrics.counter("graph_queries_by_type", query_type)
metrics.counter("solver_optimizations_by_type", solver_type)
# Errors
metrics.counter("query_failures_by_reason", reason)
metrics.counter("timeout_errors", path)
metrics.counter("graph_missing_data_errors", entity_type)
Example dashboard queries
-- What % of queries hit cache?
SELECT cache_hits / (cache_hits + cache_misses) as hit_rate
FROM metrics
WHERE timestamp > now() - interval '1 hour'
-- What's the p95 latency?
SELECT PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY latency_ms)
FROM query_latencies
WHERE timestamp > now() - interval '1 hour'
-- Which models are we calling most?
SELECT model, COUNT(*) as call_count
FROM llm_calls
WHERE timestamp > now() - interval '1 day'
GROUP BY model
ORDER BY call_count DESC
Failure modes and handling
Failure mode 1: Graph is missing data
try:
graph_result = query_graph(parsed_query)
if not graph_result or graph_result.empty:
# Graph doesn't have the answer
# Fall back to LLM alone
return call_claude(query)
except GraphConnectionError:
# Graph is down, use LLM only
log.warning("Graph unavailable, falling back to LLM")
return call_claude(query)
Failure mode 2: Solver times out
try:
solution = solver.solve(problem, timeout=5000)
if solution is None:
# Solver couldn't find optimal solution in time
# Return best so far
return solver.best_solution()
except SolverError:
# Solver crashed, fall back to greedy algorithm
return greedy_algorithm(problem)
Failure mode 3: LLM hallucinates
def verify_llm_output(llm_output, facts):
"""Verify LLM answer makes sense given facts"""
# Check 1: Are referenced entities in facts?
entities = extract_entities(llm_output)
for entity in entities:
if entity not in facts:
return False # LLM made up an entity
# Check 2: Does answer contradict rules?
# (use logic engine to verify)
if not logic_engine.verify(llm_output, facts):
return False
return True
llm_output = call_claude(prompt)
if not verify_llm_output(llm_output, facts):
# LLM hallucinated, request a different output
llm_output = call_claude(prompt + "\nEnsure your answer only uses facts provided.")
return llm_output
Real deployment: Financial advisory agent
Problem: Give customers personalized investment advice based on their portfolio, risk tolerance, regulations.
Architecture:
- Router: Classify query (recommendation, risk assessment, compliance check)
- Fast path: “What’s my current allocation?” (cached answer)
- Hybrid path: “Should I buy stock X?” (LLM + graph + rule engine)
- Batch path: “Optimize my portfolio” (constraint solver)
Latency targets:
- Fast path: 200ms
- Hybrid path: 1s
- Batch path: Return immediately, compute in background
Monitoring:
Dashboard shows:
- 70% queries hit fast path (cached)
- 25% queries use hybrid path
- 5% queries sent to batch
P95 latency: 800ms
Cache hit rate: 68%
Error rate: 0.1%
Production checklist
Before shipping a neuro-symbolic system:
- Router classifies queries by complexity
- Fast path for simple queries (cache everything)
- Hybrid path for medium queries (LLM + graph)
- Batch queue for complex queries (don’t block users)
- Fallback for graph/solver failures (use LLM alone)
- Verification of LLM outputs (check for hallucinations)
- Comprehensive monitoring (track all layers)
- On-call runbook (what to do if parts fail)
- SLAs documented (latency targets, error rates)
The bridge to the final post
This post covered production patterns. The final post pulls it all together with real case studies and what comes next:
- This post: Deploying neuro-symbolic at scale
- Final post: Real systems, lessons learned, the future
The takeaway
Neuro-symbolic systems in production require:
- Routing: Not all queries need the full stack
- Caching: Your biggest speed improvement
- Fallbacks: Systems fail; have a plan
- Monitoring: Measure everything
- Verification: Verify LLM outputs before returning them
Build for the common case (fast path), optimize later.
Most of your queries will be simple. Handle those in 200ms. For the 5% that are complex, use the full neuro-symbolic stack and batch them.
Final in this series: The Road Ahead: Neuro-Symbolic AI for Enterprise Reasoning Systems