Your AI agent is about to cost you $10k because it made a bad decision
You built a procurement agent that decides which vendors to buy from. It uses RAG to search historical vendor performance, reasons about cost vs. quality, and makes recommendations. You tested it on 50 cases and it seemed great.
In production, it recommended a vendor that was 10x over budget because it misread a field in the retrieval results. The vendor got paid before anyone caught the error.
If you’d tested this in simulation first — where a “mistake” costs nothing — you would have caught it.
Two sides of the same coin
Simulation-Assisted AI: Use simulation to train/test AI systems before they touch the real world.
AI-Assisted Simulation: Use AI to generate simulation scenarios that are hard to hand-code.
Most teams do neither. They skip simulation and go straight to production.
Simulation-Assisted AI: Before your agent touches real data
How it works
- Build a simulator of your domain (what happens when an agent takes an action?)
- Have the agent make decisions in the simulator
- Check if the decisions were good
- Log failures
- Feed failures back to improve the agent
Example: Procurement agent
class ProcurementSimulator:
def __init__(self):
self.vendors = [
{"name": "Vendor A", "cost": 100, "quality": 0.9, "delivery_days": 5},
{"name": "Vendor B", "cost": 50, "quality": 0.7, "delivery_days": 10},
{"name": "Vendor C", "cost": 200, "quality": 0.99, "delivery_days": 2},
]
def agent_recommends(self, agent_decision):
"""Agent recommends a vendor. Does it make sense?"""
recommendation = agent_decision # e.g., "Vendor A"
vendor = self.get_vendor(recommendation)
# Score the decision
cost_vs_budget = vendor['cost'] / BUDGET
quality_vs_target = vendor['quality'] / QUALITY_TARGET
delivery_vs_deadline = vendor['delivery_days'] / DEADLINE_DAYS
score = (cost_vs_budget * 0.5 +
quality_vs_target * 0.3 +
delivery_vs_deadline * 0.2)
return {
'decision': recommendation,
'score': score,
'passed': score > 0.8,
'reason': f"Cost: {cost_vs_budget:.2f}, Quality: {quality_vs_target:.2f}"
}
# Test the agent
simulator = ProcurementSimulator()
for i in range(100):
agent_decision = agent.decide(context=simulator.get_context())
result = simulator.agent_recommends(agent_decision)
if not result['passed']:
print(f"FAIL: {result['reason']}")
Why this matters
Without simulation:
- You test on 50 real cases
- You miss edge cases
- You deploy
- You lose $10k
- You realize you should have tested harder
With simulation:
- You test on 1000 simulated cases
- You find edge cases (budget overruns, delivery failures, etc.)
- You improve the agent
- You deploy with confidence
- No surprises
Common failures to simulate
- Budget overruns: Did the agent respect cost constraints?
- Quality misses: Did it prioritize quality enough?
- Timing failures: Did it meet deadlines?
- Conflicting constraints: What happens when no vendor meets all constraints?
AI-Assisted Simulation: Generating test scenarios
Here’s the hard part: writing realistic simulators is tedious. You could hand-code 1000 procurement scenarios, but that’s a week of work.
AI-assisted approach: Have Claude generate realistic scenarios.
# Ask Claude to generate 100 procurement scenarios
prompt = """
Generate 100 realistic procurement scenarios. Each scenario should have:
- A set of vendors with different costs, quality, delivery times
- A budget constraint
- A quality requirement
- A deadline
- A "ground truth" best vendor
Format as JSON. Here's an example:
{
"scenario_id": 1,
"budget": 500,
"quality_target": 0.85,
"deadline_days": 7,
"vendors": [...],
"best_vendor": "Vendor B",
"explanation": "Balances cost and quality within timeline"
}
"""
scenarios = call_claude_with_json_schema(prompt, schema=ProcurementScenario)
Now you have 100 realistic test cases without hand-coding them.
Why this works:
- Claude understands procurement logic
- Claude generates realistic edge cases (vendor goes down, budget changes, etc.)
- You get natural variation (not all scenarios are obvious)
- Testing is now “did your agent match Claude’s reasoning?” vs. “is your agent reasonable?”
Combining both: The feedback loop
1. Generate 100 scenarios with AI (AI-assisted)
2. Run your agent through them (simulation-assisted)
3. Log failures
4. Have Claude analyze failures: "Why did the agent fail here?"
5. Improve agent prompt or logic
6. Generate new scenarios around the failure case
7. Re-test
8. Repeat until error rate is < 0.1%
This is how autonomous vehicles work: simulate millions of scenarios, find edge cases, improve the model, repeat.
Real-world example: Financial trading
You built an AI agent that decides when to buy/sell stocks based on market conditions.
Without simulation:
- You test on historical data
- You deploy
- You lose $50k on a market anomaly you didn’t predict
- You realize simulation would have been cheaper than one mistake
With simulation:
- Generate 1000 different market scenarios (normal, crashes, surges, etc.)
- Run agent through each scenario
- Log: “Agent lost 15% in this scenario, why?”
- Improve agent logic
- Re-run: “Agent now loses 5% in that scenario”
- Find new failure cases
- Repeat until agent is robust
Cost of simulation: $100 (Claude API) Cost of one mistake in production: $50,000
How to build a simulator for your domain
- Identify the entities (vendors, customers, accounts, etc.)
- Identify the actions (buy, approve, reject, etc.)
- Identify the outcomes (success, failure, cost, quality, etc.)
- Write the scoring function (how do you know if a decision was good?)
class DomainSimulator:
def __init__(self):
self.entities = [...] # Real or generated
def agent_acts(self, action):
"""Agent takes an action. Score the outcome."""
entity = self.get_entity(action.target)
result = self.simulate_outcome(entity, action)
score = self.score_result(result)
return {
'action': action,
'result': result,
'score': score,
'passed': score > THRESHOLD
}
def score_result(self, result):
"""How good was this outcome?"""
return (result.efficiency * 0.5 +
result.quality * 0.3 +
result.cost_effectiveness * 0.2)
Testing AI reasoning without simulation
Even if you don’t have a simulator, you can verify AI reasoning:
Method 1: Counterfactual testing
- Agent decides: “Buy from Vendor A”
- You ask: “Why not Vendor B?”
- Agent explains: “Vendor B is 2x more expensive”
- You verify: Is that true? (Check the data)
- If agent made up a reason, it’s hallucinating
Method 2: Chain-of-thought verification
- Agent shows its reasoning step by step
- You verify each step
Agent: "I chose Vendor A because:
1. Cost is $100 (within $200 budget) ✓
2. Quality is 0.9 (meets 0.85 target) ✓
3. Delivery in 5 days (meets 7-day deadline) ✓
All constraints satisfied, lowest cost."
Method 3: Adversarial testing
- You ask the agent to explain failure cases: “What would make you choose wrong?”
- Agent: “If I misread the cost field, I might overspend”
- You test: Deliberately corrupt the cost field, see if it fails
- If it does, you found a bug
The bridge to neuro-symbolic AI
Simulation is how you verify that pure statistical models (LLMs) are safe to use for critical decisions.
The next posts go deeper:
- Post 8 (neuro-symbolic intro): Why pure LLMs aren’t enough
- Post 9 (knowledge graphs): How to add structure to reasoning
- Post 10+ (advanced reasoning): How to combine LLMs with formal logic
All of those require validation, and simulation is how you do it.
The takeaway
Before your AI agent makes real decisions, test it in a simulator. This costs nothing and can save your company money.
- Build a simulator of your domain
- Generate test scenarios with Claude (AI-assisted)
- Run your agent through scenarios (simulation-assisted)
- Log and analyze failures (why did it fail?)
- Improve and re-test until error rate is acceptable
Then you can deploy with confidence.
Most teams skip this step and learn it the hard way (expensive mistake in production). Don’t be that team.
Next in this series: What Is Neuro-Symbolic AI, and Why Now