4 ms·
Nice approach. The "don't trust an LLM to guard another LLM" principle is sound for tool-call safety specifically, where the threat model is well-defined (destr
by oliver_dr 6mo ago
Nice approach. The "don't trust an LLM to guard another LLM" principle is sound for tool-call safety specifically, where the threat model is well-defined (destructive commands, SSRF, path traversal, etc.) and pattern matching gives you deterministic guarantees.
Where this gets interesting is at the boundary between tool-call safety and output quality. Vigil solves the "agent tries to rm -rf /" problem, but there's a whole class of failures where the agent makes safe tool calls that produce wrong results - querying the right database but misinterpreting results, calling an API correctly but hallucinating the summary it returns to the user, following the workflow but giving a factually incorrect final answer.
For that layer, deterministic rules don't scale because you'd need rules for every possible factual claim. That's where LLM-based evaluation actually makes sense - not as the guard for tool execution, but as the quality check on the final output. Think of it as two layers:
1. Vigil-style - deterministic, sub-millisecond checks on tool calls and actions (is this safe to execute?)
2. Semantic evaluation - LLM-based scoring on the output (is this correct, complete, and grounded in the provided context?)
The combination of both is what production agent systems actually need. We've been building the second layer at DeepRails (evaluate + auto-remediate when quality checks fail), and something like Vigil would complement it well as the first layer.
One thought on false positives: have you considered a "soft block" mode where flagged-but-borderline calls get routed through human approval rather than hard-blocked? For long-running agent tasks, a hard block with no fallback can leave workflows in broken states.