10 ms·
I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CL
by Syntaf 26d ago
I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience.
We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents.
We paired this with skills initially, but found that the way folks built skills was often too prescriptive and limited to the authors own specific function in the company. A 2k line long skill suffers from the same gaps as we do, if an agent is just following a laundry list it’s less likely to reason about the request it’s doing.
So we instead asked ourselves: what if we just _let_ the agent reason about the work to be done and only provided the tools + guardrails to gather context and perform accounting work?
Turns out frontier models are GOOD at what they do, they outperformed our highly prescriptive skills and were able to work across a larger set of tasks even without instruction on how to do those tasks.
It’s a breath of fresh air from the decade of CRUD I’ve worked on, harness engineering is very neat.
- pdhborges 26d agoSo you still have CLIs but they have I presume an help command that describes the capabilities right. Could you give an example of an accounting guardrail you created?
- dpritchett 26d agoI’ve also found that Claude and friends are eerily good at using classic Unix CLI tools so I build mine in the same style, not unlike the `gh` CLI from GitHub, though with an agent-first design shape. Usually I’m returning TSV as a default format and I add a `help-all` subcommand to list every available command at once when needed. Another thing that helps is adding just-in-time context-sensitive hints, such as: user has just run a list query with at least one result. Add a one-liner to the response explaining the command shape for getting the detail view of the first response. In terms of skill files, I like to have my CLI generate them dynamically at runtime by walking their own current command tree and then feeding that through a text template. Examples from a public project: https://github.com/radiusmethod/gitlab-kiosk/blob/main/skill/SKILL.md.tmpl https://github.com/radiusmethod/gitlab-kiosk/blob/main/skill...
- Syntaf 26d agoYeah the CLI can provide schema for commands via the usual ‘—help’ syntax, so agents are able to discover + explore commands on their own. As for an example: if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed, if it attempts to do so without the requisite information we deny the tool call and ask the agent to escalate back to the client for proof of receipt. Often times this results in the agent not doing the work and instead sending a message back to the client asking for proof of the transaction. For humans on our platform there may be valid situations where we’d want to allow this, but for our agent this is a hard guardrail thus why it’s not just standard validation for any JE posting on our platform.
- pdhborges 26d agoif our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed And that rule is encoded in the CLI?
- Syntaf 26d agoIt’s actually encoded in an abstraction that we call “gates” which run before any tool call an agent makes, this allows us to prevent the tool call from happening and return a cited code + explanation on why their tool call was not executed https://docs.agno.com/tools/overview https://docs.agno.com/tools/overview
- newsomix9xl 26d agoCan you post a generic version of code for this somewhere (e.g. codeberg or whatever)? I find your description intriguing but I'd like to see it to make sure I understand it.
- rpastuszak 26d agoJust came here to say the same :)
- Syntaf 26d agoSorry I can't share what we're doing here directly! I will however say that this page alone does a pretty good job of illustrating what an agent harness might look like: https://docs.agno.com/tools/overview https://docs.agno.com/tools/overview * System prompt * Tool calls * Model definition Everything else (guards / etc) can just exist as code abstractions between the agent layer and the tool layer.
- ljm 26d agoI've been building a harness (on top of Pi for that matter) and have had similar experiences. Pi itself helps a lot with it being extensible by design but it's definitely been a challenge to make certain things work in an expected way. The native app I'm building on top, which I hope people who are less technical (or not technical at all) will use, is even more interesting because it's not just supposed to shell out to the CLI for everything and needs its own state.
- YZF 26d agoThis is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the model, it fills the context with conflicting instructions, it reduces the ability of the agent to respond to novel situations (and really in real life most situations are going to be novel). If you want to follow a process or a checklist you probably shouldn't use an LLM, or you should use it for some sub-tasks in the checklist/process but something more deterministic to work through the list.
- wonnage 26d agoThat works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly
- gmadsen 26d agoIt really doesn’t need to be that much more specified, give it context to the tools and level of analysis you expect then “why app slow” is a reasonable prompt
- hombre_fatal 26d agoI wouldn't agree. Sota models can do self-directed sampling, profiling, benchmarking, read call trees, etc. to give you a report of the app's bottlenecks and then recommend solutions that can be vetted. I do this constantly. As the upstream comment points you, you don't need to specify. Sota models are that good. And by being overprescriptive you can accidentally shut off branches that they would've taken, downgrading the quality of their work.
- 26d ago
- rush86999 26d agoI think you've really hit the mark on how the harness should be structured: 1. Guardrails - deterministic, social intelligence, team alignment & accountability 2. Learn by doing 3. make it stupid easy for the agent to research and access data 4. DRY Research supports this. Try picking up some ideas from my harness: https://github.com/rush86999/atom https://github.com/rush86999/atom
- polotics 26d agoHi. This is very interesting, could you link to the research? There is a dearth of proper research studies that A/B test what approach is best in terms of harness structure based on repeatable benchmark data with relevant sample uses-cases.
- rush86999 26d agoReasoning / self-consistency (voter in core/llm/self_consistency_voter.py): - Wang et al. Self-Consistency Improves Chain-of-Thought — ICLR 2023, Google Brain, 4k+ cites — https://arxiv.org/abs/2203.11171 https://arxiv.org/abs/2203.11171 — N-sample majority vote we use verbatim - Chen et al. Universal Self-Consistency — ICML 2024 — https://arxiv.org/abs/2311.17311 https://arxiv.org/abs/2311.17311 — judge fallback when no hash collides - Soft Self-Consistency — ACL 2024 — https://aclanthology.org/2024.acl-short.28.pdf https://aclanthology.org/2024.acl-short.28.pdf - Too Consistent to Detect — EMNLP 2025 — https://aclanthology.org/2025.emnlp-main.238/ https://aclanthology.org/2025.emnlp-main.238/ — why SC doesn't fix systematic bias - Self-Consistency Falls Short — TACL — https://direct.mit.org/tacl/article/doi/10.1162/TACL.a.625/ https://direct.mit.org/tacl/article/doi/10.1162/TACL.a.625/ — position-bias failure mode Multi-agent / org (core/agent_radio/, core/fleet_orchestration/): - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551, Zou Lab — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — 37k agents, CSO->scientists->reviewer->re-delegation, Merck external validation of B7-H3 design. Basis for VFS + hierarchy. - Debate or Vote (Choi & Li) — NeurIPS 2025 — https://arxiv.org/abs/2508.17536 https://arxiv.org/abs/2508.17536 — MAD gains = majority vote, not debate (why we didn't build debate) Sandbox / eval: - DABstep — arXiv:2506.23719 — https://arxiv.org/abs/2506.23719 https://arxiv.org/abs/2506.23719 — 450 real Adyen tasks, justifies code-interpreter + sandbox isolation - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 https://arxiv.org/abs/2403.14720 — provenance delimiters cut injection ASR 50% -> <2% Governance: - OWASP Top 10 for Agentic Applications 2026 — globally peer-reviewed by 100+ experts, Dec 2025 — https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ https://genai.owasp.org/resource/owasp-top-10-for-agentic-ap... — HIGH. Atom maps 1:1 (Goal Hijack -> match-confidence + oracle, Tool Misuse -> sandbox whitelist/caps, Privilege Abuse -> capability bindings, Memory Poisoning -> verified-episode graduation, etc.) docs/marketing/RESEARCH_NOTES.md:130 - NIST AI Agent Standards Initiative — Feb 17 2026, NIST CAISI — https://www.nist.gov/artificial-intelligence/ai-agent-standards-initiative https://www.nist.gov/artificial-intelligence/ai-agent-standa... + RFI summary May 2026 https://www.nist.gov/publications/summary-analysis-responses-request-information-regarding-security-considerations-ai https://www.nist.gov/publications/summary-analysis-responses... — HIGH (US gov standard). Defines the 4 enterprise minimums Atom implements: identification, authorization, access delegation, logging. - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551 — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — CSO -> 4 divisions -> 8 scientists -> reviewer -> re-delegation, no debate, no SFT — HIGH (Stanford Zou lab + Merck external validation). Basis for Atom's fleet hierarchy core/agent_radio/ and why maturity is routing not security. - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 https://arxiv.org/abs/2403.14720 — HIGH — provenance delimiters <provenance type="tool_output"> cut indirect injection ASR 50% -> <2%, used in core/provenance.py:10 - IntentGuard — https://arxiv.org/abs/2512.00966 https://arxiv.org/abs/2512.00966 + OpenReview — HIGH — intent tracing ASR 100% -> 8.5% on AgentDojo/Mind2Web, basis for sandbox egress allowlist + core/sandbox_tripwire.py
- trzy 26d agoThis doesn’t seem to work when the harness feeds images and asks the agent to do things in the real world. It fails to devise ways to keep track of its progress and fails to utilize its tools effectively.
- soulofmischief 26d agoThis couldn't have been said a year ago. It's amazing to watch. I have been building harnesses and applying networked agents to various domains since the GPT-3 API came out, and even two years ago, frontier models were just not at acceptable quality to make these harnesses useful. Everything changed overnight near the end of 2025. What will next year hold?
- eliranlevi 26d agoI’ve noticed it’s the performance that suffers when agents are paired with more than a single CLI and non-prescriptive skills. Since it seems to be out of its training data, anything non-trivial and the model just tries to brute force its way to a solution. Maybe it’s also about building them as self-improving, though I’ve been doing it manually for a CLI we don’t own. It seems to be art at the moment.
- floatrock 25d agoWhy a CLI over an MCP or even straight restful API with appropriate schema docs?
- Syntaf 25d agoI don't have a strong argument for or against using MCP, it honestly comes down to familiarity. In my own opinion, a CLI tool is going to be much more familiar ground for engineers -- I wouldn't expect the 200+ engineers at my company to all have read and understood the paradigms of the MCP protocol but I _would_ expect all of us to have a strong understanding of CLI tools and what a good/bad tool is.
- flumes_whims_ 25d agoMCP clutters the context. Much better to have cli that has --help on subcommands so it can get the parts in needs in the current context.
- dbrecht_ 22d agoFWIW there's work being done on progressive tool discover to help mitigate this, but it's still early and relies on client implementations. https://modelcontextprotocol.io/docs/2026-07-28/develop/clients/client-best-practices#progressive-tool-discovery https://modelcontextprotocol.io/docs/2026-07-28/develop/clie...