What I learned experimenting with OpenClaw for analyst workflows

I wanted to see what an agentic harness could do inside an analyst's day. So I gave OpenClaw a broad set of tools and enough room to spin up its own agents, skills and task-specific workflows. The assumption was the obvious one: more capability should mean more useful agents.

What happened was closer to the opposite. As the system got more capable, it got harder to control. This note is what I took away from that: why an agentic setup drifts into chaos, why stepping in to help can make it worse, and why the fix in the end wasn't better prompting but tighter constraints.

The sandbox: letting agents explore

The experiment was open-ended on purpose. Rather than prescribe one fixed workflow, I wanted to see what the analysts would build for themselves given a capable environment. OpenClaw ran as a generic harness (broad tool access, the ability to create new agents, the ability to write new skills), and I handed the keys over and told people to explore.

At first this was exactly the point. New workflows appeared fast. Tasks that used to mean manually stitching together five steps could be handed to an agent and mostly left alone. The freedom was doing what I'd hoped.

But that flexibility came with a cost.

When the sandbox became a junk drawer

The problem was accumulation. More agents, more skills, more instructions, all growing on top of the same generic foundation. OpenClaw spreads its instructions across several files (an AGENTS file, a SOUL, an IDENTITY), and once a whole team is each adding to that, the directives start to overlap and then to contradict each other outright. Context kept growing, and a lot of it was pulling in different directions.

The agents didn't just make the odd mistake. Their behaviour got inconsistent in a way that was hard to pin down: the same kind of task could come back noticeably different depending on which pile of instructions and context happened to be live that run. They hallucinated, wandered off the request, and produced something confident and wrong.

I had started with a sandbox. Somewhere along the way it had become a junk drawer.

Two-panel claymation meme. Top, more capability multiplied the ambiguity: a Shiba stands lost in a field of signposts for every possible tool and technique, going nowhere. Bottom, constraints created clarity: a bounded scope and a short allowed-tools list send the Shiba running straight down a clear path to the goal.
More tools meant more paths and no direction. A bounded scope and a short tool list gave the agent somewhere to go.

Don't interrupt the cook

The way people reacted made it worse, and this is the part I didn't see coming.

When an agent started producing something that looked off, the natural instinct was to jump in: drop a correction into the chat, add a new point, tweak the scope, all while the model was still mid-run. It feels like helping. It isn't.

Two-panel claymation meme. Top, the stress of multi-tasking: a frazzled Shiba at a laptop buried in sticky notes as new points pile in mid-task, progress stuck at 38% with context switching and inconsistent output. Bottom, the payoff of parking new tasks: a calm Shiba finishes at 100%, consistent and easier to review, with new points filed in a holding tray.
Every mid-task correction forces a context switch on an agent that was already struggling to hold the thread.

Every one of those interruptions forced a context switch onto an agent that was already struggling to hold the thread. It didn't fix the hallucination; it just made the run slower, the output messier, and the result harder to trust. An agent that's low on coherence doesn't get more reliable when you pour more context into it halfway through.

The habit I ended up asking for was simple: if a new idea or a correction shows up, park it. Let the current run finish, look at what it produced, and then decide what needs to change against that baseline. You can't evaluate a run you kept knocking off course.

The turning point: I was solving the wrong problem

My first instinct was to blame the context. Too many instructions, too many skills, too much inherited noise. Trim the bloat and the agents would settle.

Trimming helped, but chasing context was chasing a symptom. The question underneath it was one I hadn't actually asked: how would I even know if an agent got the task right? An agent could be told "research this topic and write a report," and it would hand back something long, polished and plausible, with no objective answer to the only thing that mattered: what does done mean? Without that, it could sail past the real task and still look like it had succeeded, and I had no clean way to say it hadn't.

Once I framed the problem as verifiability rather than capability, the fixes sorted into two kinds: make the agent's world smaller and cleaner, and make "done" something you can check. It's the same lesson Loop engineering keeps landing on from the other direction: most failures are a fuzzy contract, not a weak model.

A customised harness, and less context per agent

The first kind of fix went straight back at the junk drawer, but it wasn't just tidying the files into one. I moved off the fragmented setup (separate AGENTS, SOUL and IDENTITY files, each a place for instructions to pile up and contradict each other) and customised a leaner harness, something like NanoClaw, around the actual work. The base is a single instructions.md the agent expands for itself, so how it behaves lives in one readable place instead of smeared across a handful of personality files. On top of that sits a /plans folder where each task's plan lives, and a repeatable loop those plans drive (loop-engineering), so the harder tasks run the same way each time instead of by improvisation. The harness stops being a generic sandbox and becomes something shaped to the job.

The other move was to stop feeding the whole context to one agent. Planning up front decides how a task breaks apart; several agents then each take a slice, so no single run has to hold the entire tangle at once. A smaller, scoped context is easier for the model to keep straight, and far easier for me to check afterwards.

Two-panel claymation meme. Top: one overwhelmed Shiba tangled in a giant ball of cables and scattered instruction files labelled AGENTS, SOUL, IDENTITY, CONTEXT, PRIORITIES and TASKS. Bottom: several small, calm Shibas at a harness bench, each holding one labelled slice of the work, driven by a single instructions.md, a /plans folder and a repeatable loop.
One agent drowning in a tangle of instruction files, against several agents each holding one scoped slice off a single instructions.md and a repeatable loop.

The verification gate: plan first, define done

So I stopped letting agents guess and stopped letting analysts wing it. Two changes, both applied before any execution started.

Force the plan. This is the same up-front plan that decides how to split the work across agents; here it also has to name the target. Before an agent runs, the analyst has to say what the actual objective is, what approach they'll take, what information is needed, and what the output should be. The plan is there to cut the ambiguity before the agent starts spending context and taking actions it can't take back.

Define done. This works at two levels. The task-level version is the obvious one: every task gets a testable Definition of Done written up front, and it isn't finished because the agent produced an answer; it's finished when the required evidence, structure and checks are all satisfied. That gives the human and the agent the same gate to check against, instead of two different guesses about what "good" looked like.

The level that's easy to skip while you're experimenting is the operational one. A task can be done while the experiment never is: there's always one more tool to add, one more prompt to tune, one more agent to try. Without an end stage defined for the operation itself, you keep experimenting forever. So the operation gets its own Definition of Done too: the point where the workflow is good enough to stop tuning and put to work.

Two-panel claymation meme. Top: a confused Shiba starting a task with no plan, asking 'what is done?' as scope and progress lines pull in different directions. Bottom: a focused Shiba at a verification gate, working through a testable Definition of Done checklist before delivering, with the goal clarified.
Clarify the goal, plan the approach, write a testable Definition of Done, then execute. The Definition of Done is the gate both sides check against.

This is the same instinct behind writing the spec first: decide what has to be true before you ask for the work, so there's something concrete to verify against afterwards rather than a vibe.

From open-ended prompts to output contracts

Defining "done" helped, but the agent was still working inside a noisy environment full of OpenClaw's contradictory background instructions. It needed something to anchor to. So I killed the open-ended prompt.

Instead of "write a report about X," the agent gets a structure it has to fill:

Two-panel claymation meme. Top, writing a report by hand: a stressed Shiba at a typewriter told to 'write a report about X' produces a messy, inconsistent document that's hard to verify and compare. Bottom, using an output contract and Jinja template: Shibas feed categorized fields into a template press that stamps out uniform, verifiable reports the same way every time.
A fixed template turns a rambling report into a contract: consistent, verifiable, and easy to compare across runs.

The template does more than format. It's a contract between the analyst and the agent, and it does the anchoring the tangled system prompts couldn't. Forcing the agent to fill a fixed structure overrides a lot of the background noise, because the structure is right there in the foreground telling it what the answer has to contain. The results finally lined up with the Definition of Done: consistent, checkable, and easy to compare from one run to the next.

Template the structure. When the formatting rules get fussy (exact heading levels, a fixed field order, numbers rounded a specific way), asking the agent to hold all of them every run is a bad bet. It gets most of them right and quietly misses one. So push the shape out of the prompt and into a real template: the agent returns the fields as data, and a Jinja template (or any templating engine) renders them into the final document. The formatting lives in code instead of getting re-derived on every run. The agent supplies the content; the template fixes the shape, the same way every time.

Three ideas ended up doing the real work, and they fit together cleanly:

Piece What it does
Definition of Done What must be true for the task to count as complete
Output contract What the finished work must contain
Verification gate How we decide whether it passes

What changed

I can't give you a clean before-and-after number, and I'm not going to invent one. But the shape of the work changed in a way that mattered more than any metric.

Before, the workflow was: prompt, agent, output, hope. After, it was: plan, define done, execute, verify against an output contract.

The biggest improvement wasn't that the agents got smarter. They didn't. It's that the whole thing got easier to reason about. Bad outputs were easier to spot, corrections were easier to make, and two outputs were finally easy to put side by side and compare. When you can tell good from bad quickly, everything downstream gets cheaper.

What I learned

I went in believing more capability would make the agents more useful. More tools, more skills, more autonomy, more freedom. What I came out with was close to the reverse:

  1. Capability without constraints just creates ambiguity. More tools don't help an agent decide what to do with them.
  2. Don't confuse intervention with control. Stepping in mid-run often just adds context to a process that was already confused.
  3. Define success before you ask for execution. A Definition of Done gives the human and the agent the same thing to verify against.
  4. Structure is an interface. An output contract isn't decoration: it constrains the space of answers the agent can give.
  5. Reliable agents need reliable workflows. This is the one I keep coming back to under Agentic Engineering: agent reliability isn't only a model problem, it's a workflow-design problem.