Audit Agent: an AI drafting assistant that has to argue with itself

Demonstration data. Scripted demonstration using invented audit content.

Ask a model to polish an audit finding and it will give you a tidier paragraph. Ask a second model call to review that paragraph as a sceptical quality reviewer and it will tell you the recommendation has no owner and no deadline. Feed that back and the third draft is the one worth reading. Audit Agent is built around that loop, and the rest of the tool is the plumbing that makes the loop safe to run on a whole engagement.

--- Round 1: polish the seed issue, then check it
model=claude-sonnet-5  structured_schema=yes  thinking=disabled
checker verdict: Needs Revision
  - [Recommendation] Name the owner and a deadline.
  - [Impact] State the business risk.

--- The auditor accepts both recommendations as instructions
AUDITOR INSTRUCTIONS - MUST BE INCORPORATED:
1. Name the owner and a deadline.
2. State the business risk.

--- Round 2: polish again with the instructions, check with prior context
polish request carries instruction block: True
checker request carries do-not-re-raise block: True
checker verdict: Approved

--- Strict QA routes the checker to a different model and adds an adversarial suffix
model=claude-opus-4-8  structured_schema=yes  thinking=disabled
adversarial suffix present: True

--- Review cap reached: the sign-off step parses a code-fenced reply
verdict: Ready to Approve | sign_off_mode: True

--- Tolerant JSON parsing: a reply with a raw line break inside a string
plain json.loads: Invalid control character at
extract_json_from_response: {"issue_title": "Example", "finding": "line one
line two"}

Figure 1. A scripted demonstration of Audit Agent’s generate, review and approval cycle using invented audit content. It shows how reviewer comments are accepted as instructions and carried into the next draft.

At a glance

• Status: Prototype.
• Runtime environment: Local Docker Compose environment.
• Platform: a React web front end and a Python (FastAPI) service with a PostgreSQL database.
• Stack: React, Vite, Tailwind, FastAPI, SQLAlchemy, Alembic, PostgreSQL, Docker Compose, pytest, GitHub Actions.
• Built with: Claude Code.
• AI inside the product: yes. Claude drafts and reviews risk assessments, control matrices, control test write-ups and audit issues. The default model in the code is Claude Sonnet 5, with Claude Opus 4.8 as an optional stricter reviewer.

An engagement in six phases

An engagement is a session. You give it a subject, such as access management for a fictional company, choose the frameworks you want to work to, and upload background documents. The tool walks through six phases.

It drafts a risk assessment as a set of control areas, and you decide which are in scope. It builds a risk control matrix (a table that links each risk to the controls meant to manage it), one area at a time. For each control it drafts a control test from the evidence you attach and any questionnaire answers: separate conclusions on design and operation, a classified list of evidence gaps, and a fraud-risk consideration. It polishes issues, or combines several into a themed one. It drafts a report narrative and management actions.

The control test has the most detailed instructions. They ask for a conclusion on whether the control objective is met, not just whether the activity happened. An evidence gap must be one of three kinds: no evidence exists, not yet provided, or access refused. The kinds matter differently in a report, and the prompt tells the model to default to “not yet provided” unless the text says otherwise. The fraud consideration is a required field on every test, and the reviewer prompt tells the checker to flag it if it reads as boilerplate.

Generate, check, decide

A tool that produces a finished-looking document and asks to be believed is a risk in audit work. So each AI step is a loop (Figure 2). The generate call drafts. A separate call, with a reviewer’s prompt, checks the draft and returns a verdict and recommendations. The auditor accepts or rejects each recommendation, and the accepted ones go into the next draft as instructions that must be reflected. The checker is told what was already accepted, so it does not raise the same points again.

Build decision. The loop stops after two rounds. After the second round a lighter sign-off call summarises what changed, and the auditor still approves or edits by hand. I put the loop’s shared pieces in one module after the same fix had to be made in several places.

There is also a per-session strict mode. It sends the check to a different, more capable model and adds an instruction to look for reasons to reject, because a reviewer that defaults to approving is not much of a review. The structured replies are enforced by the API against a JSON schema, which replaced a function that tried to repair broken JSON. The old repair code remains for the free-text sign-off call (the last block in Figure 1 shows it parsing a reply with a raw line break).

Long jobs that survive a crash

A control matrix for a big engagement can take minutes, and a model call can fail halfway. The first version started a new thread for every request. Now four of the AI steps (matrix generation, control test, issue polish, themed issue) are rows in a jobs table.

job = db.scalars(
    select(models.Job)
    .where(models.Job.status == "queued")
    .order_by(models.Job.created_at)
    .with_for_update(skip_locked=True)
    .limit(1)
).first()

Figure 3. The claim step from the job queue. The row lock is held only long enough to mark the job running; the long model call happens afterwards.

A worker claims one job, commits straight away so the lock is released, and then makes the model call. The result and the “completed” flag are written in one transaction, so a crash either leaves the job to run again or leaves a finished one, never a half-written version. I chose to re-run a crashed job from the start rather than resume it, because resuming needs durable state for every step and that is a bigger machine than a tool like this needs. A job that keeps crashing the process is failed after three attempts.

The batch button follows the same thinking. It queues only controls not yet tested, and skips any control where the auditor has recorded reliance on another assurance source, and any that failed, because a person should look at those first.

Under the hood

I built Audit Agent with Claude Code as a React and FastAPI application, using PostgreSQL to hold the engagement state and a worker queue for the longer AI jobs. A status register of tasks and requirements tracks what is built and what is next.

Layer	What I used
Interface	React, Vite, Tailwind
Backend	Python, FastAPI, SQLAlchemy, with a four-thread worker pool for queued jobs
Data	PostgreSQL, migrations with Alembic, a jobs table used as the queue
Runtime AI	Claude Sonnet 5 by default, Claude Opus 4.8 as an optional stricter reviewer, via the Anthropic API with JSON-schema structured output
Hosting	Local Docker Compose environment
Test and operational tooling	pytest, an eval harness against the live API, GitHub Actions

The project also has an eval harness that runs each phase against the live API and asserts on the output: that it matches the schema, that severity is one of three values and that a root cause is present.

What it shows

Audit Agent turns AI drafting into a workflow an auditor can review. Structured outputs keep every draft in a known shape, an independent checking pass challenges it, the auditor makes each decision explicitly, review rounds are bounded so the loop always finishes, and durable background jobs mean a long engagement is never left half written.