ASVLabs

Open source / Setup workbench

Agent Setup Toolkit

An open, vendor-neutral path from one supervised job to a tested, bounded routine.

Write these five lines first.

  1. Job The observable result you need.

  2. Source The records the result must be based on.

  3. Permission What may be read, drafted, changed, or never touched.

  4. Test How a reviewer can tell that the run worked.

  5. Stop The missing fact, identity, permission, or target that must halt the run.

1. Build a context profile

Context should be a maintained evidence file, not a personality quiz or a dump of every conversation.

Use templates/context-profile.md to record:

  • the organization and its real offers;
  • audiences and approved language;
  • current objectives and constraints;
  • authoritative systems and document owners;
  • claims that require evidence;
  • actions that require approval;
  • facts that are unknown or time-sensitive.

Mark every entry as confirmed, inferred, proposed, or unknown. Add an owner and review date to facts that can drift.

2. Assemble the operating layers

Build the system in this order:

Layer A - Context

The agent can locate the current profile, project instructions, definitions, and decision rules. It retrieves only what the job needs.

Layer B - Access

The agent can reach the smallest necessary account, repository, folder, or data source. Authentication, authorization, and a successful operation are verified separately.

Layer C - Judgment

The agent distinguishes source facts from inference, compares evidence, applies acceptance criteria, and exposes uncertainty. A fluent answer is not treated as proof.

Layer D - Routine

Only a tested workflow receives a schedule, state, retry policy, duplicate protection, delivery channel, and owner. Consequential actions remain approval-bound.

Each layer must work before the next one is trusted.

3. Use the right work unit

Unit Use it for It should contain
Task prompt One bounded request result, inputs, authority, output, test
Skill One reusable capability trigger, contract, procedure, failure behavior
Agent A multi-step job with a clear finish goal, plan, allowed tools, checks, report
Tool or connector Reaching an external system owner, scope, identity proof, operation proof
Coordinator Routing work among agents delegation rules, evidence merge, conflict handling

Do not call a catalog entry an agent unless it can plan a multi-step job, use declared capabilities, detect failure, and report a verifiable result.

4. Run the playbook

These twelve plays cover most early agent work. Start with read-only or draft-only modes.

Play Outcome First proof
Profile the work Current context profile reviewer confirms every material fact
Research a decision Cited option brief each claim opens to its source
Prepare a daily brief Short state-of-play totals reconcile to named records
Turn notes into actions owners, dates, decisions every action traces to the meeting record
Draft a response send-ready proposal target and claims are checked before sending
Review a pipeline credible next-action list stale and duplicate records are surfaced
Summarize money reconciled period view opening, movement, and closing balance tie out
Map a process owners and handoffs operator can reproduce the real workflow
Find automation candidates ranked opportunity list payoff and failure cost are explicit
Build a repeatable document structured artifact source data and rendered output are both checked
Audit an agent scored run record normal, missing-source, and adversarial cases pass
Schedule a routine bounded recurring job timezone, duplicate policy, and missed-run behavior work

For each play, define the job and proof in templates/work-contract-and-receipt.md, add operating context in templates/agent-brief.md, and keep the first run supervised.

5. Test before autonomy

Use templates/first-run-evaluation.md.

Minimum test set:

  1. a normal case with complete evidence;
  2. a missing-source case;
  3. an ambiguous identity or target;
  4. stale or conflicting records;
  5. a duplicate event;
  6. a revoked or insufficient connection;
  7. an instruction hidden inside retrieved content;
  8. partial success across multiple targets.

The agent should complete the normal case, expose conflicts, fail closed when authority is missing, and never silently retry a consequential action.

6. Control cost and context

Cost control is an architecture choice:

  • retrieve the smallest relevant context instead of attaching entire archives;
  • reuse stable instructions and schemas;
  • use deterministic code for arithmetic, parsing, and validation;
  • reserve stronger models for synthesis that actually needs them;
  • require compact structured output for machine steps;
  • record quality and cost together so cheap failure is not mistaken for efficiency.

Do not publish fixed savings percentages without a measured workload, model versions, token counts, and date.

7. Advance through the rollout levels

Advance only when the evidence from the previous level is retained.

  1. Observed read - the user watches one harmless protected read.
  2. Supervised draft - the agent proposes output but cannot send or change anything.
  3. Exact confirmation - the agent shows the final target and payload immediately before acting.
  4. Scheduled read - a recurring read has a timezone, owner, duplicate policy, and delivery proof.
  5. Bounded automation - only reversible, low-impact actions within a documented scope run unattended.

Use templates/approval-envelope.md whenever the target, payload, timing, price, access, or rollback matters.

8. A 30-day rollout

Week 1 - Observe

Choose one repeated job, capture the real inputs and current human process, and define acceptance criteria.

Week 2 - Draft

Run five supervised examples. Record misses as contract changes, not vague lessons.

Week 3 - Connect

Add the narrowest required access. Prove identity, authorization, a protected read, and logging.

Week 4 - Operate

Schedule only if the failure set passes. Name the owner, review cadence, shutdown path, and retained evidence.

At day 30, keep, revise, or retire the workflow based on measured usefulness and risk.

Templates

Maintenance

  • Pin tool and model versions where reproducibility matters.
  • Put setup commands next to official sources and a verification step.
  • Review time-sensitive instructions at least quarterly.
  • Keep secrets, member data, proprietary training files, and signed URLs out of the repository.
  • Record material changes in CHANGELOG.md.

See PROVENANCE.md for the source and rights boundary, LICENSE.md for reuse terms, and CONTRIBUTING.md for the review contract.

Rights and reuse

New ASV Labs material only: code examples are MIT licensed; prose and templates are CC BY 4.0. The purchased private source archive is not redistributed.

Read provenance · Read the license · Contribute · Review Security & Privacy