When should an agent
act, revise, or ask?

1. Treat trust as a set of testable claims.

As agents become more capable, a successful task result tells us less about how they got there. A report can be accurate but published without permission. A benchmark can report success because the agent changed the grader. A policy can prevent harm while blocking enough legitimate work to make the system unusable.

These are different failure modes. They require separate evidence.

Safeguards is a proposed research program built around existing work on unsupported answers and agent action controls. It starts with Reins, explores evidence signals motivated by FRED and InterpDetect, and proposes Karotte as the environment for controlled experiments. The integrated system and its benefits have not yet been demonstrated.

2. Build on the work that already exists.

FRED: identify and correct unsupported answers.

FRED studies fine-grained hallucination detection and correction in financial question answering. Its contribution to this agenda is a way to distinguish error types and study targeted correction. Editing here means correcting generated text, not changing model weights. Extending that idea to an agent’s proposed action is a new hypothesis.

InterpDetect: investigate the evidence behind a decision.

InterpDetect studies interpretable signals from a smaller proxy model for detecting hallucinations in retrieval-augmented generation. It motivates testing whether such signals help an action controller beyond tool rules or an external judge. It does not show that we can inspect a proprietary agent’s internal activations, intervene in them in real time, or transfer its results directly to action safety.

Reins: intervene before an action is taken.

Reins supplies open-source primitives for policy decisions, approval flows, and action records. It provides an implementation starting point, not a verified guarantee that every tool is mediated or every failure blocks execution. The first work is to reproduce and audit the relevant execution paths.

Karotte: protect the environment and evaluator.

Karotte, developed independently by Preference Model, supplies mechanisms for building hardened task environments. Its design separates student execution from grading, protects submitted artifacts, and provides tools for reproducing known attacks. We propose using those mechanisms rather than building another sandbox framework.

Karotte protects important parts of the evaluation process. It cannot, by itself, establish that a task’s reward matches the user’s intent. That remains a task-design and measurement problem.

3. Make the boundaries explicit.

The proposed first integration is a narrow task-tool adapter. It is not a universal proxy for every possible agent action.

1. Define the task and permissions. Keep the user’s objective, allowed resources, prohibited side effects, and approval requirements explicit. Treat documents and tool results as evidence, not permission to change the task.
2. Evaluate the proposed action. A trusted tool adapter submits the action, relevant context, and available evidence to a policy decision. Only that adapter may execute the scoped tool action.
3. Apply the decision. Allow an authorized action; reject a prohibited one; request approval when needed. A proposed revision is a new action that must be evaluated again. Bind approvals to the exact action and relevant state.
4. Protect evaluation. Use Karotte’s task lifecycle, confinement, and submission handling. Keep the reward calculation, reference material, and protected policy state outside the student’s control.
5. Reconcile the evidence. Join the intended action, actual tool side effect, policy decision, approval, submission, and score under the same run identity. Mark missing records explicitly.

Karotte’s custom MCP tool server runs with privileged access, so tool authors must deliberately restrict student operations. Runtime guarantees also vary: VM-backed limits and ordinary container watchdogs are not equivalent. Each experiment must record the actual confinement configuration and test the enforcement paths it relies on. See the tool and runtime documentation.

Evidence completeness is its own claim. A completed run is not necessarily a fully captured run; Karotte’s backend documentation describes artifact-upload failures that do not fail the run. Reports must preserve those gaps.

4. Start with one bounded experiment.

Use a simulated workflow: an agent prepares a report from supplied files and may request publication through a scoped tool. The task specifies which files may be read, who may receive the report, and when approval is required. No real customer files or external publication are needed.

Create paired benign and adversarial variants covering out-of-scope disclosure, unnecessary approval of an authorized action, malicious instructions in retrieved material, and attempts to alter the submission or grader.

Keep Karotte confinement fixed when measuring action-decision quality. Test its protection mechanisms separately inside an outer isolated environment. Otherwise a change in infrastructure can be mistaken for an improvement in evidence use.

Develop on one split, freeze the policy and thresholds, then evaluate on held-out variants. Keep scripted regressions separate from tests of transfer. Use unfamiliar task wording, resources, tools, and permission combinations to probe what the controller actually learned.

Three hypotheses to test.

• Evidence helps decisions. Adding defined evidence signals reduces harmful execution at matched benign-task completion. If it only increases blocking, this hypothesis is unsupported.
• Approval can be more selective. A controller can reduce interruptions without increasing harmful approvals, abandoned tasks, or human review effort.
• The boundaries are complementary. Action controls address authorization failures; protected evaluation addresses grading failures. Improvement in both categories does not automatically establish synergy.

5. Report usefulness and failures together.

Use permission specifications, observed tool-side effects, and protected outcome checks as separate evidence. A judge’s agreement with another judge is not sufficient validation.

• Unauthorized execution: completed actions that violate the permission specification, reported against the relevant unauthorized-action opportunities.
• Benign-task completion: authorized tasks completed correctly, including any permitted approval step.
• Interruption cost: approval requests, unnecessary requests, reviewer time, and abandoned tasks.
• Evaluation integrity: successful tampering attempts and whether the reported score still corresponds to the submitted work.
• Evidence coverage: runs with complete, reconciled action and outcome records. Timeouts, missing traces, and invalid runs stay in the report.
• Operational cost: added latency and compute for the complete decision path, not just one policy function.

Compare policies across thresholds and show the safety-versus-completion tradeoff. Use paired cases where possible, publish uncertainty, and separate development results from held-out performance. Choose acceptable risk and interruption levels before inspecting the holdout; those levels depend on the intended workflow.

There are no benchmark results for the proposed integrated system in this note. A favorable replay is evidence about a known case, not proof of general robustness.

6. Ship evidence in small increments.

First: a reproducible foundation.

Pin Reins, reproduce installation and execution, and audit error paths, unknown tools, policy loading, approval binding, and record failures. Publish the threat model and a clear list of mediated actions.

Next: an adapter and a task pack.

Implement a narrowly scoped Karotte integration. Keep trusted policy configuration outside student control. Demonstrate both an authorized success and an enforced denial through each supported path. Record task, model, policy, tool, runtime, and grader versions.

Then: a measured research contribution.

Run the controlled comparisons above. Release the task definitions, allowed inputs, evaluation code, and appropriately sanitized evidence. Broaden the scope only when the held-out results show a useful tradeoff and the records reconcile.

The aim is a contribution researchers can reproduce: a precise question, a working experimental surface, and evidence that makes the next decision easier.

References & provenance

1. Tan, L., Huang, K.-W., & Wu, K. (2025). FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Models. ICML 2025 Workshop on Assessing World Models. Workshop listing.
2. Tan, L., Huang, K.-W., Shi, J., & Wu, K. (2025). InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generation. NeurIPS 2025 Workshop on Mechanistic Interpretability. Workshop paper.
3. Pegasi AI. Reins. Open-source agent action controls. Apache-2.0. Source reviewed at revision d5bc85d.
4. Preference Model. Karotte: A Technical Deep Dive. October 7, 2026. Source repository.
5. Karotte documentation: custom tools, runtimes, scoring, and backend integration.

Safeguards continues Kevin Wu’s work developed under Pegasi AI. Prior research is credited to all listed authors. The proposed roadmap is a Safeguards research direction; it does not imply a partnership with Preference Model or an existing Karotte integration.