An AI agent is useful when it can complete a bounded class of work under explicit constraints. It is not useful merely because it generates a plan, calls several tools, or produces a long transcript that resembles autonomous reasoning.

The practical question is whether the system can make progress without exceeding its authority. Tool use, memory, planning, and autonomy matter only when the surrounding workflow can validate actions, stop unsafe continuation, request approval, and preserve enough evidence for later review.

This article treats an agent as a controlled engineering system rather than as an unconstrained model loop. The main thesis is that useful autonomy comes from matching permissions to risk. The agent should continue only while its observations, action budget, and policy checks justify the next step.

The business problem is delegated authority

Consider a support agent that investigates duplicate tickets and closes redundant cases.

The intended outcome appears simple. The agent should identify two tickets describing the same incident, preserve the ticket containing the most complete discussion, close the duplicate, and record the relationship between them.

The workflow becomes risky when the available evidence is incomplete. Similar wording does not necessarily mean that two tickets describe the same failure. One ticket may concern a different customer, environment, product version, or billing period.

Closing the wrong ticket can hide an unresolved problem, distort support metrics, and remove a case from an employee's active queue. The action is reversible in principle, but reversal still creates correction work and can delay the original request.

The system therefore needs to answer more than whether two descriptions look similar. It must determine whether the evidence is sufficient, whether the action is allowed, and whether a human must approve the state change.

The useful agent is not the one that closes the largest number of tickets. It is the one that closes valid duplicates and stops when the evidence or authority is insufficient.

The standard agent loop can fail while appearing productive

A common agent architecture follows a repeated sequence:

$$\text{observe} \rightarrow \text{plan} \rightarrow \text{act} \rightarrow \text{observe}$$

The ReAct framework demonstrated the value of interleaving reasoning with actions and observations. Tool results can provide information that was unavailable in the original prompt, while new observations can change the next action.

This loop is a useful foundation, but it does not define the operational boundary. A model can continue selecting actions after the plan becomes invalid. It can repeat a failed tool call, interpret incomplete output as confirmation, or attempt a state-changing action without authorization.

The loop can therefore make visible progress while moving toward the wrong outcome.

The standard loop is not wrong. It becomes unsafe when task completion is treated as the only terminal objective.

The right question is how much authority the evidence justifies

The weak design question is: How autonomous can this agent be?

The stronger question is: Which actions may this agent perform automatically under the current evidence, permissions, and failure costs?

This reframes autonomy as a policy decision.

A research assistant may search public sources and draft a summary without approval because its intermediate actions are read-only and the final output can be reviewed before use. An agent that sends payments, changes production configuration, or deletes customer records requires a much narrower permission boundary.

The same model and planning architecture can therefore be acceptable in one workflow and unacceptable in another. Risk comes from the combination of model behavior, tool authority, environment state, and recovery cost.

An agent is a policy-controlled state machine

A practical agent can be represented as a state transition system.

At step \(t\), the agent has state \(s_t\). The state can contain the user goal, verified observations, remaining action budget, completed steps, pending approvals, and relevant task memory.

The policy proposes an action:

$$a_t = \pi(s_t)$$

The action is not executed immediately. A control function evaluates whether it is permitted:

$$g(s_t, a_t) \in \{ allow, pause, deny \}$$

If the action is allowed, a tool executes it and returns observation \(o_t\). The state is then updated:

$$s_{t+1} = T(s_t, a_t, o_t)$$

If the control function returns pause, the workflow requests human approval or additional information. If it returns deny, the action is rejected and the run ends or selects an allowed alternative.

The agent also needs explicit terminal states. A useful result taxonomy is:

A step limit alone does not create this behavior. The system must know why it is stopping.

Conditions required for bounded autonomy

A controlled agent requires more than a capable language model.

Useful additions include sandboxed execution, automatic policy validation, durable workflow state, and specialized evaluators. The exact implementation depends on the cost and reversibility of the actions.

A bounded baseline based on evidence and permissions

The original confidence-based example increased an internal score after every step. That is not a reliable control mechanism. A model's self-reported confidence is not automatically calibrated, and an arbitrary increase does not show that new evidence was obtained.

The example below uses observable controls instead. A support agent searches for a possible duplicate, compares two tickets, and considers closing one of them.

The comparison service returns a synthetic duplicate score of 0.72. The workflow requires at least 0.90 before a closing action can be considered. The write tool also requires approval, although the run stops before reaching that check because the evidence threshold already fails.

The score and threshold are illustrative policy values. They are not derived from a real support dataset.

from dataclasses import dataclass


@dataclass(frozen=True)
class ToolContract:
    side_effect: str
    requires_approval: bool
    idempotent: bool


TOOLS = {
    "search_tickets": ToolContract(
        side_effect="read_only",
        requires_approval=False,
        idempotent=True,
    ),
    "compare_tickets": ToolContract(
        side_effect="read_only",
        requires_approval=False,
        idempotent=True,
    ),
    "close_ticket": ToolContract(
        side_effect="writes_state",
        requires_approval=True,
        idempotent=False,
    ),
}


def run_agent(
    max_steps: int = 4,
    approved_actions: frozenset[str] = frozenset(),
) -> dict:
    plan = [
        (
            "search_tickets",
            {"query": "billing export fails"},
        ),
        (
            "compare_tickets",
            {
                "left": "T-104",
                "right": "T-091",
            },
        ),
        (
            "close_ticket",
            {
                "ticket_id": "T-104",
                "duplicate_of": "T-091",
            },
        ),
    ]

    trace = []
    duplicate_score = None
    seen_actions = set()

    for step, (
        tool_name,
        arguments,
    ) in enumerate(
        plan[:max_steps],
        start=1,
    ):
        signature = (
            tool_name,
            tuple(sorted(arguments.items())),
        )

        if signature in seen_actions:
            trace.append(
                {
                    "step": step,
                    "tool": tool_name,
                    "status": (
                        "blocked_repeated_action"
                    ),
                }
            )

            return {
                "status": "needs_human_review",
                "reason": "repeated_action",
                "trace": trace,
            }

        seen_actions.add(signature)
        contract = TOOLS[tool_name]

        if tool_name == "compare_tickets":
            duplicate_score = 0.72

        if tool_name == "close_ticket":
            if (
                duplicate_score is None
                or duplicate_score < 0.90
            ):
                trace.append(
                    {
                        "step": step,
                        "tool": tool_name,
                        "status": (
                            "blocked_low_evidence"
                        ),
                    }
                )

                return {
                    "status": (
                        "needs_human_review"
                    ),
                    "reason": (
                        "duplicate_score_"
                        "below_0.90"
                    ),
                    "trace": trace,
                }

            if (
                contract.requires_approval
                and tool_name
                not in approved_actions
            ):
                trace.append(
                    {
                        "step": step,
                        "tool": tool_name,
                        "status": (
                            "blocked_missing_approval"
                        ),
                    }
                )

                return {
                    "status": (
                        "needs_human_review"
                    ),
                    "reason": (
                        "approval_required"
                    ),
                    "trace": trace,
                }

        trace.append(
            {
                "step": step,
                "tool": tool_name,
                "status": "executed",
            }
        )

    return {
        "status": "completed",
        "reason": "plan_finished",
        "trace": trace,
    }


result = run_agent()

print(f"status={result['status']}")
print(f"reason={result['reason']}")

for event in result["trace"]:
    print(
        f"step={event['step']} "
        f"tool={event['tool']} "
        f"status={event['status']}"
    )

The following output was produced by executing the code:

status=needs_human_review
reason=duplicate_score_below_0.90
step=1 tool=search_tickets status=executed
step=2 tool=compare_tickets status=executed
step=3 tool=close_ticket status=blocked_low_evidence

The safe outcome is an escalation

The agent executes two read-only actions. It searches for related tickets and compares the candidate pair.

The third action would change external state. Before calling the tool, the controller checks whether the duplicate evidence meets the required threshold. The observed value of 0.72 is below 0.90, so the closing action is not executed.

The result is needs_human_review. This is not a failed task in the same sense as an exception or crash. It is a correct policy outcome for a case that exceeds the agent's evidence boundary.

The example does not establish that a similarity score of 0.90 is sufficient for real duplicate detection. A production policy would require validation on reviewed ticket pairs, analysis of false closures, and separate treatment of unusual customer or product contexts.

It also does not prove that every write operation requires manual approval. A reversible, low-cost action may be allowed automatically after the system demonstrates sufficiently low error rates. The permission rule should follow the failure cost rather than a universal preference for or against autonomy.

Tool contracts define the operational surface

Function calling connects a language model to external systems. The model proposes a tool and arguments, while application code validates and executes the call. The current OpenAI function-calling documentation similarly treats the tool definition and application-side execution as separate parts of the workflow.

A tool contract should describe more than an input schema.

A tool returning 200 OK is not enough when only part of a multi-resource operation succeeded. The contract must allow the controller to distinguish complete success, partial success, and unknown state.

Retries also need contract awareness. Automatically retrying an idempotent search is usually less risky than retrying a payment or deployment request whose first result is uncertain.

Stopping conditions should be designed before prompts

Prompts often explain how an agent should work but leave stopping behavior vague.

A robust loop should stop when the goal is verified, when a required action is blocked, or when reliable progress is no longer possible. The controller should not rely on the model to invent the appropriate terminal state during an unusual case.

Repeated-action detection is especially useful. If the agent proposes the same tool call with the same arguments after receiving the same result, another attempt is unlikely to create new information.

The controller can stop immediately, request a revised plan, or escalate the case. It should not interpret repetition as persistence by default.

Planning should produce commitments that can be checked

A plan is useful when it decomposes the task into verifiable milestones. It is less useful when it becomes a long narrative that cannot be compared with environment state.

Instead of storing unrestricted prose, the workflow can represent a plan as structured steps containing an objective, expected evidence, allowed tool class, and completion condition.

The plan should be revised when an observation invalidates an assumption. Revision is not permission to expand the task indefinitely. New steps remain subject to the original goal, tool policy, and action budget.

Memory should not become an unreviewed source of truth

Agent memory can refer to several different mechanisms.

The Reflexion framework showed how verbal feedback stored in episodic memory can improve later attempts without changing model weights. That result does not imply that arbitrary self-generated reflections should be treated as verified production knowledge.

A memory write should record its source, scope, creation time, and review status. Derived notes should remain distinguishable from external observations and approved facts.

Memory also needs invalidation. A tool result describing yesterday's deployment state should not silently control today's rollback decision.

An audit trace is not a private reasoning transcript

Inspectability does not require storing unrestricted internal reasoning or hidden chain-of-thought.

A useful audit trace records observable and decision-relevant events:

The current OpenAI Agents SDK observability documentation describes traces containing model calls, tool calls, handoffs, guardrails, and custom spans. This kind of structure is useful because it separates workflow events rather than reducing the run to an unstructured conversation.

A trace should also protect sensitive information. Tool outputs may contain personal data, credentials, proprietary content, or regulated records. Auditability requires retention controls and access rules, not indiscriminate logging.

Evaluation must reward correct stopping

An agent benchmark that rewards only task completion teaches the wrong operational objective.

A system that completes 90 percent of tasks by taking unauthorized actions may be less useful than a system that completes 75 percent and correctly escalates the remaining cases.

Agent evaluation should therefore preserve several outcomes.

Task success should be verified from the environment where possible. A model stating that a ticket was closed is not evidence that the ticket state actually changed.

Benchmarks such as AgentBench evaluate agents in interactive environments rather than through static question answering alone. For a production system, the benchmark still needs local cases reflecting its tools, policies, and failure costs.

The test set must contain situations where the agent should not act

I would divide evaluation cases into three groups:

• Executable tasks: the instruction is clear, evidence is sufficient, and the required action is permitted.
• Escalation tasks: the goal is plausible, but information, approval, or domain judgement is missing.
• Blocked and failure tasks: the request violates policy, a tool fails, or continuation would exceed the action budget.

The evaluation should also inject ambiguous tool responses, partial success, stale memory, repeated failures, and conflicting observations. These cases reveal whether the controller responds to environment state or merely follows the original plan.

The ToolEmu study demonstrated a method for testing tool-using agents in emulated high-stakes scenarios and identified failures with potentially serious consequences. Its results support the broader point that agent safety must be evaluated through interactions, not only through isolated model responses.

Human review needs a defined decision interface

Adding a human approval step does not automatically make a workflow safe.

The reviewer needs enough information to make the decision without reconstructing the entire run. An approval request should identify the proposed action, affected resource, supporting evidence, uncertainty or conflict, and expected side effect.

The current OpenAI guidance on guardrails and human review separates automatic validation from approval decisions that pause a run. This is a useful architectural distinction: deterministic policies should not require manual review, while sensitive judgement should not be hidden inside automatic execution.

Approval should also be scoped. Permission to close ticket T-104 should not become general permission to close all tickets during the session.

Autonomy should increase through evidence, not ambition

A practical deployment can begin with read-only assistance. The agent gathers information, compares records, and prepares a proposed action for review.

After the team measures error rates and reviewer corrections, selected low-risk actions can be automated. Higher-impact or weak-evidence cases remain gated.

This progression is not mandatory for every application, but it prevents authority from being granted before the team understands the failure distribution.

A broader agent is not automatically more valuable. The relevant question is whether additional permissions reduce total work after review, correction, and incident costs are included.

When an agent is not the right architecture

An LLM agent is useful when the task requires interpretation, adaptation to observations, and selection among several tools or paths.

A deterministic workflow is often preferable when the inputs are structured, the action sequence is stable, and the rules can be encoded directly.

Replacing a clear state machine with an LLM loop can increase variance without adding meaningful capability.

How I would operationalize the agent

I would organize the production design into three control layers:

1. Execution control: typed tools, permissions, idempotency rules, budgets, and sandboxing.
2. Decision control: evidence requirements, approval policies, stopping conditions, and scoped memory.
3. Evaluation control: structured traces, failure injection, task verification, and correction-cost measurement.

The layers can be implemented in different services or frameworks. What matters is that the language model does not define its own authority.

Before deployment, I would test the complete workflow with the same tool schemas, policy rules, and approval boundaries used in production. A model-only benchmark does not test serialization errors, stale resources, partial tool results, or incorrect permission configuration.

After deployment, reviewed failures should become regression cases. The goal is not only to improve the prompt. A failure may require a narrower tool, a stronger validator, a different terminal state, or removal of an unnecessary permission.

What the controls cannot guarantee

A bounded architecture reduces avoidable operational risk, but it does not prove that the system will behave correctly in every environment.

Evidence thresholds may be poorly calibrated. A human reviewer may approve an invalid action. Tool descriptions may omit an important side effect. A safe action in isolation may become unsafe when several actions are combined.

Automated evaluators can also miss qualitative failures. Trace grading can identify recurring patterns, but high-impact and unusual cases still require direct inspection.

The purpose of the control architecture is not to eliminate uncertainty. It is to make uncertainty visible before the system converts it into an external action.

Key takeaways

• Agent autonomy should be defined as permission to perform specific actions under explicit evidence, policy, and recovery conditions.
• Correct escalation, blocked unsafe actions, and reviewable traces are part of agent quality rather than exceptions to task completion.
• Useful agents combine a capable model with typed tools, deterministic controls, bounded execution, and evaluation based on real environment outcomes.

Sources

Primary publications

1. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations.
2. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Advances in Neural Information Processing Systems, 36.
3. Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., and Hashimoto, T. (2024). Identifying the Risks of LM Agents with an LM-Emulated Sandbox. International Conference on Learning Representations.

Agent evaluation

1. Liu, X., Yu, H., Zhang, H., et al. (2024). AgentBench: Evaluating LLMs as Agents. International Conference on Learning Representations.

Official documentation

1. OpenAI. Function Calling. OpenAI API documentation.
2. OpenAI. Guardrails and Human Review. OpenAI API documentation.
3. OpenAI. Evaluate Agent Workflows. OpenAI API documentation.