An AI engineering stack should be selected by responsibility, not by tool popularity. The important question is not how many products appear in the architecture. It is whether every component reduces a specific engineering risk or satisfies a constraint that the existing system cannot handle.

AI projects often accumulate tools faster than they accumulate operational clarity. A system may contain an orchestration framework, vector database, prompt platform, model gateway, evaluation service, and several observability products while still lacking a reliable deployment gate or rollback procedure.

The resulting architecture looks modern but remains difficult to explain. Engineers cannot identify which component owns input validation, where retrieval quality is evaluated, which model version produced an answer, or how a failed request can be reconstructed.

The main argument of this article is that the stack should be designed as a map of responsibilities. Tool names should appear only after data flow, interfaces, quality controls, ownership, and operating constraints are explicit.

The practical problem is operating the complete request path

Consider an internal support assistant that answers technical questions using company documentation.

A user sends a question through an API. The system validates the request, retrieves relevant passages, constructs a model input, generates an answer, attaches source references, records the execution trace, and returns the response.

The first prototype may implement this process in one Python function. That can be appropriate while the team is testing whether retrieval and generation create useful answers.

Production introduces additional requirements. The API contract must remain stable. Retrieved evidence must be inspectable. Model and prompt versions must be recorded. Evaluation must prevent known regressions. Sensitive content must not appear in unrestricted logs. A previous release must remain available if the new version fails.

These requirements define the stack more reliably than a preferred list of frameworks.

A stack is useful when these responsibilities are covered by understandable components and interfaces. It is not useful merely because each layer has a specialized vendor.

A tool list is not an architecture

A typical stack diagram may contain Python, FastAPI, LangGraph, PostgreSQL, pgvector, a model API, an evaluation framework, Kubernetes, and an observability platform.

The diagram still leaves important questions unanswered. It does not show whether prompts are versioned, whether the retrieval index can be rebuilt, whether evaluation blocks deployment, or whether state-changing tool calls require approval.

Architecture begins with boundaries and contracts.

A component should have a defined input, output, owner, failure behavior, and replacement boundary. Without those properties, the system contains dependencies but not controlled responsibilities.

Stack completeness means covering required responsibilities

Let \(R\) be the set of responsibilities required by the system and \(S\) the selected stack.

Define \(cover(s, r)\) as an indicator showing whether component \(s\) provides responsibility \(r\) under an explicit contract.

A simplified completeness condition is:

$$Complete(S, R) = \mathbf{1}\left\{\forall r \in R,\ \exists s \in S : cover(s, r) = 1\right\}$$

This condition does not require one tool per responsibility. One component may cover several responsibilities. A PostgreSQL deployment may store transactional metadata, evaluation records, and vector embeddings for a small system.

The condition also does not reward additional tools. Once all required responsibilities are covered, another component should be added only when it resolves a measured constraint or reduces an important risk.

A practical justification rule can be expressed as:

$$Add(t) = \mathbf{1}\left\{Benefit(t) > BuildCost(t) + OperatingCost(t) + MigrationCost(t)\right\}$$

This is an engineering heuristic rather than a measurable universal law. Its purpose is to force the team to include operational and migration cost in the decision, not only development convenience.

The runtime layer should make execution repeatable

The runtime layer includes the programming language, dependency environment, configuration, model client, and execution process.

Python is common in AI systems because the machine learning ecosystem is concentrated around it. Selecting Python does not by itself make execution repeatable. The system still needs locked dependencies, explicit configuration, tested startup behavior, and separation between development and production credentials.

A runtime boundary should make it possible to identify which code revision, dependency set, model configuration, and environment produced a result.

Containerization can help create a controlled deployment unit, but it does not replace dependency management or artifact versioning. A container built from an unpinned environment can remain difficult to recreate.

The API layer should expose a stable contract

The external API should define request fields, response fields, authentication requirements, error states, timeouts, and compatibility rules.

FastAPI uses Python type declarations to support request conversion, validation, and OpenAPI documentation. These capabilities can reduce manual API boilerplate, but they remain useful only when the schemas describe the real service contract.

An AI endpoint should not return only an unrestricted text field when the consuming system also needs source references, model status, review flags, or structured errors.

The API boundary should separate external compatibility from internal implementation. Replacing the retrieval engine or model provider should not require every client to change its integration.

The retrieval layer owns evidence access

A retrieval layer is justified when the system must search a corpus, filter results by metadata, or attach supporting evidence to a response.

Its responsibility includes more than storing embeddings. It also includes document identity, chunking, metadata filters, index construction, deletion, versioning, and offline retrieval evaluation.

pgvector adds vector storage and similarity search to PostgreSQL. Its official documentation describes exact search and approximate indexes such as HNSW and IVFFlat.

PostgreSQL with pgvector can be a reasonable choice when the corpus is manageable, the team already operates PostgreSQL, and vector records need strong relationships with transactional metadata.

It is not a universal recommendation. A separate retrieval service may become justified when corpus scale, query throughput, index-management requirements, or latency targets exceed what the current deployment can meet.

The decision should be driven by measured retrieval and operational constraints rather than by the assumption that every RAG system requires a specialized vector database.

Orchestration is justified by workflow state

Orchestration frameworks are useful when the workflow contains branching, retries, persistent state, human approvals, resumable execution, or several cooperating steps with different failure policies.

A simple sequence of retrieval followed by one model call may not require a graph framework. Adding one can create another state model, persistence layer, deployment concern, and debugging interface without improving the task.

LangGraph focuses on capabilities such as durable execution, streaming, and human-in-the-loop workflows. Its persistence layer can checkpoint graph state and support resuming interrupted workflows.

These capabilities solve real problems for long-running or stateful agents. They are unnecessary when the workflow has no state worth preserving and no branching policy that requires explicit control.

Evaluation belongs inside the stack

Evaluation is often treated as an activity performed during model selection. In an AI service, it should also be release infrastructure.

A deployment candidate should be tested against representative tasks, known failures, safety constraints, retrieval requirements, latency limits, and comparison baselines.

The evaluation layer should preserve the dataset version, evaluator version, model and prompt configuration, results, uncertainty, and promotion decision.

For a RAG assistant, generation quality and retrieval quality should be evaluated separately. A poor answer can result from missing evidence, incorrect ranking, prompt behavior, model limitations, or unsupported synthesis. One aggregate score cannot identify which layer failed.

The evaluation suite does not need to begin as a separate platform. Versioned test data, executable evaluation code, stored results, and a deployment gate may provide enough control for the first production version.

Observability should reconstruct the request path

Observability makes the running system inspectable through telemetry rather than through assumptions about what the code should have done.

OpenTelemetry defines signals including traces, metrics, and logs. A trace can represent the path of one request, metrics can summarize runtime measurements, and logs can record discrete events.

For an AI request, the trace may include API validation, retrieval, reranking, model execution, tool calls, policy checks, and response construction.

The trace should use identifiers that connect model, prompt, index, and deployment versions. Without those references, the team may know that a request was slow or incorrect without knowing which system state produced it.

Observability also requires data controls. Prompts, retrieved passages, tool outputs, and model responses may contain sensitive information. Logging everything is not a safe substitute for designing useful telemetry.

A stack review should detect missing and unjustified components

The following Python example reviews two stack configurations for a bounded support assistant.

The context requires an API, vector retrieval, evaluation, and production observability. It does not require stateful orchestration because the workflow contains one retrieval step followed by one generation step.

The policy is deliberately simple. It demonstrates how a stack review can detect both missing responsibilities and components that do not have a current constraint.

from dataclasses import dataclass


@dataclass(frozen=True)
class StackContext:
    needs_api: bool
    needs_retrieval: bool
    needs_stateful_orchestration: bool
    needs_production_observability: bool


REQUIRED_BY_CONTEXT = {
    "api": "needs_api",
    "retrieval": "needs_retrieval",
    "orchestration": "needs_stateful_orchestration",
    "observability": "needs_production_observability",
}

ALWAYS_REQUIRED = {"runtime", "evaluation"}


def review_stack(context: StackContext, stack: dict[str, str]) -> dict[str, list[str] | str]:
    missing = sorted(
        responsibility
        for responsibility in ALWAYS_REQUIRED
        if responsibility not in stack
    )

    missing += sorted(
        responsibility
        for responsibility, flag in REQUIRED_BY_CONTEXT.items()
        if getattr(context, flag) and responsibility not in stack
    )

    unjustified = sorted(
        responsibility
        for responsibility, flag in REQUIRED_BY_CONTEXT.items()
        if not getattr(context, flag) and responsibility in stack
    )

    status = "approved" if not missing and not unjustified else "review"
    return {"status": status, "missing": missing, "unjustified": unjustified}


context = StackContext(
    needs_api=True,
    needs_retrieval=True,
    needs_stateful_orchestration=False,
    needs_production_observability=True,
)

candidate_stack = {
    "runtime": "Python",
    "api": "FastAPI",
    "retrieval": "Postgres + pgvector",
    "orchestration": "LangGraph",
    "observability": "OpenTelemetry",
}

revised_stack = {
    "runtime": "Python",
    "api": "FastAPI",
    "retrieval": "Postgres + pgvector",
    "evaluation": "pytest + task-specific evals",
    "observability": "OpenTelemetry",
}

for name, stack in [("candidate", candidate_stack), ("revised", revised_stack)]:
    result = review_stack(context, stack)
    print(
        f"{name}: status={result['status']} "
        f"missing={result['missing']} "
        f"unjustified={result['unjustified']}"
    )

The following output was produced by executing the code:

candidate: status=review missing=['evaluation'] unjustified=['orchestration']
revised: status=approved missing=[] unjustified=[]

The first configuration optimizes appearance instead of coverage

The candidate stack contains recognizable tools for runtime, API delivery, vector retrieval, orchestration, and observability.

It still fails the review because evaluation is missing. The system can execute and trace requests but has no explicit quality layer preventing known regressions from reaching production.

The candidate also includes orchestration even though the declared workflow does not require persistent state, branching, or human approval. The framework may still be technically usable, but the context does not justify its current operational cost.

The revised stack removes the unjustified orchestration component and adds task-specific evaluation. It is approved under the simplified policy because all declared responsibilities are covered without an unsupported layer.

This result does not prove that the revised stack is production-ready. Security, deployment, rollback, load testing, data governance, and ownership still require separate review. The example evaluates only the responsibilities encoded in the policy.

An architecture decision record should include an exit condition

A tool decision should preserve the constraint, considered alternatives, operating consequences, and condition for review.

decision_record = {
    "responsibility": "vector_retrieval",
    "choice": "Postgres + pgvector",
    "constraint": "shared transactional metadata and moderate corpus size",
    "accepted_cost": "Postgres index tuning and embedding migrations",
    "revisit_when": "recall or latency targets fail under representative load",
}

The revisit condition prevents the decision from becoming permanent doctrine.

A component may be appropriate for the current corpus and traffic while becoming unsuitable after volume, latency, security, or availability requirements change.

The record should be updated when the constraint changes, not when a new tool becomes fashionable.

Clean boundaries make components replaceable

Replaceability does not mean that every tool can be exchanged without work. Models, indexes, orchestration frameworks, and observability systems expose different capabilities and data formats.

The objective is to confine the change.

A retrieval interface can return document identifiers, passages, metadata, and scores without exposing the complete storage implementation to the generation layer. A model gateway can normalize request metadata while preserving provider-specific controls internally.

Evaluation should run against stable system behavior rather than one vendor's response schema. Observability should use shared identifiers that survive a model or retrieval migration.

A useful replacement test is whether the team can identify which interfaces, data migrations, evaluations, and deployment steps would change before beginning the migration.

The stack should grow with system risk

A prototype, an internal service, and a high-impact production system should not have identical operational requirements.

The stack should grow because failures or constraints require additional control. Premature infrastructure can slow experimentation and create maintenance work before the task is stable.

Under-engineering creates the opposite problem. A notebook or synchronous script may remain in production after traffic, ownership, and failure cost have exceeded its original assumptions.

Evaluation and observability should share identifiers

Offline evaluation and production telemetry are often implemented as separate systems. They should still refer to the same versions and cases.

A failed production request should identify the model, prompt, retrieval index, configuration, and deployment version. The same case should be convertible into an evaluation example or regression test.

Without shared identifiers, production failures become screenshots and manually copied prompts. The team loses the context needed to reproduce them reliably.

This feedback path is one of the main reasons evaluation and observability belong in the core stack rather than in optional supporting tools.

Operational cost includes human comprehension

A component creates cost through infrastructure, upgrades, security review, data migration, on-call procedures, and developer learning.

It also increases the amount of system behavior that engineers must understand during an incident.

An orchestration framework may reduce custom control-flow code while introducing new state semantics. A specialized vector service may improve query performance while creating another backup, access-control, and deployment boundary.

These costs may be justified. They should appear in the architecture decision instead of being treated as free consequences of using a library.

Onboarding is an architecture test

A new engineer should be able to trace one request from input to output and identify where quality, security, and operational decisions occur.

The documentation does not need to describe every class or configuration option. It should explain the main request path, evaluation path, deployment path, and ownership boundaries.

If onboarding depends on oral history, the architecture is too implicit. If only one engineer understands how retrieval, prompts, deployment, and rollback connect, the stack contains a concentration-of-knowledge risk.

A practical stack review has three stages

1. Map responsibilities. Define data flow, contracts, evaluation, deployment, observability, security, and ownership before selecting specialized products.
2. Justify components. Connect each non-obvious tool to a measured constraint, accepted operating cost, and review condition.
3. Test the complete paths. Trace requests, reproduce failures, run evaluation, deploy a candidate, and verify rollback through the real interfaces.

The review should be repeated when traffic, corpus size, model behavior, risk, team structure, or external requirements change.

The stack cannot replace clear engineering policy

No framework can decide the correct evaluation threshold, acceptable failure cost, data-retention policy, or human approval boundary for every application.

A vector database does not make retrieval relevant. An agent framework does not make actions safe. An observability platform does not make an alert actionable. A model registry does not justify promotion.

Tools implement capabilities. The engineering team still has to define the policies governing those capabilities.

The strongest stack is not the one with the most layers. It is the one whose request path, quality controls, ownership, and recovery behavior remain understandable when the system fails.

Key takeaways

• Choose stack components only after defining the responsibilities, constraints, interfaces, and owners of the complete system.
• Evaluation and observability are core production layers, while specialized retrieval or orchestration tools should be introduced only when the workflow justifies them.
• Every non-obvious tool decision should record its operating cost, replacement boundary, and condition for future review.

Sources

1. FastAPI. Python Types and API Validation. Official FastAPI documentation.
2. LangChain. LangGraph Overview and Persistence. Official LangGraph documentation.
3. pgvector and OpenTelemetry. pgvector Official Repository and OpenTelemetry Signals. Official documentation.