AI Product Engineering

System architecture where the AI capability is a first-class dependency, not a feature you call from the application layer. The architectural decisions that determine whether an AI product works at scale are made at this layer, not discovered after launch.

Data model for AI inputs and outputs: the prompt context, retrieved documents, model response, evaluation score, and user feedback action are all first-class entities stored in the database, not transient API responses that disappear after the UI renders. This matters because prompt regression testing, cost attribution, and the improvement cycle all depend on having a retrievable record of what the model was asked and what it returned.

Latency architecture: for interactive AI products (chat, copilot, real-time analysis), the p95 latency target is defined before architecture is chosen. GPT-4o averages 1-3s for short responses via streaming; for non-streaming or long outputs, p95 latency can reach 8-15s. Architecture decisions that affect latency: response streaming (Server-Sent Events from the backend to the browser, so the user sees output arriving rather than waiting for the full response); async job processing for non-interactive tasks (document analysis, batch classification); model selection (GPT-4o mini at 1-2s vs GPT-4o at 3-8s for the same query); and context window management (fewer tokens in = lower latency and cost).

Provider abstraction layer: the application calls a provider-agnostic service interface, not the OpenAI SDK directly. If you switch from OpenAI GPT-4o to Anthropic Claude 3.5 Sonnet or Google Gemini 1.5 Pro, one service adapter changes, not 40 call sites across the codebase. This is critical for AI products because model pricing, capability, and availability evolve faster than the product.

Fallback architecture: when the primary model provider is unavailable or rate-limited, the system falls back to a secondary provider automatically (OpenAI primary, Anthropic secondary) via the abstraction layer; circuit breaker pattern with a 60-second retry window prevents cascading failures from amplifying latency under load.

The data infrastructure that feeds the AI, ingestion pipelines, retrieval systems, and the feedback loop that keeps the data current. Most RAG products fail not because the model is wrong but because the retrieval layer returns stale, irrelevant, or truncated context. We engineer retrieval as a product concern with measurable quality targets, not a configuration problem solved once and forgotten.

Document ingestion pipeline: PDF extraction via pdfminer.six (structured PDFs with embedded text) or PyMuPDF (scanned PDFs via OCR); HTML content via crawl pipeline with readability extraction stripping navigation and boilerplate; database content via scheduled Fivetran or Airbyte sync; API content via incremental polling with cursor-based extraction. All sources normalised to a common document format (title, source URL, last-modified timestamp, content body) before chunking.

Chunking strategy: fixed-size chunks with overlap (512 tokens, 50-token overlap) as the baseline; recursive character splitting with sentence boundary respect for narrative content; semantic chunking (grouping sentences by embedding similarity) for documents where topic boundaries don't align with paragraph breaks. Section-level chunking for structured documents (contracts, technical manuals) where a section header provides critical context for the chunk content. Chunk metadata (document ID, section title, page number, source URL) stored alongside the embedding for attribution.

Embedding models: OpenAI text-embedding-3-large (3072 dimensions, best performance for English-dominant enterprise knowledge bases); Cohere embed-v3.0 (multi-lingual, strong for mixed-language corpora); sentence-transformers all-mpnet-base-v2 (self-hosted for air-gapped or compliance-restricted deployments). Vector stores: Pinecone Serverless for managed hosting with namespacing for multi-tenant isolation; pgvector on PostgreSQL for organisations that want to keep data in their existing database; Weaviate for multi-modal (text + image) retrieval.

Retrieval optimisation: hybrid search combining BM25 sparse retrieval and dense vector retrieval, with Reciprocal Rank Fusion (RRF) to merge the two ranked lists, outperforms pure vector search on domain-specific terminology that the embedding model hasn't seen in pre-training. Cohere Rerank API as a second-stage reranker: the top 20 BM25+vector results are re-ranked by a cross-encoder model that considers query-document relevance more precisely than cosine similarity. Retrieval quality measured as recall@5 (fraction of ground-truth relevant documents appearing in the top 5 results) and MRR@5 (mean reciprocal rank of the first relevant document), tracked on every pipeline change.

Automated evaluation pipelines that run your test dataset against every model or prompt change, catching quality regressions before they reach users. Evaluation is a first-class engineering deliverable, not a spreadsheet maintained by the ML team.

Evaluation dataset construction: minimum 100-200 labelled test cases covering the distribution of query types expected in production, including adversarial cases (queries designed to elicit hallucination), edge cases (ambiguous queries with multiple valid answers), and refusal cases (queries the model should decline). The dataset is version-controlled in Git alongside the codebase; when a regression is found in production, the failing case is added to the dataset so it is tested in every future evaluation run.

Evaluation metrics by task type: for RAG question-answering, context precision (fraction of retrieved context relevant to the query), context recall (fraction of ground-truth information present in the retrieved context), answer faithfulness (fraction of answer claims grounded in retrieved context, not hallucinated), and answer relevance (how directly the answer addresses the question). For classification tasks, F1 score, precision, recall by class. For generation tasks, ROUGE-L for extractive content, BERTScore for semantic similarity, and LLM-as-judge scoring (GPT-4 evaluating correctness and relevance on a 1-5 scale) for subjective quality.

Evaluation tooling: LangSmith for LLM application tracing and evaluation (traces every LangChain and LangGraph call with the full context, model parameters, and output); Langfuse for self-hosted tracing in compliance-sensitive deployments; Ragas for RAG-specific metric calculation (context precision, faithfulness, answer relevance). CI/CD integration: evaluation pipeline runs on every pull request via GitHub Actions; a PR that reduces answer faithfulness below 0.80 or increases hallucination rate above 5% blocks merge. Production quality monitoring: daily evaluation run on a random sample of production queries (with PII stripped) against the ground-truth dataset, with Slack alert when metrics fall below threshold.

Designing the product experience around the actual properties of AI output, uncertainty, latency, and occasional incorrectness, rather than treating AI as a deterministic data source that always returns a correct answer instantly. Users who experience a confident-sounding wrong answer with no correction path lose trust in the entire product. The UX engineering layer handles this.

Streaming output implementation: Server-Sent Events (SSE) from the FastAPI or Node.js backend to the browser, rendering the model response token-by-token as it arrives. This transforms a 4-second wait for a complete response into visible content appearing within 0.5 seconds, the single most impactful latency improvement for chat and copilot interfaces. React state update batching configured to avoid re-render thrashing on high-frequency token events (debounced to 60fps render cadence).

Loading state design calibrated to expected wait time: for queries with expected latency under 2 seconds, a skeleton screen with pulse animation matching the output layout; for 2-8 second queries, an animated "Thinking..." indicator with elapsed time; for queries over 8 seconds (document analysis, batch operations), a progress bar with stage labels (Retrieving documents... Analysing... Generating...). Users who understand that work is being done abandon at a fraction of the rate of users watching a spinner with no context.

Citation display for verifiable claims: every factual statement in the AI output is linked to the source document chunk that contains the supporting evidence. The citation expands on click to show the exact passage from the source, so users can verify without leaving the product. Citation design matters most when the product serves professionals (lawyers checking case law citations, analysts verifying data claims, doctors reviewing clinical evidence) where unverified AI output creates liability.

Correction and feedback flows: a thumbs-down interaction on any AI output opens a structured feedback panel, incorrect, irrelevant, harmful, or other, with an optional free-text note. The feedback is logged with the full trace (query, retrieved context, model parameters, output) and surfaced in the quality dashboard. Correction flows for structured outputs (where the model fills form fields or generates data tables) allow inline editing with the corrected value stored as a ground-truth label for the next evaluation run.

Engineering the product to hit your cost-per-query and latency targets requires measuring both at realistic production query distributions before selecting models and designing the inference architecture. Benchmarks from marketing pages are not the right input, your specific query length distribution, context window usage, and output verbosity determine your actual cost and latency.

Cost profiling methodology: instrument the development environment to log input token count, output token count, model used, and latency for every inference call during a 2-week sprint. Analyse the distribution: p50/p95/p99 latency, mean and p95 token count per query type, and cost per query at target volume (e.g., 100,000 queries/month). This produces a cost model showing what the product costs to run before it has users, not as a surprise at month 3.

Model selection decision: GPT-4o at $2.50/million input + $10.00/million output tokens vs GPT-4o mini at $0.15/million input + $0.60/million output, the 10-17x cost difference is significant at scale, but only if GPT-4o mini's quality meets the evaluation bar for your specific task. We run both models against your evaluation dataset and measure quality delta on your task, not on MMLU benchmarks. For many document-grounded Q&A tasks, GPT-4o mini achieves 90-95% of GPT-4o quality at 15% of the cost.

Semantic caching with Redis: queries that produce semantically similar (not just string-identical) outputs are served from cache. Embedding the query at inference time and comparing cosine similarity against cached query embeddings (similarity threshold 0.95+) yields cache hit rates of 20-40% for enterprise knowledge base products where users frequently ask the same question in different phrasings. Each cache hit eliminates a model API call entirely.

Prompt compression: LLMLingua or LLMLingua-2 applied to long retrieved contexts, compressing retrieved chunks by removing low-importance tokens while preserving the factual content the model needs. Typical compression ratio 2-3x with under 5% quality impact, directly reducing per-query cost and latency proportionally. Model routing: a lightweight classifier (fine-tuned DistilBERT or GPT-4o mini prompt classification) categorises each incoming query as simple (single-hop factual retrieval) or complex (multi-step reasoning, synthesis across multiple documents). Simple queries route to GPT-4o mini; complex queries route to GPT-4o. This routing layer typically reduces cost by 35-50% versus routing all queries to the larger model.

The improvement loop is the engineering infrastructure that converts production usage into measurable quality improvements, without it, an AI product plateaus at launch quality. We build this loop as a product component, not a post-launch operations task.

Production signal capture: every AI interaction is instrumented with a structured event (query_id, session_id, model_used, prompt_version, input_token_count, output_token_count, latency_ms, retrieved_chunks, output_text). User interaction events logged on top: output_accepted (user used the AI output without modification), output_edited (user modified the output, the edited version stored as a ground-truth correction), output_rejected (user dismissed or regenerated), and query_reformulated (user followed up with a clarification, indicating the first response missed intent). These signals are distinct quality indicators: acceptance rate measures precision; reformulation rate measures whether users got what they asked for the first time.

Failure pattern analysis: weekly review of the 50 most common failure patterns from production logs, queries where output_rejected or query_reformulated rates exceed 20% for a given query cluster. Query clusters identified by semantic embedding similarity (k-means on query embeddings with k=20-30 clusters). Each high-failure cluster gets a root cause analysis: is the retrieval returning irrelevant context? Is the prompt not handling this query type? Is the model confidently wrong on this specific domain? The root cause determines the fix, retrieval tuning, prompt update, or fine-tuning.

Prompt versioning and A/B testing: prompts stored in the database as versioned records (not hardcoded in the codebase). A/B tests run by splitting incoming queries by user cohort (50/50 or 80/20 depending on risk tolerance) and measuring quality metrics per prompt version. Statistical significance threshold: 95% confidence before declaring a winner, with minimum 500 queries per variant to avoid underpowered tests. Winning prompt version promoted to production via a database update, no code deployment required.

Evidently AI monitoring for distribution shift: Population Stability Index (PSI) calculated weekly on the query embedding distribution to detect when production queries are drifting away from the training distribution the prompts were optimised for. PSI above 0.2 triggers a review of whether the existing prompt and retrieval configuration still covers the new query types. This prevents silent quality degradation from distribution shift going unnoticed until user complaints surface.