Semantic Search Development

Product search that understands natural language queries and buyer intent, "comfortable running shoes for flat feet" finds the right products through semantic similarity even when the exact words don't appear in any product title. Built on text-embedding-3-small or text-embedding-3-large embeddings indexed in Pinecone, pgvector (PostgreSQL extension), or Weaviate depending on your scale and infrastructure preferences; Pinecone for large catalogues above 10 million vectors where approximate nearest neighbour (ANN) performance is critical, pgvector for existing PostgreSQL stacks where operational simplicity matters more than ANN latency at extreme scale. Attribute-aware retrieval that treats structured product attributes (colour, size, material, price range) as filters applied on top of semantic retrieval rather than as separate faceted search, users get semantically relevant results narrowed to the attribute constraints they specified. Visual similarity search for fashion, furniture, and home goods: CLIP (Contrastive Language-Image Pre-training) model encodes product images and query text into the same embedding space, enabling "find similar to this image" and "show me products that look like this description" without separate image metadata. Personalised ranking using user session history: a lightweight re-ranking pass boosts results from categories the user has interacted with, reducing zero-click searches on subsequent sessions. Merchandising controls: business rules pinning specific products to the top position for priority SKUs, excluding out-of-stock items from the ranking pool, and boosting promoted items by a configurable multiplier applied after semantic retrieval. Zero-result rate target below 2% for product catalogues, the metric that directly predicts lost revenue from search failure.

Search for customer-facing knowledge bases and help centres that surfaces the answer rather than a list of articles, built to deflect support tickets by helping users self-serve instead of opening a ticket when they cannot find what they need. Users asking questions in natural language ("how do I reset my password", "why was my order cancelled", "what is included in the pro plan") find the most relevant article through semantic matching, even when their vocabulary doesn't match the article title or the exact phrasing your support team used. Chunk-level retrieval rather than article-level: long articles broken into overlapping 512-token chunks with semantic search operating at the chunk level so the most relevant passage is returned, not just the most relevant article as a whole. Passage highlighting in the search results preview so users can see the relevant answer before clicking into the full article, reducing friction for simple questions. Integration with Zendesk Guide, Intercom Articles, Freshdesk, Notion, and custom CMS knowledge bases via their content APIs or direct database connections; embedding pipeline runs nightly or on article publish webhook to keep the index current. Answer synthesis layer as an optional add-on: the top 3 retrieved chunks passed to GPT-4o mini to synthesise a direct answer, shown above the article links, the same pattern as Intercom Fin but built on your content and deployed in your infrastructure. Ticket deflection rate measurable from your help desk analytics: tracked as the percentage of users who searched, found a result, and did not subsequently open a ticket within 24 hours.

Unified search across your organisation's internal content: Confluence, Notion, SharePoint, Google Drive, Slack, internal databases, and ticketing systems, one search interface that retrieves from all sources ranked by relevance to the query rather than forcing users to remember which tool holds what. Connector architecture that pulls content from each source via its API on a configurable sync schedule, converts documents to a normalised text representation, chunks and embeds them, and writes the vectors plus source metadata to a central index. Metadata attached to each document at index time: source system, document type, author, last modified date, and access permission group, used for filtering (search only Confluence spaces relevant to my team) and access control enforcement at retrieval. Permission-aware retrieval using permission groups fetched from each source system at index time: a user's search only returns documents from sources and spaces they have access to in the originating system, with permissions re-evaluated on a configurable refresh cycle so permission changes propagate within hours rather than days. Cross-source relevance fusion: results from Confluence, Google Drive, and Slack ranked on the same relevance scale using dense vector similarity, with source-specific tuning to prevent one high-volume source from dominating the results. Slack message search for organisations where decisions and context are trapped in channel history: message chunks indexed with thread context preserved, so a search for a specific project decision surfaces the thread where it was discussed rather than an isolated message. Search latency target under 300ms at the 95th percentile for indices up to 5 million documents, the threshold below which users perceive search as instant rather than slow.

Semantic search for technical documentation, API references, code repositories, and internal engineering wikis, built for the query patterns developers actually use, which are conceptual descriptions of problems rather than exact function or class names. A developer asking "how do I handle rate limiting in the API client" finds the relevant documentation and code examples through semantic similarity, even if the documentation calls it "throttling" and the relevant class is named RequestThrottleMiddleware. Documentation chunking strategy optimised for technical content: function-level chunks for API references (each function or method as a separate retrievable unit with its signature, parameters, and description), section-level chunks for conceptual guides, and paragraph-level chunks for prose documentation, different granularities serve different query types. Code-aware embedding using code-specialised models where code retrieval is the primary use case: CodeBERT or OpenAI's code embedding models produce more accurate similarity scores for code search than general-purpose text embedding models trained on prose. Tree-sitter parsing for code snippet extraction: function and class definitions extracted from source repositories using language-aware AST parsing rather than naive line splitting, so retrieved code snippets are syntactically complete. Integration with documentation platforms (Docusaurus, GitBook, ReadTheDocs, Notion, Confluence) via their APIs or static export pipelines. Search widget as an npm package or web component for embedding into your existing documentation site without rebuilding the site around a new platform. Mean Reciprocal Rank (MRR) at 10 above 0.75 for documentation search, the retrieval quality metric that corresponds to developers finding the right answer in the first page of results.

Production-grade hybrid retrieval combining dense vector semantic search with sparse BM25 keyword retrieval, the combination that outperforms either approach alone for most real-world query distributions, because semantic search handles intent and paraphrase while BM25 handles exact identifiers, product codes, proper nouns, and technical terms that embedding models generalise over when specificity is required. Reciprocal rank fusion (RRF) to merge the ranked lists from dense and sparse retrieval: RRF does not require tuning a weight parameter because it combines ranks rather than scores, making it robust to distribution shifts in query volume and content without manual retuning. Cross-encoder re-ranking as a second-stage pass: the top 20-50 candidates from the initial retrieval passed to a cross-encoder model (Cohere Rerank, BGE reranker, or a fine-tuned model) that scores query-document pairs jointly rather than independently, producing a final ranked list with significantly higher precision than the first-stage retrieval alone, typically 10-20% improvement in Precision@3 on domain-specific corpora. Query expansion for queries that retrieve poorly: HyDE (Hypothetical Document Embeddings) generates a hypothetical ideal answer to the query, embeds it, and uses that embedding for retrieval, effective for abstract or under-specified queries. Query reformulation via an LLM rewrite step for natural language questions that contain search-irrelevant phrasing. Retrieval evaluation framework built at project start: a labelled evaluation set of 100-200 representative query-document pairs used to measure Recall@5, Recall@10, Precision@3, MRR, and NDCG before deployment and after each configuration change, so retrieval improvements are measured rather than assumed.

Search performance monitoring built into the system from day one, because production search behaviour rarely matches what you tested during development, and retrieval quality degrades as your content changes and query patterns evolve. Query logging pipeline capturing every search with its timestamp, user session ID (anonymised), the query string, the result set returned (document IDs and positions), and whether the user clicked a result and at which position, the raw data for all downstream analytics. Zero-result rate tracked per query volume bucket and surfaced in a daily dashboard: zero-result queries above 3% indicate content gaps or vocabulary mismatch between your users' language and your indexed content, and the specific queries driving zero results tell you exactly what content to add. Click-through rate (CTR) by result position: a result appearing at position 1 should have CTR above 40% for navigational queries; if it doesn't, the top-ranked result is wrong. Position-weighted CTR across the query population identifies systemic retrieval problems, low CTR on high-volume queries is the highest-priority fix. Failed search pattern analysis: queries with high volume and low engagement (below 10% CTR, high reformulation rate, or high back-click rate) cluster into topics where your current retrieval is failing, each cluster becoming a retrieval improvement task. A/B testing framework for retrieval configurations: traffic split at the session level between retrieval variants, with statistical significance testing on CTR and engagement metrics, so a proposed improvement to the re-ranking model is validated on real traffic before full rollout. Alerting for sudden quality degradation: a spike in zero-result rate or a drop in CTR that breaches a configured threshold triggers an alert, typically caused by a content ingestion failure, an embedding index rebuild issue, or a query pattern shift after a product launch.