Hybrid research retrieval
Keyword, vector and semantic ranking find commentary while preserving the passages behind the answer.
New York equity hedge fund · $1bn–$5bn AUM · 25–50 staff
Years of research became an Azure system the firm could query, control and extend.
Internal notes, attachments, analyst email and external research moved through a governed Azure pipeline into one source-aware corpus for analysts and approved AI agents.
What was built
An Azure AI research data lake
Storage, extraction, ETL, metadata, search, identity and audit inside the client's Azure environment.
- Internal and external research records
- 10,000+
- Searchable, source-linked passages
- Tens of thousands
- Source figures retained in the measured extraction test
- 99.9%
- Storage, search, identity and audit under firm control
- Client Azure
Some material lived in a research management system. Other knowledge remained in files, attachments or individual communication records. Finding the answer depended on knowing where to look and who might have saved or written it.
An earlier data-mapping and discovery project established what belonged in a shared archive, what should be fetched live and which architecture performed best on the fund's own documents.
The next engagement was an engineering build: implement the selected design inside the fund's Azure environment and create a controlled service that analysts and agents could query with citations.
The Azure architecture
A governed path from source document to cited answer.
The architecture prepared the corpus once, preserved each transformation and routed questions through the evidence path they required.
Azure AI research data lake
Sources → governed ingestion → cited answers
Research sources
Internal notes
Attachments + PDFs
Analyst email
External research
01
Land
Azure Blob Storage
Raw files, extracted content, clean records and trace history
02
Process
Event-driven ETL
Route by format, deduplicate and preserve the parent document
03
Extract
Document Intelligence
Financial PDFs and tables; native paths for email and HTML
04
Enrich
Research metadata
Company, date, author, type, themes, entities and source
05
Retrieve
Azure AI Search
Hybrid, semantic and filtered search with cited passages
Query service
Azure Container Apps
Routes each question to search, relationships or deterministic analysis.
Access + secrets
Entra ID · Key Vault
Approved identities, managed access and no infrastructure keys in code.
Reviewable result
Answer + citations + audit
Supporting passages remain visible and each query leaves an identity-aware record.
Azure Blob Storage remains the durable record for source files, transformations and query traces.
Source → passage → answerBuild inside the client boundary
The fund retained control of the research asset and its operating environment.
Azure Blob Storage became the durable source of record for raw files, extracted content, cleaned research objects and query traces. Azure Document Intelligence handled complex PDFs. Azure OpenAI created vector representations in the client's Azure environment, and Azure AI Search provided the retrieval layer.
Azure Container Apps hosted the query service and processing workers. Microsoft Entra ID governed access. Managed identities connected Azure services without embedding credentials in code, while Azure Key Vault held the remaining secrets.
The design allowed the fund to extend the source set and connect approved AI clients without moving its research archive into a separate hosted product.
A repeatable ingestion path
Build an ETL pipeline, not a one-time document upload.
Historical and new research needed to pass through the same controlled transformation, with enough durability to change the search logic without paying to extract every document again.
- Step 01
Land and identify
Files enter Azure Blob Storage, receive a content hash for deduplication and are routed according to format.
- Step 02
Extract once
Azure Document Intelligence processes complex PDFs. Email and HTML use native parsing where OCR adds no value.
- Step 03
Clean and enrich
The pipeline removes repeated noise while retaining analytical text, then adds company, date, author, type, themes and source metadata.
- Step 04
Embed and index
Azure OpenAI creates vector representations and Azure AI Search serves hybrid, semantic and filtered retrieval.
Raw files, extraction outputs and cleaned records stay in Blob Storage. Each indexed passage retains its parent document and the source location required to move from a generated answer back to the original evidence.
Financial-document extraction
Readable text was not a sufficient quality bar.
Investment research contains dense tables, shifting units and figures that can remain plausible even after moving into the wrong column. During discovery, several extraction paths were tested on the same financial documents.
Azure Document Intelligence retained 99.9% of source figures in the measured reference set while preserving clean table structure and keeping processing inside Azure. It became the production PDF route.
The raw document and extraction artifact remained available behind the cleaned record. The result was a reviewable chain from cited passage to structured content and original file.
The query layer
Azure AI Search found evidence. The wider system decided how to use it.
Questions about prose, numerical records and company relationships require different retrieval paths. The query service selected the method before composing a response.
Metadata-aware filtering
Company, date, author, provider, source type and recency narrow the evidence before generation begins.
Deterministic analysis
Ratings, price-target ranges and dispersion use calculation over records rather than model arithmetic over prose.
Entity and relationship retrieval
Connected companies can be surfaced with the evidence supporting each relationship, then joined back to the research corpus.
The architecture also stated when the corpus could not support an answer. An unloaded source, incomplete calculation set or weak relationship remained visible as a limitation rather than being turned into confident prose.
Azure-native controls
Identity and audit were part of the query path.
Microsoft Entra ID validated the user and limited the service to the permitted access group. In testing, the sign-in layer correctly refused an account outside that group, showing that access depended on the client's identity policy rather than possession of an endpoint.
Managed identities allowed the service to reach Azure AI Search and Blob Storage without infrastructure credentials in application code. Query-level rate limits and identity-aware logs created a record of who used which research tool and when.
Each query wrote its identity, tool, timing and retrieval activity to an append-only audit record in Blob Storage. Operational health could be monitored separately while the detailed quality record remained available for evaluation.
What the build delivered
A working Azure research foundation, ready to accept additional governed sources.
More than 10,000 internal and external research records were processed into tens of thousands of searchable passages. The service answered questions across the research history with citations back to the underlying material.
The same controlled path can ingest additional research through defined Blob landing zones, deduplicate it against the existing corpus and carry it through extraction, enrichment and indexing.
The build kept the documents, search index, access controls and query audit in the client's Azure environment while exposing the research tools to approved analysts and agents.
Not every external feed considered during discovery was represented as loaded. Licensed research still depended on transfer rights, and some structured data was better suited to a warehouse or live connection.
Evidence boundary: this case study describes the working Azure foundation and cited demonstrations. It makes no firm-wide adoption, time-saving or investment-performance claim.
The service behind this work
Data & AI Foundation
Governed, portable data, retrieval and firm memory that make research usable by people and approved agents.
See how the Data & AI Foundation works →Read next
Data & AI foundation
Research spread across four systems, made queryable
A measured discovery proved a multi-layer query architecture and led to approval for the full Azure build.
Data & AI foundation
An inbox of research turned into an indexed history
More than 500 daily research items became one queryable record across time, topics and tickers, with the original source one click away.
Is your research estate ready for AI agents?
A useful research agent needs a durable source of record, repeatable ingestion, accurate extraction, metadata, identity controls and a way for people to inspect the evidence.