The DSP Evaluation Matrix for Real-Time Prompt Intent Scoring Depth

Real-time prompt intent analysis for programmatic advertising requires a DSP to translate conversational context into commercial intent and use that signal in retrieval and bidding at auction speed. Keyword matching is not enough. The DSP evaluation matrix tests six dimensions of scoring depth, including semantic representation, conversation history, auction design, inferred intent, latency architecture, and disclosure, to distinguish relevant placement from wasted spend.

Why keyword-level intent scoring fails inside a conversational AI interface

Legacy ad retrieval extracts keywords from a query, then fetches ads against those keywords. Tencent’s Real-time Ad REtrieval (RARE) paper documents how this process amplifies the gap between what a user said and the manually chosen keywords an advertiser bid on, producing systematic missed-retrieval failures. The mismatch begins before an auction can rank candidates. An ad that retrieval misses cannot compete on relevance, regardless of how sophisticated the later scoring stage appears.

Conversational input makes that mismatch more pronounced because one exchange can contain a budget, constraints, a shortlist, and prior rejections. Those details qualify the user’s intention. A keyword can name a product without expressing why the user has ruled it out. The arXiv ad-insertion framework identifies fleeting, context-dependent intent as something static keywords fail to capture. A prompt is therefore not simply another query field, but a window into reasoning already underway. Because the signal has changed, platform evaluation must change with it.

What genuine prompt intent scoring requires — the three properties any platform must satisfy

The ad-insertion framework sets three simultaneous requirements for effective scoring, which are contextual coherence, computational efficiency, and adherence to ethical and transparency standards. Contextual coherence means matching an ad to the user’s transient intention. Efficiency means resolving that match without adding user-facing latency. Privacy preservation and explicit ad disclosure belong in the same design. None can compensate for the absence of another.

Deeper semantic analysis usually takes more processing work, while full real-time personalization puts privacy at greater risk. Explicit disclosure can also reduce engagement, creating a tension that engagement figures alone cannot resolve. A platform’s architecture reveals its actual compromise. For buyers, the evaluative question is how the system preserves context while constraining computation and protecting the user, rather than whether it can demonstrate one impressive capability in isolation. A detailed semantic score is insufficient if its computation delays the exchange or its inputs exceed acceptable privacy boundaries.

Dimension one — how the platform represents intent, from keyword tokens to commercial intention

The representation between prompt and ad determines what the scoring engine can express. The spectrum includes numeric document IDs, content-extracted DocIDs, and LLM-generated semantic tokens. These are not interchangeable ways to label the same understanding. RARE describes numeric DocIDs as producing one-to-few mappings between identifiers and candidates, limiting scalable generation while leaving commercial intent unmined. Extracting content gives the system a different representation, but does not by itself establish that it understands the user’s commercial purpose.

RARE uses LLM-generated Commercial Intentions (CIs) as an intermediate representation. A knowledge-injected LLM generates CIs for the ad corpus offline, while a separate online LLM generates them from incoming queries. Retrieval then matches those intentions against a pre-built CI-to-ads index through a one-to-many relationship. The buyer’s diagnostic question follows directly from that design. Does the platform generate a semantic representation of commercial intent, or match prompt text against predefined topics and keywords? The distinction determines whether retrieval can reach an intention the user has not explicitly named.

Tencent and Renmin University of China’s LEADRE system illustrates a different point on this spectrum. Its Intent-Aware Prompt Engineering module constructs intent-aware prompt-response pairs to generate ads tailored to personal interests. Here, an explicit search query is not available. Behavioral signals must help fill in that missing context. RARE and LEADRE are therefore useful architectural cases.

Dimension two — conversation depth as a scoring variable, multi-turn context versus single-prompt snapshots

A platform that scores only the latest prompt treats a conversation as unrelated queries. That discards the constraints and preferences established earlier, even when the latest question depends on them. Consider a user who spends several turns refining a product requirement before asking a closing question. The words in that last question may say little on their own. Scoring it in isolation misses the accumulated funnel-stage signal and can retrieve an ad based only on what the user asks at the end, overlooking the developed requirement.

Microsoft Copilot’s contextual conversation ads consider the entire conversational session. This makes cumulative context part of the placement decision. Conversation depth also creates natural bid stratification because a first exchange and a long refinement thread can express different buying mindsets. Turn count alone is not the substantive distinction. The relevant capability is reading how intent develops across those turns.

For evaluation, the question is whether the auction-time score incorporates full session history, a rolling window, or only the triggering prompt. Advertiser control over the number of turns read is a separate capability to establish. The ad-insertion framework uses genre-based bidding to put this context from across turns to work. Stable semantic clusters let context from the conversation shape placement without per-turn re-auctioning, which reduces latency while keeping placement sensitive to context.

Dimension three — are bids matched to conversational intent or individual queries?

A platform can understand a conversation during retrieval yet still allocate spend through a keyword-shaped bidding mechanism. Depth must survive the handoff to pricing. Otherwise, the richer interpretation has little influence on which advertiser wins or which context receives the budget.

The ad-insertion framework proposes high-level semantic clusters, called “genres,” as the bidding unit. Advertisers bid on stable categories instead of sensitive real-time responses, which cuts the computing work and privacy risk while keeping contextual targeting. Its VCG mechanism gives approximate guarantees for dominant-strategy incentive compatibility and individual rationality. Those properties concern predictable bidding behavior.

ChatGPT’s live ad system uses natural-language context hints for each ad group in place of a keyword layer. A relevance-weighted second-price auction considers ad and landing-page relevance to the conversation as well as the bid. Buyers need to check whether bid relevance scoring uses the meaning of the conversation and whether advertisers can describe the contexts they want beyond topic keywords. A lower bid against a highly relevant intent cluster can outrank a higher bid against a generic one, making the quality of contextual specification part of spend allocation.

Dimension four — how the platform handles the display-advertising problem, scoring intent when no explicit query exists

An explicit prompt gives a scoring system a useful starting point, but not every AI placement includes a purchase-intent statement. LEADRE formalizes the display version of this problem as a low-intent environment in which explicit queries are absent. The prompt must draw on user data within the advertising system and supplement it with external behavioral signals, such as content domain behaviors. Inferring intent from these signals tests a different capability from interpreting a clearly expressed shopping request.

The distinction determines which surfaces a platform can address. A system limited to explicit prompts covers query-driven AI placements, while one that combines in-session signals with behavioral context can address a broader range of AI-native placements. That broader inference also returns the evaluation to privacy. Having more context does not always make that context better.

For a buyer, the practical question is what the engine uses when the current session contains no explicit purchase intention. The evaluation also needs to establish whether advertisers can configure or suppress those signals for brand-safety or privacy reasons. This evaluation reveals how much the scoring layer understands context and how much it depends on the user to state an intention in a directly usable form.

Dimension five — latency architecture and the real-time constraint on scoring depth

Deep scoring that cannot resolve at auction speed is not a viable real-time scoring layer. RARE’s design, documented in the EMNLP 2025 proceedings, separates commercial knowledge preparation from live retrieval. The online LLM generates CIs for an incoming query, and matching runs against the precomputed index. Offline work makes online depth more feasible.

The buyer’s central request is therefore concrete. Can the platform show an architecture diagram that separates offline precomputation from online scoring? The diagram should make its latency claims intelligible, rather than leave “real time” as an unexplained label. Related questions concern the end-to-end scoring budget, whether the engine is co-located with the LLM inference pipeline or operates through a separate call, and how scoring depth degrades under load.

The ad-insertion framework’s synthetic experiment clears its VCG auction in approximately 1.25 seconds on a consumer-grade laptop, providing a reference for feasibility under synthetic conditions. Dedicated infrastructure would differ, and auction computation is not a substitute for explaining the full scoring path. The matrix therefore evaluates the allocation of work.

Dimension six — transparency and disclosure as a scoring-quality signal, not a compliance footnote

Disclosure design reveals how advertising is integrated with the conversational response. It also exposes a limit of apparent relevance. The University of Michigan study on personalized LLM ads, a between-subjects experiment with 179 participants, found that participants had trouble spotting embedded ads and even preferred responses with hidden advertisements. Well-integrated ads can be harder to recognize without explicit labeling, so user preference alone cannot validate insertion quality.

The same study found that participants who wanted to change their advertising experience used natural-language queries to ask the chatbot, leaving disclosure links unused. That behavior casts doubt on whether web-advertising controls work the same way in a conversation. For buyers, the questions concern whether labeling is readable in the exchange and whether understanding an ad requires a separate click.

ChatGPT keeps ad matching separate from response generation and reads the conversation content at the same time as it generates the response. Advertisers cannot pay to influence what the LLM actually says, a design that makes clear labeling structurally easier to enforce. The evaluation should therefore check whether ads are inserted separately or embedded during generation. Neither apparent fluency nor contextual relevance answers that architectural question. Undisclosed or poorly disclosed placements expose both the advertiser and the platform to regulatory risk, making transparency a buyer-quality criterion.

Applying the matrix — what a structured DSP evaluation looks like in practice

A structured assessment starts with semantic representation, then looks at multi-turn depth and the auction mechanism. That order establishes whether the system can express intent, retain its development, and use it to allocate spend. Weak representation undermines the downstream dimensions. Display inference, latency architecture, and disclosure then test the scope and operating conditions of that capability.

The assessment produces a trade-off profile. Generalist DSPs with broad reach but no access to conversational context cannot demonstrate the conversational capabilities in dimensions one through four. Single-surface networks may have deep context on one AI surface but lack cross-surface reach. Neither strength answers the other limitation. A DSP that combines conversational-context scoring with direct supply relationships across multiple AI publisher surfaces is in a different category, if it can show how its approach works across all six dimensions.

Attribution remains a separate constraint on the buying decision. OpenAI launched its conversion pixel and Conversions API in May 2026, but attribution in assistant-mediated discovery remains partially unsolved. Scoring depth does not establish measurement completeness. Buyers should therefore include explicit questions about what a platform can and cannot attribute, as well as the conversations it can read. The strongest evaluation identifies where context changes retrieval, ranking, and spend, while making the limits of reach, speed, privacy, and measurement visible.