Practical LLM Inference and RAG

This module establishes the mechanics and tradeoffs underlying LLM inference, from token processing and probability distributions to decoding, caching, and serving performance.

• ·Course Introduction

  Welcome and orientation to practical LLM inference and RAG systems. The course covers model inference, embeddings, vector stores, retrieval, prompt design, evaluation, and production tradeoffs involving quality, latency, scalability, and cost.

• ·Tokenization

  Tokenization maps text to the discrete token IDs processed by a language model. Token boundaries, vocabulary choices, and token counts influence context usage, latency, and pricing.

• ·Autoregressive Generation

  Autoregressive generation predicts the next token conditioned on the tokens already in context, then appends the selected token and repeats the process. This sequential dependency is central to understanding generation latency.

• ·Logits and Probabilities

  A model produces a logit for each possible next token. Softmax transforms these relative scores into probabilities that expose the model's uncertainty and provide the basis for token selection.

• ·Decoding Strategies

  Decoding converts next-token probabilities into an output sequence. Different selection rules change determinism, diversity, and the likelihood of low-probability continuations.

• ·KV Caching

  During generation, previously computed attention keys and values can be retained and reused for later tokens. KV caching reduces repeated work but consumes memory that grows with context and generated sequence length.

• ·Batching

  Batching processes multiple sequences together so model computation and hardware resources are shared across requests. Larger or dynamic batches can increase throughput while introducing scheduling and latency considerations.

• ·Latency and Throughput

  Latency measures the time experienced by an individual request, while throughput measures completed work over time. Prompt length, generated length, batching, hardware utilization, and memory constraints influence these metrics in different ways.

• Inference Synthesis

  This capstone assessment integrates the module's foundations into a coherent explanation of how an inference request becomes generated text. It evaluates whether learners can connect model behavior and serving mechanisms to quality, latency, throughput, and memory outcomes.

This module develops a systematic approach to designing prompts for reliable LLM behavior. Learners examine prompt structure, instruction priority, specificity, examples, output constraints, decomposition, retrieved context, and prompt injection before synthesizing these concepts in an end-to-end reasoning task.

• ·Prompt Anatomy

  Prompt anatomy describes how instructions, context, inputs, and output requirements work together to communicate a task to an LLM. Understanding these components provides a foundation for diagnosing unclear or incomplete prompts.

• ·Instruction Hierarchy

  Instruction hierarchy concerns the relative authority of instructions supplied through different parts of a model interaction. It helps explain why some directions should take precedence when messages or embedded content conflict.

• ·Task Specificity

  Task specificity defines how precisely a prompt communicates the desired operation and boundaries. Specific prompts reduce ambiguity by stating relevant constraints and what constitutes an acceptable response.

• ·Delimiters

  Delimiters mark boundaries between a prompt's instructions and the content the model must analyze or transform. Clear boundaries make the intended relationship between directions and data easier for the model to interpret.

• ·Few-Shot Examples

  Few-shot prompting provides examples of inputs paired with desired outputs so the model can infer task behavior from demonstrations. Example quality depends on relevance, consistency, and coverage of important cases.

• ·Output Schemas

  Output schemas define the shape and constraints of the response rather than leaving formatting implicit. Structured requirements improve consistency and make responses easier to validate or use downstream.

• ·Task Decomposition

  Task decomposition separates a demanding operation into smaller reasoning steps that can be stated and checked independently. It can reduce ambiguity by clarifying dependencies and the role of each step.

• ·Grounded Context

  Grounded-context prompting defines how retrieved or supplied evidence should be used when generating an answer. It establishes the relationship between the question, available context, citations or evidence, and uncertainty when the context is insufficient.

• ·Prompt Injection

  Prompt injection occurs when untrusted content attempts to alter the instructions governing the model's behavior. Reliable prompt design treats external content as data, maintains clear authority boundaries, and avoids allowing retrieved text to redefine the task.

• Prompt Design Synthesis

  This synthesis assesses the ability to reason about prompt behavior as a coordinated system rather than as a collection of isolated techniques. Learners must identify competing design issues, justify revisions, and predict how the revised prompt affects reliability and response quality.

This module explains how text becomes searchable numerical representations and how retrieval systems select useful context for language models. Learners reason about embedding spaces, chunking, similarity, filtering, indexing, query formulation, reranking, and retrieval quality.

• ·Embedding Representations

  An embedding model maps text into a vector space where related content can have similar representations. The usefulness of an embedding depends on how well the model captures the distinctions relevant to the retrieval task.

• ·Similarity Measures

  Similarity measures such as cosine similarity, dot product, and Euclidean distance quantify relationships between vectors. Their interpretation depends on vector normalization and the assumptions of the embedding model and index.

• ·Document Chunking

  Chunking divides source content into units that can be embedded and retrieved independently. Chunk size, overlap, and boundary placement influence both whether relevant information is found and whether retrieved passages contain enough context.

• ·Metadata Filtering

  Metadata such as source, date, language, or access scope can be stored alongside each chunk and used as a structured retrieval constraint. Filtering can improve relevance and enforce boundaries before or alongside vector similarity search.

• ·Vector Indexes

  A vector index organizes embeddings so a query can find nearby vectors without comparing against every stored item. Approximate nearest-neighbor methods trade some exactness for lower latency and greater scalability.

• ·Dense Retrieval

  Dense retrieval embeds a query and searches for content whose vectors are close under a selected similarity measure. It can recover conceptually related wording even when the query and source use different terms.

• ·Hybrid Retrieval

  Hybrid retrieval combines term-based matching with semantic vector matching. The combination can preserve exact matches for names, identifiers, and rare terms while also recovering paraphrases and conceptually related content.

• ·Query Formulation

  The form of a retrieval query determines which aspects of the user’s information need are represented in the search operation. Rewriting, expansion, or decomposition can improve coverage but may also introduce drift from the original intent.

• ·Reranking

  A reranker applies a more expressive relevance model to a limited candidate set after fast retrieval. This two-stage design can improve ranking quality while avoiding the cost of applying the expensive model to the entire corpus.

• ·Retrieval Metrics

  Metrics such as recall@k, precision@k, and mean reciprocal rank evaluate different aspects of retrieval quality. Selecting and interpreting a metric requires identifying whether missing relevant content, excess irrelevant content, or ranking position is the primary concern.

• Retrieval System Synthesis

  A retrieval system must balance semantic coverage, exact matching, ranking quality, latency, and context usefulness. This synthesis task requires tracing retrieval behavior from source preparation through result evaluation and identifying which design choice explains observed strengths or failures.

This module examines vector stores as operational systems for storing, indexing, querying, and managing embedding records. Learners reason about data organization, record lifecycles, index configuration, durability, consistency, maintenance, and distributed scaling tradeoffs.

• ·Vector Store Architecture

  A vector store combines an API, record storage, metadata handling, vector indexes, and query execution. Understanding these boundaries helps distinguish storage concerns from retrieval and model concerns.

• ·Collections

  A collection groups records that share dimensions, distance behavior, and indexing expectations. Collection boundaries influence administration, query scope, and schema compatibility.

• ·Namespaces

  Namespaces provide logical partitioning within a vector store, often allowing queries and operations to target a restricted subset of data. They can support tenant, environment, or lifecycle separation without requiring entirely separate stores.

• ·Record Lifecycle

  Record lifecycle management determines how vectors, identifiers, metadata, and source versions are created and changed over time. Correct lifecycle semantics prevent stale, duplicated, or orphaned records from affecting retrieval.

• ·Index Configuration

  Index settings determine how aggressively a system approximates nearest-neighbor search and how much memory or computation it consumes. Configuration choices also affect rebuild requirements and the cost of incorporating new records.

• ·Persistence

  Persistence determines whether vectors, metadata, and index state survive beyond the lifetime of a running process. Durable storage, snapshots, and recovery procedures create different guarantees and operational costs.

• ·Consistency

  A vector store may expose newly written records immediately, after propagation, or only after indexing completes. These guarantees determine whether a query can observe stale vectors, metadata, or deletions.

• ·Compaction

  Repeated inserts, updates, and deletes can leave fragmented storage or obsolete index entries. Compaction reclaims space and restores efficient access, but it can consume resources and compete with serving workloads.

• ·Sharding

  Sharding spreads records and index work across multiple partitions to increase capacity or parallelism. A query may need to search several shards and merge their local nearest neighbors into a global ranking.

• ·Replication

  Replicas maintain additional copies of vector-store data or indexes so reads can continue during failures and workloads can be distributed. Replication requires coordinating updates and defining behavior when copies temporarily disagree.

• Vector Store Synthesis

  This synthesis evaluates whether a vector-store design satisfies retrieval and operational requirements under realistic quality, latency, scale, durability, and cost constraints. Learners must trace how choices in one subsystem affect the behavior of the whole system.

This module connects ingestion, retrieval, context preparation, and grounded generation into an end-to-end RAG pipeline. Learners reason about how information flows through the pipeline, how context is selected and constrained, how answers remain attributable and appropriately uncertain, and how overall system quality is evaluated.

• ·RAG Architecture

  A RAG architecture connects source ingestion, indexing, query processing, retrieval, context preparation, and answer generation. Understanding these boundaries helps distinguish retrieval problems from generation problems.

• ·Ingestion Pipeline

  An ingestion pipeline extracts source content, preserves useful structure and metadata, creates chunks and embeddings, and writes records to a retrieval system. Each transformation can affect what information is later retrievable.

• ·Retrieval-Generation Handoff

  The retrieval-generation handoff defines what evidence, metadata, scores, and ordering are available after search. A clear handoff preserves the information needed for context selection and grounded response behavior.

• ·Context Assembly

  Context assembly determines which retrieved items are included, how they are ordered, and how their boundaries and source information are represented. Good assembly makes relevant evidence easier for the model to use without obscuring distinctions between sources.

• ·Context Budgeting

  Context budgeting allocates a finite token window among instructions, conversation history, retrieved evidence, and the model's output. Increasing context can improve coverage but may increase cost, latency, distraction, and truncation risk.

• ·Grounded Generation

  Grounded generation directs the model to use retrieved evidence as the basis for claims and to distinguish supported information from inference or missing information. Grounding reduces unsupported answers but does not guarantee correctness when the evidence is incomplete or wrong.

• ·Citation Attribution

  Citation attribution connects generated claims to specific passages or source records. Useful attribution requires both citation correctness—the source supports the claim—and citation completeness—the important supported claims are not left unreferenced.

• ·Abstention

  Abstention is a controlled response to insufficient, conflicting, stale, or low-confidence evidence. Appropriate abstention prevents the system from presenting unsupported conclusions as established facts.

• ·Index Freshness

  Index freshness measures how closely searchable records reflect the current source state. Update frequency, processing delay, consistency behavior, and deletion handling determine whether retrieval can return current and valid evidence.

• ·End-to-End Evaluation

  End-to-end evaluation examines whether a system retrieves useful evidence, uses it faithfully, answers the question correctly, and meets latency and cost requirements. Separating these dimensions helps identify which pipeline stage needs improvement.

• RAG Pipeline Synthesis

  This synthesis task integrates pipeline architecture, ingestion, retrieval handoff, context assembly, token budgeting, grounded generation, attribution, abstention, freshness, and evaluation. Learners reason about how a weakness at one stage propagates through the final answer and how system tradeoffs affect reliability, latency, and cost.

This module develops the practices needed to evaluate and operate reliable LLM and RAG systems. Learners distinguish evaluation targets and metric types, diagnose failures, detect regressions, and reason about observability, latency, cost, and alerting in production.

• ·Evaluation Targets

  Evaluation targets define what success means for a system, such as retrieval effectiveness, groundedness, answer quality, safety, latency, or cost. Separating these targets prevents one aggregate score from hiding important failures.

• ·Evaluation Datasets

  An evaluation dataset defines the cases used to measure system behavior and may include queries, relevant sources, reference answers, labels, or expected refusal conditions. Its coverage and quality determine how meaningfully results generalize.

• ·Reference-Based Metrics

  Reference-based metrics assess outputs against a human- or expert-defined target using measures such as exact match, overlap, or task-specific correctness. They are useful when acceptable answers can be specified, but may penalize valid alternatives that differ from the reference.

• ·Reference-Free Metrics

  Reference-free metrics evaluate characteristics such as relevance, groundedness, completeness, or consistency without requiring one predetermined answer. They broaden coverage but require careful interpretation because proxy measures can disagree with human judgments.

• ·LLM-as-Judge

  LLM-as-judge evaluation uses structured criteria and comparison prompts to produce scalable judgments of generated responses. Judge bias, sensitivity to phrasing, position effects, and weak calibration can make its scores unreliable without validation.

• ·Error Attribution

  Error attribution separates failures caused by missing or misranked evidence, context assembly, generation, citation, or operational conditions. This distinction connects an evaluation result to the component whose behavior should be investigated.

• ·Regression Testing

  Regression testing compares current behavior with a prior baseline on stable evaluation cases and monitored targets. It helps reveal improvements in one dimension that cause degradation in another.

• ·Tracing

  Tracing records the causal path of a request, including retrieval queries, selected passages, prompts, model calls, outputs, and timing. It provides the context needed to investigate why a particular response occurred.

• ·Latency Monitoring

  Latency monitoring measures response timing at both the end-to-end and component levels, including retrieval, queueing, time to first token, and generation. Percentile measurements reveal slow-tail behavior that averages can conceal.

• ·Cost Monitoring

  Cost monitoring relates token usage, model selection, retrieval activity, caching, traffic, and hardware utilization to the cost of serving requests. It makes resource tradeoffs visible alongside quality and latency.

• ·Alerting

  Alerting turns monitored signals into notifications when defined thresholds, baselines, or error patterns indicate action is needed. Effective alerts balance sensitivity against noise and focus attention on user-impacting changes.

• Evaluation and Operations Synthesis

  This synthesis connects evaluation design, metric interpretation, judge reliability, error attribution, regression detection, tracing, and operational monitoring. Learners reason from observed quality and production signals to distinguish causes, tradeoffs, and appropriate actions.