Prompt Engineering & Evaluation — AWS AIF-C01

Prompt engineering is the practice of crafting input text — instructions, context, constraints, and examples — to steer a foundation model toward consistent, high-quality outputs without modifying model weights. On the AIF-C01 exam it sits at the lowest-cost end of the customization hierarchy: no training data, no compute, no redeployment, making it the first lever to try in Amazon Bedrock before RAG or fine-tuning. Do not confuse it with fine-tuning, which updates model parameters to teach new knowledge or style; prompt engineering only shapes how the model applies pre-trained knowledge at inference. Few-shot examples, chain-of-thought, and output-format directives are testable techniques.

Zero-shot prompting instructs a foundation model to complete a task using only the prompt itself, with no labeled examples provided. The model draws on patterns learned during pre-training to generalize to the new task. In Amazon Bedrock, any inference call without embedded examples is effectively zero-shot, making it the default starting point. The key exam distinction is between zero-shot and few-shot: few-shot prompting inserts one or more input-output examples directly in the prompt to steer the response’s format and style. Zero-shot suits straightforward, well-represented tasks but can underperform on highly specific or format-sensitive outputs where few-shot examples provide critical signal.

Few-shot prompting places a few input-output examples inside the prompt, giving a foundation model a concrete pattern to follow before the real task. Because the examples live in the context window rather than the model weights, no retraining or fine-tuning occurs — the model generalizes from the demonstrations at inference time. Contrast it with zero-shot prompting, which gives no examples, and fine-tuning, which permanently adjusts weights using a labeled dataset. In Amazon Bedrock, few-shot examples improve format consistency and domain vocabulary without a fine-tuning job’s cost or latency. Watch out: too many examples can exceed token limits.

Chain-of-thought (CoT) prompting instructs a foundation model to produce explicit intermediate reasoning steps before its final answer. By surfacing the model’s “work,” CoT improves accuracy on multi-step logic, arithmetic, and causal problems where direct-answer prompts tend to skip steps. On AIF-C01, distinguish zero-shot CoT (adding “think step by step,” no examples needed) from few-shot CoT (including worked reasoning chains). Both are prompt-engineering techniques usable with any model invoked through Amazon Bedrock. CoT is not fine-tuning, RAG, or retrieval augmentation — confusing it with those is a common exam trap.

A system prompt is a high-level instruction block telling a foundation model how to behave — its persona, tone, scope boundaries, and rules to follow throughout the session. Unlike a user message, which changes each turn, it persists across the conversation and is usually invisible to the end user. In Amazon Bedrock you set it at the API call level, and it works alongside Guardrails for Amazon Bedrock, a Bedrock feature enforcing content policies. The exam trap is confusing it with a user prompt: role and constraint definitions belong in the system prompt, not in each user turn.

Prompt injection occurs when malicious content embedded in user input, retrieved documents, or external data causes a large language model to ignore its system instructions and follow attacker-controlled commands instead. In direct injection the attacker speaks to the model directly; in indirect injection the hostile instructions arrive through data the model retrieves, such as a web page or tool output, making it especially dangerous in agentic Amazon Bedrock pipelines that call external APIs. Amazon Bedrock Guardrails can filter inputs and outputs to reduce exposure, but guardrails alone are not a complete defense. For the exam, distinguish prompt injection (overriding instructions via crafted input) from jailbreaking (coaxing the model past its safety training).

Jailbreaking is the practice of crafting adversarial prompts that manipulate a foundation model into producing output it was trained or configured to refuse, such as harmful instructions or policy-violating text. Unlike prompt injection, which hijacks the application’s own system instructions, jailbreaking attacks the model’s underlying safety alignment directly, often through roleplay framing, hypothetical scenarios, or encoded language. Amazon Bedrock Guardrails, a feature of Amazon Bedrock, provides a dedicated defense layer that filters inputs and outputs, enforces denied topics, and applies content policies to resist these attempts rather than relying solely on the model’s built-in alignment.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures how much of a human-written reference summary is captured by a machine-generated output. The most common variants are ROUGE-N, which counts overlapping n-grams, and ROUGE-L, which finds the longest common subsequence. Because ROUGE emphasizes recall, it rewards outputs that cover the key ideas in the reference, making it well-suited for summarization tasks.

The key exam distinction is ROUGE versus BLEU: BLEU is precision-oriented and designed for translation, while ROUGE is recall-oriented and designed for summarization. On AIF-C01, expect questions asking which metric to use when evaluating a summarization model in Amazon Bedrock or Amazon SageMaker — ROUGE is the standard answer for that scenario.

BLEU (Bilingual Evaluation Understudy) scores how well generated text matches one or more human reference texts by counting overlapping n-grams — typically sequences of one to four consecutive words — and applying a brevity penalty to discourage overly short outputs. Scores range from 0 to 1, where 1.0 means an exact match. On AIF-C01 it appears when evaluating foundation model outputs, such as comparing candidate translations against references in Amazon Bedrock workflows. Remember that BLEU emphasizes precision, making it well suited to translation, whereas ROUGE emphasizes recall, making it better for summarization.

BERTScore evaluates generated text by computing cosine similarity between contextual embeddings of candidate and reference tokens, using a pretrained language model such as BERT. Unlike BLEU or ROUGE, which require exact n-gram matches, BERTScore rewards paraphrases and synonyms that carry the same meaning, making it stronger for tasks like summarization and translation where wording naturally varies. On the AIF-C01 exam, the key distinction is that BERTScore is a model-based metric rather than a surface-level string-matching one. When evaluating foundation model outputs, it is more computationally expensive than ROUGE but substantially more sensitive to semantic quality.

Human evaluation has trained raters or domain experts directly assess foundation model outputs for fluency, factual accuracy, relevance, and safety — dimensions automated metrics like BLEU or ROUGE cannot reliably capture. In Amazon Bedrock, its model evaluation jobs support human-based reviews, which matter most during model selection and fine-tuning validation. The key exam distinction is human evaluation versus human-in-the-loop: human evaluation is a periodic quality assessment that informs model readiness, while human-in-the-loop embeds ongoing human review into live production workflows. It complements, rather than replaces, runtime oversight.

An agent is an LLM-based system that autonomously breaks a goal into steps, decides which tools or APIs to call, interprets results, and loops until the task is done. Agents for Amazon Bedrock is the managed implementation: it runs the orchestration loop, session memory, and invocation of action groups (Lambda functions plus OpenAPI schemas) without custom code. The key exam distinction is agent versus knowledge base: a knowledge base retrieves stored documents to ground responses (managed RAG), while an agent takes actions against live systems. The two combine, since a Bedrock agent can query a knowledge base as a tool.