ML Metrics & AWS AI Services — AWS AIF-C01

Overfitting occurs when a model learns the noise and specific patterns of its training data so thoroughly that it loses the ability to generalize to unseen examples. It essentially memorizes rather than learns, producing excellent training performance but poor results on validation or production data. Practitioners combat it with regularization, dropout, early stopping, cross-validation, and more training data. The key exam distinction is from underfitting: an overfit model shows high training accuracy but low test accuracy (too complex), while an underfit model performs poorly on both (too simple). Adding model capacity typically worsens overfitting.

Underfitting occurs when a model lacks the capacity or training to learn the underlying patterns in data, resulting in high error on both training and test sets. Unlike overfitting, where the model memorizes noise, an underfit model fails even on the data it was trained on — it is too rigid to generalize because it never learned enough to begin with. On the AIF-C01 exam, the key distinction is that underfitting signals a model complexity problem or insufficient training, not a data-leakage problem. In Amazon SageMaker, common remedies include choosing a more expressive algorithm, training for more epochs, or engineering richer features — rather than applying regularization, which addresses overfitting.

Computer vision is an AI discipline that trains models to extract meaning from images and video, covering tasks such as object detection, image classification, facial analysis, and text extraction (OCR). On AWS, Amazon Rekognition delivers these capabilities as a fully managed API requiring no model training — a key distinction from building a custom vision model in Amazon SageMaker. The AIF-C01 exam often tests matching a use case to the right service: Rekognition for pre-built tasks (content moderation, celebrity recognition, label detection), and SageMaker when a custom model trained on proprietary image data is required. Confusing managed inference with custom training is a common error.

Natural Language Processing (NLP) is a branch of AI that lets machines read, interpret, and generate human language, combining linguistics with statistical and neural techniques. On AIF-C01, NLP underpins several AWS services: Amazon Comprehend performs managed tasks like sentiment analysis, entity recognition, key-phrase extraction, and topic modeling without custom model training, while Amazon Bedrock foundation models handle open-ended text generation and summarization. A common exam trap is treating NLP and generative AI as interchangeable. NLP broadly covers understanding and analyzing language, whereas generative AI specifically produces new text. Comprehend is NLP; a Bedrock large language model doing free-form writing is generative AI built on NLP foundations.

Accuracy measures the proportion of all predictions — both positive and negative — that a model gets right, calculated as correct predictions divided by total predictions. Though intuitive, it only tells the full story when the classes are roughly balanced. The critical exam distinction is between accuracy and more robust metrics like F1-score or AUC-ROC. SageMaker reports accuracy alongside precision, recall, and F1 in its model-quality metrics. For fraud detection or rare-disease diagnosis, where one class dominates, high accuracy can mask a model that simply ignores the minority class — making F1 or AUC the safer choice.

The F1 score is the harmonic mean of precision and recall, combining both into one value that punishes imbalance between them. Because it uses the harmonic mean, a model with perfect precision but zero recall scores 0, not 0.5 — forcing real balance between false positives and false negatives. On the AIF-C01 exam, the key contrast is with accuracy: accuracy looks deceptively high when one class dominates (95% on a fraud set that is 5% fraud), while F1 exposes weak minority-class detection. SageMaker reports F1 among its built-in classification metrics, making it the go-to signal for imbalanced datasets.

AUC (Area Under the ROC Curve) measures a binary classifier’s ability to rank positive examples above negative ones across all decision thresholds, giving a single score from 0 to 1. Because it is threshold-independent, it reflects overall discrimination rather than performance at one cutoff, making it more informative than accuracy when classes are imbalanced. The key exam distinction: a model can score high accuracy by always predicting the majority class yet still have low AUC. Amazon SageMaker’s XGBoost and linear learner report AUC as an objective metric, and Automatic Model Tuning can optimize for it.

Amazon SageMaker is a fully managed platform covering every stage of the machine learning lifecycle: data labeling with Ground Truth, training on managed compute, hyperparameter tuning, bias detection and explainability with SageMaker Clarify, and real-time or batch inference hosting. It provisions infrastructure so teams focus on models, not clusters. The key AIF-C01 distinction is between SageMaker and AWS’s purpose-built AI services: SageMaker builds and trains custom models when no pre-built service fits, while Rekognition, Comprehend, and Transcribe deliver ready-to-call capabilities with no training. Proprietary data needing a tailored model points to SageMaker; adding image recognition or sentiment analysis points to the purpose-built services.

Amazon Comprehend is a fully managed NLP service that analyzes unstructured text to detect sentiment, extract named entities (people, places, organizations), identify key phrases, determine the dominant language, and classify documents into custom categories — all without ML expertise. It is API-driven, so adding language understanding to pipelines processing reviews or support tickets requires no model training. The key exam distinction is Comprehend versus Comprehend Medical: the base service handles general-purpose text, while Comprehend Medical extracts clinical entities — conditions, medications, dosages, and protected health information — from medical notes. Match the service to the text domain.

Amazon Rekognition is a fully managed computer vision service that analyzes images and videos using deep learning models pre-trained by AWS. It handles object and scene detection, facial comparison and search, celebrity recognition, text extraction from images, content moderation, and activity detection in video—no model training or ML expertise required. For the AIF-C01 exam, the key distinction is that Rekognition is a purpose-built vision API, not a model-building platform like Amazon SageMaker, which trains custom models. Pick Rekognition whenever a scenario describes extracting visual insights from images or video without custom training.