Data Science and Machine Learning 101: Deep Learning and NLP Computer Vision Tasks Object Detection Segmentation Image Generation

What This Is

Computer vision (CV) tasks turn raw pixels into structured information. Object detection finds where and what objects are in an image, segmentation paints each pixel with a class label (semantic) or a unique instance ID (instance), and image generation creates new pictures from learned distributions (e.g., GANs, diffusion models). In a data‑science workflow these tasks replace manual labeling, enable downstream analytics (e.g., counting defective parts on a production line), and power products such as autonomous‑driving perception stacks or AI‑augmented medical diagnostics.

Key Terms & Formulas

• Bounding‑Box Regression – Predicts box coordinates ((x_{min}, y_{min}, x_{max}, y_{max})) via a loss such as Smooth L1: (\text{L}_{\text{smooth}}(d)=0.5d^{2}) if (|d|<1) else (|d|-0.5).
• Intersection‑over‑Union (IoU) – (\displaystyle \text{IoU}= \frac{\text{area}(B_{p}\cap B_{gt})}{\text{area}(B_{p}\cup B_{gt})}); measures overlap between predicted and ground‑truth boxes.
• Mean Average Precision (mAP) – Average of AP over all classes; AP is the area under the precision‑recall curve for a given IoU threshold.
• Fully Convolutional Network (FCN) – Replaces fully‑connected layers with convolutions so the network can output a spatial map (used for segmentation).
• Dice Coefficient – (\displaystyle \text{Dice}= \frac{2|P\cap G|}{|P|+|G|}); a segmentation similarity metric (especially for medical masks).
• Mask R‑CNN – Extends Faster R‑CNN by adding a small FCN head that predicts a binary mask for each detected instance.
• Generative Adversarial Network (GAN) – Two networks (Generator (G) and Discriminator (D)) play a minimax game: (\displaystyle \min_{G}\max_{D}\; \mathbb{E}{x\sim p}}}[\log D(x)] + \mathbb{E{z\sim p[\log(1-D(G(z)))]). }
• Diffusion Model – Learns to reverse a forward noising process; loss is usually a simple MSE between predicted and true noise: (\displaystyle \mathcal{L}= | \epsilon - \epsilon_{\theta}(x_t, t) |^{2}).
• Non‑Maximum Suppression (NMS) – Post‑process step that removes overlapping boxes: keep the highest‑scoring box, discard any with IoU > threshold (e.g., 0.5).
• Anchor Boxes – Pre‑defined box shapes/sizes tiled over the image; each anchor is classified/regressed during detection.
• Pixel‑wise Cross‑Entropy – Segmentation loss: (\displaystyle \mathcal{L}= -\sum_{c}\,y_{c}\log(\hat{y}_{c})) summed over all pixels and classes.
• Style Transfer (Neural) – Optimizes an image (x) to minimize (\displaystyle \mathcal{L}= \alpha |F_{\text{content}}(x)-F_{\text{content}}(c)|^{2} + \beta |G_{\text{style}}(x)-G_{\text{style}}(s)|^{2}) where (c) is content, (s) is style.

Step‑by‑Step / Process Flow

1. Load & Inspect Images
   python
import cv2, glob, pandas as pd
paths = glob.glob('data/train/*.jpg')
img = cv2.imread(paths[0]) # BGR → RGB if needed
plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
2. Label / Convert Annotations
3. Detection: CSV with xmin, ymin, xmax, ymax, class.
4. Segmentation: PNG masks where pixel value = class ID (or separate instance IDs).
5. Generation: Collect a clean folder of images; no labels needed.
6. Pre‑process & Augment
   python
from torchvision import transforms as T
aug = T.Compose([
T.RandomResizedCrop(512),
T.RandomHorizontalFlip(),
T.ColorJitter(brightness=0.2, contrast=0.2),
T.ToTensor(),
T.Normalize(mean=[0.485,0.456,0.463], std=[0.229,0.224,0.225])
])
7. Choose a Baseline Architecture
8. Detection: Faster R‑CNN (ResNet‑50 backbone) or YOLOv8 for speed.
9. Segmentation: U‑Net (medical) or DeepLabV3+ (general).
10. Generation: StyleGAN2‑ADA (high‑fidelity) or DDPM (diffusion).
11. Train / Validate
    python
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.9)
for epoch in range(num_epochs):
train_one_epoch(model, train_loader, optimizer)
evaluate(model, val_loader) # compute mAP or Dice
12. Post‑process & Deploy
13. Apply NMS, threshold masks, convert logits → class IDs.
14. Export to ONNX/TensorRT for low‑latency inference.
15. Wrap in a Flask/ FastAPI endpoint that receives base64 images and returns JSON with boxes/masks.

Common Mistakes

Data Science Interview / Practical Insights

1. “Explain the trade‑off between Faster R‑CNN and YOLOv8.” – Faster R‑CNN gives higher mAP (two‑stage, region proposals) but is slower; YOLOv8 is single‑stage, optimized for real‑time with slightly lower AP.
2. “When would you prefer Dice over IoU?” – Dice is more sensitive to small objects and is the de‑facto metric in medical segmentation where foreground is tiny.
3. “How does a diffusion model differ from a GAN in training stability?” – Diffusion models use a simple MSE loss and a forward‑noising schedule, avoiding the adversarial min‑max game that often causes mode collapse in GANs.
4. “What is the role of a ‘feature pyramid network’ (FPN) in detection?” – FPN merges high‑resolution (low‑level) and low‑resolution (high‑level) features, enabling detection of objects at multiple scales without extra anchors.

Quick Check Questions

1. Q: Your detector’s mAP is high on the validation set but drops dramatically on a new camera feed. What is the most likely cause?
   A: Domain shift – the new images have different lighting/color distribution; you need data‑augmentation or fine‑tuning on a few samples.

2. Q: A U‑Net model for tumor segmentation shows a Dice of 0.92 on training but 0.55 on test. Which remedy should you try first?
   A: Increase regularization (e.g., add dropout or weight decay) and augment the training set to reduce over‑fitting.

3. Q: You need to generate 256×256 avatars conditioned on a user’s age. Which architecture fits best?
   A: A conditional GAN (cGAN) where the age label is concatenated to the noise vector or injected via AdaIN layers.

Last‑Minute Cram Sheet (10 one‑liners)

1. IoU ≥ 0.5 is the classic threshold for “correct” detection in COCO‑style mAP.
2. Smooth L1 loss = Huber loss; less sensitive to outliers than plain L2.
3. NMS runs in O(N log N) time; use vectorized implementations (torchvision.ops.nms).
4. U‑Net halves spatial size 4× (down‑sample) then restores it with skip connections – great for limited data.
5. Dice = 2·TP / (2·TP + FP + FN) – equivalent to F1 for binary masks.
6. GAN training tip: update the discriminator more times than the generator early on to avoid vanishing gradients.
7. Diffusion models need ≈ 1000 timesteps; DDIM can reduce to ~50 with little quality loss.
8. Anchor‑free detectors (e.g., FCOS) eliminate the need to hand‑craft anchor scales → fewer hyperparameters.
9. Transfer learning rule: freeze backbone for first N epochs, then unfreeze and fine‑tune with a lower LR (≈ 1e‑4).
10. ⚠️ ⚠️ Using BatchNorm in a small‑batch (≤ 4) setting can cause noisy statistics; replace with GroupNorm or SyncBatchNorm.