Data Science and Machine Learning 101: Deep Learning and NLP Transformers and Attention Mechanism BERT GPT SelfAttention Tokenization

What This Is

Transformers are deep‑learning architectures that replace recurrent or convolutional layers with self‑attention, letting the model weigh every token (word, sub‑word, or image patch) against every other token in a single pass. This makes them exceptionally good at capturing long‑range dependencies in text, code, or multimodal data. In a data‑science workflow they are the go‑to for any NLP problem—e.g., building a BERT‑based churn‑prediction model that reads free‑form support tickets and flags at‑risk customers, or fine‑tuning GPT‑2 to generate personalized product descriptions for a recommendation engine.

Key Terms & Formulas

• Self‑Attention – Computes a weighted sum of values V using similarity scores between queries Q and keys K:
  [ \text{Attention}(Q,K,V)=\text{softmax}!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V ]
  (d_k) = dimensionality of the key vectors; softmax turns scores into probabilities.

• Multi‑Head Attention – Runs h parallel self‑attention layers (heads) and concatenates their outputs:
  [ \text{MHA}(Q,K,V)=\text{Concat}(\text{head}_1,\dots,\text{head}_h)W^O ]
  Each head learns a different representation subspace; (W^O) projects back to model dimension.

• Positional Encoding – Adds deterministic or learned vectors to token embeddings so the model knows token order:
  [ PE_{(pos,2i)}=\sin!\left(\frac{pos}{10000^{2i/d_{model}}}\right),\; PE_{(pos,2i+1)}=\cos!\left(\frac{pos}{10000^{2i/d_{model}}}\right) ]

• BERT (Bidirectional Encoder Representations from Transformers) – A stack of encoder layers trained with Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) objectives. Use bert-base-uncased for downstream fine‑tuning.

• GPT (Generative Pre‑trained Transformer) – Decoder‑only stack trained with causal (autoregressive) language modeling: predict next token given all previous tokens.

• Tokenization – Splits raw text into atomic units. WordPiece/BPE builds a vocabulary of sub‑words; e.g., “un‑##affordable” → ["un", "##affordable"].

• Masking Ratio (MLM) – Fraction of tokens replaced by [MASK] during pre‑training (commonly 15%).

• Layer Normalization – Normalizes across the feature dimension per token:
  [ \text{LN}(x)=\frac{x-\mu}{\sigma}\gamma+\beta ]
  (\mu,\sigma) are mean & std of the token’s features; (\gamma,\beta) are learnable scale/shift.

• Feed‑Forward Network (FFN) – Position‑wise MLP applied after attention:
  [ \text{FFN}(x)=\text{GELU}(xW_1+b_1)W_2+b_2 ]
  Typically expands dimension 4× then projects back.

• Adam Optimizer – Default for transformer training:
  [ m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t,\; v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,\; \theta_{t+1}=\theta_t-\alpha\frac{m_t}{\sqrt{v_t}+\epsilon} ]
  (g_t) = gradient; (\alpha) = learning rate.

• Fine‑Tuning – Freeze most layers, replace the final classification head, and train on task‑specific data (often with a lower LR, e.g., 2e‑5).

Step‑by‑Step / Process Flow

1. Load & Clean Text – Read raw logs/tickets, drop nulls, and normalize Unicode.
   python
df = pd.read_csv('tickets.csv')
df['text'] = df['text'].astype(str).str.normalize('NFKC')
2. Tokenize & Encode – Use a pre‑trained tokenizer (AutoTokenizer) to convert strings to token IDs, add attention masks.
   python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
enc = tokenizer(df['text'].tolist(),
padding='max_length', truncation=True,
max_length=128, return_tensors='pt')
3. Split & Create Datasets – Stratify on the target (e.g., churn) and wrap tensors in a torch.utils.data.Dataset.
   python
train_idx, val_idx = train_test_split(
range(len(df)), test_size=0.2, stratify=df['churn'])
train_ds = TensorDataset(enc['input_ids'][train_idx],
enc['attention_mask'][train_idx],
torch.tensor(df['churn'][train_idx]))
4. Load Pre‑trained Model & Add Head – Load BertModel and attach a linear layer for binary classification.
   python
from transformers import AutoModel
bert = AutoModel.from_pretrained('bert-base-uncased')
class ChurnClassifier(nn.Module):
def __init__(self):
super().__init__()
self.bert = bert
self.head = nn.Linear(bert.config.hidden_size, 1)
def forward(self, ids, mask):
out = self.bert(ids, attention_mask=mask).pooler_output
return torch.sigmoid(self.head(out))
5. Train – Use AdamW with a linear LR scheduler, early‑stop on validation loss.
   python
optimizer = AdamW(model.parameters(), lr=2e-5)
scheduler = get_linear_schedule_with_warmup(optimizer,
num_warmup_steps=0,
num_training_steps=total_steps)
6. Evaluate & Interpret – Compute ROC‑AUC, plot attention heatmaps for a few tickets to show why the model flagged churn.

Common Mistakes

• Mistake: Feeding raw text directly into a transformer without tokenization.
  Correction: Always run the text through the matching tokenizer; it handles sub‑word vocab, adds special tokens, and creates attention masks.

• Mistake: Using the maximum sequence length of 512 for every dataset, causing unnecessary memory use.
  Correction: Profile the typical length of your documents and set max_length just above the 95th percentile (e.g., 128 for short tickets).

• Mistake: Fine‑tuning with a high learning rate (≥1e‑3) and expecting convergence.
  Correction: Start with 1e‑5 – 5e‑5; transformers are sensitive to LR because the pre‑trained weights are already near a good optimum.

• Mistake: Ignoring the [CLS] vs. mean‑pooling debate and always using the first token.
  Correction: For classification, the [CLS] token is standard, but for regression or similarity tasks, mean‑pool the last hidden states.

• Mistake: Forgetting to set torch.no_grad() during inference, leading to huge GPU memory consumption.
  Correction: Wrap evaluation loops with model.eval(); with torch.no_grad(): ….

Data Science Interview / Practical Insights

1. “Explain the difference between encoder‑only (BERT) and decoder‑only (GPT) architectures.” – Expect you to discuss bidirectional context vs. causal masking and typical downstream tasks (classification vs. generation).
2. “Why do we add positional encodings if self‑attention is permutation‑invariant?” – Show understanding that attention alone cannot infer order; sinusoidal or learned encodings inject sequence information.
3. “How would you reduce inference latency for a large transformer in production?” – Answers may include distillation (e.g., DistilBERT), quantization, pruning, or using a smaller max‑seq length.
4. “What’s the purpose of the ‘masking ratio’ in MLM, and what happens if you set it to 0.5?” – Demonstrate knowledge that too many masks make the task too hard, hurting pre‑training quality.

Quick Check Questions

1. Scenario: Your BERT fine‑tune on churn data overfits after 2 epochs.
   Answer: Reduce learning rate and add dropout (e.g., model.bert.config.hidden_dropout_prob = 0.3).
   Why: Smaller LR lets the model adjust gently; dropout regularizes the large hidden layers.

2. Scenario: You need to generate a product description given a few bullet points.
   Answer: Use a decoder‑only model (GPT‑2/3) with causal masking and prompt engineering.
   Why: GPT is built for autoregressive generation, while BERT cannot generate text beyond masked tokens.

3. Scenario: Your tokenized sequences contain many [PAD] tokens, and validation loss is high.
   Answer: Ensure the attention mask is passed to the model so padded positions are ignored.
   Why: Without the mask, the model treats pads as real tokens, corrupting gradients.

Last‑Minute Cram Sheet (10 one‑liners)

1. Self‑Attention = softmax(QKᵀ/√dₖ)·V – core of every transformer layer.
2. Multi‑Head = parallel attentions + linear projection – captures diverse relations.
3. Positional Encoding – sinusoidal formula; no learned parameters needed for vanilla BERT.
4. BERT → Encoder only, bidirectional; GPT → Decoder only, causal.
5. MLM masking = 15 % tokens; 80 % → [MASK], 10 % → random, 10 % → unchanged.
6. AdamW = Adam + weight decay (L2) applied to parameters, not moments.
7. Fine‑tune LR ≈ 2e‑5; batch size 16–32 for GPU memory ≤ 12 GB.
8. Attention mask = 1 for real tokens, 0 for pads; always pass it!
9. DistilBERT ≈ 40 % smaller, 60 % faster, 97 % of BERT’s accuracy – go for production.
10. ⚠️ Do NOT truncate sequences before tokenization; truncation must happen after adding special tokens (otherwise you may drop [CLS] or [SEP]).