Data Science and Machine Learning 101: Deep Learning and NLP Recurrent Neural Networks RNN LSTM GRU for Sequences

What This Is

Recurrent Neural Networks (RNNs) are a family of neural architectures that process sequential data by maintaining a hidden state that “remembers” information from previous time steps. Variants such as LSTM (Long‑Short‑Term Memory) and GRU (Gated Recurrent Unit) add gating mechanisms to mitigate the vanishing‑gradient problem, letting the network learn long‑range dependencies. In a data‑science workflow they turn raw time‑ordered signals (e.g., click‑streams, sensor readings, text) into predictive features, enabling tasks like next‑purchase prediction, speech‑to‑text transcription, or anomaly detection in IoT devices.

Key Terms & Formulas

• RNN hidden update – h_t = φ( W_hh·h_{t‑1} + W_xh·x_t + b )
h_t = hidden state at time t; φ = activation (tanh/ReLU).

• Vanishing / Exploding Gradient – gradients shrink or blow up exponentially with depth; LSTM/GRU gates address this.

• LSTM cell –
  i_t = σ(W_i·[h_{t‑1},x_t] + b_i) # input gate f_t = σ(W_f·[h_{t‑1},x_t] + b_f) # forget gate o_t = σ(W_o·[h_{t‑1},x_t] + b_o) # output gate g_t = tanh(W_c·[h_{t‑1},x_t] + b_c) # candidate c_t = f_t ⊙ c_{t‑1} + i_t ⊙ g_t # cell state h_t = o_t ⊙ tanh(c_t) # hidden output
  σ = sigmoid, ⊙ = element‑wise product.

• GRU cell –
  z_t = σ(W_z·[h_{t‑1},x_t] + b_z) # update gate r_t = σ(W_r·[h_{t‑1},x_t] + b_r) # reset gate h̃_t = tanh(W_h·[r_t ⊙ h_{t‑1}, x_t] + b_h) h_t = (1‑z_t) ⊙ h_{t‑1} + z_t ⊙ h̃_t

• Back‑Propagation Through Time (BPTT) – unroll the RNN for T steps, compute gradients w.r.t. each time slice, then sum:
  ∂L/∂θ = Σ_{t=1}^{T} ∂L/∂h_t · ∂h_t/∂θ.

• Sequence‑to‑Sequence (Seq2Seq) – encoder RNN → context vector → decoder RNN; used for translation, summarization.

• Teacher Forcing – during training, feed the true previous token to the decoder instead of the model’s own prediction to speed convergence.

• Bidirectional RNN – concatenate forward and backward hidden states: h_t = [→h_t ; ←h_t]; captures past and future context (useful for POS tagging).

• Dropout on Recurrent Connections – h_t = dropout(h_t, p) applied after the recurrent update to regularize temporal dynamics.

• Gradient Clipping – g = clip(g, -c, c) (commonly c = 1–5) to prevent exploding gradients.

• Loss for Sequences – typically Cross‑Entropy per time step:
  L = - Σ_{t=1}^{T} Σ_{k=1}^{K} y_{t,k} log(p_{t,k}).

Step‑by‑Step / Process Flow

1. Load & Align Data – read CSV/Parquet, parse timestamps, and pad/truncate sequences to a uniform length max_len (or use torch.nn.utils.rnn.pack_padded_sequence).
2. Exploratory & Pre‑processing – plot autocorrelation, compute lag features, encode categorical tokens (e.g., Tokenizer for text), and scale numeric inputs (StandardScaler on per‑feature basis).
3. Train‑Validation Split – use time‑aware split (e.g., last 20 % of timestamps as validation) to avoid leakage.
4. Build Baseline – a simple ARIMA or Logistic Regression on lagged aggregates; record baseline metrics (accuracy, ROC‑AUC).
5. Define RNN Model – choose LSTM vs GRU vs plain RNN; set hidden_dim, num_layers, bidirectional=True/False, and dropout. Example (PyTorch‑style pseudocode):

```python
class SeqModel(nn.Module):
def init(self, vocab, emb_dim, hidden, n_layers, bidir):
super().init()
self.emb = nn.Embedding(vocab, emb_dim)
self.rnn = nn.LSTM(emb_dim, hidden, n_layers,
batch_first=True,
bidirectional=bidir,
dropout=0.3)
self.fc = nn.Linear(hidden * (2 if bidir else 1), out_dim)

   def forward(self, x, lengths):
       x = self.emb(x)
       packed = pack_padded_sequence(x, lengths, batch_first=True,
                                     enforce_sorted=False)
       packed_out, (h_n, _) = self.rnn(packed)
       out, _ = pad_packed_sequence(packed_out, batch_first=True)
       # use last hidden state (or attention) → classification
       logits = self.fc(h_n[-1])
       return logits

`` 6. Train with BPTT – useAdamoptimizer,nn.CrossEntropyLoss, and gradient clipping (torch.nn.utils.clip_grad_norm_`). Monitor validation loss; early‑stop if it stops improving.
7. Evaluate & Tune – compute sequence‑level metrics (e.g., BLEU for translation, F1 for token classification). Hyper‑parameter search (hidden size, learning rate, batch size) via Optuna or scikit‑learn’s GridSearchCV wrapper.
8. Deploy – export to ONNX/TensorFlow SavedModel, wrap in a REST endpoint, and batch‑process new sequences with the same padding logic.

Common Mistakes

• Mistake: Feeding raw timestamps directly into the RNN.
  Correction: Convert timestamps to relative features (e.g., time‑since‑last‑event, day‑of‑week sin/cos) or embed them; raw large values saturate activations.

• Mistake: Ignoring variable‑length sequences and padding with zeros without masking.
  Correction: Use pack_padded_sequence (PyTorch) or Masking layer (Keras) so the network ignores padded timesteps during loss/gradient computation.

• Mistake: Training a plain RNN on long texts and blaming “bad performance”.
  Correction: Switch to LSTM/GRU or add attention; plain RNNs struggle with long‑range dependencies due to vanishing gradients.

• Mistake: Setting a huge learning rate and never seeing convergence.
  Correction: Start with 1e-3 (Adam) or 1e-2 (SGD) and use a learning‑rate scheduler (e.g., ReduceLROnPlateau).

• Mistake: Evaluating only the last timestep for tasks that need per‑step predictions (e.g., POS tagging).
  Correction: Align loss and metrics with each timestep; use TimeDistributed wrapper or reshape logits to (batch, seq_len, n_classes).

Data Science Interview / Practical Insights

1. “When would you prefer a GRU over an LSTM?” – GRU has fewer gates (no separate cell state), so it’s faster and uses less memory; choose it when training time or model size is a bottleneck and performance is comparable.
2. “Explain why bidirectional RNNs can’t be used for real‑time inference.” – They require future context (the backward pass) which isn’t available at prediction time; they’re only suitable for offline or batch tasks.
3. “What is teacher forcing and why can it cause exposure bias?” – Feeding the true previous token speeds up training but the model never learns to recover from its own mistakes, leading to error accumulation at inference.
4. “How do you handle class imbalance in sequence labeling?” – Use weighted cross‑entropy (class weights ∝ 1/frequency) or focal loss; optionally oversample rare token classes with torch.utils.data.WeightedRandomSampler.

Quick Check Questions

1. Q: Your validation loss keeps rising after a few epochs, but training loss continues to drop. What do you try?
   A: Apply early stopping and increase dropout or use gradient clipping; the model is over‑fitting.

2. Q: You need to predict the next 7 days of electricity demand from hourly readings. Which architecture is most appropriate?
   A: A stacked LSTM (or GRU) with a seq‑to‑seq decoder, because it can model long temporal dependencies and produce multi‑step forecasts.

3. Q: During inference you notice the model outputs the same class for every timestep. What is a likely cause?
   A: The hidden state is being reset incorrectly (e.g., h_0 set to zeros each timestep) or the output layer has a bias toward one class; ensure hidden state is carried across timesteps.

Last‑Minute Cram Sheet (10 one‑liners)

1. RNN hidden state: h_t = φ(W_hh·h_{t‑1} + W_xh·x_t + b).
2. LSTM gates: input i, forget f, output o; cell state c_t = f⊙c_{t‑1} + i⊙g.
3. GRU simplifies: only update z and reset r gates → fewer parameters.
4. BPTT = unroll → backprop → sum gradients across time steps.
5. Gradient clipping (c≈5) prevents exploding gradients.
6. Bidirectional RNN = forward ⊕ backward hidden vectors; not for online prediction.
7. Teacher forcing ratio = p (0‑1); lower p reduces exposure bias.
8. Mask padded timesteps (Masking layer or pack_padded_sequence) to avoid learning from zeros.
9. Cross‑entropy per step: L = - Σ_t Σ_k y_{t,k} log(p_{t,k}).
10. ⚠️ Using a plain RNN on long sequences → vanishing gradients; switch to LSTM/GRU or add attention.