Data Science and Machine Learning 101: Deep Learning and NLP Neural Networks Fundamentals Perceptron Activation Functions Backpropagation Loss Functions

What This Is

A neural network is a stack of perceptrons (tiny linear classifiers) linked by activation functions that let the model learn non‑linear patterns. During training the network adjusts its weights with back‑propagation, minimizing a loss function (e.g., cross‑entropy). In a data‑science workflow the NN replaces a hand‑crafted feature pipeline when the relationship between inputs and the target is too complex for linear models—think predicting churn from dozens of behavioural logs, classifying skin‑lesion images for early melanoma detection, or generating personalized product recommendations from click‑stream data.

Key Terms & Formulas

• Perceptron (single neuron) – computes z = w·x + b and outputs a = φ(z). w = weight vector, x = input vector, b = bias, φ = activation.
• Sigmoid activation – σ(z) = 1 / (1 + e^{-z}); maps any real‑valued input to (0, 1), useful for binary‑output layers.
• ReLU activation – ReLU(z) = max(0, z); introduces sparsity and mitigates vanishing gradients in deep nets.
• Softmax – softmax_i(z) = e^{z_i} / Σ_j e^{z_j}; converts logits to a probability distribution over K classes.
• Cross‑entropy loss (binary) – L = -[y·log(p) + (1‑y)·log(1‑p)], where y∈{0,1} and p = σ(z).
• Cross‑entropy loss (multiclass) – L = - Σ_{k=1}^K y_k·log(softmax_k(z)), with one‑hot y.
• Gradient of loss w.r.t. weights – ∂L/∂w = (a - y)·x for a single perceptron with sigmoid; generalizes to matrix form ∇_W L = δ·A^{T} where δ is the error signal.
• Back‑propagation – recursively compute δ^{(l)} = (W^{(l+1)})^{T} δ^{(l+1)} ⊙ φ'(z^{(l)}) (⊙ = element‑wise product) from output layer back to input layer.
• Learning rate (α) – scalar step size in weight update: W := W - α·∇_W L.
• Weight initialization (He / Xavier) – set W ∼ N(0, 2/fan_in) for ReLU (He) or N(0, 1/fan_in) for tanh/sigmoid to keep activations in a healthy range.
• Regularization (L2) – add λ·||W||_2^2 to the loss; λ controls penalty strength, helping prevent over‑fitting.

Step‑by‑Step / Process Flow

1. Load & preprocess – df = pd.read_csv(...); handle missing values, encode categoricals, and scale numeric features (StandardScaler or MinMaxScaler).
2. Train‑test split – X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, stratify=y).
3. Build the model – using TensorFlow/Keras:

python
model = tf.keras.Sequential([
tf.keras.layers.Dense(128, activation='relu', input_shape=(n_features,)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid') # binary churn
])
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
loss='binary_crossentropy',
metrics=['AUC'])

1. Fit with early stopping –

python
es = tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)
history = model.fit(X_train, y_train,
validation_data=(X_val, y_val),
epochs=100, batch_size=256,
callbacks=[es])

1. Evaluate & iterate – plot loss/metric curves, compute confusion matrix, adjust hyper‑parameters (layers, units, λ, α) or add dropout (tf.keras.layers.Dropout(0.3)) and repeat until validation performance stabilises.

Common Mistakes

Data Science Interview / Practical Insights

1. “Explain the difference between a perceptron and a neuron in a deep network.” – Expect you to discuss linear vs. non‑linear activation, single‑layer vs. multi‑layer composition, and the role of back‑propagation.
2. “When would you prefer ReLU over tanh?” – Look for points on computational efficiency, non‑saturation, and the need for zero‑centered activations (tanh) in shallow nets.
3. “How does the choice of loss function affect gradient flow?” – Demonstrate knowledge that cross‑entropy yields gradients proportional to (p‑y), avoiding the flat regions of MSE for classification.
4. “What are the signs of a vanishing gradient and how do you fix it?” – Mention tiny weight updates, training loss plateauing, and remedies like ReLU, proper initialization, or batch normalization.

Quick Check Questions

1. Scenario: Your churn model’s validation loss stops decreasing after epoch 3, but training loss keeps falling.
   Answer: Apply early stopping and increase regularization (e.g., add dropout or L2).
   Why? The gap signals over‑fitting; early stopping prevents further memorisation.

2. Scenario: You need a multi‑class classifier for 10 disease categories and want calibrated probabilities.
   Answer: Use a softmax output layer with categorical cross‑entropy loss.
   Why? Softmax guarantees a proper probability simplex; cross‑entropy directly optimises likelihood.

3. Scenario: After adding a new hidden layer, training becomes extremely slow and loss stays near 0.5.
   Answer: Switch the hidden activation to ReLU and re‑initialize weights with He initialization.
   Why? Sigmoid saturates; ReLU + He keep gradients alive.

Last‑Minute Cram Sheet (10 one‑liners)

1. Perceptron output: a = φ(w·x + b).
2. Sigmoid derivative: σ'(z) = σ(z)·(1‑σ(z)).
3. ReLU derivative: ReLU'(z) = 1 if z>0 else 0.
4. Back‑prop rule: δ^{(l)} = (W^{(l+1)})ᵀ δ^{(l+1)} ⊙ φ'(z^{(l)}).
5. Weight update: W ← W – α·∇_W L.
6. Cross‑entropy (binary): L = -[y·log(p) + (1‑y)·log(1‑p)].
7. Softmax for K classes: p_k = e^{z_k} / Σ_j e^{z_j}.
8. He init variance: Var(W) = 2 / fan_in.
9. Dropout: randomly zeroes a fraction p of activations during training; scale by 1/(1‑p) at inference.
10. ⚠️ ⚠️ Never use a sigmoid activation in hidden layers of deep nets unless you have a very specific reason; it kills gradient flow.