~/learn/Machine Learning · Production ML & modern concepts
Deep learning essentials
Activations, backprop, vanishing gradients, and regularizers like dropout and batch norm.
why this matters
Even non-DL roles get asked about vanishing gradients, ReLU and dropout to check fundamentals.
A neural network stacks linear layers with non-linear activations; without the non-linearity, any depth collapses into one linear model. Training uses backpropagation (the chain rule) to compute gradients and an optimizer like SGD or Adam to update weights.
Sigmoid and tanh saturate, so gradients shrink layer after layer — the vanishing gradient problem. ReLU, residual connections, careful initialization and normalization layers fix it. Exploding gradients are handled with gradient clipping.
Forward and backward
input → [linear → ReLU] → [linear → ReLU] → linear → softmax
↑ backprop sends gradients right-to-left ↑Regularize with dropout (randomly zeroing activations during training), weight decay, data augmentation and early stopping. Batch norm stabilizes training by normalizing layer inputs.
Interview trap
Dropout and batch norm behave differently at training and inference time — forgetting to switch the model to eval mode is a classic bug.
key takeaways
- Non-linear activations give depth its power.
- ReLU and residual connections fight vanishing gradients.
- Switch to eval mode for inference.
check your understanding
Why does a deep network with sigmoid activations train slowly?