Deep Learning Glossary 2 | Practice Test / Quiz / MCQs

Home > Deep Learning 101 > Quizzes > Deep Learning Glossary 2

Deep Learning Glossary 2

Fast practice, instant feedback. Timer auto-submits when time’s up.

Avg score: 0% • Most missed: “ICML”

Study First Challenge a Friend Stats

Deep Learning Glossary 2

Time left 00:00

25 Questions

1. Adam

An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.An open source C++/Python software library for numerical computation using data flow graphs - particularly Deep Neural Networks. It was created by Google. In terms of design - it is most similar to Theano - and lower-level than Caffe or Keras.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.

2. Highway Layer

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An open source C++/Python software library for numerical computation using data flow graphs - particularly Deep Neural Networks. It was created by Google. In terms of design - it is most similar to Theano - and lower-level than Caffe or Keras.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.

3. Channel

An open source C++/Python software library for numerical computation using data flow graphs - particularly Deep Neural Networks. It was created by Google. In terms of design - it is most similar to Theano - and lower-level than Caffe or Keras.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.

4. TensorFlow

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.An open source C++/Python software library for numerical computation using data flow graphs - particularly Deep Neural Networks. It was created by Google. In terms of design - it is most similar to Theano - and lower-level than Caffe or Keras.

5. Recurrent Neural Network (RNN)

A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Models sequential interactions through a hidden state - or memory. It can take up to N inputs and produce up to N outputs. For example - an input sequence may be a sentence with the outputs being the part-of-speech tag for each word (N-to-N). An input could be a sentence - and the output a sentiment classification of the sentence (N-to-1). An input could be a single image - and the output could be a sequence of words corresponding to the description of an image (1-to-N). At each time step - an RNN calculates a new hidden state (“memory”) based on the current input and the previous hidden state. The “recurrent” stems from the facts that at each step the same parameters are used and the network performs the same calculations based on different inputs.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.

6. Adagrad

An adaptive learning rate algorithms that keeps track of the squared gradients over time and automatically adapts the learning rate per-parameter. It can be used instead of vanilla SGD and is particularly helpful for sparse data - where it assigns a higher learning rate to infrequently updated parameters.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.

7. Adadelta

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.A gradient descent based learning algorithm that adapts the learning rate per parameter over time. It was proposed as an improvement over Adagrad - which is more sensitive to hyperparameters and may decrease the learning rate too aggressively. Adadelta It is similar to rmsprop and can be used instead of vanilla SGD.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.

8. Neural Turing Machine (NTM)

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.Neural Network architectures that can infer simple algorithms from examples. For example - a NTM may learn a sorting algorithm through example inputs and outputs. NTMs typically learn some form of memory and attention mechanism to deal with state during program execution.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.

9. Deep Belief Network (DBN)

An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.A type of probabilistic graphical model that learn a hierarchical representation of the data in an unsupervised manner. DBNs consist of multiple hidden layers with connections between neurons in each successive pair of layers. DBNs are built by stacking multiple RBNs on top of each other and training them one by one.

10. Softmax

An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Typically used to convert a vector of raw scores into class probabilities at the output layer of a Neural Network used for classification. It normalizes the scores by exponentiating and dividing by a normalization constant. If we are dealing with a large number of classes - a large vocabulary in Machine Translation for example - the normalization constant is expensive to compute. There exist various alternatives to make the computation more efficient - including Hierarchical Softmax or using a sampling-based loss such as NCE.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.

11. Activation Function

An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.To allow Neural Networks to learn complex decision boundaries - we apply a nonlinear activation function to some of its layers. Commonly used functions include sigmoid - tanh - ReLU (Rectified Linear Unit) and variants of these.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.

12. Seq2Seq

A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.This model eads a sequence (such as a sentence) as an input and produces another sequence as an output. It differs from a standard RNN in that the input sequence is completely read before the network starts producing any output. Typically - seq2seq models are implemented using two RNNs - functioning as encoders and decoders. Neural Machine Translation is a typical example of a seq2seq model.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.

13. Neural Machine Translation (NMT)

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.Uses Neural Networks to translate between languages - such as English and French. NMT systems can be trained end-to-end using bilingual corpora - which differs from traditional Machine Translation systems that require hand-crafted features and engineering. NMT systems are typically implemented using encoder and decoder recurrent neural networks that encode a source sentence and produce a target sentence - respectively.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.

14. VGG

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Convolutional neural network model that secured the first and second place in the 2014 ImageNet localization and classification tracks - respectively. The VGG model consist of 16-19 weight layers and uses small convolutional filters of size 3×3 and 1×1. Very Deep Convolutional Networks for Large-Scale Image Recognition

15. Noise-contrastive estimation (NCE)

A sampling loss typically used to train classifiers with a large output vocabulary. Calculating the softmax over a large number of possible classes is prohibitively expensive. Using NCE - we can reduce the problem to binary classification problem by training the classifier to discriminate between samples from the “real” distribution and an artificially generated noise distribution.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.

16. Categorical Cross-Entropy Loss

Also known as the negative log likelihood. It is a popular loss function for categorization problems and measures the similarity between two probability distributions - typically the true labels and the predicted labels. It is given by L = -sum(y * log(y_prediction)) where y is the probability distribution of true labels (typically a one-hot vector) and y_prediction is the probability distribution of the predicted labels - often coming from a softmax.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.

17. Batch Normalization

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A technique that normalizes layer inputs per mini-batch. It speed up training - allows for the usage of higher learner rates - and can act as a regularizer. Batch Normalization has been found to be very effective for Convolutional and Feedforward Neural Networks but hasn’t been successfully applied to Recurrent Neural Networks.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.

18. Backpropagation Through Time (BPTT)

An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.The Backpropagation algorithm applied to Recurrent Neural Networks (RNNs). BPTT can be seen as the standard backpropagation algorithm applied to an RNN - where each time step represents a layer and the parameters are shared across layers. Because an RNN shares the same parameters across all time steps - the errors at one time step must be backpropagated “through time” to all previous time steps - hence the name. When dealing with long sequences (hundreds of inputs) - a truncated version of BPTT is often used to reduce the computational cost. Truncated BPTT stops backpropagating the errors after a fixed number of steps.

19. Average-Pooling

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.A pooling technique used in Convolutional Neural Networks for Image Recognition. It works by sliding a window over patches of features - such as pixels - and taking the average of all values within the window. It compresses the input representation into a lower-dimensional representation.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.

20. ReLU

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Short for Rectified Linear Unit(s). ReLUs are often used as activation functions in Deep Neural Networks. They are defined by f(x) = max(0 - x). The advantages of ReLUs over functions like tanh include that they tend to be sparse (their activation easily be set to 0) - and that they suffer less from the vanishing gradient problem. ReLUs are the most commonly used activation function in Convolutional Neural Networks. There exist several variations of ReLUs - such as Leaky ReLUs - Parametric ReLU (PReLU) or a smoother softplus approximation.

21. Vanishing Gradient Problem

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Arises in very deep Neural Networks - typically Recurrent Neural Networks - that use activation functions whose gradients tend to be small (in the range of 0 from 1). Because these small gradients are multiplied during backpropagation - they tend to “vanish” throughout the layers - preventing the network from learning long-range dependencies. Common ways to counter this problem is to use activation functions like ReLUs that do not suffer from small gradients - or use architectures like LSTMs that explicitly combat vanishing gradients. The opposite of this problem is called the exploding gradient problem. On the difficulty of training recurrent neural networks

22. RMSProp

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A gradient-based optimization algorithm. It is similar to Adagrad - but introduces an additional decay term to counteract Adagrad’s rapid decrease in learning rate.

23. Momentum

Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.An extension to the Gradient Descent Algorithm that accelerates or damps the parameter updates. In practice - including a momentum term in the gradient descent updates leads to better convergence rates in Deep Networks.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.

24. Gradient Clipping

An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.A technique to prevent exploding gradients in very deep networks - typically Recurrent Neural Networks. There exist various ways to perform gradient clipping - but the a common one is to normalize the gradients of a parameter vector when its L2 norm exceeds a certain threshold according to new_gradients = gradients * threshold / l2_norm(gradients).

25. ILSVRC (ImageNet Large Scale Visual Recognition Challenge)

Evaluates algorithms for object detection and image classification at large scale. It is the most popular academic challenge in computer vision. Over the past years - Deep Learning techniques have led to a significant reduction in error rates - from 30% to less than 5% - beating human performance on several classification tasks.A type of Neural Network layer that uses a gating mechanism to control the information flow through a layer. Stacking multiple Highway Layers allows for training of very deep networks. Highway Layers work by learning a gating function that chooses which parts of the inputs to pass through and which parts to pass through a transformation function - such as a standard affine layer for example. The basic formulation of a Highway Layer is T * h(x) + (1 - T) * x - where T is the learned gating function with values between 0 and 1 - h(x) is an arbitrary input transformation and x is the input. Note that all of these must have the same size.An adaptive learning rate algorithm similar to rmsprop - but updates are directly estimated using a running average of the first and second moment of the gradient and also include a bias correction term.Input data to Deep Learning models can have multiple channels. The canonical examples are images - which have red - green and blue color channels. A image can be represented as a 3-dimensional Tensor with the dimensions corresponding to channel - height - and width. Natural Language data can also have multiple channels - in the form of different types of embeddings for example.

⚡ Recently practiced quizzes in this topic

Live quiz activity

Deep Learning Glossary Deep Learning through Transformative Pedagogy Deep Learning Knowledge Test Artificial Neural Networks

25 Questions

❤ If you liked Fatskills, consider supporting us by checking out The Life Manuals You Never Got.

About | Explore | User Guide | Topics | Subjects | Doubt Solver | Career Aptitude Test | Answers | Free Tools | OSHA Basics Quiz | What Should We Know?
Privacy | Terms |

Without work one finishes nothing. - Ralph Waldo Emerson
© 2026 Fatskills.com

All trademarks, logos and brand names are the property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, trademarks and brands does not imply endorsement.

25 Questions

❤ If you liked Fatskills, consider supporting us by checking out The Life Manuals You Never Got.

About | Explore | User Guide | Topics | Subjects | Doubt Solver | Career Aptitude Test | Answers | Free Tools | OSHA Basics Quiz | What Should We Know? Privacy | Terms |

Without work one finishes nothing. - Ralph Waldo Emerson© 2026 Fatskills.com

All trademarks, logos and brand names are the property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, trademarks and brands does not imply endorsement.

About | Explore | User Guide | Topics | Subjects | Doubt Solver | Career Aptitude Test | Answers | Free Tools | OSHA Basics Quiz | What Should We Know?
Privacy | Terms |

Without work one finishes nothing. - Ralph Waldo Emerson
© 2026 Fatskills.com