Deep Learning in NLP§

Go to [[Week 2 - Introduction]] or back to the [[Main AI Page]] Part of the page on [[Artificial Intelligence/Introduction to AI/Week 2 - Introduction/Natural Language Processing]] For more details see [[Deep Learning]]

Deep learning refers to neural networks with many layers (the deep part) that takes as features as input and extracts higher-level features from this data. 

Much like in CNNs (Convolutional Neural Networks.)

Deep learning networks are able to learn a hierarchy of representations and different levels of abstractions of their input. Deep learning networks can use supervised learning or unsupervised learning and can be formed as hybrids of other approaches (such as incorporating a recurrent neural network with a deep learning network).

Long Short Term Memory cells used as neurons in NLP are more complicated than traditional perceptron neurons, as they have multiple gates and the ability to forget their current informational state.

An interesting application is where a CNN is used to analyse an image and then an LSTM is used to generate a textual representation of the image.

Modern Approaches in NLP§

Go to [[Week 2 - Introduction]] or back to the [[Main AI Page]] Part of the page on [[Artificial Intelligence/Introduction to AI/Week 2 - Introduction/Natural Language Processing]]

Modern approaches to NLP primarily focus on neural network architectures. As neural network architectures rely on numerical processing, an encoding is required to process words.

Two common methods are:

• one-hot encodings
  • Essentially a binary encoding of the word’s position for efficient training by neural networks
• word vectors
  • A numeric encoding of a range of dimensions of the word to which operations can be performed

Another encoding is word vectors, which represent words as highly dimensional vectors where units of the vector are real values. But rather than assign each unit a word (as in one-hot), each unit represents categories for the word (such as singular vs. plural or noun vs. verb) and can be 100-1,000 units wide (dimensionality).

What makes this encoding interesting is that words are now numerically related, and the encoding opens up applying mathematical operations to word vectors (such as adding, subtracting, or negating).

Recurrent Neural Networks in NLP§

Go to [[Week 2 - Introduction]] or back to the [[Main AI Page]] Part of the page on [[Artificial Intelligence/Introduction to AI/Week 2 - Introduction/Natural Language Processing]] See [[RNNS - Recurrent neural networks]] for more detail.

As words move through the neurons, the next batch of inputs enter the network in such a way that late-stage words form part of the processing of early-stage words.

This ensures context is taken into account when processing language.

Traditional RNNs are trained through back-propagation through time (BPTT).

Reinforcement Learning in NLP§

Go to [[Week 2 - Introduction]] or back to the [[Main AI Page]] Part of the page on [[Artificial Intelligence/Introduction to AI/Week 2 - Introduction/Natural Language Processing]] For more details see [[Reinforcement - supervised learning]]

The ‘change vs do nothing’ and ‘new information vs old information’ calculations come into play heavily here. Mainly in terms of determining if a translation requires further information before the current output should be expressed as the answer.

This is especially important in translating languages like Japanese where the verb comes last, but is required in all langauges as a full sentence is required before its context is fully known.