1980s: Artificial Neural Networks

In This Lesson

Lesson Objectives and Outcomes

• Students will be able to explain the basic structure of artificial neural networks (input layer, hidden layers, output layer).

• Students will be able to describe how neural networks are used to power language applications like chatbots (including concepts like Transformers and self-attention).

Introduction

This lesson covers one of the main technologies that allow chatbots to understand what you're saying and respond in ways that feel almost human. This is the power of artificial neural networks, which are computational systems that mimic the structure of our brains.

We'll uncover the building blocks of neural networks, learn how they adapt based on information, and discover how this amazing technology allows chatbots to have conversations that feel surprisingly natural.

Why Are Artificial Neural Networks (ANN) Important?

Neural networks are one of the key foundational technologies that allow chatbots to process and produce language dynamically, nuanced, and increasingly sophisticatedly. They are crucial for developing chatbots that can truly understand and respond to human users in a seamless and natural manner.

At their core, neural networks are computational models designed to mimic how the human brain operates, allowing for complex patterns in data to be learned and used for prediction and decision-making.

Key facts to remember;

• Neural networks can learn and adapt from vast datasets. Chatbots use these learning capabilities to understand and generate human language. By processing millions of text examples, they learn how to respond in a contextually appropriate and human-like way.

• Through neural networks, chatbots can grasp and understand the intricacies of language, including semantics, syntax, and even cultural nuances. This deep understanding is essential for bots to engage in meaningful dialogues, answer questions, and provide assistance.

• As chatbots interact with users, the underlying neural networks can generate continuous improvement and improve over time. This means that the conversation quality can improve as the system gains more experience.

• Neural networks enable chatbots to be scalable and flexible and handle a wide variety of topics and tasks. They are not limited to predefined scripts but can generate responses on the fly, which makes them highly versatile.

• Using neural networks, chatbots can tailor conversations to incorporate personalization to individual users by recognizing patterns in previous interactions. This can enhance user satisfaction and engagement as the chatbot appears more attuned to the user's preferences and history.

Neural networks are the foundation upon which large language models (LLMs) are built. In future lessons, we'll explore how these language models use neural networks to understand and produce incredibly realistic text.

What Are Artificial Neural Networks (ANNs)

Artificial neural networks (ANNs) are computational systems inspired by the structure and functioning of the human brain. They consist of interconnected artificial neurons, also known as nodes or units, which mimic the behavior of biological neurons. These neurons receive input signals from other neurons via connections or links and transmit output signals, which subsequent neurons can further process in the network.

Boiling it down, artificial neural networks leverage the concepts of artificial neurons, connections, weights, thresholds, and layers to process input signals and generate output signals. They find wide-ranging applications in diverse fields, such as computer vision, speech recognition, and data analysis.

Artificial Neural Networks Explained

Imagine a neural network as a web of interconnected processing units inspired by the human brain. Here's a simplified view:

Deep artificial neural network, schematic structure with layers.
An artificial neural network consists of a series of interconnected nodes (Circles),
modeled after the simplified structure of neurons in the brain.
Each node in this network acts as an artificial neuron,
while the Lines represent the connections that relay
the output from one neuron to the input of another2.

Building Blocks

Neurons and Connections are the building blocks of neural networks. Neurons are the basic units, like tiny calculators, that process information. Neurons connect to each other, sending signals that influence each other's activity. It's in these connections that learning happens.

Learning Process

Neural networks learn through training with data sets. They adjust the strengths of connections between neurons based on the data. Imagine strengthening some connections like well-worn paths and weakening others.

Layers of an ANN

Input Layer: This is the first layer of the network and acts as the entry point for data. The data can be anything from numbers and images to text and sounds. Each node in the input layer represents a single element of the data. For example, in an image recognition task, the input layer might have one node for each pixel in the image.

Hidden Layers: These layers are the heart of the ANN and are responsible for most of the computation. A typical ANN can have several hidden layers, each containing a number of nodes. The nodes in a hidden layer receive signals from the previous layer, process them using an activation function, and then send the results to the next layer. The activation function helps the network learn non-linear relationships between the input data and the output. The more hidden layers and nodes an ANN has, the more complex relationships it can learn.

Output Layer: The final layer of the network produces the final output. The number of nodes in the output layer depends on the specific task the network is designed for. For example, in an image recognition task, the output layer might have one node for each possible class of object (e.g., cat, dog, car). The activation function of the output layer determines how the activation from the hidden layer is mapped to the final output value.

AI Applications are built using Neural Networks

Neural networks are powerful tools used in various AI applications, including image recognition, speech recognition, and powering LLMs used in AI Chatbots.

Remember that an artificial neural network (ANN) is a computational model inspired by the structure and function of the human brain. Just like our brains are made up of interconnected neurons, ANNs consist of artificial neurons called nodes arranged in layers. These layers work together to process information and learn from data.

How ANNs Work

ANNs learn through a process called training. During training, the network is presented with a set of labeled data examples. The network processes the input data, generates an output, and compares it to the correct label. The error is the difference between the predicted output and the actual label.

The network then uses an algorithm to adjust the weights of the connections between the nodes in the network. This process of adjusting weights is called backpropagation. By iteratively feeding the network data and adjusting the weights, the network gradually learns to map the input data to the desired output.

In the example below, the inputs are "The Car Is" and based on the weights of the connections between the nodes, the given output is "Red".

Author Note: I am adding this next section as we will revisit this topic in the "Early 2020 Large Language Models (LLMs)

How Neural Networks Make Large Language Models (LLMs) Work

Large language models (LLMs) are a type of artificial intelligence (AI) that uses deep learning techniques, a subset of machine learning algorithms that rely on ANNs. LLMs are trained on massive amounts of text data, which allows them to identify patterns in language and generate human-quality text in response to a wide range of prompts and questions. The hidden layers of the ANNs used in LLMs play a crucial role in learning these complex patterns in language.

LLMs are built on top of specific neural network architectures, commonly called Transformers. These networks excel at handling sequences, like words in a sentence.

The power of transformers comes from their ability to work in Parallel. Unlike older models, Transformers can analyze entire sequences of words simultaneously. This allows them to grasp complex relationships between words and better understand context.

There is a training advantage. Using neural networks, LLMs can learn from massive datasets, identify patterns, and make predictions about language. This core foundation enables them to understand and generate human-like text.

One of the key techniques is Self-Attention. This special technique allows the network to focus on the most relevant parts of a sequence when making predictions. It's like a spotlight that highlights important words within a sentence.

Machine learning-based attention is a mechanism which intuitively mimics cognitive attention. It calculates "soft" weights for each word, more precisely for its embedding, in the context window. These weights can be computed either in parallel (such as in transformers) or sequentially (such as recurrent neural networks). "Soft" weights can change during each runtime, in contrast to "hard" weights, which are (pre-)trained and fine-tuned and remain frozen afterwards.1

Attention (machine learning) - Wikipedia

Footnotes

1. Attention (machine learning) - Wikipedia - Definition from Wikipedia

2. Neural network (machine learning) - Wikipedia - Further Reading