Decoding 🤖 Jargons 101

In the current day and age of AI, it does get quite overwhelming hearing new AI jargon every now and then. The terminologies are so scary that you feel like you are going to be replaced by AI for sure. It might not be the case, though, if you spend some time understanding what all this is about. Don’t worry, I am here to help you understand some key jargon that would change your perspective towards AI 👨🏻‍🏫.

Before we get into the nuances, let’s try to understand what AI is. Consider yourself when you were born you were blank, you did not know anything about the world. Then you are fed with a lot of things through your sense organs. Once you gathered some good amount of data repeatedly, you started connecting the dots and started interacing with the world by gestures or speaking some words which most probably were not accurate. Since your interactions were maybe incorrect or inappropriate at times, you were corrected by your parents after a good amount of repeated training, and you have become what you have become (I hope the training went well 😝). So, taking this analogy, let’s try to understand what these complex-sounding jargons in AI

GPT

GPT stands for Generative Pretrained Transformer. Let’s 💔 it down:
Generative: Having an ability to generate text, image, audio, video, etc. (Similar to how you can create different things)
Pretrained: This means that it has been fed and trained beforehand, which also means that it can only generate based on the extent it has been trained (as to how you can only think about things you have already been exposed to)
Transformer: This is the core that understands your input, basically an algorithm. (Similar to your brain, which processes the external input)

So, looking at what GPT is all about, we can clearly see that the capability of the GPT model is to an extent of data it has been trained, how well the transformer is written to understand the nuances, and how well it can articulate the response.

Very much like how an individual can only understand the languages they have been trained on (Pretrained), based on their intellect, different people might interpret the same input in different ways (Transformer) and different people might have different atriculation skills (Generative).

Transformer

🗣️Let’s get back to reality. We are not talking about the movie Transformer. The Transformer we are talking about is the core section where the machine understands the context of your input. It has couple of steps to achieve it, let’s explore one by one:

Encoder Block

The encoder block processes the user input and understands the input context. There are different steps to achieve this:

Tokenization: Unlike humans, machines do not understand languages and words hence, we have to represent the user input in the form of numbers, also known as tokens. We take the user input, break it into smaller chunks (chunking generally depends on the algorithm), and map them to their respective token.

Input: I love cats
Token: ["I", "love", "cats"] # Tokenization depends on the algorithm

Vocabulary: This is the map for user input to token. This is typically built from the training dataset. It determines how a language is understood and processed by an AI model. (Like a human having understanding of a large number of vocabulary can understand and interact in a much better way than a person having understanding of less number of vocabulary)

Vocab Size: This represents the size of vocabulary.

Input Embedding: Token by themself do not make any sense, so they are converted into vector in this step. By doing so, the model can understand some level of input context.

I → [0.1, 0.5, -0.2]  
love → [0.6, -0.1, 0.3]  
cats → [0.2, 0.7, -0.5]

Vectors: Vectors are lists of numbers representing the token in a multidimensional graph where the dimension is represented by the length of the vector (in the above example, it’s 3 dimensional). To put it simply, vector ensured similar or more related tokens appear closer to each other in the graph where whereas dissimilar words will be farther apart, which helps understand the meaning, context, or usage of the token.

Positional Embedding: If we would consider the sentence “The cat sat on the mat” and change the sequence to “The mat sat on the cat” the token that get’s generated would be the same but the meaning would not be has changed which a transformer would not be able to distinguish hence, positional embedding solves this by adding positional information to the token embedding.

🫵Self-Attention: This is the step where we let all the tokens talk to each other, where each token will be asking, "What should I focus on this input when I do my job?”. This was,y each token will create an attention score (list of scores) in order to indicate how much attention it should pay to every other token. This helps the transformer figure out the deeper context of each token.

For example, in the sentence “The cat sat on the mat”
For “cat”, the model might pay attention to “sat” (because it’s an action the cat is doing) and “mat” (because it’s where the cat is sitting). It gives less attention to “The”.

Multi-Head Attention: This is an extension of self-attention where we run the self-attention step multiple times with different perspectives. This helps the model build a more comprehensive understanding of the input.
For example: If we tell a group of people who are blindfolded and have not seen an elephant to touch different part of an elephant and ask them how an elephant should look like everyone might have a different understanding of an elephant where the one who touched the tail might think an elephant is like a snake where the other who touched the ear might think it’s like a giant leaf etc. But when we consider the input of all the people, then we will get a much clearer picture of what an elephant looks like. This is what multi-headed attention does.

Softmax: Softmax takes the vector generated so far and converts it into probability where the value ranges from 0 to 1 where the number closer to one has having higher probability. This helps the transformer predict the next word based on probability.

🌡️Temperature: Temperature is a value that tells the transformer to be creative or precise. The value ranges between 0 to 2, where <1 means more precise and >1 means more creative.
This is not indicative that <1 is always right and >1 is always wrong. It only indicates how predictable the output would be.