Source: my notes from Intro to Large Language Models by Andrej Karpathy.

Just listened to this awesome presentation by Andrej Karpathy (top researcher at OpenAI) on The Busy Person’s Intro to LLMs. I found it extremely helpful and was making notes throughout. Sharing them below for anyone who wants to understand AI and LLMs in the simplest way possible:

1/ What is a Large Language Model (LLM)?

LLMs are intelligent pieces of code that can learn from a vast universe of data, make inferences using it, and use those inferences to answer a user’s questions or perform certain tasks.

An LLM typically consists of 2 files – (1) Parameters file and (2) Run file.

a) Parameters are the weights that go into the LLM and power the neural network within it.

b) The Run file is some sort of code to run logic on the neural network within the model.

LLMs can be of 2 types:

a) Proprietary – users don’t have access to the actual parameters/ weights within the model, and can only use it via a web interface or API. Eg. OpenAI’s ChatGPT, Google’s Bard etc.

b) Open source – users have access to the full model, including the parameters/ weights. They can easily modify the model for their own specific purpose. Eg. the Llama-2 model by Meta.

Here’s the example of Llama-2 given by Andrej, wherein the Parameters file is 140GB and the Run file is just 500 lines of code written in any language like C, Python etc.

Interestingly, you just need a standard computer to be able to run an LLM like Llama-2 and derive inferences. It runs locally on the machine and hence, even Internet connectivity isn’t a requirement.

However, training an LLM is a significantly more computationally heavy task that requires thousands of GPUs and needs millions of dollars in investment.

2/ What is the LLM’s neural network really doing?

Very simply, a neural network tries to predict the next word in a sequence with the highest confidence. Eg., for the sequence “cat sat on a…”, the next word is predicted to be “mat” with 97% confidence.

The next word prediction task forces the neural network to learn a lot about the world and assimilate tons of knowledge on the Internet, including the interconnections within it.

The outcome of this basic learning and word-prediction capability is the ability to create various kinds of documents eg. a piece of Java code, an Amazon product description, or a Wiki article.

The now-famous Transformer architecture is what powers this neural network.

3/ How the neural network “really” works is still a mystery

While we can input Billions of parameters into a network, and iteratively adjust them for better predictions, we really don’t know how the parameters are exactly collaborating within the network

We know that these Parameters build and maintain some kind of knowledge database, but it’s a bit strange and imperfect. For eg., the model can answer “who is Tom Cruise’s mother?” as “Mary Lee Pfeiffer” but if you ask “who is Mary Lee Pfeiffer’s son?”, it answers “I don’t know”. This is popularly called a Recursive Curse.

LLMs aren’t like a car, where we know each part and how they exactly work with each other.

Think of LLMs as mostly inscrutable artifacts. The only thing we truly know about them is whether they work or not, and with what probability.

We can give LLMs inputs, and empirically evaluate the quality of outputs. We can only observe their behavior.

Andrej Karpathy (paraphrased from Intro to Large Language Models)

4/ How do you obtain the secret sauce of an LLM – the Parameters? Stage 1: Pre-training

Getting the Parameters requires essentially “compressing the Internet”.

To illustrate how Llama-2 was trained, imagine taking a chunk of the Internet (~10TB of text, procured by crawling the Internet). You then compress this large chunk of Internet data dump using 6,000 GPUs* for 12 days. FYI, this would cost ~$2Mn as GPUs are super expensive. [*A graphics processing unit (GPU) is an electronic circuit that can perform mathematical calculations at high speed. This is what Nvidia makes and AI is the reason behind its stock ripping.]

What this GPU computation throws out is the Parameters file – a compression of all this Internet data, almost like an intelligent distillation of it.

5/ Stage 2 of training – Finetuning

The document generation capability of a base-trained LLM is of limited use. What we want is a Q&A-type assistant model. The way to achieve this is to take the model through the next stage of training called Finetuning.

In this stage, we swap out the generic, large Internet dataset, and replace it with a specific Q&A dataset that has been collected manually.

This step of finetuning is done by companies like OpenAI and Anthropic, who hire people that create ideal answers to questions that users might typically ask. This human-curated dataset of conversations then becomes labeling instructions for the LLM to be finetuned upon.

It’s important to note that at this stage, quality is more important than quantity.

After finetuning, your base LLM becomes an assistant model. Interestingly, and again we don’t understand how it really happens, but the model is able to use the generic Internet knowledge from the pre-training stage and combine it with the conversational datasets from the finetuning stage.

The below slide summarizes both parts of LLM training. Pretraining is very expensive so update training typically happens once a year. Finetuning is more of a model alignment exercise and can happen as frequently as once a week.

6/ Stage 3 of training – comparisons

There is another subsequent stage of finetuning that can happen where for each question, human labelers compare multiple answers from the assistant model and label the best one. OpenAI calls this step Reinforced Learning from Human Feedback (RLHF).

7/ How do labeling instructions given to humans look?

Here is an excerpt from InstructGPT paper by OpenAI. It shows how human trainers should craft responses that are helpful, truthful, and harmless. These instructions can run into hundreds of pages.

Rather than being totally manual, labeling is fast becoming a human-machine collaboration exercise where LLMs can create drafts for humans to splice and put together, LLMs can review and critique labels based on instructions etc.

8/ How to compare various LLMs and find the best one for your use case?

There are platforms like Chatbot Arena (managed by Berkeley) where models are compared against each other and ranked as per an Elo rating, kind of like comparing and ranking chess players.

As seen below, closed proprietary models are ranked much higher than open-source models. However, the former can’t be modified for your use while the latter can.

9/ LLM scaling laws

Performance of LLMs is a smooth, well-behaved, predictable function of (1) N – no. of parameters in the network and (2) D – the amount of text we train on.

And the trends don’t show signs of topping out.

We can expect “more intelligence for free” by scaling.

Andrej Karpathy (paraphrased from Intro to Large Language Models)

Essentially, algorithmic progress isn’t necessary for better models. We can just throw more N and D via more compute power and get better intelligence.

Another interesting point – we can get better ‘general capability’ simultaneously across all areas of knowledge by training a bigger model for longer. Essentially, with more training, we should expect the performance of models to rise more across all areas for free.

This is the reason driving the gold rush for (1) data and (2) compute, as the more we throw of it at LLMs, we can get exponentially better models. Algorithmic progress then just becomes a bonus.

10/ Future Idea #1: System 1 vs System 2 thinking by LLMs

Currently, LLMs are System 1 thinkers, similar to what Daniel Kahneman calls the intuitive, fast-thinking part of the human brain (like speed chess). Where researchers are trying to go is converting them into System 2 thinkers, mimicking the slow, deliberate, rational, reflective part of the human brain (like professional chess players thinking through decision trees on the spot).

Turning LLMs into System 2 thinkers is about trading off time for accuracy.

11/ Future Idea #2: self-improvement

Another area of future improvement is making LLMs capable of self-improvement. The AlphaGo program by DeepMind has already demonstrated that in a narrow, sandbox environment with a clear reward function (telling the program whether how it played is good or bad), AI can self-improve to become better than the best human player. Can this be achieved in other, more open contexts?

12/ Future Idea #3: custom LLMs

This is already being done by OpenAI via GPT agents that can do custom tasks and are available on the GPT Store (more details in my post on the OpenAI Dev Day).

Bringing it all together – LLM OS

I don’t think it’s correct to think of LLMs as just chatbots or word predictors. Think of it as the kernel process of an emerging operating system.

This process is coordinating a lot of resources like memory and computational tools, for problem solving.

Andrej Karpathy (paraphrased from Intro to Large Language Models)

As per Andrej, this is how the LLM OS framework will likely look:

There is another similarity between today’s OS ecosystem and the LLM OS view. Today, there are proprietary OS like Windows, iOS, and Android, with a fragmented ecosystem of open-source products co-existing. Similarly in LLM OS, there are proprietary LLMs like ChatGPT, Claude, and Bard, with a burgeoning parallel ecosystem of open-source LLMs, many of which are presently powered by Meta’s Llama series.

Appendix: LLM Security

As per Andrej, one of the major security threats facing LLMs is various ways to jailbreak them and get answers to undesirable questions. For eg. a user can role-play within a question to make it sound harmless to the LLM. In other cases, users can create encoded versions of harmful questions and bypass the LLM’s security check.

Another type of security threat is prompt injection, where bad actors can infect images, documents, and web pages with harmful prompts that can cause undesirable behavior by LLMs when the user tries to input these materials.

The final type of threat is data poisoning where attackers can inject trigger words within training data that corrupts the model and catalyzes specific undesirable behavior.

Subscribe

to my weekly newsletter where in addition to my long-form posts, I will also share a weekly recap of all my social posts & writings, what I loved to read & watch that week + other useful insights & analysis exclusively for my subscribers.