Generative AI¶

LLM¶

Large Language Models

Limitations¶

• Bias
• Hallucinations
• Expensive to build & run

ChatGPT¶

1. Train supervised policy
2. Provide prompt
3. Labeler demonstrates desired output behavior
4. Fine-tune model
5. Collect comparaison data & train reward model
6. Prompt and several model outputs are samples
7. Labeler ranks outputs from best to worst
8. Data used to train reward model
9. Policy optimization

GAN¶

Generative Adversarial Networks

flowchart LR
n[/Noise/] ---> g[Generator] --> d
rd[Real Data] -->
d[Discriminator] -->
rf{Real/Fake} -.->
|Backpropagation| d & g

Fine-Tuning¶

Process of training model using specific data, usually with a significantly smaller learning rate

Disadvantages¶

• requires copy of the model
• associated costs of hosting it
• Risk of “catastrophic forgetting”: model forgets previously learnt information

RAG¶

Retrieval Augmented Generation

Makes use of a source of knowledge, usually vector store of embeddings and associated texts

By comparing predicted embeddings of query to embeddings in the vector store, we can form a prompt for the LLM that fits inside its context and contains the information needed to answer the question

Advantages¶

• Does not require re-training
• No need to deal with internal workings of model
• Just adjust the data that the model “cheats” off
• Reduces the amount a model “hallucinates”

Difficulties¶

• Finding relevant data to give the model

Keywords¶

• Data organization:
• Vector creation: Unique index that points right to a chunk of information
• Querying: Prompting
• Retrieval
• Prompt goes through embedding model and transforms into a vector
• Systems uses this to get the chunks most relevant to the question
• Prepending the context: Most relevant chunks are served up as context
• Answer generation

Types of Questions¶

Comments