What is fine-tuning?

• Taking a generalized large language model and fine-tuning them to specific use cases

What does fine-tuning do?

• Let’s you put mode data into the model than what fit’s the prompt
• Get’s the model to learn the data rather than just access it
• Steers the model towards consistent outputs
• Reduces Hallucinations
• Customizes the model to your application or use-case

The process of fine-tuning a model is similar to the process of initial model training

Pros of fine-tuning:

• Can train using unlimited data and this new data can be used to update previously trained information in the model.
• Enables the model to learn new information
• Less cost of querying the model (compared to just prompting with data)
• can use RAG

Fine-tuning your own LLM can have several benefits like performance, privacy, cost, reliability, customization, stability, etc.

llama library (Lamini): [INST] and [/INST] tags for sending instructions to llama2 model.

<aside> 💡

Pre-training → Fine-tuning, the fine-tuning process is mostly done after the pre-training.

</aside>

In Pre-training, initially the model has zero knowledge of the world and thus can’t form english sentence structures, and can only predict next tokens leading to gibberish outputs.

However, as the model trains on a large corpus of unlabelled data, it keeps getting better at predicting the right tokens and thus the next word.

Once the model is trained using self-supervised learning, it learns the language and gains knowledge of sentence structures.

ElutherAI: “The Pile” is a dataset for representing the entire scraped data from the internet. (https://github.com/EleutherAI/the-pile)

The pre-training process results in a base model that can predict next token sequences, however the limitation is that it is not suitable for a chatbot interface.

Fine-tuning can be used to train/update the model on new data which can either be self-supervised unlabelled data, curated labelled data with much less cost. The process of fine-tuning updates the entire model and not just part of it (whereas in other traditional machine learning cases fine-tuning is often performed on a subset of the weights).

Outcomes of fine-tuning:

• Behavior Change: respond more consistently, more focus (moderation), better at conversation (easy to extract relevant information).
• Gain Knowledge: learning new concepts (specifically trained on), correcting old information.

Some of the tasks where you would want to fine-tune a model are:

• Extraction: text in, less text out; like summarization, topics, keywords, routing (when you need a yes/no to determine the next route)
• Expansion: text in, more text out; writing (emails, messages, chats)

Having a clarity on the task to be performed will help in fine-tuning the model well.

Steps for fine-tuning first time: