AI315: Fine-tuning for local LLMs NEW

Agenda:

• Deciding: Is fine-tuning worth it, and which model should you use?

  • Prompting, RAG, or fine-tuning: Decisions based on typical business use cases such as domain-specific language, fixed response formats, classification, and extraction

  • Selecting open models for on-premises deployment: Size, licensing, quality in German, and hardware requirements

  • Estimating GPU memory: Model size, quantization, LoRA compared to QLoRA

  • Data privacy and sovereignty: Training data and model remain in your own infrastructure
    
    

Hands-on lab: Testing the base model without training and establishing a measurable baseline

• Building training data and training with LoRA/QLoRA

  • Creating training examples from your own documents, tickets or sample responses: format, scope and quality

  • Applying chat templates and instruction format correctly

  • Generating synthetic training data using a larger model and performing spot checks

  • Understanding LoRA and QLoRA: Rank, Alpha and Target Modules

  • Training with Hugging Face TRL and Unsloth on a single GPU

  • Monitoring the training process: loss, overfitting, and the right time to stop
    
    

Hands-on lab: Fine-tuning a small model on a custom domain-specific dataset

• Evaluating and deploying locally

  • Before-and-after comparison using a custom test set: formatting consistency, subject-matter accuracy, and capabilities lost during training

  • Using LLM-as-a-Judge specifically and securing with manual spot-checks

  • Merging the LoRA adapters with the base model and exporting it as GGUF

  • Running the fine-tuned model using Ollama or llama.cpp and connecting via an OpenAI-compatible API

  • Outlook: Preference tuning with DPO, if supervised fine-tuning is not sufficient
    
    

Hands-on lab: Evaluating, exporting, and deploying the fine-tuned model in Ollama

Objectives:

Upon completion of the workshop AI315 Fine-tuning for local LLMs participants will be able to:

• make informed decisions about whether prompting, RAG, or fine-tuning is appropriate for a given use case

• select an open model based on the task, license, and available hardware

• estimate GPU memory requirements for LoRA and QLoRA

• create a usable training dataset from their own materials

• train an open-source model with LoRA or QLoRA on a single GPU

• evaluate the benefits of the training by comparing performance against the base model using a test set

• export the result and run it locally using Ollama or llama.cpp

Target audience:

The training AI315 Fine-tuning for local LLMs is aimed at:

• Software developers who want to adapt an open language model to the domain terminology, formats, or tasks of their company

• Data scientists and ML engineers who want to practically learn fine-tuning

• IT managers and architects who are planning AI solutions that are not dependent on the cloud

Prerequisites:

For participation in the course AI315 Fine-tuning for local LLMs the following prerequisites are required:

•      good Python skills

•      basic understanding of how language models work (tokens, prompts, context windows)

•      some experience with Jupyter notebooks or the command line is helpful

The exercises run in a provided GPU environment; own GPU hardware is not required.

Description:

Many companies want to use language models, but do not want to give their data to a cloud provider. The course AI315 Fine-tuning for local LLMs shows how an open-source model can be adapted to custom tasks with manageable effort and then runs completely in-house.

At the beginning is the question whether fine-tuning is the right approach at all. Participants distinguish it from prompting and RAG, select a suitable model and estimate the hardware requirements. Afterwards, they build a training dataset from their own materials and train a small model with LoRA or QLoRA on a single GPU.

In the last part, the results are evaluated: A test set compares the fine-tuned model with the base model and also identifies which capabilities have been lost. Finally, participants export the model as a GGUF and run it using Ollama or llama.cpp. By the end of the course “Fine-Tuning Local LLMs,” participants will have a complete workflow that they can replicate using their own data.