Local LLMs: Run AI Privately on Your Own Laptop

The Landscape of Private AI

• Why Run LLMs Locally? 30min
• Hardware Requirements and Bottlenecks 45min
• Understanding Model Architectures 60min

Environment Setup

• Installing Python and Virtual Environments 💻 30min
• Configuring CUDA and Metal Drivers 💻 45min
• Setting up Ollama and LM Studio 💻 30min

Navigating Hugging Face

• Exploring Model Repositories 🔒
• Understanding Model Sizes and Parameters 🔒
• Downloading Models via CLI 🔒

Quantization Techniques

• Introduction to GGUF and EXL2 Formats 🔒
• Comparing 4-bit vs 8-bit Quantization 🔒
• Measuring Perplexity and VRAM Usage 🔒

Running Inference

• Using llama.cpp for High Performance 🔒
• Exposing Local Models via OpenAI-Compatible API 🔒
• Managing Context Windows and KV Caching 🔒

Prompting Strategies

• System Prompting and Roleplay 🔒
• Chain-of-Thought and Few-Shot Prompting 🔒
• Handling Model-Specific Chat Templates 🔒

Building a Local Knowledge Base

• Vector Databases: ChromaDB and FAISS 🔒
• Embedding Models for Local Search 🔒
• Connecting RAG to Local LLMs 🔒

Parameter Efficient Fine-Tuning

• Introduction to LoRA and QLoRA 🔒
• Preparing Datasets for Fine-Tuning 🔒
• Training a Custom Model on Consumer Hardware 🔒

Final Implementation

• Project Scoping: Building a Private Assistant 🔒
• Developing the RAG Pipeline 🔒
• Deployment and Performance Tuning 🔒
• Final Presentation and Code Review 🔒