Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Addressing cloud LLM risks: data retention policies, potential training on user inputs, and foreign jurisdiction issues.
- Examining Ollama’s architecture: the model server, registry system, and OpenAI-compatible API.
- Comparative analysis with alternatives such as vLLM, llama.cpp, and Text Generation Inference.
- Reviewing licensing terms for popular models including Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Installing Ollama on Linux systems with support for both CUDA and ROCm.
- Implementing CPU-only fallbacks and optimizing for AVX/AVX2 instructions.
- Deploying via Docker and managing persistent volume mappings.
- Configuring multi-GPU environments and strategizing VRAM allocation.
Model Management
- Retrieving models from the Ollama registry, e.g., using 'ollama pull llama3'.
- Importing GGUF format models from HuggingFace and TheBloke.
- Evaluating quantization levels (Q4_K_M, Q5_K_M, Q8_0) and their respective trade-offs.
- Managing model switching and understanding limits on concurrent model loading.
Creating Custom Modelfiles
- Mastering Modelfile syntax, including FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Tuning generation parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to enforce role-specific behaviors.
- Publishing custom models to a local registry.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON output modes.
- Integrating with frameworks like LangChain and LlamaIndex, or custom applications.
- Implementing authentication and rate limiting using reverse proxies.
Performance Optimization
- Managing context window sizes and KV cache efficiency.
- Implementing batch inference and handling parallel requests effectively.
- Optimizing CPU thread allocation and ensuring NUMA awareness.
- Monitoring GPU utilization and managing memory pressure.
Security and Compliance
- Establishing network isolation for model serving endpoints.
- Implementing input filtering and output moderation pipelines.
- Maintaining audit logs for all prompts and completions.
- Verifying model provenance through hash validation.
Requirements
- Intermediate proficiency in Linux and container administration.
- A high-level understanding of machine learning concepts and transformer architectures.
- Working knowledge of REST APIs and JSON data structures.
Target Audience
- AI engineers and developers looking to replace cloud-based LLM APIs with local solutions.
- Organizations where data sensitivity restricts the use of cloud-hosted models.
- Government and defense teams operating under air-gapped requirements for language models.
