How to serve a large language model from your local workstation

Setting up a large language model on a local workstation is no longer just a whim for AI geeks: it's perfectly feasible if you have a good computer, some patience, and a clear idea of ​​what you want to achieve. From a powerful laptop to a desktop with a serious GPU, or, as in your case, a machine with an AMD Ryzen AI MAX+ and an iGPU with a good amount of shared VRAM, you can create your own "private ChatGPT" accessible from other devices on your network.

The challenge isn't just installing tools like Ollama, LM Studio, or GPT4All , but understanding how to leverage your hardware (CPU, GPU, and iGPU), how to expose a remote AI server that you can connect to from your phone or laptop, and how to choose the right model and quantization to avoid crashing your machine on the first question. We'll look at all of this calmly, but practically, without losing sight of the important technical details.

What does it mean to deliver an LLM from your local workstation?

When we talk about "serving a large language model from a local workstation," it's not just about opening a chat and that's it. It's about setting up a service accessible via HTTP (usually with an OpenAI-type API) that can be used from other devices: Python scripts, web applications, mobile apps, or even low-code tools. Your machine acts as a mini inference server within your home or office network.

This has several implications: firstly, you need the model to run smoothly on your hardware (whether CPU, dedicated GPU, or iGPU), and secondly, the service needs to listen on an IP address accessible to other computers (not just 127.0.0.1). Tools like Ollama and LM Studio greatly simplify this process, as they expose a local server compatible with the OpenAI API or with simple HTTP endpoints that you can consume from any modern programming language.

On systems like a Framework Desktop with Ryzen AI MAX+ and Pop!_OS (or another Linux distribution), the goal is usually to leverage the 98 GB of shared VRAM allocated to the iGPU to accelerate LLM processes, rather than relying solely on the CPU. This is where drivers, ROCm (in the case of AMD), and the GPU acceleration options of each tool come into play.

Beyond the technical aspects, there's a strategic side: a local LLM gives you more control over costs, privacy, and latency . You don't pay for tokens, you're not dependent on an internet connection, and sensitive data stays within your network. However, you are responsible for system maintenance, updates, and security.

Hardware and system requirements for local LLMs

Before you get bogged down with installations, it's a good idea to review what hardware you need to run an LLM from your local workstation with reasonable efficiency. You don't need a massive data center setup, but you do need to understand where the bottlenecks are: RAM, VRAM, CPU, and disk space.

For running large language models, RAM is usually the critical resource. Models are loaded into memory (RAM and/or VRAM), so if you run out of RAM, you'll see "Out of Memory" errors or even crashes. With 8 GB of RAM, you can try very small models (1B, 3B, 7B quantized), although you'll notice a lack of fluidity. With 16 GB , you can easily handle well-quantized 7B and 13B models. With 32 GB or more , you can start seriously working with large models (30B, 40B, 70B) without too many problems.

As for CPUs , modern desktop processors handle local inference quite well, especially with quantized models. Ideally, you should have at least 4 physical cores , and if possible, 8 or more cores for reasonably fast text generation, particularly if you're handling multiple concurrent requests from different devices on your network.

The GPU (or iGPU) makes all the difference in performance when you want smooth responses and decent tokens per second. A dedicated GPU with at least 8 GB of VRAM (for example, an RTX 3060 or similar) can easily handle mid-range models. In your case, with a Ryzen AI MAX+ and an iGPU with 98 GB of shared VRAM , the trick is getting the AI ​​stack (Ollama, llama.cpp, ROCm, etc.) to detect and utilize that iGPU. Sometimes you have to adjust environment variables like HIP_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES , check drivers, and review the logs to see if the GPU is being used.

Storage also matters: model files range from a couple of GB to hundreds of GB. A small quantized model might be around 2 GB, a medium one around 5 GB, and large models 40 GB or more. There are even LLMs that exceed one terabyte. It's wise to reserve between 20 and 100 GB of free space if you plan to test several models without constantly deleting files.

Finally, the operating system . Windows 10/11, Linux, and macOS can all run local LLMs. For environments like yours, with Pop!_OS and AMD hardware, it's usually more practical to stick with Linux to take better advantage of ROCm and GPU integrations with Ollama or llama.cpp. If a particular distribution gives you too many problems with drivers, it's sometimes worth switching to one that's more compatible with your hardware (Ubuntu, Fedora, etc.).

Basic concepts: LLMs, Transformers and quantization

To avoid going in blind, it helps to have a basic understanding of what's behind large language models . An LLM is a neural network trained in a self-supervised manner on enormous amounts of text to predict the next word in a sequence, and from there generate responses, translate, summarize, or program.

Most modern LLMs use the Transformer architecture , which works by converting words into numerical vectors (embeddings) . These vectors are processed through layers of attention and feed-forward, learning statistical patterns from the language: basically, which words tend to appear together and in what contexts. At each step, the model estimates the probability of the next word given the previous sequence and other configurable parameters.

The Transformer is typically divided into two conceptual blocks: an encoder , which is responsible for "understanding" the input text and generating a contextual representation, and a decoder , which takes that representation and generates output tokens one by one. While large-scale chat systems currently use decoder-only architectures, the underlying idea remains the same: the sequence is processed and tokens are generated until the response is complete.

During generation, parameters such as temperature , top-k, top-p, etc., come into play, influencing how much the model "risks" when choosing less probable words. Changing these settings can make the AI ​​respond more creatively or more conservatively, which is crucial if you're setting up a remote service for other users or applications.

A vital concept for running LLMs on a workstation is quantization . Instead of storing network weights as 16- or 32-bit floating-point numbers, they are converted to 8-, 4-, or even 2-bit integers. This significantly reduces the model file size and the memory required to run it, at the cost of a slight loss of accuracy in the responses.

In practice, converting a 32-bit model to a format like Q4_K_M or similar can reduce a 40 GB model to half its size or more. Metrics like perplexity help measure the impact: a lower value indicates better quality, and you'll see it increase slightly when quantization is applied, but often the user barely notices the difference in real-world use while performance improves dramatically.

Advantages of delivering an LLM from your own machine

Setting up an AI server on your own PC or workstation has several clear advantages over constantly relying on cloud-based APIs. The first is privacy and complete control over your data . Everything you send to the model (prompts, documents, chat histories) stays on your computer, without passing through external servers, which is crucial if you work with sensitive medical, legal, or financial information.

There's also the issue of cost and performance . Each call to an API like GPT-4, Claude, or other commercial models has a price per million tokens. If your application makes a lot of requests, the bill can skyrocket. With a local model, beyond the cost of hardware and electricity, you don't pay per query . And by not depending on the network, you usually gain in latency: the time to the first token and the generation speed are typically better, especially on slow or unstable networks.

Another powerful advantage is customization . Many local execution tools allow you to adjust model parameters, load specific prompt templates, or even reuse weights from other models for fine-tuning. A huge community of derived models has been created for different use cases, from programming assistants to creative chatbots or models specialized in legal documents.

From a technology strategy perspective, running LLMs on your own infrastructure reduces dependence on specific vendors . You're not exposed to price changes, usage limits, content restrictions, or service outages. And, if you later decide to combine this with cloud services (for example, for peak loads), you can design hybrid architectures that distribute the load between your on-premises hardware and the cloud.

However, it's not all sunshine and roses: gaining control also means taking on responsibilities for model governance, updates, security, and monitoring . If you're going to use this server for anything more serious than just tinkering at home, it's wise to plan from the outset how you'll log information, manage model versions, and audit its behavior.

Key tools for delivering LLMs locally

To set up an AI service accessible from other devices, you have several tools at your disposal. Some are geared towards users who don't want to deal with terminals, while others are more technical but offer greater control. The most relevant for your case are Ollama, LM Studio, GPT4All, and llama.cpp/ROCm-based environments , as well as platforms like Comfy for image generation.

Ollama is a user-friendly layer on top of llama.cpp that allows you to download, run, and manage models directly from the command line, also offering an HTTP API on port 11434 by default. It supports Windows, macOS, and Linux, and is especially powerful on machines with well-configured NVIDIA or AMD GPUs. You can run it on Pop!_OS without any problems as long as you have the correct drivers.

LM Studio is more visual, with a graphical interface for searching for models on Hugging Face, downloading them, and chatting with them. It also offers a local server mode compatible with the OpenAI API , perfect for integrating it into your applications as if you were calling GPT-4, but without paying and without using your network. It supports Windows, macOS, and Linux, and works quite well with modern GPUs.

GPT4All , on the other hand, combines a simple graphical interface with command-line tools. It's designed for users who want something in between: the ability to use a local chat, but also more advanced features via the terminal or integrations with other projects. It works well without a GPU, relying solely on the CPU, making it useful for systems without powerful graphics acceleration.

In the graphics field, Comfy is a very popular open-source platform for AI-powered image generation and editing . While not specifically designed for LLMs, it fits well into a more comprehensive local AI setup: you use Ollama or LM Studio for text, and Comfy for images, all running from your workstation.

If you want to take it to the extreme, platforms based on AMD Ryzen AI Max+ with ROCm and llama.cpp can even run models with up to a trillion parameters in a distributed manner across local, quantized, and hyper-optimized clusters. This scenario is more typical of tech startups with serious ambitions than a home user, but it demonstrates that large-scale local inference is perfectly achievable.

Ollama on Linux: installation, configuration and use as a server

Returning to a practical example and aligned with your environment, Ollama on Linux is a very solid option for serving an LLM from your workstation. The official installation, on distributions like Pop!_OS or Ubuntu, is usually done with a single command:

`curl -fsSL https://ollama.com/install.sh | sh` downloads and installs the service, which runs in the background using systemd. Once installed, Ollama downloads the models to an internal folder (on Linux, this is usually `/usr/share/ollama /.ollama/models` unless you change the path) and starts an API at 127.0.0.1:11434.

To adapt it to your needs, you can configure several environment variables by editing the systemd unit: with sudo systemctl edit ollama.service you add a [Service] block where you define options such as OLLAMA_MODELS (custom path of models), OLLAMA_HOST (for example, 0.0.0.0:11434 so that other devices on the network can connect), OLLAMA_ORIGINS to control CORS, or OLLAMA_KEEP_ALIVE to decide how long a model remains in memory after the last request.

After saving, run ` sudo systemctl daemon-reload` and then `sudo systemctl restart ollama` to apply the changes. It's important to check the Ollama logs , which on Linux are accessed with `sudo journalctl -u ollama` (you can add `-f` to monitor in real time or `-n 100` to see only the last 100 lines), because there you'll know if it's detecting the GPU, if there are CUDA/ROCm errors, or if it's only using the CPU.

Within those logs, you'll see a line with the message "looking for compatible GPUs," and, if everything is working correctly, another line indicating that a usable GPU has been found. If, however, you see "no compatible GPUs were discovered," it means that Ollama is relying solely on the CPU , and in that case, you'll need to check AMD drivers and environment variables (HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, etc.).

With Ollama running, you can manage models with commands like `ollama pull llama3.2:1b` (unload), `ollama run llama3.2:1b` (interactive run), `ollama list` (model list), `ollama show` (details), `ollama rm` (delete), or `ollama ps` (models loaded into memory and whether they use CPU or GPU). The `ollama stop <model>` command lets you unload a model from memory if you no longer need it.

Choosing and labeling models: size, quantification, and variants

Once your server is up and running, you need to decide which model to use for your applications and devices. On the Ollama website or in the LM Studio catalogs, you'll find a huge list of models with labels that might seem a bit confusing at first, but are actually quite informative.

On one hand, there's size , usually expressed in "B"s (billions of parameters): 1B, 3B, 7B, 13B, 32B, 70B, etc. More parameters generally mean more knowledge and better performance, but also higher RAM and VRAM consumption . If your workstation is running on fumes or you need low latency with many users simultaneously, it might be wiser to use a well-tuned 7B or 13B model rather than a sluggish, resource-intensive one.

Then there's quantization , with labels like Q2_K, Q3_K_M, Q4_K_S, q8_0, fp16, fp32, etc. The lower the number (Q2 versus Q8), the more compressed the model is and the less memory it uses, but there's also a loss of precision. Variants with a K suffix (K_S, K_M, K_L) are generally considered a balance between size and quality , while fp16 or fp32 means virtually unquantized, which gives maximum quality at the cost of consuming resources.

Additionally, you'll see variations based on usage type : models labeled as instruct (designed for following instructions), chat (general conversation), code (programming), vision (multimodal with images), etc. To serve a general assistant to other devices, you'll typically want something like chat or instruct, while for a code generation backend, a code model like CodeLlama or DeepSeek Coder would be better.

Finally, you'll often see a " latest" label . This doesn't mean it's the latest model, but rather a kind of "recommended default version": it's usually a middle ground between size, quality, and hardware requirements. If you're unsure where to start, "latest" is a good first option for testing.

LM Studio and OpenAI-compatible server mode

If you prefer a more visual approach and are particularly interested in exposing an OpenAI-type API from your workstation, LM Studio is a very convenient tool. After downloading the Windows, macOS, or Linux version from its website, you install it like any other application, and upon opening it, you'll find a built-in search engine for models hosted on platforms like Hugging Face.

In LM Studio, you can filter by size, task type, and hardware compatibility. For systems with only a CPU and 8 GB of RAM , you'll need to use small, highly quantized models. If you have more than 16 GB of RAM , you can more easily move up to 7B or 13B. With a GPU and 16 GB of RAM (or a Mac with an M1/M2 drive), you can try medium-sized models, and with a GPU and 32 GB of RAM, you're entering 40B territory without it being a problem, always adjusting the quantization accordingly.

Once you've downloaded a model, LM Studio lets you chat directly from the interface, providing statistics on token usage and speed. You can open multiple chats, change generation parameters in the side panel, and see how each model behaves with real prompts. This is useful before deciding which model to expose as a remote service for your apps.

The most interesting part comes with the "Local Server" option . By activating it with the "Start Server" button, LM Studio starts a server on your machine that emulates the OpenAI API . This means that from your Python, JavaScript, or other code, you can continue using the OpenAI client, but pointing to http://localhost:1234/v1 (or another port you configure) instead of the actual OpenAI servers.

For example, in Python you would use something like this: define an OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed") and then create completions with the model "local-model" or whatever name you've configured. From the outside, it looks like you're calling GPT-4, but you're actually communicating with your local LLM, running on the workstation and enforcing your privacy and cost rules.

This compatibility with the OpenAI format allows for the easy implementation of hybrid architectures : you can have your code use the local model first and, in case of errors, unacceptable latency, or the need for a very large model, temporarily delegate to a paid external API. This way, you maintain flexibility without sacrificing the cost savings and control that local inference provides.

Performance, typical problems and how to fine-tune them

Once you have the model up and running and the server exposed, it's time to grapple with real-world performance and common errors . The first thing you'll usually encounter are "Insufficient Memory" or "Out of Memory" messages when trying to load models that are too large for your RAM/VRAM. The solution typically involves closing resource-intensive applications , choosing a more aggressive quantization (for example, Q4 instead of Q8), or simply reducing the model size.

If you've opted to use the GPU or iGPU of your Ryzen AI MAX+ and notice the system repeatedly reverting to CPU-only mode, there's likely a driver or GPU detection issue . On Windows, ensure your NVIDIA/AMD drivers are up to date and that your tool (LM Studio, Ollama, etc.) has GPU acceleration enabled in its settings. On Linux, check ROCm, environment variables, and logs to verify that the GPU is actually being detected.

Another source of headaches is corrupted or incomplete model downloads . If a model fails to load or freezes immediately upon launch, sometimes simply deleting it from the tool (e.g., `ollama rm`, delete folder in LM Studio) and downloading it again is enough. Keep in mind that a 30 or 40 GB model that's only partially downloaded isn't going to magically start.

In CPU-only environments, the most common complaint is slow generation . This is normal: LLMs are resource-intensive. To improve this, focus on smaller, well-quantized models, increase the number of threads the tool can use, and ensure the CPU isn't thermally throttling. Sometimes a change of thermal paste or an improvement in case ventilation makes a bigger difference than you might think.

If you're looking to serve an LLM to other devices in your home or office, don't forget to also review your network and security settings . Exposing the API at 0.0.0.0 means any device on the network can reach your server, which is useful but carries risks. You can restrict access with firewalls, separate networks, or application-level authentication, especially if you handle sensitive data or if your workstation is exposed to the internet.

Beyond these basic issues, more serious deployments for startups or enterprises require consideration of monitoring, structured logging, model backups, and update strategies . Playing around at home is not the same as building a commercial product where models become a critical business component.

In short, serving a large language model from a local workstation involves combining good hardware, the right tools, and some careful configuration . With a modern CPU, sufficient RAM, a well-supported GPU or iGPU, and platforms like Ollama or LM Studio, you can build your own private AI infrastructure, accessible from all your devices, with fine-tuned control over costs, privacy, and latency that 100% cloud solutions can't always match.