Exo AI software: home cluster for large language models

Running large language models (LLMs) locally is often associated with massive machines, expensive graphics cards, and inefficient power consumption. However, in recent months an alternative has emerged that is attracting considerable attention: Exo, software that transforms several home devices into an AI cluster capable of running very demanding models without relying on the cloud.

With Exo, the idea is to pool the power of computers, laptops , mobile phones, and other connected devices so that, by combining their resources, they can run LLM (Local Lifecycle Management) applications that we would normally only see in professional data centers. This opens the door to setting up a kind of "mini data center" at home, ideal for projects like local assistants such as Home Assistant, SMEs that want to experiment with AI without breaking the bank, or curious developers who want to go beyond typical cloud testing.

What is Exo and what does it aim to solve?

Exo is an open-source , experimental software maintained by Exo Labs, whose main goal is to allow you to set up your own AI cluster at home using everyday devices. Instead of relying on a single GPU with a lot of VRAM, Exo distributes the processing power across multiple devices connected to the same network.

Exo's premise is simple yet powerful : your everyday devices—a MacBook, an iPhone , an iPad, a Linux PC, even certain single-board computers (SBCs)—can collaborate to run a large AI model. Each contributes memory and computing power, and the software orchestrates everything to function as if it were a single, larger machine.

Exo was born out of a very common limitation : not everyone has access to a high-end GPU like an NVIDIA RTX 3090 with 24 GB of VRAM and tens of gigabytes of RAM. Running a "modest" model like Llama 3.1 70B with decent local performance requires hardware that is beyond the budget of many developers, makers, or small businesses.

Instead of forcing you to invest in a massive piece of equipment or relying solely on the cloud, Exo proposes combining the available memory of several smaller devices so that the complete model fits within the cluster. This significantly democratizes access to experimentation with advanced AI, which is especially appealing to startups and SMEs.

How Exo works on a technical level

The core of Exo lies in its ability to distribute LLM models across all devices on a home or office network. Instead of loading the entire model onto a single machine, Exo breaks it down and distributes it, leveraging the combined memory and computing power of each node.

To achieve this, Exo uses a distributed shared memory approach , adapting the portion of the model that is placed on each device based on the amount of memory (RAM and, on macOS, GPU VRAM) available. Thus, a model that would normally require a very powerful GPU can run across laptops, desktops, and even other devices, provided the total cluster memory is sufficient.

The process begins with installing Exo on compatible devices . In many cases, this requires Python 3.12 or higher, as well as other tools such as a dependency manager and compilers. Once the service is running, devices running Exo automatically discover each other within the same network, eliminating the need to manually configure IPs or ports.

Internal communication between nodes is based on peer-to-peer connections , which favors an architecture where each machine contributes what it can without depending on a single, rigid central server. Based on this topology, Exo performs a resource and latency analysis to decide how to partition the model and where to place each fragment.

One of the key features is dynamic model partitioning . Exo gains a real-time view of the cluster topology, the resources of each device (memory, CPU, GPU), and the quality of the network links (latency, bandwidth between nodes). With this information, it automatically decides how to divide the model to balance the load and minimize bottlenecks.

Main technical features of Exo

The Exo project incorporates several features designed to maximize the available hardware and make life easier for the user. Among the most notable are:

Automatic device discovery : Devices running Exo find each other automatically without you having to touch your network settings. This avoids the typical nightmare of dealing with static IPs, manual ports, or custom scripts to get nodes to see each other.

RDMA support over Thunderbolt : One of the most striking features is that Exo comes with day-one compatibility for RDMA (Remote Direct Memory Access) over Thunderbolt 5. This approach allows for latency reduction between devices of up to 99% compared to more traditional communication methods, which is critical when distributing parts of the same model among multiple machines.

Tensor parallelism : Exo implements tensor parallelism techniques , meaning it divides the model's tensors across multiple devices. Thanks to this, in certain configurations, speed improvements of up to 1,8 times using two devices and up to 3,2 times with four have been reported, compared to running the model on a single device with the same resources.

Topology-aware and auto-parallelization : The system automatically calculates the best way to partition the model and distribute it across the cluster, taking into account both the power of each node and the quality of the connections between them. This "topology-aware auto-parallel" feature reduces the need for manual adjustments by the user.

MLX integration and GPU support on macOS : On macOS, Exo uses MLX as its inference backend and MLX Distributed for distributed communication. This allows it to take advantage of the GPUs integrated into Apple Silicon chips, which is especially useful on devices like the latest generation Mac Studio or MacBook Pro.

Practical use cases and performance on Macs

Exo has been tested in very demanding scenarios by combining several high-end Macs, taking advantage of both tensor parallelism and RDMA over Thunderbolt 5. In these setups, gigantic models have been run that we would normally only see in data center environments.

One example discussed in the community is the use of four Mac Studio M3 Ultra processors connected via Thunderbolt 5 and RDMA. Together, they provide approximately 15 TB of effective VRAM to support incredibly large desktop models.

Among the models that have been run in these types of setups are Qwen3-235B (8-bit) , DeepSeek v3.1 671B (also 8-bit quantized) , and even Kimi K2 Thinking (native 4-bit) . In all these cases, each Mac hosts a portion of the model, and communication between nodes is optimized using RDMA.

These experiments have been documented by creators like Jeff Geerling , who has shown how it's possible to set up a home configuration with a huge amount of effective video memory. In this scenario, Exo acts as the orchestration layer that allows the model to function coherently despite being distributed across multiple machines.

Beyond these extreme configurations, the same principle can be applied to more modest uses , for example, to power a large language model that serves as the brain of an advanced home automation system, a private conversational assistant, or text analysis tools within a small business.

Using Exo with Home Assistant and advanced home automation

In the Home Assistant community, Exo is already being discussed as an option for locally running large LLMs that can serve as the reasoning engine for home automation. Although many people haven't yet tried it directly, the project has garnered attention because it promises to allow very resource-intensive models on relatively modest hardware , provided enough hardware is grouped together.

The idea is that, instead of relying on a powerful GPU or the cloud , you can set up a cluster with multiple computers, even if they don't have dedicated GPUs. Exo would handle distributing the model across all of them, allowing Home Assistant to consult a local LLM to make more complex decisions or provide more natural responses.

This approach is especially appealing to those who prioritize privacy , as household data doesn't have to leave the local network. All inference and natural language processing is performed within the cluster you've set up, without relying on external servers.

The project has garnered attention on tech blogs like CNX Software, which highlights that Exo can even run on computer clusters, smartphones, or development boards . However, in practice, the most robust support is currently found on macOS and Linux, with some limitations still present on iOS.

For many home automation enthusiasts, the Exo + Home Assistant combination could be the missing piece to have an advanced, always-on, and completely local conversational assistant, without paying for cloud tokens or relying on third parties.

Compatible AI models and limitations with proprietary models

Exo is compatible with a good number of popular AI models , especially those that are part of the open ecosystem and can be downloaded and run locally. These include families such as LLaMA, Mistral, LLaVA, Qwen, and DeepSeek, among others.

In the specific case of Meta's Llama 3 , Exo works particularly well with these architectures, allowing the use of different-sized variants depending on the cluster's resources. This makes it easier for both individual experimenters and small businesses to have a modern language model without incurring licensing costs or depending on third-party APIs.

Another interesting advantage is the ability to expose a ChatGPT -compatible API . With some adjustments, Exo allows models running on your cluster to accept requests in a format similar to the OpenAI API , which greatly simplifies integration with applications that already work with that standard.

However, there is a clear limitation when we talk about proprietary models like GPT-4 . GPT-4 is a closed model, hosted on OpenAI's infrastructure and not available for download. This means that, by definition, it cannot be run on Exo or any other local environment outside of OpenAI.

Therefore, when working with Exo, you should stick to open-source models or those available for local use . For most practical applications (chatbots, assistants, text analysis, basic and advanced reasoning), there is already a sufficiently powerful catalog of open-source models, especially with the recent evolution of models like Llama 3.x, Qwen, and DeepSeek.

Main advantages of using an Exo cluster

Exo's biggest advantage is that it allows you to drastically reduce your hardware investment . Instead of having to buy a workstation with a massive GPU, you can reuse computers and devices you already have at home or in the office. Many users could set up a cluster simply by leveraging their Macs, an older PC, and other underutilized equipment.

This approach democratizes access to advanced AI , enabling startups and SMEs with tight budgets to compete in a league previously reserved for well-resourced corporations. Not all projects justify renting expensive GPUs in the cloud, and that's where a local cluster makes perfect sense.

Furthermore, Exo is very flexible in terms of device types . Although the most mature support is currently on macOS (using GPUs) and Linux (currently on CPUs), the project aims to integrate other types of hardware as well and tactically improve compatibility. This ability to mix different machine profiles allows you to design the cluster according to your specific needs.

Another key advantage is that, thanks to its distributed architecture, the cluster can be scaled relatively easily . If you need more performance at any point, simply add another device running Exo. Each new node will be automatically detected and its resources added to the total, without you having to redesign the system from scratch.

Finally, dynamic partitioning and topology awareness ensure that even machines with modest hardware can contribute something useful. While an older laptop won't run the entire model on its own, it can handle some of the workload or perform some of the calculations, contributing to overall performance.

Current limitations and points to consider

Not everything is perfect, and Exo is still clearly experimental software . This means that bugs, unstable behavior, and frequent changes in how the system is configured and run may still occur. It's not an "install and forget" solution like a mature commercial product.

On iOS devices like iPhones and iPads, support is still limited and requires further development to become truly usable for the general public. While the project envisions leveraging these devices as well, currently the majority of the smooth experience is found on computers.

Another key factor is that the overall efficiency of the cluster will depend heavily on your home network . If you have congested Wi-Fi, outdated routers, or slow connections, distributing a large model across multiple nodes can become a significant bottleneck. In serious environments, it's almost a given that you'll use high-quality wired connections or, in the case of Macs, Thunderbolt 5 with RDMA.

Furthermore, while the combined memory of many modest devices can achieve impressive figures, they won't always compete in raw performance with a cluster of high-end GPUs . There's a delicate balance between coordination costs, network latencies, and effective computing power that must be considered.

Nevertheless, Exo's value proposition remains very interesting : transforming existing equipment, at virtually zero hardware cost, into a cluster capable of running models that until recently seemed exclusive to specialized data centers.

Exo installation, requirements and execution methods

To run Exo on macOS, it is common to use tools such as Homebrew for package management, as well as uv for Python dependency management, macmon for monitoring hardware on Apple Silicon chips, Node.js for building the web control panel, and Rust (currently in nightly version) for compiling certain bindings.

The typical workflow on macOS involves cloning the official GitHub repository , installing the dependencies, and running the corresponding command to start Exo. This launches both the API and a web dashboard accessible locally at http://localhost:52415 , from which you can view the cluster status, models, instances, and more.

In addition to the terminal version , Exo offers a native macOS application that runs in the background. This app requires macOS Tahoe 26.2 or later and is distributed as a .dmg file (for example, as EXO-latest.dmg). During installation, it may ask for permission to modify certain system settings and add a network profile, which is necessary to enable advanced features like RDMA.

In Linux environments, installation typically involves using the system's package manager (such as apt in Debian/Ubuntu) or Homebrew for Linux if you prefer. Keep in mind that some utilities, such as macmon, are specific to macOS and are not used in Linux.

Currently, on Linux, Exo runs on CPUs only , without full support for GPU accelerators. The development team is working on expanding compatibility with different hardware platforms, so it's recommended to check the issues and feature requests on GitHub if you want to know if your specific device might be supported in the future.

Configuring RDMA over Thunderbolt on macOS

One of the most striking new features in the latest versions of macOS is the addition of RDMA support, especially over Thunderbolt 5 ports. This feature is essential for minimizing latency between physically connected Macs, which is very valuable when distributing an LLM model.

RDMA is available starting with macOS 26.2 and works on computers with Thunderbolt 5 , such as the Mac mini M4 Pro, Mac Studio M4 Max, MacBook Pro M4 Max, and Mac Studio M3 Ultra. However, in the specific case of the Mac Studio, you should avoid using the Thunderbolt 5 port located next to the Ethernet port, as it has certain limitations for RDMA.

To activate RDMA, you need to follow a process in Recovery Mode . The basic steps are to shut down your Mac, start it up while holding the power button for about 10 seconds until the startup menu appears , choose the "Options" option to enter the recovery environment, and open Terminal from the Utilities menu.

Within the Recovery Terminal, execute the command specified in the Exo documentation to enable system-level RDMA. After pressing Enter and restarting the Mac, the functionality is activated, and Exo can use it to establish very low-latency links between different Macs connected via Thunderbolt 5.

From there, Exo handles the complex part , leveraging RDMA to minimize delays in data transfer between nodes. This improvement in internal communication is especially important when working with models containing hundreds of billions of parameters fragmented across multiple machines.

Interact with Exo via API and web panel

In addition to the graphical dashboard at http://localhost:52415 , Exo exposes a REST API that allows you to manage virtually the entire lifecycle of models: choosing where to place them, creating instances, launching chat requests, and releasing resources when they are no longer needed.

A typical workflow might begin by querying the /instance/previews endpoint . By sending the parameters of the model you want to deploy, this service returns all valid placements for that model in your cluster . This is essentially a list of possible "deployment plans" based on the current topology and resources.

Based on the preview's response, you choose the layout that best suits your needs (for example, the first one in the list, which you can select using tools like jq from the terminal) and use it to send a POST request to the /instance endpoint . The payload of this request must conform to the types defined in CreateInstanceParams, including parameters such as the model, hardware configuration, and allocated nodes.

Once the instance is created, you can interact with it using an OpenAI API-compatible endpoint , such as /v1/chat/completions . The JSON request structure is virtually the same as that used with the ChatGPT API, making it very easy to adapt existing applications to point to your Exo cluster instead of the cloud.

When you no longer need a specific model, you can release it by sending a DELETE request with the instance ID , which you can find by querying endpoints like /state or /instance . Additionally, it's helpful to have quick access to signals like /models , which lists all locally available models, or /state itself, which details the current deployment status.

Project status and development community

Exo is hosted on GitHub under the repository exo-explore/exo and is developed openly, with contributions from both the core Exo Labs team and the community. The CONTRIBUTING.md file outlines the guidelines for contributing, submitting pull requests, reporting bugs, and proposing new features.

Since this is a live and constantly evolving project , new versions, compatibility improvements with hardware models and platforms, and changes in how certain components are configured are common. Therefore, it's advisable to frequently review the documentation and open issues.

Regarding accelerator compatibility, macOS currently enjoys more advanced support thanks to the use of the GPU and MLX , while Linux currently relies on the CPU. The development team has encouraged users interested in seeing support for new types of hardware to open or support existing requests with reactions on GitHub, so they can prioritize which platforms are most in demand.

From an economic standpoint, Exo is completely free and open source . There are no paid licenses associated with using the software, which makes it a great fit for personal, educational, or small business projects that want to seriously experiment with AI without a large budget.

Exo is establishing itself as a key player in the democratization of advanced AI , enabling anyone with multiple devices at home or in the office to set up their own language model cluster. While it still has a way to go, the possibilities it opens up are vast for home automation, personal assistants, data analytics, and testing environments for modern models.