Can Two Macs Run Cutting-Edge AI? Realizing a Dream Distributed Setup Through "Role Separation"
Introduction
In the world of AI, balancing text generation speed and quality is one of the greatest challenges. Currently, NVIDIA is drawing massive attention for proposing an groundbreaking mechanism called the "TwoTower" architecture to solve this issue.
According to NVIDIA’s reports, this technology has successfully accelerated inference speed by 2.42x compared to traditional AI while maintaining 98.7% of the quality. Furthermore, it boasts high practicality, as it can be introduced by training on just 8% of existing AI data without needing to retrain the AI from scratch.
However, to actually run this TwoTower architecture, the model size reaches about 60 billion parameters (requiring about 126GB of storage), necessitating an environment that is impossible for individuals: a machine equipped with two ultra-high-performance 80GB-class GPUs (like NVIDIA H100 or A100).
Therefore, as a purely technical hypothesis, I considered the following idea:
"If we make the relationship between Diffusion LLMs and AR LLMs ‘loosely coupled,’ couldn’t we build a pseudo-TwoTower just by connecting two 64GB MacBook Pros via a LAN?"
This article is an explanation that organizes whether this dream-like idea could actually work, based on the technical mechanics of NVIDIA’s TwoTower.
1. What is TwoTower in the First Place?
There are two major approaches to how AI generates text.
- AR (Autoregressive) Model: Predicts the continuation of the previous word, one word at a time. Accurate but slow.
- Diffusion Model: A technique used in image generation AI. It creates a rough block of words from noise or masks and refines them. Fast because it can create multiple words at once, but context tends to collapse.
The currently mainstream AR model requires reading massive data from GPU memory every time it generates a single word. As a result, the time spent "waiting for data movement" is longer than the "calculation time," which has been the bottleneck causing slow speeds.
The idea behind TwoTower is to take the "best of both worlds" from AR and Diffusion models to eliminate this bottleneck.
Technical Mechanics of TwoTower
NVIDIA’s TwoTower is a mechanism that builds two independent "towers" within a single massive AI model (the backbone).
- AR Context Tower (The Proofreader):
It leverages the existing knowledge of the AR model as-is to understand the context. This tower is "completely frozen" during training, meaning there is no need to retrain it from scratch. - Diffusion Denoiser Tower (The Drafter):
This is the newly trained tower. It receives blocks of words containing noise and refines them to create candidate sentences.
Because the drafter creates multiple words at once, the number of processing steps decreases, achieving dramatic speedups.
2. Impossible for Individuals? The Wall of "Tight Coupling"
So why can’t the current TwoTower be run in a personal environment? It is because the two towers are "tightly coupled" (stuck closely together).
According to NVIDIA’s TwoTower paper, when the Diffusion denoiser tower generates text, it continuously references the internal data of the frozen AR tower via a technique called "cross-attention" at every single layer (hierarchy).
To use human work as an analogy, it’s like the drafter directly accessing the proofreader’s brain data to check their work every time they write a single word.
To do this on a computer, ultra-high-speed data communication (hundreds to thousands of GB per second) occurs inside the AI model at every step.
Therefore, it requires special ultra-high-speed NVIDIA cables (NVLink) and two data-center-grade GPUs equipped with ultra-high-performance memory (HBM) of the 80GB class. Personal Macs simply do not have such ultra-high-speed communication capabilities.
3. The Solution: Completely Separating Roles via "Loose Coupling"
Let’s flip the perspective here. What if we stop the "tight coupling" where they reference each other at every layer, and make it "loosely coupled" (a looser relationship)?
Using human work as an analogy again, we change the system to: "The drafter creates a draft of about 16 to 64 words before handing it over to the proofreader."
If the "context data" created by the AR tower is fixed as a single bundle and passed to the Diffusion tower, ultra-high-speed communication is no longer necessary. A standard LAN cable or a Thunderbolt cable connecting two Macs should be fast enough for the communication.
4. Dream Setup Image Using Two Macs
If this "loose coupling" works well, we could coordinate two Macs in the following configuration.
Configuration Image
- Mac A (Proofreader / AR Role): Reads the prompt (instruction) and generates data that understands the context.
- Mac B (Drafter / Diffusion Role): Receives the data from Mac A as a condition and generates a batch of 16 to 64 word candidates at once.
Workflow
- Mac A summarizes the current context data and sends it to Mac B.
- Mac B generates multiple word candidates at once and sends them back to Mac A.
- Mac A selects the correct sentences from the candidates and finalizes them (final verification).
- Repeat.
In short, it is an approach of "distributing work roles across two machines" rather than "splitting the AI model in two".
5. Three Conditions and Challenges for Realization
For this configuration to work, several conditions must be met.
- Communicate in Bundled Quantities
Communicating word-by-word will cause freezes due to latency, so data is exchanged in "block" units of "16 to 64 words." - "Fixate" Intermediate Data
Instead of passing data back and forth at every layer, the context data created by Mac A is fixated once, and Mac B uses that fixed data. - Tolerance for Strict Sequentiality (Trade-off)
NVIDIA’s paper also points out that parallel generation by Diffusion models loses accuracy in tasks requiring strict left-to-right order dependency, such as "Python code generation" or "mathematical reasoning." By making the system loosely coupled, this tendency is likely to become stronger.
6. Why It Could Still Work
Despite technical hurdles (orders of magnitude difference in communication speed and latency), there are reasons why this two-Mac setup could still work.
- Diffusion Models Only Need "Approximations"
Diffusion generation is good at producing "mostly correct (approximate)" candidates rather than hitting the exact right answer. - AR Handles Final Verification
Ultimately, Mac A’s AR model selects the correct answer from the candidates, so the final quality of the text is guaranteed even if the draft is somewhat rough. - Affinity with Self-Speculative Decoding
In another NVIDIA study, a method called "Self-Speculation" is proposed, where the same AI model acts as both the "drafter" and the "verifier." Dividing these roles between two Macs is theoretically very sound.
Conclusion
To summarize the idea of this article:
If the relationship between AR and Diffusion models can be designed as "loosely coupled," it may be possible to establish a highly advanced distributed LLM using just two Macs, without expensive data-center GPUs.
At this stage, this is purely a "hypothesis" and has not been actually programmed and run. Also, because it omits the precise coordination of "cross-attention at every layer" inherent to the original TwoTower, some degradation in accuracy is unavoidable.
However, as AI models continue to grow in size, the transition from an era of "doing everything on a single supercomputer" to an era of "distributing roles across multiple machines" may be an unavoidable path.
If any engineers are intrigued by this idea, I highly encourage you to actually verify it!