Running large AI models locally usually means buying a lot of expensive memory. A new open-source project called Edge0 gets around that by predicting, one step ahead, which parts of the model it will need next - so it can pull them off a hard drive fast enough to keep up.
The model in question is a 35-billion-parameter mixture-of-experts (MoE) system, a design built from many smaller sub-networks that only activates a handful of them per word it generates. That sparsity should make MoEs cheap to run, but it doesn't shrink the file: even compressed to 4-bit precision, the weights still take up 19.5GB, more than fits comfortably in memory on typical consumer hardware. The catch is that you don't know which sub-networks a layer needs until the previous layer has finished computing, so simply streaming weights from disk normally stalls the whole process. Edge0's fix is a small prediction module that guesses the next layer's routing a token in advance and then treats that guess as the actual decision, while a bolt-on adapter claws back the accuracy lost to compression and prediction. The result: 20 tokens per second on a single 24GB machine, using just 3GB of active memory, and scoring within a few points of the uncompressed model across five public benchmarks.
That's a meaningfully bigger model running on meaningfully smaller hardware than most local-AI setups manage, and the code, checkpoints, and adapters are all released openly. For anyone trying to run large open models without renting a GPU cluster or paying per-token API fees, this closes a real gap between what labs ship and what a laptop can actually hold in memory.
Worth noting: this is a fresh arXiv preprint, not a peer-reviewed or independently benchmarked release, and the whole approach hinges on the routing predictor guessing correctly - the paper doesn't say how often it's wrong, or what happens to speed and accuracy when it is.