|
 |
Local AI Is a Routing Problem, Not a Privacy Feature
I’ve had a longstanding interest in local AI for a simple reason. I would rather not send proprietary code, unpublished plans, or knowledge about how a business operates to an outside model provider when I have a viable alternative. I want more control over where that information is processed.
A company does not need a model on every employee’s laptop to achieve that. A properly secured internal server can keep information away from outside providers. I admit there’s a bit of wishcasting here. But if my friends are any guide, I’m far from the only one who wants a capable model living on my own laptop.
What has changed is that more tasks can now run on individual devices. As local models improve, I’ve started thinking more seriously about what belongs on a laptop or phone, what needs an internal server, and what still calls for an outside provider. Privacy drew me to local AI. I think the more consequential question is where each task belongs.
The Device Becomes a Place to Run AI
Local AI is already appearing in consumer products. Supported Pixel phones can summarize recordings on the device, while Windows can generate live captions without an internet connection. People use these features without downloading a model or running an inference server. That is part of what makes them useful. Developers can build similar features with open-source projects such as Moonshine Voice, which supports on-device transcription and voice interaction.
 |
I expect more business applications to follow. Tools for running local models can select suitable hardware, download models, and provide interfaces similar to hosted AI services. Developers no longer have to assemble every component themselves.
The hardware is improving too, but I would be careful about generalizing. Microsoft’s Project Zenith targets developer-class PCs with at least 64 GB of memory and 250 GB/s of memory bandwidth. That tells us something about where powerful laptops are heading. It does not mean the typical company laptop can run the same workloads.
 |
There is also a cost to using computers people already own. Models compete with employees’ applications for memory and processing power. Laptops sleep and disconnect. A fleet is worth considering as AI capacity, but it is not a free or continuously available computing cluster.
Which Work Goes Where?
I would start with well-defined tasks whose results are easy to check. Transcription, document classification, local search, and focused coding assistance are candidates for laptops. Shared business data or demanding, always-available applications may call for an internal server. Outside providers remain an option when information is approved for external processing.
An evaluation of local coding models found that narrow assignments with specific instructions worked better than extensive code exploration. Long conversations were less reliable. Phones have tighter memory and heat constraints, making short, specialized tasks a more practical starting point.
 |