- 📱 Over 70% of global smartphones run on Arm CPUs. This means a lot of AI can run on these devices.
- 🔌 KleidiAI enables LLM inference under 1GB RAM, even on Raspberry Pi 5.
- ⚡ ExecuTorch allows mobile-optimized, quantized LLMs to run serverlessly.
- 🔒 On-device AI significantly improves user privacy and reduces latency.
- 🌍 Local LLMs help more people get digital access for underserved and offline communities.
Generative AI on Arm CPUs: Is It Really Possible?
You no longer need expensive GPUs or always-on cloud services to use generative AI. Thanks to new ideas in edge computing, tools like ExecuTorch and KleidiAI let developers run small large language models (LLMs) right on Arm CPUs. These are the same low-power chips in Android phones, Raspberry Pi devices, and billions of other small systems worldwide. If you build AI bots for places with no internet, or if you have a small budget, or you just want fast, local AI, this new way of doing things changes how and where AI can be used.
Traditional LLMs Weren’t Designed for the Edge
Large language models such as OpenAI’s GPT-4, Meta’s LLaMA, or Google’s PaLM have changed what generative AI can do, but they were made for cloud computers first. Running these models typically requires:
- Advanced hardware: High-performance GPUs like NVIDIA A100 or H100, which use a lot of energy and cost too much.
- A lot of RAM and storage: Models consuming up to hundreds of gigabytes depending on their size.
- Always-on internet: APIs need cloud connection and don't work well with delays.
These needs make it much harder for more people to use AI, especially on devices like phones, factory sensors, in distant places, and on hobby devices. More specifically:
- Limited Resources: Devices like smartphones or Raspberry Pi have little computing power and memory. Most Pis only have 1–8GB of RAM and no GPU to speed things up.
- Cloud Dependence: Using cloud services means slower answers, higher costs, and possible worries about data privacy.
- Thermal and Power Limits: Phone CPUs slow down when they work hard to keep from getting too hot. This makes them not good for long, heavy AI tasks.
So, most older LLMs are stuck behind paywalls, in data centers, and need fast internet. This leaves out developers or users who need smart features right on their devices.
ExecuTorch: Lightweight Runtime for Local GenAI
ExecuTorch is an open-source runtime designed by the PyTorch team to put AI inference on small devices — specifically for phones and small, built-in devices. It’s more than just a simpler PyTorch. It's a system made specifically to bring machine learning models to devices with limits, without giving up too much.
Main Features of ExecuTorch
- Selective Execution: Unlike full-blown PyTorch models, ExecuTorch supports operator fusion and pruning to only run what is needed. This makes it use less power when running.
- TorchScript Compatibility: Models are turned into an intermediate format using TorchScript, making them quicker and lighter when running AI tasks.
- Quantization Support: Changes models to 8-bit or even lower formats, making memory use and computing needs much smaller.
- Transformer-Compatible: LLMs use transformer architecture. ExecuTorch can now handle these, thanks to tools like Torch-MLIR.
ExecuTorch takes big models that use a lot of resources and makes them simpler. This means you can run AI tasks cheaply and without a server, which is great for devices with Arm chips. For developers who used to need cloud AI tools, this creates a new possibility: AI that truly runs on the device.
Optimizing for Arm CPUs With ExecuTorch
Arm CPUs are most common in phones and built-in devices because they use little power, can grow, and work well. Platforms like Android, Raspberry Pi, and IoT controllers run mostly on Arm Cortex-A, Cortex-M, or Cortex-R architectures. Each has different amounts of processing power, instructions they can use, and how well they handle heat.
ExecuTorch uses everything the Arm system can do:
Using Arm’s Hardware Speed-Up
- Neon SIMD Extensions: Arm’s Neon (Single Instruction, Multiple Data) instructions are important for making math operations faster. This is great for the calculations LLMs do.
- Arm Compute Library: A set of programs made to run faster, used to improve performance in computer vision and deep learning.
- Efficient Memory Usage: Quantized tensors mean the same AI tasks can run using much less RAM. This is perfect for devices with 1GB or less memory.
Benefits of Backend Compilers
- Torch-MLIR Integration: A next-gen compiler that makes models much better for use by turning TorchScript into operations a machine can understand.
- Execution Planning: Breaks models into steps that run one after another or at the same time. This allows for ThreadPool and using many cores on a CPU.
All these things together mean models can run in real time (or close to it) even on devices without GPUs. This expands what AI on a device can do.
KleidiAI: Bringing LLMs to Smartphones and Raspberry Pi
ExecuTorch gives the best runtime for models, but KleidiAI helps make LLMs work on devices with very few resources. It makes this possible and efficient. Hugging Face helped make it. It's a custom runtime that gives a simple way to use transformer models like TinyLlama, Phi-2, and Falcon-RW on devices with very little RAM, sometimes as little as 512MB.
Important Performance Details
- Raspberry Pi 5: Can run LLM tasks with only 1GB RAM, using smaller versions of common models.
- Midrange Android Phones (e.g., Pixel 6): Get 2–4 tokens per second for AI tasks. This works for making text, sorting things, and finding out what someone wants.
KleidiAI supports streaming inference and works well with ExecuTorch, giving a system that is fast and easy to move to other places. And what's more, it doesn't need special, proprietary speed-up systems. It runs only on the CPU.
KleidiAI has open-source parts and a modular design. This gives developers making offline bots or small device solutions the tools they need. They don't get stuck with one vendor or rely on hidden systems.
Billions of Devices Could Soon Run GenAI Locally
Arm CPUs are used everywhere. According to Statista, over 70% of smartphones worldwide run on Arm chips. This means over 6 billion devices could now run local LLMs using tools like ExecuTorch and KleidiAI.
Why Local Generative AI is Important
- Cost Efficiency: Users save money each month by not using cloud AI APIs from companies like OpenAI or Google.
- Offline Operations: Devices in country areas or disaster zones can use advanced AI tools even without internet.
- Faster Response Times: When AI runs locally, there are no delays from sending data to far-off servers and getting it back.
- Data Controls and Privacy: Your personal or sensitive data stays only on your device. This lowers risks with rules and data theft.
This offers big chances for areas without good service, specific uses, and business apps that care most about data control or running on their own.
Real-World Use Cases for Edge-Based GenAI
What ExecuTorch and KleidiAI can do is not just an idea. They make possible a new kind of portable AI assistants, learning tools, and factory automation systems.
Key Uses
- Raspberry Pi Bots: Bots made for museums, kiosks, or school lessons that work offline. They are perfect for classrooms with bad internet.
- Multilingual Android Translators: These tools are good for healthcare or customer service. They give local translation even when the internet is down.
- Retail Assistants: Smart checkout systems or stock bots that give quick information without needing to contact main servers.
- Medical Apps in the Field: Local LLMs for finding problems or keeping records. Doctors can use them in places with no phone signal or internet.
These uses cut costs and make things more reliable. They also greatly spread AI to places that cloud solutions ignored before.
Implications for No-Code Platforms Like Bot-Engine
No-code tools like Bot-Engine, Make.com, or GoHighLevel are getting a huge demand for custom, smart automation. Putting LLMs on devices into these systems is a big step forward.
How On-Device AI Makes Automation Platforms Much Better
- Reduced Hosting Requirements: Bots can be made and used without needing more servers.
- Massive Scalability: One model can go with thousands of devices, and running costs stay the same.
- Offline Autonomy: Bots can keep working without needing to connect to cloud services again.
- Enterprise-Ready Privacy: Local AI tasks meet rules (like HIPAA, GDPR) because sensitive talks stay on the device.
This means people and businesses using these platforms can now use AI everywhere, even where there is no Wi-Fi or ethernet.
The Limitations of Today's Local LLMs
The progress is great, but local generative AI has some drawbacks that are good to know about:
- Smaller Models, Less Contextual Power: Fewer parameters mean shorter memory and less understanding of complex user input.
- Slower Inference: Running AI tasks on a CPU, even a well-tuned one, is slower than on a GPU or TPU. This is true especially for tasks that need to process a lot of data quickly.
- Manual Model Tuning: To get the best results, you often need to make models smaller, trim them, or fine-tune them. These tasks need some knowledge of machine learning.
But for tasks that don't need very advanced language understanding, like making summaries, tagging keywords, or understanding commands, local LLMs work well enough.
How to Get Started With LLMs on Arm Devices
You don't need an AI PhD to get started. Follow these steps to set up a local LLM solution:
- Pick a Lightweight Model: Look at Hugging Face's models. Models like phi-2, TinyLlama, or Falcon-RW give good performance for their size.
- Convert to ExecuTorch Format: Use PyTorch’s tools to make models smaller and export them to TorchScript or Torch-MLIR.
- Run It on KleidiAI: Install KleidiAI on devices like Raspberry Pi or Android to run the model well.
- Integrate With No-Code Tools: Put your model into APIs or safe interfaces to connect it to systems like Make.com.
- Test and Fine-Tune: Check how much memory and time it uses. Then, change the model size to fit what you need.
In just a few hours, even small teams or single developers can build strong, offline AI tools.
Innovating for Global Inclusion
Running generative AI on Arm CPUs is more than just a tech achievement. It helps make things fair for everyone. Billions of people in poorer areas or places with bad infrastructure can now use AI tools. Before, only big companies could use these.
Benefits for Everyone
- Bridges the Connectivity Gap: Gives smart tools where relying on the cloud doesn't work.
- Empowers Solopreneurs: Start AI products without paying for server time or API calls.
- Supports Regional Languages: Developers can tune LLMs for local ways of speaking, native languages, or cultural uses.
- Boosts Public Sector Innovation: Small AI can run kiosks, public apps, or emergency tools in projects with tight budgets.
Edge LLMs are an innovation that not only pushes tech forward but brings its good parts to everyone, everywhere.
From Lab to Living Room — AI Is Now Local
More tools like ExecuTorch and KleidiAI mean a big change. Generative AI doesn't have to live in huge cloud groups or cost hundreds of thousands of dollars anymore. Instead, it can be right in your pocket, on your robot, or in a Raspberry Pi working quietly on a shelf.
This move to local AI is not just about the hardware. It's a new way to think about intelligence: always there, private by nature, and needing no server.
AI is not just a service anymore. It's a part, and it's getting built into every device we use.
Citations
Statista. (2023). Share of smartphones with Arm-based chips worldwide from 2018 to 2023. Retrieved from https://www.statista.com/statistics/1264114/global-smartphone-arm-based-processor-usage/
Hugging Face. (2023). phi-2 runs on ExecuTorch + KleidiAI: LLM on edge devices. Retrieved from https://huggingface.co/blog/phi-scrolls
