Local AI Experience on M1 Mac: Why the Expected Performance Wasn't Achieved?

Local AI Experience Pushes Hardware Limits

As artificial intelligence technologies rapidly advance, interest in running these systems locally on personal computers is also growing. However, recent experiences show that this process can be more challenging than expected, especially on devices with older or mid-range hardware.

Local Model Trial with Ollama

ZDNET senior contributing writer Tiernan Ray attempted to run a local AI model on his M1 chip MacBook Pro using the open-source Ollama tool. Despite the 2021 model device with 16 GB RAM and 1 TB storage performing flawlessly in daily use, it proved inadequate under AI workloads.

Ray selected the 30-billion parameter GLM-4.7-flash model developed by Chinese AI startup Z.ai. It was noted that the model occupies 19 GB on disk and is considered 'small'. Although the download process reached speeds of 45 MB/s on a gigabit internet connection, serious performance issues were encountered during the model execution phase.

Unexpected Slowness and Hardware Challenges

The model took a full hour and sixteen minutes to respond to the first query. It was observed that the model 'thought' for 5,197.3 seconds to generate a response to the question "What kind of large language model are you?". During this process, a noticeable slowdown occurred in the MacBook's overall performance.

The experiment revealed that running local AI models requires current and high-capacity hardware. Experts recommend at least 36 GB RAM in next-generation machines for similar attempts. This situation also explains why cloud-based solutions like the Moltbot AI Assistant are more widely adopted.

Advantages and Disadvantages of Local AI

Using local AI models offers several key advantages. These include processing sensitive data without sending it to the cloud, avoiding paid API usage costs, and providing greater control over the model. However, these advantages are balanced by high hardware requirements.

While more optimized options like OpenAI's open-source GPT-OSS:20b model are said to run faster, the local AI experience is generally observed to be far from user-friendly. These developments also explain why cloud-based solutions like OpenAI's new projects and Google's Chrome integrations stand out.

Expectations for the Future

Hardware manufacturers developing next-generation processors optimized for AI workloads could improve the local AI experience. However, for now, average users are expected to continue preferring cloud-based solutions. This situation also brings along privacy and security concerns, similar to those in wearable technology devices.

Technology experts emphasize that using local AI remains a valuable skill for professional development, but attempting it without suitable hardware can lead to disappointment. It is noted that more optimization is needed on both the software and hardware sides for the democratization of AI technologies.