Apple releases eight small AI language models aimed at on-device use

🌈 Abstract

The article discusses the growing popularity of "small language models" in the world of AI, which can be run on local devices instead of requiring data center-grade computers in the cloud. It focuses on Apple's introduction of a set of tiny source-available AI language models called OpenELM, which are small enough to run directly on a smartphone.

🙋 Q&A

[01] Apple's OpenELM Models

1. What are the key features of Apple's OpenELM models?

• The OpenELM models are a set of tiny AI language models that can run directly on a smartphone, rather than requiring data center-grade computers in the cloud.
• There are 8 distinct OpenELM models, ranging from 270 million to 3 billion parameters.
• The models come in two flavors: "pretrained" (raw, next-token version) and "instruction-tuned" (fine-tuned for instruction following).
• OpenELM features a 2048-token maximum context window and was trained on around 1.8 trillion tokens of data from publicly available datasets.
• Apple's "layer-wise scaling strategy" reportedly allows OpenELM to achieve better performance with fewer training tokens compared to other small language models.

2. What are Apple's goals in releasing the OpenELM models?

• Apple aims to "empower and enrich the open research community" by releasing the source code, model weights, and training materials for OpenELM.
• Transparency is a key goal, as Apple states that the "reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks."
• However, Apple also cautions that the models may produce "outputs that are inaccurate, harmful, biased, or objectionable" due to the publicly sourced datasets used for training.

[02] Comparison to Other Language Models

1. How do the OpenELM models compare to other large language models?

• The largest model in Meta's Llama 3 family includes 70 billion parameters, and OpenAI's GPT-3 from 2020 had 175 billion parameters.
• In comparison, the OpenELM models range from 270 million to 3 billion parameters, making them much smaller.
• Parameter count serves as a rough measure of AI model capability and complexity, but recent research has focused on making smaller AI language models as capable as larger ones were a few years ago.

2. How do the OpenELM models compare to Microsoft's Phi-3 models?

• Like the OpenELM models, Microsoft's Phi-3 models also aim to achieve a useful level of language understanding and processing performance in small AI models that can run locally.
• The Phi-3-mini model features 3.8 billion parameters, which is larger than the smallest OpenELM models but smaller than the largest ones.

[03] Potential Integration into Apple's Products

1. How might Apple integrate the OpenELM models into its consumer devices?

• While Apple has not yet integrated this new wave of AI language model capabilities into its consumer devices, the upcoming iOS 18 update is rumored to include new AI features that utilize on-device processing to ensure user privacy.
• However, the article suggests that Apple may potentially hire Google or OpenAI to handle more complex, off-device AI processing to give Siri a long-overdue boost.