📱 Small Language Models: Survey, Measurements, and Insights

This research paper reviews small language models (SLMs), which are optimized for use on devices with limited resources, such as smartphones. It covers recent advancements in SLM architectures, training datasets, and algorithms, and benchmarks their performance on tasks like commonsense reasoning, problem-solving, and mathematics. The paper also assesses the models' runtime efficiency on various hardware platforms and the effect of quantization techniques on performance. The authors highlight future research areas, including co-designing SLM architectures with device processors, creating high-quality synthetic datasets, and developing scaling laws that account for deployment constraints.

📎 Link to paper