The Beginner’s Guide to Small Language Models

Recently, small language models have emerged as an interesting and more accessible alternative to their larger counterparts. What are they, how do they work, and how do they compare to LLMs?

By:Arthur Team

March 29, 2024

Large language models have been top of mind since OpenAI’s launch of ChatGPT in November 2022. From LLaMA to Claude 3 to Command-R and more, companies have been releasing their own rivals to GPT-4, OpenAI’s latest large multimodal model.

However, because large language models are so immense and complicated, they are often not the best option for more specific tasks. Say, for instance, that you need to cut a piece of paper. You could use a chainsaw to do so, but in reality, that level of intensity is completely unnecessary. Scissors would work just as well in this case—probably even better.

Recently, small language models have emerged as an interesting and more accessible alternative to their larger counterparts. In this blog post, we will walk you through what small language models are, how they work, the benefits and drawbacks of using them, as well as some examples of common use cases.

What Are Small Language Models and How Do They Work?

While small language models are similar to large language models in that they have complex architectures and are trained on trillions of tokens of data, one of the main differences is that SLMs are less computationally intensive—they run smaller and faster. An LLM as a computer file might be hundreds of gigabytes, whereas many SLMs are less than five.

This smaller size and efficiency is achieved via a few different techniques including knowledge distillation, pruning, and quantization. Knowledge distillation transfers knowledge from a pre-trained LLM to a smaller model, capturing its core capabilities without the full complexity. Pruning removes less useful parts of the model, and quantization reduces the precision of its weights, both of which further reduce its size and resource requirements.

Unsurprisingly, there is a tradeoff here. It’s been observed that small language models cannot be generalists in the way that the largest of the large language models can. SLMs are more task-specific, and can only really be effective if they are prompted and fine-tuned for a specific job. (More on this below!)

Benefits and Drawbacks of Small Language Models

Like we mentioned above, there are some tradeoffs to consider when opting for a small language model over a large one. But, there are also a number of ways in which SLMs are superior to LLMs.

Compared to LLMs, SLMs are:

• More Lightweight & Efficient: This makes them more suitable for situations where computational resources are limited or where real-time inference is required.
• More Accessible: SLMs lower the barrier to entry for people who want to experiment with language models. Anyone who has access to a laptop or mobile device can train and deploy an SLM, whereas training and deploying an LLM would likely require expensive cloud services or specialized hardware.
• Better for Domain-Specific Tasks: SLMs can be fine-tuned for specific domains or tasks, resulting in better performance and understanding within those particular areas. One example of this could be organizing/extracting data from documents a company just received. You can use tools like Arthur Bench to make an SLM good enough for that data job.
• More Secure: Since SLMs have smaller codebases and fewer potential surfaces for security breaches, they are also less vulnerable to malicious attacks.
• Better for the Environment: SLMs use less energy and memory than LLMs, which makes them more environmentally friendly and sustainable. They also have smaller footprints and faster inference times, which makes them more suitable for edge computing and real-time applications.

Most of the drawbacks of SLMs stem from the fact that they have fewer parameters, making them less flexible and generally useful than LLMs. This can cause problems like:

• Lower levels of accuracy and nuance in responses due to a limited understanding of language and context
• Reduced performance on complex tasks
• Being more prone to bias
• Limited creativity and originality
• Difficulty with following instructions and learning long-term dependencies

Examples of Small Language Model Use Cases

Some examples of popular small language models are Mistral’s 7B, Microsoft’s Phi-2, and Google’s Gemma.

As far as use cases go, small language models are often used in applications like chatbots, virtual assistants, and text analytics tools deployed in resource-constrained environments.

Another use case might be data parsing/annotating, where you can prompt an SLM to read from files/spreadsheets. It can then (a) rewrite the information in your data in the format of your choice, and (b) add annotations and infer metadata attributes for your data.

The Future of Small Language Models

What small language models might lack in size, they more than make up for in potential. They have the ability to greatly impact the AI landscape. In a world where AI has not always been equally available to everyone, they represent its democratization and a future where AI is accessible and tailored to diverse needs.

With improvements in training techniques, hardware advancements, and efficient architectures, the gap between SLMs and LLMs will continue to narrow—which will open doors to new, exciting applications and allow AI development to reach a broader population.

FAQ

What are the common architectures used in Small Language Models?

Small Language Models often utilize architectures like Transformer, LSTM, or Recurrent Neural Networks, but with a significantly reduced number of parameters compared to Large Language Models. Some popular SLM architectures include distilled versions of GPT, BERT, or T5, as well as models like Mistral's 7B, Microsoft's Phi-2, and Google's Gemma. These architectures are designed to balance performance, efficiency, and accessibility.

How do Small Language Models compare to Transfer Learning approaches?

While Small Language Models and Transfer Learning are both techniques to make language models more accessible and efficient, they differ in their approach. Transfer Learning involves fine-tuning a pre-trained large language model on a specific task or dataset, whereas Small Language Models are trained from scratch using knowledge distillation, pruning, and quantization to create a more lightweight and task-specific model. SLMs can often outperform transfer learning approaches for narrow, domain-specific applications due to their enhanced focus and efficiency.

What are the typical hardware requirements for deploying and running Small Language Models?

One of the key benefits of Small Language Models is their reduced hardware requirements compared to Large Language Models. Typically, SLMs can be run on standard laptop or desktop computers, often requiring only a few gigabytes of RAM and basic GPU acceleration. This makes them much more accessible for deployment in resource-constrained environments, edge devices, or personal computing setups, where the computational and memory demands of large models would be prohibitive. The lightweight nature of SLMs opens up a wider range of real-world applications and democratizes access to advanced language AI capabilities.