Why your company may need a small language model, not a bigger LLM
For narrow, high volume work, a small model trained on your own data can be cheaper, faster and safer than the biggest model money can buy. Here is how to decide, and how to build one.

Every few months a bigger, smarter large language model (LLM) arrives. It is tempting to plug it into everything. But most of the AI work inside a company is not open ended reasoning. It is the same few jobs, done thousands of times a day: read this email and route it, pull these fields out of this form, answer this question from our policy, tag this ticket.
For work like that, a small language model (SLM), usually a few billion parameters instead of hundreds of billions, trained or tuned on your own data, often does the job as well as a giant model. Sometimes better.
LLM or SLM: the honest comparison
| Large model through an API | Small model you own | |
|---|---|---|
| Best at | Open ended reasoning, writing, broad knowledge | One narrow task, done the same way every time |
| Cost per request | Higher, and grows with every token | Much lower once it is running |
| Speed | Depends on the provider | Fast, and you control it |
| Data | Leaves your network | Can stay on your servers or in your cloud |
| Setup effort | Hours | Weeks |
| Control | Provider can change the model | You decide when it changes |
The short version: use a large model when the task is broad and the volume is low. Use a small model when the task is narrow and the volume is high.
Five signs you should build an SLM
- The task is repetitive and well defined. Classification, extraction, routing, summarising into a fixed format, answering from a known set of documents.
- Volume is high. Thousands or millions of requests a month, where API bills grow with every new customer.
- Data cannot leave your walls. Banks, insurers, hospitals and government teams often cannot send customer data to an outside API. In India, the Digital Personal Data Protection Act and sector regulators make this a board level question.
- Speed matters. Live chat, voice agents and in app features need answers in a fraction of a second.
- You need it to behave the same way every time. A model you own does not change underneath you after a provider update.
If none of these apply, a good prompt on a large model is probably the right answer for now.
How we build a small model for a company
1. Pick one task and write the test first. Collect a few hundred real examples with the correct answers. This evaluation set is the most important thing you will build, because it tells you whether the small model is good enough.
2. Measure a large model on that test. This is your baseline. It tells you the best quality you can expect, and what it costs today.
3. Choose an open model as the starting point. Families like Llama, Gemma, Mistral, Qwen and Phi come in sizes from about 1 billion to around 14 billion parameters. Smaller is cheaper to run. Start with the smallest that comes close to your baseline.
4. Prepare the training data. Use your own history: past tickets, emails, chats, forms, with personal data removed. Where you do not have enough, a large model can generate and label more examples for you. This is called distillation: the big model teaches the small one.
5. Fine tune it. Techniques such as LoRA let you tune a model on a single GPU in hours, not weeks, by training a small set of extra weights instead of the whole model.
6. Test it against the baseline. Run the same evaluation set. If the small model matches the large one on your task, you have a winner. If not, add data and tune again.
7. Shrink it and serve it. Quantisation makes the model smaller and faster with little loss in quality. Serve it with tools like vLLM or Ollama, on your own servers, in your cloud, or even on a laptop or phone.
8. Watch it and retrain. Log every answer, review the mistakes, and add them to the training data. A small model gets better with each round.
The best setup is usually both
You rarely have to choose only one. The pattern we recommend most often is a router: a small model handles the common, predictable requests, and anything unusual or complex is passed to a large model. Most of the traffic, and most of the cost, goes through the cheap path, while the hard cases still get the best model available.
Where to start
Pick the single most repetitive AI task in your company, the one with the biggest bill or the strictest data rules. Build the evaluation set this week. Within a month you will know, with numbers, whether a small model can take it over.
If you want help scoping it, talk to us. We can help you pick the task, build the test and decide with real numbers.

