Small Language Models: Edge AI That Fits in Your Pocket

The AI That Runs Without Internet

The hiking app needed trail information. But hikers don't have cell service in the backcountry. Cloud APIs weren't an option.

We deployed a 1B parameter model directly on the phone. 50ms inference. Zero API costs. Works in airplane mode. The model fits in 500MB - smaller than most game assets.

Small Language Models (SLMs) unlock use cases cloud AI can't touch: offline operation, guaranteed privacy, zero latency, and no per-query costs.

Why Small Models?

BenefitCloud LLMOn-Device SLM Latency500ms-2s<100ms PrivacyData leaves deviceData stays local CostPer-query feesZero marginal cost OfflineRequires internetWorks anywhere

> If you only remember one thing: SLMs trade capability for deployment flexibility. They won't match GPT-4, but they run anywhere.

Model Selection Guide

Phi-3-mini (3.8B): Best quality/size ratio for general tasks

Llama-3.2-1B: Ideal for mobile - fits in 500MB quantized

Gemma-2B: Strong for code and structured output

Qwen-1.5-0.5B: Ultra-small for constrained devices

Quantization: Making It Fit

4-bit quantization shrinks models 4x with ~5% quality loss:

ModelFP16 Size4-bit SizeQuality Loss Phi-3-mini7.6GB1.9GB~3% Llama-3.2-1B2GB500MB~5%

> Pro tip: Test quantized models on YOUR tasks. Quality loss varies by use case. Structured output tasks lose more than generation.

Deployment Options

llama.cpp: CPU-optimized, works everywhere

ONNX Runtime: Cross-platform with hardware acceleration

MLX: Apple Silicon optimized (M1/M2/M3)

TensorRT: NVIDIA GPU maximum performance

Mobile Optimization

> Watch out: Memory is the constraint, not compute.

Profile actual memory usage - it exceeds model file size

Cache KV pairs to avoid recomputation

Batch inference when possible

Use streaming output to feel faster

Best Practices Checklist

[ ] Profile on target hardware - Dev machine ≠ production device

[ ] Quantize appropriately - 4-bit for mobile, 8-bit for desktop

[ ] Cache KV pairs - Avoid recomputing context

[ ] Test quality - Quantization affects different tasks differently

[ ] Plan for memory - Leave headroom for OS and other apps

FAQ

Q: Can SLMs replace cloud APIs?

For some tasks, yes. FAQ answering, text classification, simple generation - SLMs handle these well. Complex reasoning still needs larger models.

Q: How do I choose between models?

Benchmark on your actual use case. General benchmarks don't predict your specific performance.

Q: What about fine-tuning SLMs?

Absolutely viable. QLoRA makes it efficient. Fine-tuned small models often beat generic large models on narrow tasks.