Most of the vision-language models (VLMs) you hear about for autonomous driving win by getting bigger. EM-VLM4AD uses a 235M-parameter T5-Base backbone. DriveLM-Agent goes all the way to 3.96 billion parameters.
A handful of recent papers (TinyDrive, MiniDrive and TS-VLM) push the other way. They show you can get similar question-answering scores with much smaller encoders, adaptors and language models. That’s the thread I picked up for this project:
How good can a driving VQA model be if it has to fit in 50M parameters, total?
Two rules made it a fair fight:
- Frozen weights count. You can’t hide a big pretrained encoder by freezing it.
- No teacher model. No distilling from a large VLM at any stage.
The task: DriveLM VQA
The benchmark is DriveLM on nuScenes. Each sample is six camera images around the car (front, front-left, front-right, back, back-left, back-right) plus a question, and the model writes a free-text answer:
Q: What are objects to the back right of the ego car? A: There are many cars, one traffic cone, and one pedestrian to the back right of the ego car.
The questions are short and fall into a few categories (perception, prediction, planning and behaviour), and they’re scored with the usual captioning metrics: BLEU-4, METEOR, ROUGE-L and CIDEr.
The short, structured questions are part of why small models can compete here. You don’t need a general-purpose chatbot, you need something that looks at the right camera and describes what’s there.
Starting point: TS-VLM
I built on TS-VLM, which had the best scores among the small models. At 35.9M parameters it matches or beats the much larger EM-VLM4AD on all four metrics.
Its key idea is a module called TGSSP (text-guided SoftSort pooling). Rather than treating all six cameras equally, it:
- Encodes each camera view with a small image encoder.
- Scores each view against the question embedding with cosine similarity.
- Uses a SoftSort relaxation to turn those scores into a differentiable soft ranking.
- Applies a learned rank prior to get one relevance weight per view.
So a question about “objects to the back right” can learn to lean on the back-right camera. The weighted views are then handed to a small T5 language model, which generates the answer.
My three changes
I started from TS-VLM-Mini and made three changes. Each came out of a literature review of more than a dozen recent driving-VQA and efficient-VLM papers. I kept an idea only if it passed three filters: there was real evidence behind it, it didn’t make inference more expensive, and it fit in the parameter budget I had left.
1. Keep a little spatial structure (DCT tokens)
TS-VLM collapses each camera view into a single pooled vector. That’s cheap, but it throws away where things are in the image, which matters a lot for “what’s to the left of the truck?” style questions.
My image encoder is MobileViTv2 (width 0.75, ImageNet-1k weights, 2.48M parameters), shared across all six views, and it produces an 8×8 feature map per view. Instead of average-pooling that map down to one vector, I apply a 2D Discrete Cosine Transform and keep the 3×3 lowest-frequency coefficients. That gives:
- 9 tokens per view, 54 tokens total
- each view’s tokens scaled by its TGSSP relevance weight before concatenation
The DCT is a fixed transform with no learned weights, so the whole change costs only about 0.15M parameters. It’s the same trick JPEG uses: most of the useful structure lives in the low frequencies.
2. A deeper language model
I swapped google/t5-efficient-mini (4 encoder + 4 decoder layers) for google/t5-efficient-mini-nl6 (6 + 6 layers, same 384-dim hidden size, same 32K SentencePiece vocabulary). That costs 9.45M extra parameters and is the single biggest spend in the budget.
3. A free grounding signal (training only)
This one is my favourite. About 60% of DriveLM questions contain a ground-truth object reference right in the text, something like <c1,CAM_FRONT,714.2,486.7>, which gives the camera and pixel coordinates of the object being asked about.
That’s free supervision that the original model ignores. So I added a tiny two-layer MLP grounding head (74K parameters). It takes the referenced camera’s visual token, predicts the object’s normalised \((u, v)\) pixel location, and adds that error to the training loss with a small fixed weight:
\[\mathcal{L} = \mathcal{L}_{\text{LM}} + \lambda \, \mathcal{L}_{\text{ground}}\]The head only runs during training. At inference it’s skipped entirely, so it nudges the visual tokens toward encoding where things are without costing anything once deployed.
The budget
| Component | Parameters |
|---|---|
| MobileViTv2-0.75 image encoder | 2.48M |
| Fusion module (TGSSP + DCT tokens + projection) | 1.62M |
| T5-efficient-mini-nl6 language model | 40.66M |
| Grounding head (training only) | 0.07M |
| Total | 44.84M |
That leaves about 5M of headroom under the 50M cap.
Results
TS-VLM has no public code, so the first step was reimplementing TS-VLM-Mini from the paper, filling in the gaps with my own assumptions. Then I applied my three changes and retrained. Everything was trained for 15 epochs on a single RTX 3090. The test-set numbers come from one evaluation on the held-out split, using the checkpoint with the lowest validation loss (epoch 12).
| Model | Params | FLOPs | BLEU-4 | METEOR | ROUGE-L | CIDEr |
|---|---|---|---|---|---|---|
| EM-VLM4AD-Base | 235M | 9.47B | 45.36 | 34.49 | 71.98 | 3.20 |
| MiniDrive-224 | 83M | 5.9B | 49.70 | 36.30 | 73.30 | 3.28 |
| TinyDrive v1.2 | 31.8M | 6.50B | 55.82 | 50.56 | 69.50 | 3.02 |
| TS-VLM-Mini (paper) | 35.9M | 17.73B | 54.77 | 40.52 | 74.06 | 3.34 |
| TS-VLM-Mini (my reproduction) | 35.3M | 7.30B | 50.66 | 38.08 | 74.53 | 3.365 |
| Mine | 44.84M | 8.26B | 51.94 | 38.71 | 74.46 | 3.347 |
Other rows are taken from their papers. I measured FLOPs for my own models with fvcore (one forward pass, batch size 1, a typical question length).
Against my own TS-VLM-Mini reproduction, which is the only apples-to-apples comparison, the three changes give:
- BLEU-4: +1.28
- METEOR: +0.63
- ROUGE-L: −0.07
- CIDEr: −0.018
So it’s a mixed result, not a clean win. The n-gram precision metrics clearly improve, while ROUGE-L and CIDEr are essentially flat (slightly behind, but well within what I’d expect from run-to-run noise).
Compared with the much bigger models, it’s a good place to be. At about a fifth of EM-VLM4AD’s size, it beats it on every metric by a wide margin.
Things worth noticing
Reproductions don’t match papers. My from-scratch TS-VLM-Mini disagrees with the paper’s reported numbers by a few points on every metric: lower on BLEU-4 and METEOR, slightly higher on ROUGE-L and CIDEr. Without public code, it’s impossible to tell which undocumented details cause the gap. That’s why I compare my changes only against my own reproduction, not against the paper’s numbers.
FLOPs are hard to compare across papers. The TS-VLM paper reports 17.73B FLOPs. My reproduction of the same architecture measures 7.30B. That’s almost certainly a difference in how FLOPs were counted, not in real compute. Most papers don’t say exactly how they count, so treat the FLOPs column as rough.
The metrics disagree with each other. TinyDrive has by far the best BLEU-4 and METEOR but the worst ROUGE-L and CIDEr in the table. Captioning metrics reward different things (exact n-gram overlap, synonyms, longest common subsequence, consensus with references), and a model can be tuned toward one family. “State of the art” depends a lot on which column you look at.
What’s next
- Ablations. Right now I only know the combined effect of the three changes. I want to switch each one on separately (DCT tokens only, deeper T5 only, grounding head only) to see which one is actually responsible for the BLEU-4/METEOR gains, and whether one of them is costing me on CIDEr.
- Category-specialised adapters. Perception, prediction and planning questions are quite different. Small per-category adapters fit easily in the roughly 5M parameters I have left.
- Metric-aware fine-tuning. Optimising more directly for CIDEr-style objectives, which has worked well in image captioning.
The broader takeaway matches what TinyDrive, MiniDrive and TS-VLM found: for a narrow domain like driving VQA, picking the right camera and keeping a bit of spatial structure buys you more than raw scale. A 45M-parameter model trained on one consumer GPU comfortably beats a model five times its size.
If you’re working on small VLMs or driving VQA and want to compare notes, leave a comment below.