How I Fine-Tuned a Vision Language Model for Drone Image Understanding

A few months ago, if someone had asked me to fine-tune a Vision Language Model (VLM), I probably would have spent the next hour browsing the internet and reading articles just to understand where to begin.

Fast forward to the project that I built during my summer break, and I found myself working on exactly that.

My project was to fine-tune a Vision Language Model that could understand drone images and answer questions related to them. Questions like:

• How many cars are visible?
• Is there a truck in this image?
• What is the most common object?

I thought it was going to be easy. It wasn't.

The Problem

Modern VLMs are incredibly capable, but most of them are trained on everyday images like people, pets, buildings, food, and street scenes. Drone imagery is a completely different world.

Objects are tiny.

The camera looks down from above.

Multiple vehicles can be packed into a small region.

Lighting and viewing angles are different.

A model that can perform well on normal photographs doesn't automatically perform well on aerial imagery.

The goal of my project was to bridge that gap.

Creating the Dataset

One of the biggest surprises was that there wasn't a ready-made Visual Question Answering dataset for drone imagery that was fit for this project.

Instead of collecting thousands of question-answer pairs manually, I thought of a different strategy.

I used the VisDrone2019 object detection dataset and converted its annotations into natural language question-answer pairs.

For each image, I programmatically generated questions such as:

• "How many cars are there?"
• "Is there a truck present?"
• "What is the most common object?"
• "How many objects are there in total?"

In the end, I was able to generate thousands of training examples without writing every question manually.

Making It Work on Limited Hardware

Like many students, I didn't have access to expensive GPUs.

I trained everything using Kaggle's free Tesla T4 GPUs.

That meant every design decision mattered.

Instead of full fine-tuning I used QLoRA, a parameter-efficient fine-tuning technique that updates only a tiny fraction of the model's parameters while keeping the rest frozen. This made it possible to fine-tune a multi-billion parameter model within the limited GPU memory available.

I guess that the job of a good engineer is to work within the given constraints, and produce the optimal result.

The Journey

The project wasn't just about training a model.

There were times when training became painfully slow, memory errors appeared out of nowhere, learning became unstable, and one unfortunate mistake even corrupted my trained adapter.

Each problem forced me to step back, investigate what was actually happening, and understand the system instead of blindly changing parameters.

Looking back, I learned more from the debugging sessions than the training itself.

The Results

After multiple iterations, the final model became noticeably better at answering questions about drone images.

It learned to answer existence questions reliably and became significantly better at counting objects compared to the original model.

The improvement wasn't perfect counting dense scenes remains challenging, but seeing a general-purpose Vision Language Model adapt to an entirely different visual domain was incredibly rewarding.

What I Learned

This project changed the way I think about AI.

Before this, I mostly viewed machine learning through tutorials and research papers. This was the first time I experienced what it actually feels like to build an end-to-end system from creating the dataset to training the model, debugging unexpected issues, evaluating results, and documenting everything.

More importantly, I realized that successful AI projects aren't just about choosing the right model. They're about understanding data, working around hardware limitations, designing reliable pipelines, and patiently solving one problem after another.

Final Thoughts

This project gave me my first opportunity to work on Vision Language Models, and it was one of the most rewarding learning experiences I've had.

There's still a long way to go before models can perfectly understand complex aerial imagery, but seeing a general-purpose VLM learn a completely new domain through fine-tuning was exciting to witness.

If you're a student thinking that projects involving large AI models require expensive hardware or massive resources, I'd encourage you to give them a try anyway. With the right techniques and a bit of persistence, you can build surprisingly capable systems using nothing more than freely available tools and a willingness to keep learning.

Sometimes, the best projects aren't the ones that work perfectly they're the ones that teach you the most.