How to Train YOLOv5 on a Custom Dataset

Last Updated on 19/11/2025 by Eran Feit

Getting Ready to Train YOLOv5 on a Custom Dataset

Training YOLOv5 on a custom dataset is one of the most powerful ways to move from “toy examples” to real-world computer vision solutions. Instead of relying only on generic datasets like COCO, you can train yolov5 on a custom dataset that reflects your exact problem: boats in aerial footage, vehicles in traffic cameras, products on store shelves, or anything else you care about. This gives you far more control over the classes you detect and the environments your model will see in production.

At a high level, the goal is simple: collect or download images, label them with bounding boxes, organize them into the YOLO format, and then fine-tune YOLOv5 on this data. Behind that simple idea there’s a clear pipeline: preparing the Python environment, choosing the right YOLOv5 model size, configuring the data.yaml file, and monitoring the training metrics such as loss and mAP. Each of these steps makes the difference between a model that “kind of works” and a model that is robust enough to trust.

When you train YOLOv5 on a custom dataset, you also design the structure of your project. You decide how many classes you want, how to split your images into train/validation/test, and how to name and store them. A clean folder layout and a correct data.yaml file mean that YOLOv5 can load your data reliably and report separate performance metrics for each class, so you instantly see which objects are easy and which are still challenging for the model.

Finally, training is not the end of the journey. Once the model has learned from your custom images, you still need to run inference, visualize predictions, and iterate. By comparing metrics like precision, recall, and mAP, and by visually inspecting detections on new images or videos, you can decide whether to collect more data, rebalance classes, or tweak hyperparameters. This full loop—from environment setup to training and evaluation—is what turns YOLOv5 into a practical, end-to-end solution for your specific use case.

If you are completely new to YOLO models, you might first enjoy my YOLOv5 Image Classification — Complete Tutorial, which explains the core ideas of YOLOv5 in a simple classification setting before moving into object detection.

Why training YOLOv5 on a custom dataset changes everything

When you train YOLOv5 on a custom dataset, you move from generic demos to a model that truly understands your world.
Instead of detecting only COCO classes, you can create detectors for boats, docks, forklifts, or any niche objects you care about.
This is the difference between playing with models and actually solving a real business or hobby problem.

The core idea is simple.
You prepare a dataset in YOLO format, point YOLOv5 to your data.yaml file, pick a model variant, and start training.
Behind this simple loop there is a clean pipeline of environment setup, dataset download, configuration, and monitoring during training.

A well organized dataset is just as important as a powerful model.
By keeping a clear folder structure for train, valid, and test, and by describing your classes correctly in data.yaml, you give YOLOv5 exactly what it needs to learn fast.
Good structure also makes it easy to share your project or move it later to another machine or cloud environment.

Once training is complete you use the resulting weights to run inference on new images or videos.
By comparing the original image with bounding boxes and labels from YOLOv5 you can quickly see what works and where the model still struggles.
This feedback loop is the heart of every strong computer vision pipeline.

If you want to generate labels faster for future projects, take a look at Boost Your Dataset with YOLOv8 Auto-Label Segmentation, where I show how to use YOLOv8 to automatically label video frames for training.

Link for the video tutorial : https://youtu.be/nEIhzAX81JU

Code for the tutorial here : https://eranfeit.lemonsqueezy.com/buy/ee38aefc-cf14-4749-beb1-724195b42919

or here : https://ko-fi.com/s/5982955852

Link for Medium users : https://medium.com/object-detection-tutorials/how-to-train-yolov5-on-a-custom-dataset-346b7c9db2b0

You can follow my blog here : https://eranfeit.net/blog/

 Want to get started with Computer Vision or take your skills to the next level ?

If you’re just beginning, I recommend this step-by-step course designed to introduce you to the foundations of Computer Vision – Complete Computer Vision Bootcamp With PyTorch & TensorFlow

If you’re already experienced and looking for more advanced techniques, check out this deep-dive course – Modern Computer Vision GPT, PyTorch, Keras, OpenCV4

Setting up a clean YOLOv5 environment for training

Before you can train YOLOv5 on a custom dataset, you need a stable environment where PyTorch, CUDA, and all required libraries work together smoothly.
Using a dedicated Conda environment keeps this project isolated from other Python setups on your machine.
This section walks through creating the environment, cloning the YOLOv5 repository, and installing all needed dependencies.

### Create a new Conda environment dedicated to YOLOv5 with Python 3.8. conda create --name YoloV5 python=3.8  ### Activate the YOLOV5 Conda environment so all next commands run inside it. conda activate YoloV5  ### Choose or create a folder on your disk where you want to keep your YOLOv5 projects, for example C:/cool-Python-stuff. ### Navigate into that folder before cloning the repository. cd C:/cool-Python-stuff  ### Clone the official YOLOv5 repository from GitHub into the current folder. git clone https://github.com/ultralytics/yolov5.git  ### Move into the YOLOv5 project folder so you can run its scripts. cd yolov5  ### Install PyTorch, Torchvision, and Torchaudio with CUDA 11.8 support from the official channels. conda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=11.8 -c pytorch -c nvidia  ### Install GitPython so YOLOv5 can access Git information when needed. pip install gitpython>=3.1.30  ### Install Matplotlib for plotting training curves and visualizing results. pip install matplotlib>=3.3  ### Install a compatible NumPy version for numerical operations. pip install numpy>=1.22.2  ### Install OpenCV for image and video processing in Python. pip install opencv-python>=4.1.1  ### Install Pillow to handle image loading and basic image operations. pip install Pillow>=10.0.1  ### Install psutil to monitor system resources such as CPU and memory usage. pip install psutil  ### Install PyYAML so YOLOv5 can read configuration files like data.yaml. pip install PyYAML>=5.3.1  ### Install Requests to handle HTTP requests inside the project if needed. pip install requests>=2.23.0  ### Install SciPy for additional scientific and numeric utilities. pip install scipy>=1.4.1  ### Install thop to compute the number of FLOPs and parameters for the model. pip install thop>=0.1.1  ### Install tqdm to get nice progress bars during training and inference. pip install tqdm>=4.64.0  ### Install the Ultralytics package so you can also use the newer YOLO interface if needed. pip install ultralytics>=8.0.147  ### Install TensorBoard to visualize training metrics in an interactive dashboard. pip install tensorboard  ### Remove opencv-python-headless if it exists to avoid conflicts with the regular OpenCV package. pip uninstall -y opencv-python-headless  ### Reinstall opencv-python to make sure the correct version is active. pip uninstall -y opencv-python pip install opencv-python>=4.1.1 

This code block creates a reliable YOLOv5 workspace and prevents the most common environment issues.
Once it finishes successfully you are ready to focus on your custom dataset.

Downloading and organizing the Aerial Maritime Drone dataset

To train YOLOv5 on a custom dataset, you first need images and labels in YOLO format.
In this tutorial you use the Aerial Maritime Drone dataset from Roboflow, which contains boats, docks, cars, and jetskis captured from above.
The goal is to place the dataset under a simple path on your disk and keep a folder layout that YOLOv5 expects.

After downloading the dataset as a YOLOv5 PyTorch ZIP file, you extract it into a data folder such as C:/Data-Sets.
Then you rename the dataset folder to a short and clear name like Aerial-Maritime so it is easy to type in your paths.
Inside that folder you will see train, valid, and test subfolders, each with images and labels.
The next step is to make sure the data.yaml file correctly points to these folders and lists all your classes.

### Define the full path to the training images folder on your machine. train: C:/Data-Sets/Aerial-Maritime/train/images  ### Define the full path to the validation images folder on your machine. val: C:/Data-Sets/Aerial-Maritime/valid/images  ### Set the number of object classes that your custom dataset contains. nc: 5  ### List the class names exactly in the same order as they appear in your label files. names: ['boat', 'car', 'dock', 'jetski', 'lift'] 

This small configuration file is the bridge between your dataset and YOLOv5.
When it is correct, YOLOv5 can automatically load images, labels, and class names for training and evaluation.

You should keep the folder structure tidy because it makes debugging much easier.
A typical layout looks like this, where each split has its own images and labels directory.

### Top level dataset folder with configuration and documentation files. Aerial-Maritime/ ├── data.yaml ├── README.dataset.txt ├── README.roboflow.txt ├── test │   ├── images │   └── labels ├── train │   ├── images │   └── labels └── valid     ├── images     └── labels 

Once your dataset looks like this and the YAML file uses full paths, you are ready to start training YOLOv5 on this custom aerial maritime data.

Training YOLOv5 on the Aerial Maritime custom dataset

Training is where YOLOv5 actually learns how to detect your objects.
You tell the training script where to find data.yaml, which model weights to start from, how large the images should be, and how many epochs to run.
In this example you run a medium sized model (yolov5m.pt) for 250 epochs and save all results under your dataset folder.

Before running the command, make sure you are inside the main YOLOv5 folder in your Conda environment.
You can also adjust batch-size based on your GPU memory if needed.
During training you will watch the loss values go down and the mAP metric go up, which indicates the quality of your custom model.

### Navigate to the main YOLOv5 project folder where train.py is located. cd C:/cool-Python-stuff/yolov5  ### Run the training script using the medium YOLOv5 model as a starting point. ### Point --data to your custom data.yaml file and --name to an output folder for results. python train.py --data C:/Data-Sets/Aerial-Maritime/data.yaml --weights yolov5m.pt --img 640 --epochs 250 --batch-size 16 --name C:/Data-Sets/Aerial-Maritime/All-results 

While training runs, pay attention to box_loss, obj_loss, and cls_loss in the logs.
These numbers should generally decrease as the model improves.

At the same time, look at the mAP@0.5 and mAP@0.5:0.95 metrics in the console or in TensorBoard.
They should go up over time, showing that YOLOv5 is learning to detect your boats, cars, docks, jetskis, and lifts more accurately.

At the end of training YOLOv5 will save two key weight files under your results folder.
last.pt contains the final state of the model after the last epoch, and best.pt stores the best performing checkpoint according to validation mAP.
In most cases you will use best.pt for inference because it usually gives you the highest accuracy on unseen images.

For another perspective on training deep detectors on custom data, you can read How to Train Detectron2 on Custom Object Detection Data, which walks through a similar custom-dataset pipeline using Detectron2 and Faster R-CNN.

Running inference with your trained YOLOv5 model

Once you have a trained model, it is time to test it on real images and see the predictions.
YOLOv5 includes a detect.py script that takes a weights file, a source image or folder, and an output name where predictions will be stored.
You can point this script to any new image of boats and docks to verify that your custom model works as expected.

In the command below you load best.pt from the training run, use a specific test image from the dataset, and write results into a new result folder.
YOLOv5 will draw colored bounding boxes with class labels like boat or jetski and save the output image to disk.
You can open this image in any viewer to quickly judge whether the detections are good enough or if you need more training or data.

### Run YOLOv5 inference using the best performing weights from your custom training. python detect.py --weights C:/Data-Sets/Aerial-Maritime/All-results/weights/best.pt --source C:/Data-Sets/Aerial-Maritime/test/images/DJI_0262_JPG.rf.5f24b2ccccf544d3bb0c3cb740be0f4b.jpg --name C:/Data-Sets/Aerial-Maritime/result 

You can change --source to a folder path to run detection on all images inside that folder.
You can also pass a video path or even a webcam index to see live detections.

This is the moment where everything connects.
You collected data, defined a clean structure, trained YOLOv5 on your custom dataset, and now you see the model detecting your objects in real images.

If you enjoy working with modern detectors, you may also like Getting Started with YOLOX for Object Detection and SSD MobileNet v3 Object Detection Explained for Beginners, which give you more options for real-time object detection in Python.

FAQ

Conclusion

Training YOLOv5 on a custom dataset turns a generic detection model into a focused tool tailored to your own images.
By building a clean environment, downloading the Aerial Maritime dataset, and wiring a simple data.yaml file, you gave YOLOv5 everything it needs to understand boats, cars, docks, jetskis, and lifts from aerial views.

The training command connects these pieces into a single pipeline.
You watched losses go down and mAP go up, and then used best.pt to run inference and visually inspect predictions on new images.
This is the practical loop that powers almost every successful computer vision project.

From here you can extend the workflow in many directions.
You can add more classes, collect additional images in challenging conditions, or fine tune hyperparameters for even higher accuracy.
You can also combine this detector with segmentation, tracking, or analytics models to build complete end to end systems.

Whether you are monitoring boats in a harbor, counting vehicles in a parking lot, or detecting equipment in an industrial site, the pattern stays the same.
Prepare strong data, train YOLOv5 on your custom dataset, validate carefully, and keep iterating.
The more you repeat this loop, the more powerful and reliable your models will become.

Connect :

☕ Buy me a coffee — https://ko-fi.com/eranfeit

🖥️ Email : feitgemel@gmail.com

🌐 https://eranfeit.net

🤝 Fiverr : https://www.fiverr.com/s/mB3Pbb

Enjoy,

Eran