How to Train YOLOv8 Object Detection on a Custom Dataset : Cards detection

Last Updated on 25/11/2025 by Eran Feit

When you train YOLOv8 on custom dataset, you turn a general-purpose object detector into a specialist that understands exactly the objects you care about.
Instead of relying on COCO’s people, cars, and dogs, you can teach YOLOv8 to recognise things like playing cards, medical instruments, or products on a shelf with high speed and accuracy.

The core idea is simple: you collect your own images, label the objects with bounding boxes, and then fine-tune YOLOv8 on this data. During training, the model learns how your objects look under your lighting, your camera angles, and your backgrounds. That’s why training on a custom dataset almost always outperforms using a generic pre-trained model for real projects.

In a cards-detection example, you might gather hundreds or thousands of images of different decks, suits, and card orientations. With consistent YOLO-format annotations, YOLOv8 can learn to detect each card and distinguish one class from another, even when cards overlap slightly or appear at odd angles. The same approach extends naturally to any domain where you can define clear classes and provide good labels.

By the time you’ve finished the full pipeline—environment setup, dataset download, label visualisation, training, and testing—you’ll have a model that doesn’t just detect “some objects,” but understands your specific problem. That’s the real power of learning how to train YOLOv8 on custom dataset: you gain a reusable workflow you can apply to new ideas again and again.

If you enjoy learning how to train YOLOv8 on custom dataset for cards, you might also like my guide on building a YOLOv8 dental object detection model, where we detect teeth in dental X-rays with a similar training pipeline.

What It Really Means to Train YOLOv8 on a Custom Dataset

Training YOLOv8 on a custom dataset starts with defining a clear target: what exactly should the model detect, and in which conditions? For cards detection, that might be individual card faces on a table, in someone’s hand, or partially occluded in a stack. Once the target is clear, you design your dataset so it reflects the real-world situations you expect the model to handle, including different backgrounds, lighting conditions, and card orientations.

Next comes annotation, which is where the “intelligence” in your model truly begins. Every image needs accurate bounding boxes and class labels in YOLO format, stored as simple text files. These labels tell YOLOv8 exactly where each object is and what it represents. For cards, that might mean separate classes for each rank and suit, or more general classes like “card front” and “card back,” depending on your use case. Good bounding boxes and consistent class naming are critical for stable training and strong results.

Once your data is organised into train/val folders and a data.yaml file, the training step connects everything. The YAML file tells YOLOv8 where the images and labels live and lists all class names. With that in place, you load a YOLOv8 model, point it to your dataset, and configure key settings: image size, batch size, number of epochs, device (CPU or GPU), and where to save the experiment. Under the hood, YOLOv8 adjusts millions of parameters so its predictions get closer and closer to your ground-truth boxes on each training step.

Finally, you evaluate and visualise what the model has learned. Running inference on test images lets you compare YOLOv8’s predictions against your original labels: for example, by drawing bounding boxes for predicted cards on one image and ground truth on another. This side-by-side view helps you quickly spot over- and under-detections, misclassified suits, or cards the model consistently misses. With that feedback, you can improve your dataset, tweak training parameters, and iterate until the detector is robust enough for real use.

Let’s Walk Through the YOLOv8 Cards Detection Code Together

This tutorial is built as a complete, end-to-end example of how to train YOLOv8 on custom dataset for a real project: detecting playing cards in images. Instead of just showing a few isolated lines, the code is organised as a practical workflow you can reuse in your own computer vision work. You start from a clean Conda environment, move through installing PyTorch with CUDA support and the Ultralytics YOLOv8 package, and finish with a trained model that can spot cards in new images and draw bounding boxes around them.

The first part of the code focuses on setting up the tools you need to work efficiently. You create a dedicated Conda environment, install a compatible version of PyTorch with CUDA 11.8, and then add the ultralytics package that provides the YOLOv8 implementation. This ensures that when you run the training code later, your GPU is actually being used and the model can train at a reasonable speed. For many beginners, this environment step is the main barrier, so having the commands clearly laid out removes a lot of friction.

Next, the code turns to the dataset. You download a playing cards dataset exported in YOLOv8 format and store it in a standard folder structure with train, valid, and test images and labels. A small OpenCV script then loads a single training image together with its .txt label file, converts YOLO coordinates into pixel coordinates, and draws the bounding boxes on top of the image. This part is crucial: it lets you visually confirm that your labels and classes are correct before you invest time in training the model.

The core training script then ties everything together. It loads the YOLOv8n model definition, points it to your data.yaml file, and sets training parameters like epochs, batch size, image size, and device. When you run it, YOLOv8 trains on your custom playing card images and saves the best weights in a project folder. Finally, a separate testing script loads the saved best.pt model, runs inference on test images, and compares the predicted boxes to the ground truth labels. You end up with two visual outputs: one image showing model predictions and another showing the original annotations, making it easy to judge how well your custom card detector is performing and where you might want to improve your dataset or training settings.

Link for the video tutorial : https://youtu.be/lw6tn3nHaj8

Code for the tutorial : https://eranfeit.lemonsqueezy.com/buy/024dd090-3a14-4d53-98ce-d735c83eac87 or here : https://ko-fi.com/s/2defcaa482

Link to the dataset : https://universe.roboflow.com/augmented-startups/playing-cards-ow27d/dataset/4

Link for Medium users https://medium.com/object-detection-tutorials/how-to-train-yolov8-object-detection-on-a-custom-dataset-cards-detection-5d99bf849987

You can follow my blog here : https://eranfeit.net/blog/

 Want to get started with Computer Vision or take your skills to the next level ?

If you’re just beginning, I recommend this step-by-step course designed to introduce you to the foundations of Computer Vision – Complete Computer Vision Bootcamp With PyTorch & TensorFlow

If you’re already experienced and looking for more advanced techniques, check out this deep-dive course – Modern Computer Vision GPT, PyTorch, Keras, OpenCV4

How to Train YOLOv8 Object Detection on a Custom Dataset: Cards Detection

Training YOLOv8 on a custom dataset is one of the fastest ways to turn a generic object detector into a specialist for your own project.
Instead of detecting only COCO classes like people, cars, or dogs, you can fine-tune YOLOv8 so it understands playing cards, medical objects, products on shelves, or anything else you can label.

In this tutorial, we’ll train YOLOv8 on a custom dataset of playing cards exported from Roboflow in YOLOv8 format.
You’ll see how to create a clean Conda environment, install PyTorch with CUDA and the Ultralytics package, download and organise the dataset, visualise YOLO labels with OpenCV, train the model, and then test its performance on new images.

The goal of the code is simple but powerful.
By the end, you’ll have a complete pipeline: from raw images and label files, through training a YOLOv8 model, to visualising both predictions and ground-truth bounding boxes side by side.
This same pattern can be reused for any custom dataset you want to train YOLOv8 on in the future.

Setting up a clean YOLOv8 Conda environment

Before you train YOLOv8 on custom dataset, it’s important to isolate dependencies in a dedicated Conda environment.
This prevents version conflicts and makes it easy to repeat the setup on another machine.

In this first block, you’ll create and activate a Conda environment, verify your CUDA toolkit, install a CUDA-enabled PyTorch stack, and then add the Ultralytics package that provides YOLOv8.

### Create a dedicated Conda environment named YoloV8 with Python 3.8 so all YOLOv8 dependencies stay isolated from your other projects.
conda create --name YoloV8 python=3.8

### Activate the new YoloV8 environment so every subsequent install and command runs inside this clean setup.
conda activate YoloV8

### Check the installed CUDA toolkit version to confirm that nvcc is available and matches the version you plan to use with PyTorch.
nvcc --version

### Install PyTorch 2.1.1, torchvision, torchaudio, and the CUDA 11.8 build so YOLOv8 can use your GPU for faster training.
conda install pytorch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 pytorch-cuda=11.8 -c pytorch -c nvidia

### Install the Ultralytics package version 8.1.0 which includes the YOLOv8 implementation and training utilities.
pip install ultralytics==8.1.0

Once these commands complete successfully, you have a solid foundation for GPU-accelerated object detection with YOLOv8.
If anything fails, it’s usually related to CUDA drivers or conflicting PyTorch builds, so double-check CUDA and PyTorch compatibility first.

Getting the playing cards dataset in YOLOv8 format

To train YOLOv8 on custom dataset, you need images and label files in a compatible format.
For this tutorial we’ll use the Playing Cards dataset created by Augmented Startups and hosted on Roboflow Universe, specifically version v4 that already includes a YOLOv8-ready export.

1. Open the dataset page:
   https://universe.roboflow.com/augmented-startups/playing-cards-ow27d/dataset/4
2. Click Download Dataset and choose the YOLOv8 export option.
3. Unzip the download into your data folder, for example C:/Data-sets/Playing Cards/.

After extraction, your folder structure should look like this (simplified view):

data/
    train/
        images/
            001198293_jpg.rf.411db15ce8a9a42a2d51a1885f7592d2.jpg
            ...
        labels/
            001198293_jpg.rf.411db15ce8a9a42a2d51a1885f7592d2.txt
            ...
    valid/
        images/
            ...
        labels/
            ...
    test/
        images/
            ...
        labels/
            ...
    data.yaml

The images folders contain your JPG files, and the labels folders contain YOLO-format .txt files where each line encodes class x_center y_center width height in normalised coordinates.
The data.yaml file tells YOLOv8 where these folders live and lists the class names (card ranks and suits) so training can start immediately.

To speed up dataset creation for other projects, check out how I use YOLOv8 and SAM to auto-label segmentation masks and turn raw video into training data with minimal manual work.

Visualizing one labeled image with OpenCV

Before you train YOLOv8 on custom dataset, it’s smart to verify that labels align correctly with the images.
This section loads one sample image and its annotation file, converts YOLO coordinates into pixel coordinates, and draws bounding boxes with card labels using OpenCV.

### Import the YOLO class from the Ultralytics package so we can later load and train YOLOv8 models.
from ultralytics import YOLO 

### Import OpenCV to handle image loading, drawing bounding boxes, and creating visualizations.
import cv2 

### Import the yaml module so we can read the data.yaml configuration file that stores class names.
import yaml 

### Define the path to a sample training image of playing cards.
img = "C:/Data-sets/Playing Cards/train/images/001198293_jpg.rf.411db15ce8a9a42a2d51a1885f7592d2.jpg"

### Define the path to the corresponding YOLO label file for the same training image.
imgAnot = "C:/Data-sets/Playing Cards/train/labels/001198293_jpg.rf.411db15ce8a9a42a2d51a1885f7592d2.txt"

### Specify the path to the data.yaml file so we can load the list of class names used for training.
data_yaml_file = "C:/Data-sets/Playing Cards/data.yaml"

### Open the YAML file in read mode so we can parse dataset configuration and class names.
with open(data_yaml_file , 'r') as file:
    ### Parse the YAML content into a Python dictionary.
    data = yaml.safe_load(file)

### Extract the list of label names from the YAML dictionary so each class id can be mapped to a human-readable name.
label_names = data['names']

### Print the list of label names to verify that the dataset classes loaded correctly.
print(label_names)

### Read the sample image from disk using OpenCV.
img = cv2.imread(img)

### Extract the image height, width, and channel count so we can convert normalised YOLO coordinates into pixel values.
H, W, _ = img.shape 

### Open the annotation file to read all label lines for this image.
with open(imgAnot, 'r') as file:
    ### Read every line in the label file which represents one bounding box annotation.
    lines = file.readlines()

### Initialize an empty list to hold parsed annotation tuples.
annotations = [] 

### Loop over each line from the annotation file to decode the YOLO bounding box format.
for line in lines:
    ### Split the line into separate values for class id and bounding box coordinates.
    values = line.split()
    ### The first value is the class label index.
    label = values[0]

    ### Convert the remaining values (x, y, w, h) from strings to floating point numbers.
    x, y, w, h = map(float, values[1:])
    ### Append a tuple of label and bounding box parameters to the annotations list.
    annotations.append((label , x, y, w, h))

### Print the list of parsed annotations to quickly inspect the raw YOLO coordinates.
print(annotations)

### Loop through each annotation to draw the corresponding bounding box and label on the image.
for annotation in annotations:
    ### Unpack the stored label index and YOLO bounding box coordinates.
    label , x, y, w, h = annotation
    ### Convert the label index into a readable class name using the label_names list.
    label_name = label_names[int(label)]

    ### Convert the YOLO x center, y center, width, and height into top-left and bottom-right pixel coordinates.
    x1 = int((x - w / 2) * W)
    y1 = int((y - h / 2) * H)
    x2 = int((x + w / 2) * W)
    y2 = int((y + h / 2) * H)

    ### Draw a rectangle on the image using the computed pixel coordinates to highlight the card region.
    cv2.rectangle(img , (x1, y1), (x2, y2), (200,200,0), 1)

    ### Draw the class name slightly above the bounding box so we can see which card was labeled.
    cv2.putText(img , label_name, (x1, y1-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (200,200,0), 2)

### Display the annotated image in a window named "img" so we can visually inspect the labels.
cv2.imshow("img", img)

### Wait indefinitely for a key press so the OpenCV window does not close immediately.
cv2.waitKey(0)

If the boxes perfectly wrap each card, your dataset and data.yaml configuration are correct.
If boxes appear shifted or off, double-check that image paths, label files, and the data.yaml configuration all match the real folder structure.

Training your custom YOLOv8 cards detector

Now you’re ready to actually train YOLOv8 on custom dataset of playing cards.
This script loads the YOLOv8n model, references the data.yaml, and launches training with key hyperparameters such as epochs, batch size, patience, and image size.

### Import the YOLO class from Ultralytics so we can build and train a YOLOv8 model.
from ultralytics import YOLO 

### Define the main entry point of the training script so it can be reused or imported cleanly.
def main():

    ### Load the YOLOv8 nano configuration file so we start from a lightweight model architecture.
    model = YOLO("yolov8n.yaml")

    ### Specify the path to the data.yaml file that describes dataset locations and class names.
    data_yaml_file = "C:/Data-sets/Playing Cards/data.yaml"

    ### Set the project directory where YOLOv8 will create experiment folders and logs.
    project = "C:/Data-sets/Playing Cards"

    ### Define a custom experiment name so you can easily distinguish this run from future trainings.
    experiment = "My-Card-Model"

    ### Choose the batch size for training which controls how many images are processed per iteration.
    batch_size = 32

    ### Start training the YOLOv8 model using your custom playing cards dataset and chosen hyperparameters.
    results = model.train(
                          ### Point YOLOv8 to the dataset configuration file.
                          data=data_yaml_file,
                          ### Train for 50 epochs so the model has enough time to converge.
                          epochs=50,
                          ### Store all outputs inside the specified project directory.
                          project=project, 
                          ### Save this run under the experiment name so it gets its own folder.
                          name=experiment , 
                          ### Use the first GPU (index 0) for training to accelerate learning.
                          batch=batch_size , 
                          ### Enable early stopping after 5 epochs without improvement to save time.
                          device=0 ,
                          ### Train with 640x640 images which is a standard resolution for YOLOv8.
                          patience=5, 
                          ### Print verbose logs to the console so you can follow the progress.
                          imgsz=640 , 
                          ### Run validation at the end of each epoch to monitor model performance.
                          verbose=True ,
                          val=True)

    ### Add a reminder comment about where YOLOv8 will store the run results and best weights on disk.
    # The results will be store in :  C:/Data-sets/Playing Cards/My-Card-Model

### Ensure the main function runs only when this script is executed directly.
if __name__ == "__main__":
    ### Call the main function to kick off YOLOv8 training on your custom playing cards dataset.
    main()

Training will create a folder like C:/Data-sets/Playing Cards/My-Card-Model containing checkpoints, metrics, and the final best.pt weights.
Watch the console for metrics such as mAP and loss to see how well the model is learning.

Once you are comfortable training YOLOv8 on a custom dataset, you can apply similar ideas to live streams, for example in my tutorial on real-time YouTube video object detection with YOLOv8 and Python.

Running inference with your trained YOLOv8 model

After you train YOLOv8 on custom dataset, the next step is to run inference on unseen test images.
This block loads one test image, applies the trained model, and draws bounding boxes and labels for all predictions above a given confidence threshold.

### Import the YOLO class from Ultralytics so we can load the trained YOLOv8 weights for inference.
from ultralytics import YOLO

### Import OpenCV to read test images and draw predicted bounding boxes and labels.
import cv2 

### Import the os module to help build file paths in a cross-platform way.
import os 

### Define the path to a sample test image of playing cards that the model has not seen during training.
imgTest = "C:/Data-sets/Playing Cards/test/images/008090758_jpg.rf.caa872ea30359a5f1cf5c6de034b8684.jpg"

### Define the path to the ground truth annotation file for the same test image.
imgAnnot = "C:/Data-sets/Playing Cards/test/labels/008090758_jpg.rf.caa872ea30359a5f1cf5c6de034b8684.txt"

### Read the test image from disk using OpenCV.
img = cv2.imread(imgTest)

### Extract the image height, width, and channel count so we can later convert YOLO coordinates to pixels.
H , W , _ = img.shape

### Create a copy of the original image that we will use to draw model predictions.
imgpredict = img.copy()

### Build the full path to the trained model weights (best.pt) inside the YOLOv8 experiment folder.
model_path = os.path.join("C:/Data-sets/Playing Cards","My-Card-Model","weights","best.pt")

### Load the trained YOLOv8 model from the best weights file.
model = YOLO(model_path)

### Set a confidence threshold so only predictions with high enough scores are drawn.
threshold = 0.5 

### Run the model on the test image and take the first (and only) result in the returned list.
results = model(imgpredict)[0]

### Print the raw results object to quickly inspect detected boxes and scores.
print(results)

### Loop over each detected bounding box in the model output.
for result in results.boxes.data.tolist():
    ### Unpack the bounding box coordinates, detection score, and class id from the list.
    x1 , y1 , x2 , y2 , score, class_id = result 

    ### Only draw detections that exceed the chosen confidence threshold.
    if score > threshold:
        ### Draw a rectangle representing the predicted bounding box for this card.
        cv2.rectangle(imgpredict, (int(x1), int(y1)), (int(x2), int(y2)), (0,255,0), 1)
        ### Put the predicted class name above the bounding box in uppercase text.
        cv2.putText(
                    imgpredict,
                    results.names[int(class_id)].upper(),
                    (int(x1), int(y1-10)), 
                    cv2.FONT_HERSHEY_SIMPLEX,
                    0.5,
                    (0,255,0),
                    1,
                    cv2.LINE_AA
                    )

This gives you a quick visual sense of how well the model can detect and classify cards it has never seen before.
If predictions look noisy, consider training longer, adjusting the confidence threshold, or improving your dataset’s labels.

If you want to explore alternative architectures for custom datasets, take a look at my YOLOX object detection tutorial and my guide on training Detectron2 on custom object detection data.

Comparing YOLOv8 predictions to ground truth labels

To truly understand how well you trained YOLOv8 on custom dataset, it helps to compare predictions with ground-truth labels on the same image.
This final block draws a second version of the image using the true annotations from the label file and saves both images for side-by-side inspection.

### Create a copy of the original image that will be used to draw ground truth bounding boxes.
imgTruth = img.copy()

### Open the ground truth annotation file to read all label lines for this test image.
with open(imgAnnot, 'r') as file:
    ### Read each line in the label file which represents one ground truth bounding box.
    lines = file.readlines()

### Initialize an empty list to store parsed ground truth annotations.
annotations = [] 

### Loop over each line and parse the YOLO format labels into structured tuples.
for line in lines:
    ### Split the line into individual components made of class id and bounding box coordinates.
    values = line.split()
    ### The first value is the integer class label index.
    label = values[0]

    ### Convert the remaining values (x, y, w, h) to floating point numbers.
    x, y, w, h = map(float, values[1:])
    ### Append the parsed annotation as a tuple to the annotations list.
    annotations.append((label , x, y, w, h))

### Print the parsed ground truth annotations so you can verify the values.
print(annotations)

### Loop through each ground truth annotation and draw it on the imgTruth image.
for annotation in annotations:
    ### Unpack the label index and YOLO bounding box coordinates.
    label , x, y, w, h = annotation
    ### Convert the label index into the corresponding class name from the model results.
    label_name = results.names[int(label)].upper()

    ### Convert YOLO center coordinates and box size into top-left and bottom-right pixel coordinates.
    x1 = int((x - w / 2) * W)
    y1 = int((y - h / 2) * H)
    x2 = int((x + w / 2) * W)
    y2 = int((y + h / 2) * H)

    ### Draw the ground truth bounding box rectangle on the image.
    cv2.rectangle(imgTruth , (x1, y1), (x2, y2), (200,200,0), 1)

    ### Write the class name above the ground truth bounding box so it is easy to compare with predictions.
    cv2.putText(imgTruth , label_name, (x1, y1-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (200,200,0), 2)

### Save the ground truth visualization to disk so you can review it later.
cv2.imwrite("c:/temp/imgTruth.png",imgTruth)

### Save the prediction visualization to disk so you can compare it next to the ground truth image.
cv2.imwrite("c:/temp/imgpredict.png",imgpredict)

### Show the OpenCV window displaying the ground truth bounding boxes.
cv2.imshow("imgTruth", imgTruth)

### Show the OpenCV window displaying the model predictions.
cv2.imshow("imgpredict", imgpredict)

### Show the original, unmodified image for reference.
cv2.imshow("Original", img)

### Wait for a key press before closing all OpenCV windows so you have time to inspect the results.
cv2.waitKey(0)

Now you have three views of the same scene: original image, predicted boxes, and ground-truth boxes.
This kind of qualitative evaluation is extremely helpful in spotting under-trained classes, missing labels, or annotation mistakes that metrics alone might hide.

FAQ :

Conclusion

Training YOLOv8 on a custom dataset turns a generic detector into a specialised tool for your own use cases.
In this tutorial, you built a full pipeline around a playing cards dataset: setting up a Conda environment, installing YOLOv8, downloading and organising data from Roboflow, visualising label quality, training a model, and checking predictions against ground truth.

Once you understand how each code block fits into the bigger picture, you can freely swap the cards dataset for your own images and labels.
The same approach works whether you want to detect industrial parts, medical tools, retail products, or sports players—just update the dataset paths, regenerate labels if needed, and retrain.

Most importantly, you now have a reusable pattern.
Whenever you want to train YOLOv8 on custom dataset again, you can follow these steps, tweak hyperparameters, and steadily improve your models without rebuilding everything from scratch.

For more advanced workflows that combine detectors and segmenters, you can also read my Segment Anything + YOLOv8 tutorial, where we generate precise masks from detection boxes in a single Python pipeline.

Connect :

☕ Buy me a coffee — https://ko-fi.com/eranfeit

🖥️ Email : feitgemel@gmail.com

🌐 https://eranfeit.net

🤝 Fiverr : https://www.fiverr.com/s/mB3Pbb

Enjoy,

Eran