Last Updated on 22/04/2026 by Eran Feit
Understanding YOLOv8 Segmentation for Real Projects YOLOv8 has quickly become one of the most powerful tools for real-time object detection and segmentation, combining speed, accuracy, and a clean developer experience into one flexible framework. With its segmentation capabilities, you can move beyond simple bounding boxes and generate precise masks that separate objects from their background at the pixel level. In this yolov8 segmentation tutorial , we focus on how to leverage these features in practical projects, so you can build production-ready workflows for images and videos with minimal hassle.
What You’ll Learn in This YOLOv8 Segmentation Tutorial In this tutorial, we walk step-by-step through a complete yolov8 segmentation tutorial focused on dog cutouts and background replacement, starting from environment setup and dataset preparation all the way to training a custom one-class segmentation model. You’ll see how to run inference on a single image, generate clean masks, and then apply the same model to video frames to replace the background dynamically. The goal is to give you a clear, hands-on pipeline you can copy, adapt, and reuse for your own projects—whether you’re segmenting pets, products, or any other objects.
Introduction to YOLOv8 Dog Segmentation and Background Replacement Dog photos and videos are everywhere, but clean, production-ready cutouts usually require tedious manual editing.YOLOv8 segmentation , you can teach a model to detect and segment only dogs, generate precise masks, and drop them onto any background you like.yolov8 segmentation tutorial : from environment setup and training a one-class dog model, to segmenting a single image and replacing the background in video.dog segmentation YOLOv8 workflows that feels fast, reliable, and friendly enough for daily creative or commercial projects.
If you want another real-world segmentation use case, check out my
YOLOv8 Segmentation Tutorial for Real Flood Detection
where I apply a similar pipeline to flooded areas on real map images.
Link to the video tutorial : https://youtu.be/eaHpGjFSFYE
You can download the code here : https://eranfeit.lemonsqueezy.com/buy/96ebfa91-9aa4-4089-acbd-721024b9b5af
or here : https://ko-fi.com/s/d1f1e5a5f6
Master Computer Vision
Follow my latest tutorials and AI insights on my
Personal Blog .
Beginner Complete CV Bootcamp
Foundation using PyTorch & TensorFlow.
Get Started → Interactive Deep Learning with PyTorch
Hands-on practice in an interactive environment.
Start Learning → Advanced Modern CV: GPT & OpenCV4
Vision GPT and production-ready models.
Go Advanced → Getting Your Environment Ready for YOLOv8 Segmentation Send me an email for the dataset
A stable environment is the backbone of any good YOLOv8 segmentation tutorial .
### Create a dedicated Conda environment so this YOLOv8 segmentation tutorial stays isolated and stable. conda create --name YoloV8 python=3.9 ### Activate the new environment so every next command installs into the correct place. conda activate YoloV8 ### Install the supervision library used later for visualizations and utilities. pip install supervision ### Uninstall the headless OpenCV build that may conflict with GUI operations. pip uninstall opencv-python-headless ### Check your CUDA toolkit version to ensure it matches the planned PyTorch build. nvcc --version ### Install PyTorch, TorchVision, and TorchAudio built for CUDA 11.8 from official channels. conda install pytorch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 pytorch-cuda=11.8 -c pytorch -c nvidia ### Install the specific Ultralytics YOLOv8 version for consistent segmentation behavior. pip install ultralytics==8.1.0 ### Uninstall any conflicting OpenCV version to avoid runtime issues. pip uninstall opencv-python ### Install a compatible OpenCV build with full image and video support. pip install opencv-python>=4.6.0 Summary of This Part You create a dedicated Python environment, verify CUDA, install compatible PyTorch, YOLOv8, and OpenCV, and remove conflicting packages.
Preparing the Dog Segmentation Dataset in a Clean Structure A good yolov8 segmentation tutorial does not stop at code snippets.C:\Data-sets\Dog segmentation YoloV8.
Make sure your dataset follows the YOLOv8 segmentation format: images in images/, labels in labels/, with matching filenames.
1. Download the Dog segmentation from my link : https://www.dropbox.com/scl/fi/1pkuozq4mmd1qnlopml08/Dog-segmentation-YoloV8.zip?rlkey=e2750ts6akmuj6x988iwmqlnt&st=xyptm465&dl=0 2. Extract the zip . ### Create a clear directory to store your dog segmentation dataset. mkdir "C:\Data-sets\Dog segmentation YoloV8" 3. copy the Dataset into : C:\Data-sets\Dog segmentation YoloV8 ### Move or extract your prepared dog segmentation images and labels into this folder. ### Ensure you have separate images and labels directories matching YOLOv8 expectations. Summary of This Part You extract the dataset once, put it into a stable folder, and ensure filenames and structure are ready for the training step.
Training a Custom YOLOv8 Dog Segmentation Model Now we move from setup into real YOLOv8 instance segmentation .yolov8m-seg.pt model and fine-tune it on your one-class dog dataset.
During training, YOLOv8 uses your config.yaml file (defined in the next section) to locate train/val images and understand that it only needs to predict one class: dog.
### Import the YOLO class from Ultralytics to handle training and inference. from ultralytics import YOLO ### Wrap the training process in a main function for clarity and reusability. def main (): ### Load the pretrained YOLOv8 segmentation model as a strong starting point. model = YOLO ( " yolov8m-seg.pt " ) ### Define the root directory where your dog segmentation dataset and runs will be stored. project = " C:/Data-sets/Dog segmentation YoloV8 " ### Name this specific training run so results are organized and easy to track. experiment = " My-model " ### Set a batch size that fits comfortably on an ~8GB GPU while keeping training efficient. batch_size = 16 ### Launch the YOLOv8 training loop with your custom configuration. results = model . train ( data = " Best-Semantic-Segmentation-models/Yolo-V8/Segment-Dog-with-New-background/config.yaml " , epochs = 100 , project = project , name = experiment , batch = batch_size , device = 0 , imgsz = 640 , patience = 40 , verbose = True , val = True ) ### Ensure the training runs only when the script is executed directly. if __name__ == " __main__ " : main () Summary : You fine-tune YOLOv8-seg on a single dog class, save the experiment in a clear project folder, and prepare a robust model that outputs clean dog masks.
For another complete deep learning workflow from raw data to production,
see
How to Build a DenseNet201 Model for Sports Image Classification
.
Understanding the config.yaml for One-Class Dog Segmentation The config.yaml file is your model’s map.yolov8 background removal workflow, you only care about one class: dog.
This file is intentionally minimal, which makes it easy to reuse for future segmentation projects.
To speed up dataset creation and polygon masks for YOLO-based projects,
explore my
Segment Anything Tutorial: Generate YOLOv8 Masks Fast
.
Here is the config.yaml :
train : C : / Data - sets / Dog segmentation YoloV8 / train / images val : C : / Data - sets / Dog segmentation YoloV8 / valid / images nc : 1 names : [ ' dog ' ] Summary of This Part You define train and validation paths plus a single class name.
Segmenting a Single Dog Image with Your Trained Model With training finished, it’s time to test your model on a real image.yolov8 segmentation tutorial becomes visually satisfying: clear edges, no manual tracing, and ready for creative use.
You also save individual masks and the final mask to disk, making it easy to debug predictions or integrate them into other tools.
Here is the test image :
Dog segmentation ### Import YOLO for inference, NumPy for arrays, and OpenCV for image handling. from ultralytics import YOLO import numpy as np import cv2 ### Set the path to your trained YOLOv8 dog segmentation weights. model_path = " C:/Data-sets/Dog segmentation YoloV8/My-model2/weights/best.pt " ### Set the path to the dog image you want to segment. image_path = " Best-Semantic-Segmentation-models/Yolo-V8/Segment-Dog-with-New-background/German_Shepherd.jpg " ### Read the input image in BGR format using OpenCV. img = cv2 . imread ( image_path ) ### Extract the image height and width for later mask processing. H , W , _ = img . shape ### Load your trained YOLOv8 segmentation model from disk. model = YOLO ( model_path ) ### Run the model on the input image to obtain predictions. results = model ( img ) ### Access the class name mapping from the model for readable labels. names = model . names print ( names ) ### Work with the first (and only) result from the predictions list. result = results [ 0 ] ### Initialize an empty mask that will accumulate all detected dog masks. final_mask = np . zeros (( H , W ), dtype = np . uint8 ) ### Extract the predicted class indices for each detected object. predicted_classes = result . boxes . cls . cpu (). numpy () print ( predicted_classes ) ### Loop through each predicted mask and merge them into one final mask. for j , mask in enumerate ( result . masks . data ): ### Move the mask tensor to CPU, convert to NumPy, and scale to [0,255]. mask = mask . cpu (). numpy () * 255 ### Read the class ID for the current mask. classId = int ( predicted_classes [ j ]) ### Print which class was detected for easier debugging. print ( " Object " + str ( j ) + " detected as " + str ( classId ) + " - " + names [ classId ]) ### Resize the mask to match the original image size. mask = cv2 . resize ( mask , ( W , H )) ### Combine the current mask with the accumulated final mask. final_mask = np . maximum ( final_mask , mask ) ### Build a filename for saving this individual mask. file_name = " output " + str ( j ) + " .png " ### Save the individual mask to disk for inspection. cv2 . imwrite ( " C:/Data-sets/Dog segmentation YoloV8/My-model2/ " + file_name , mask ) ### Save the merged final mask that includes all detected dogs. cv2 . imwrite ( " C:/Data-sets/Dog segmentation YoloV8/My-model2/final_mask.png " , final_mask ) ### Ensure the combined mask is binary so it behaves cleanly in bitwise operations. _ , threshold_mask = cv2 . threshold ( final_mask , 128 , 255 , cv2 . THRESH_BINARY ) ### Convert the mask to uint8 type required by OpenCV functions. threshold_mask = threshold_mask . astype ( np . uint8 ) ### Apply the mask to the original image to extract only the dog region. result = cv2 . bitwise_and ( img , img , mask = threshold_mask ) ### Display the original image for comparison. cv2 . imshow ( " img " , img ) ### Display the final merged mask to visualize segmentation quality. cv2 . imshow ( " final mask " , final_mask ) ### Display the masked result showing only the segmented dog. cv2 . imshow ( " Result " , result ) ### Wait for a key press before closing the windows. cv2 . waitKey ( 0 ) ### Close all OpenCV display windows cleanly. cv2 . destroyAllWindows ()
Summary of This Part You run YOLOv8 inference on a single image, build a combined mask, threshold it, and apply it to extract only the dog.video background replacement YOLOv8 workflows.
To compare this trained-model approach with a zero-training method,
read
Segment Anything Python — No-Training Image Masks
.
Replacing the Video Background with YOLOv8 Dog Segmentation This final step turns your model into a practical video editing engine.best mask with the highest confidence, you avoid noisy predictions and keep the dog crisp against the fresh background.video background replacement YOLOv8 pipelines shine for content creators, advertisers, and social clips.
The logic centers the dog on the new background, keeps only non-black pixels from the segmented result, and displays live previews.
You can download the video file from here : https://ko-fi.com/s/d1f1e5a5f6
Background test image :
Background image ### Import the YOLO model class along with NumPy and OpenCV utilities. from ultralytics import YOLO import numpy as np import cv2 ### Define the path to the input video containing the dog. video_path = " Best-Semantic-Segmentation-models/Yolo-V8/Segment-Dog-with-New-background/dog.mp4 " ### Define the path to your trained YOLOv8 segmentation weights. model_path = " C:/Data-sets/Dog segmentation YoloV8/My-model2/weights/best.pt " ### Define the path to the background image you want to place behind the dog. background_image_path = " Best-Semantic-Segmentation-models/Yolo-V8/Segment-Dog-with-New-background/new_background.jpg " ### Load the new background image using OpenCV. background_image = cv2 . imread ( background_image_path ) ### Open the input video file for reading frames. cap = cv2 . VideoCapture ( video_path ) ### Verify that the video file was opened successfully. if not cap . isOpened (): print ( " Error open video file. " ) exit () ### Process each frame until the video ends or the user quits. while True : ### Read the next frame from the video. ret , frame = cap . read () ### If no frame is returned, the video has ended. if not ret : print ( " Video playback completed. " ) break ### Scale down the frame to speed up YOLOv8 inference. scale_precent = 30 width = int ( frame . shape [ 1 ] * scale_precent / 100 ) height = int ( frame . shape [ 0 ] * scale_precent / 100 ) dim = ( width , height ) ### Resize the frame according to the chosen scale. frame = cv2 . resize ( frame , dim , interpolation = cv2 . INTER_AREA ) ### Retrieve height and width of the resized frame. H , W , _ = frame . shape ### Load the trained YOLOv8 model for segmentation. model = YOLO ( model_path ) ### Run the model on the current frame to get segmentation predictions. results = model ( frame ) result = results [ 0 ] ### Access the class name mapping from the model. names = model . names ### Initialize variables to store the best mask and its confidence score. best_mask = None best_confidence = 0.5 ### Extract predicted class indices and confidence scores. predicted_classes = result . boxes . cls . cpu (). numpy () confidence_values = result . boxes . conf . cpu (). numpy () ### Iterate over each predicted mask. for j , mask in enumerate ( result . masks . data ): ### Convert the mask to a NumPy array in [0,255] range. mask = mask . cpu (). numpy () * 255 ### Read the class ID for this detection. classId = int ( predicted_classes [ j ]) ### Read the confidence score for this detection. confidence = confidence_values [ j ] ### Resize the mask to match the frame size. mask = cv2 . resize ( mask , ( W , H )) ### Keep the mask with the highest confidence above the threshold. if confidence > best_confidence : best_mask = mask best_confidence = confidence ### Apply a binary threshold to the best mask to clean it. _ , threshold_mask = cv2 . threshold ( best_mask , 128 , 255 , cv2 . THRESH_BINARY ) ### Convert the mask to uint8 type for OpenCV operations. threshold_mask = threshold_mask . astype ( np . uint8 ) ### Use the mask to extract only the dog region from the frame. result = cv2 . bitwise_and ( frame , frame , mask = threshold_mask ) ### Compute the center of the background image. center_y , center_x = background_image . shape [ 0 ] // 2 , background_image . shape [ 1 ] // 2 ### Compute placement coordinates to center the dog on the background. start_y , start_x = center_y - result . shape [ 0 ] // 2 , center_x - result . shape [ 1 ] // 2 end_y , end_x = start_y + result . shape [ 0 ], start_x + result . shape [ 1 ] ### Build a boolean mask for non-black pixels in the segmented result. non_black_pixels = ( result != [ 0 , 0 , 0 ]). any ( axis = 2 ) ### Copy the background image to avoid modifying the original. final = background_image . copy () ### Paste the segmented dog onto the background wherever the mask is non-black. final [ start_y : end_y , start_x : end_x ][ non_black_pixels ] = result [ non_black_pixels ] ### Optionally resize the final composite for display. scale_precent = 60 width = int ( background_image . shape [ 1 ] * scale_precent / 100 ) height = int ( background_image . shape [ 0 ] * scale_precent / 100 ) dim = ( width , height ) ### Resize the final frame to the chosen display resolution. final = cv2 . resize ( final , dim , interpolation = cv2 . INTER_AREA ) ### Show the original resized frame. cv2 . imshow ( " original " , frame ) ### Show the masked dog result. cv2 . imshow ( " result " , result ) ### Show the final composited frame with the new background. cv2 . imshow ( " Final " , final ) ### Exit the loop if the user presses the 'q' key. if cv2 . waitKey ( 25 ) & 0x FF == ord ( ' q ' ): break ### Release the video capture resource. cap . release () ### Close all OpenCV windows. cv2 . destroyAllWindows ()
For more advanced and instant masking workflows you can plug into this pipeline,
check
Segment Anything Tutorial: Fast Auto Masks in Python
and
One-Click Segment Anything in Python (SAM ViT-H)
.
Summary of This Part You open a video, run YOLOv8 segmentation on each frame, build a binary mask, blend the dog onto a new background, and preview the output.yolov8 background removal system powered entirely by your own model.
FAQ : What is YOLOv8 segmentation used for in this guide? It is used to generate precise dog masks so you can create clean cutouts and background replacements with a single trained model.
Why focus on a one-class dog segmentation model? A one-class model reduces noise and teaches YOLOv8 to specialize in detecting and segmenting dogs only.
Do I need a powerful GPU for this YOLOv8 segmentation tutorial? A GPU is recommended for faster training, but you can run smaller models and fewer epochs on CPU if needed.
What role does config.yaml play? It configures dataset paths, class count, and names so YOLOv8 can correctly load and train your segmentation dataset.
How are final masks created from YOLOv8 outputs? All masks are resized, merged, and thresholded into a single binary mask that cleanly isolates the dog.
Can this workflow handle video background replacement? Yes, the tutorial shows how to segment each frame and composite the dog over a new background in real time.
Is it possible to reuse this pipeline for other animals? You can replace the dataset and class name, then retrain YOLOv8 to segment any object you choose.
How do I avoid unstable masks between frames? Use a confidence threshold, keep the best mask, and optionally apply smoothing to stabilize results.
Can I export the final composited video? Yes, replace the preview windows with a VideoWriter that saves each final frame into a new video file.
What should I do if segmentation accuracy is low? Improve labels, train longer, adjust learning rate, and increase image resolution for sharper masks.
Conclusion By now, you’ve walked through a full yolov8 segmentation tutorial that mirrors real production needs: you built a clean environment, organized a dataset, trained a one-class dog model, validated it on single images, and deployed it into a frame-by-frame video background replacement pipeline.config.yaml to mask post-processing—so debugging and improving results becomes straightforward.
This workflow also scales naturally.
Most importantly, this approach turns segmentation and background removal from a “magic button” into a transparent, reproducible process you can trust.
You can find more practical segmentation pipelines in my
Image Segmentation tutorials collection
.
Connect : ☕ Buy me a coffee — https://ko-fi.com/eranfeit
🖥️ Email : feitgemel@gmail.com
🌐 https://eranfeit.net
🤝 Fiverr : https://www.fiverr.com/s/mB3Pbb
Enjoy,
Eran
Skip to content 
