When looking at a picture, what do you see? Perhaps you see a person, a dog or a tree. Can a computer be taught to see the same thing? That is the task of semantic segmentation. It is the process of assigning a label to every pixel in an image. In the fully supervised setting, computer algorithms need expensive pixel-level annotations to learn how to segment images. However, in the weakly-supervised setting, algorithms can learn from less expensive annotations such as object tags or labels.
Fully Supervised Semantic Segmentation
In fully supervised semantic segmentation, a machine learning model is fed a set of images along with their corresponding pixel-level class-specific annotations. This means that every single pixel in the image is labeled with the class that it belongs to. For example, in an image of a dog, every pixel that belongs to the dog would be labeled "dog".
This method provides incredibly accurate results, but it is time-consuming and expensive to create such a dataset. Every single image would need to be annotated pixel-by-pixel, and that's no small task!
Weakly-Supervised Semantic Segmentation
Weakly-supervised semantic segmentation aims to reduce the amount of annotation needed, making the process easier and cheaper. In this paradigm, the labeled data consists of images and corresponding tags or labels that describe objects in the image. The image in itself is not labeled, just the objects that it contains.
For instance, if we have a picture of a dog in a park, the labels provided could be "dog" and "park". These labels do not provide as much information as pixel-level annotations but are easier to obtain in bulk.
The Challenge of Weakly-Supervised Semantic Segmentation
While the weakly-supervised approach is much less expensive, it comes with its own set of challenges.
First and foremost, the labels provided only give information on the overall scene in the picture. In our previous example, simply having the labels "dog" and "park" does not tell us which parts of the image correspond to the "dog" or the "park".
Secondly, different objects can share the same label. Consider the following image of a dog and a cat in a park:
Labelling this image with just "dog" and "park" would not help us differentiate between the pixels that belong to the dog and the pixels that belong to the park.
Thirdly, the labels might not be accurate. Suppose someone labels a picture of a cake in a park with "cake" and "park". However, the label "park" might not add any useful information as it is already obvious by the picture itself, and it might only add confusion to the model.
Techniques for Weakly-Supervised Semantic Segmentation
To overcome the challenges of weakly-supervised semantic segmentation, different techniques have been proposed. One such approach is the seed growing method.
Seed growing is an unsupervised method that can perform segmentation by requiring only a seed point. It works by starting with a seed point provided in the label, and then expanding the region based on some similarity criteria.
Here's how it works:
- Start with the seed point provided in the label.
- Find all pixels that are similar to the seed point according to a similarity metric.
- Assign the same label to all similar pixels.
- Repeat the process by expanding the region in all directions from these new pixels.
- Stop when there are no more similar pixels.
The seed growing method is one example of unsupervised semantic segmentation that can be used when only weak labels are available. It doesn't require any fully or weakly annotated training data and instead, uses the seed point as the sole source of information.
Semantic Segmentation is one of the most important computer vision tasks. It has a wide range of applications in fields such as self-driving cars or medical imaging. While the fully supervised method is accurate in its results, it is expensive and time-consuming to create the dataset. In contrast, weakly-supervised semantic segmentation is less expensive, but it comes with its own set of challenges. Different techniques such as seed growing methods have been proposed to overcome these challenges, and further research is needed to improve the methods.
