Localization and Object detection are two of the core tasks in Computer Vision ,
as they are applied in many real-world applications such as Autonomous vehicles
and Robotics. So, if you want to work in these industries as a Computer vision
specialist or you want to build a relative product , you better have a good
grasp of them. But what are they? What Object detection and localization means?
And why we group them as they are one thing?
First things first. Let’s do a quick recap of the most used terms and their
meaning to avoid misconceptions:
-
Classification/Recognition: Given an image with an object , find out
what that object is. In other words, classify it in a class from a set of
predefined categories. -
Localization : Find where the object is and draw a bounding box around
it -
Object detection: Classify and detect all objects in the image. Assign a
class to each object and draw a bounding box around it. -
Semantic Segmentation: Classify every pixel in the image to a class
according to its context, so that each pixel is assigned to an object -
Instance Segmentation: Classify every pixel in the image to a class so
that each pixel is assigned to a different instance of an object
Remember, though, that these terms are not clearly defined in the scientific
community, so you may encounter one of them in a different meaning. In my
understanding, these are the correct interpretations.
As we get the basic terms straight, it is time to do some localization and
object detection. How do we do it? Well there have been many approaches over the
years, but since the arriving of Deep Learning, Convolutional Neural Networks
became the industry standard. Remember our goal is to classify the object and
localize it. But are we sure that there is only one object? Is it possible
that there are two or three or fifteen objects? In fact, most of the time it is.
That’s why we can split our problem into two different problems. In the first
case , we know the number of objects (we will refer to the problem as
classification + localization) and in the second we don’t (object detection). I
will start with the first one as it is the most straightforward.
Stanford University School of Engineering
Classification + Localization
If we have only one object or we know the number of objects, it is actually
trivial. We can use one convolutional neural network and train it not only to
classify the image but also to output 4 coordinates for the bounding box. In
that way we treat the localization as a simple regression problem.
For example, we can borrow a well-studied model such as ResNet or Alexnet which
consists of a bunch of convolutional, pooling and other layers, and repurpose
the fully connected layer to produce the bounding box apart from the category.
It is so simple that make us question whether or not it will give results. And
it actually works pretty well in practice. Of course, you can get fancy with it
and modify the architecture for serving specific problems or enhance its
accuracy, but the main idea remains.
Be sure to note that in order to use this model, we should have a training set
with images annotated for the class and the bounding box. And it is not the most
fun to do such annotations.
But what if we do no know the number of objects a priori? Then we need to get
into the rabbit’s hole and talk about some hardcore stuff. Are you ready? Do you
want to take a break before? Sure, I understand but I warn you not to leave.
This is where the fun begins.
Object Detection
I am kidding. There is nothing hardcore about the architectures which will be
discuss. All there is, are some clever ideas to make the system intolerant to
the number of outputs and to reduce its computation cost. So, we do not know the
exact number of objects in our image and we want to classify all of them and
draw a bounding ox around them. That means that the number of coordinates that
the model should output is not constant. If the image has 2 objects , we need 8
coordinates . If it has 4 objects, we want 16. So how we build such a model?
One key idea to traditional computer vision is regions proposal. We generate a
set of windows that are likely to contain an object using classic CV algorithms,
like edge and shape detection and we apply only these windows( or regions of
interests) to the CNN. To learn more about how regions are proposed, make sure to check
here.
This is the basis on a fundamental
paper , which introduced a new architecture
called RCNN.
R-CNN
Given an image with multiple objects , we generate some regions of interests
using a proposal method( in RCNN’s case this method is called selective search)
and warp the regions into a fixed size. We forward each region to Convolutional
Neural Network (such as AlexNet), which will use an SVM to make a classification
decision for each one and predicts a regression for each bounding box. This
prediction comes as a correction of the region proposed, which may be in the
right position but not at the exact size and orientation.
Although the model produces good results, it suffers from a main issue. It is
quite slow and computational expensive. Imagine that in an average case, we
produce 2000 regions, which we need to store in disk, and we forward each one of
them into the CNN for multiple passes until it is trained. To fix some of these
problems, an improvement of the model comes in play called ‘fast-RCNN’
Fast RCNN
The idea is straightforward. Instead of passing all regions into the
convolutional layer one by one, we pass the entire image once and produce a
feature map. Then we take the region proposals as before ( using some external
method) and sort of project them onto the feature map. Now we have the regions
in the feature map instead of the original image and we can forward them in some
fully connected layers to output the classification decision and the bounding
box correction.
Note that the projection of regions proposal is implemented using a special
layer( ROI layer) ,which is essentially a type of max-pooling with a pool size
dependent on the input, so that the output always has the same size. For more
details on the ROI layer check this great article.
Faster RCNN
And we can take this a step further. Using the produced feature maps from the
convolutional layer, we infer regions proposal using a Region Proposal network
rather than relying on an external system. Once we have those proposal , the
remaining procedure is the same as Fast-RCNN (forward to ROI layer, classify
using SVM and predict the bounding box). The trick part is how to train the
whole model as we have multiple tasks that need to be addressed:
-
The region proposal network should decide for each region if it contains an
object or not -
And it needs to produce the bounding box coordinates
-
The entire model should classify the objects to categories
-
And again predict the bounding box offsets
If you want to learn more about the training part you should check the original
paper, but to give you an overview we need
to utilize a multitask loss to include all 4 tasks and back propagate this loss
to the network.
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
As the name suggests, FasterRCNN turns out to be much faster than the previous
models and is the one preferred in most real-world applications.
Localization and object detection is a super active and interesting area of
research due to the high emergency of real world applications that require
excellent performance in computer vision tasks (self-driving cars , robotics).
Companies and universities come up with new ideas on how to improve the accuracy
on regular basis.
There is another class of models for localization and object detection, called
single shot detectors, which have become very popular in the last years because
they are even faster and require less computational cost in general. Sure, they
are less accurate, but they are ideal for embedded systems and similar
power-hungry applications.
But to learn more , you have to wait for my next article.
Ok I’m kidding. You don’t have to wait. There is also this amazing course by Coursera. But seriously, you should also wait for me. Don’t forget me.
* Disclosure: Please note that some of the links above might be affiliate links, and at no additional cost to you, we will earn a commission if you decide to make a purchase after clicking through.
