Image Segmentation
[fcn
skip
mask-rcnn
unet
refinenet
segnet
deeplab
deep-learning
dilated-convolutions
image-segmentation
pspnet
scribble
]
There are two types of image segmentation:
- semantic segmentation: assign labels to each pixel
- instance segmentation: extract the bounary/mask for each instance in the image
Examples
- Fully Convolution Neural Network (FCN): FCN convert the fully layers in traditional network used for image classifcation, e.g., Alexnet, VGG16, to convolution layer. Thus is could generate a probability map of each pixel.
- UNet: it combines two parts: left part uses convolution and max-pooling to extract feature; right part uses upsampling and skip (input from lower layer of left part) to generate the label map.
- SegNet: similar to UNet, but it doesn’t use skip to combine the input from lower layer of left part (refered encoder network).
- Dilated Convolutions: the problem in FCN is that, using pooling and then up-sampling will cause data loss. Dilate convolution resolves this problem by adding
dilate
to convolution, which increases the field size while doesn’t reduce the output size as pooling dose. - RefineNet: similar to UNet, but utilizes the ResNet as the base.
- PspNet: is applies the idea of spatial pyramid pooling to image segmentation,
- DeepLab: combines Atrous Convolution (similar to Dilated Convolutions) with PspNet.
- Mask-R-CNN: uses the idea of object detection for semantic segmentation, where the probability of each boundary box is used as a response map, where softmax is then applied to generate the mask.
Common Techniques
- transposed convolution: a conjugate pair of convolution operator, whose forward propgation is the backward propagation of convolution operation and vice versa
- skip: combine the output of intermidiate layers to have multiple level features
Labels for Training Data
- Scribble uses a few simple scribbles as the label of the training image. Cost function is \(\sum_{i}\psi _i^{scr}\left(y_i|X,S\right)+\sum i-logP\left(y_i| X,\theta\right)+\sum{i,j}\psi _{ij}\left(y_i,y_j|X\right)\)
- Image-level label the label if provides to image level and there is no pixel level label, like image classificaition case. Cost function is \(\underset{\theta ,P}{minimize}\qquad D(P(X)||Q(X|\theta ))\ subject\to\qquad A\overrightarrow{P} \geqslant \overrightarrow{b},\sum_{X}^{ }P(X)=1\)
- Bounding box and label: the label is some bounding boxes and their labels, as object detection case. Cost function is \(P\left ( x,y,z;\theta \right ) = P\left ( x \right )\left (\prod_{m=1}^{M} P\left ( y_m|x;\theta \right )\right )P\left ( z|y \right )\)
Written on April 2, 2019