スタンフォード/CS131/宿題5-3 定量的評価

前回のStanford University/CS131/宿題5-2 画素レベル特徴(色・位置特徴)の続きをやる。今回の宿題は、Quantitative Evaluation(定量的評価)をカバーする。

スポンサーリンク

Quantitative Evaluation

Looking at images is a good way to get an idea for how well an algorithm is working, but the best way to evaluate an algorithm is to have some quantitative measure of its performance.
画像を見ることは、アルゴリズムがどの程度うまく機能しているのかを知るための良い方法であるのだが、アルゴリズムを評価する最も良い方法は、その性能を定量的に測ることだ。

For this project we have supplied a small dataset of cat images and ground truth segmentations of these images into foreground (cats) and background (everything else). We will quantitatively evaluate different segmentation methods (features and clustering methods) on this dataset.
今回のプロジェクトに関しては、猫画像の小データ・セットと、これら画像を前景(猫)、背景(その他全て)に分割した正解データを提供している。このデータ・セットを使用して、さまざまな分割メソッド(特徴とクラスタ化メソッド)を定量的に評価する。

We can cast the segmentation task into a binary classification problem, where we need to classify each pixel in an image into either foreground (positive) or background (negative). Given the ground-truth labels, the accuracy of a segmentation is $(TP+TN)/(P+N)$.
分割タスクは、画像の各画素を前景(正)か背景(負)に分類する必要がある、二値分類問題と位置付けることができる。正解ラベルを前提にすれば、分割の正確度は$(TP+TN)/(P+N)$になる。

Implement compute_accuracy in segmentation.py.
compute_accuracy関数を実装せよ。

from time import time
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rc
from skimage import io,color
import random
from segmentation import hierarchical_clustering,kmeans_fast,color_features
from scipy.spatial.distance import squareform, pdist
from skimage.util import img_as_float
from __future__ import print_function

%matplotlib inline
plt.rcParams['figure.figsize'] = (15.0, 10.0) # set default size of plots
plt.rcParams["font.size"] = "17"
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
def compute_accuracy(mask_gt, mask):
    """ Compute the pixel-wise accuracy of a foreground-background segmentation
        given a ground truth segmentation.
    Args:
        mask_gt - The ground truth foreground-background segmentation. A
            logical of size H x W where mask_gt[y, x] is 1 if and only if
            pixel (y, x) of the original image was part of the foreground.
        mask - The estimated foreground-background segmentation. A logical
            array of the same size and format as mask_gt.
    Returns:
        accuracy - The fraction of pixels where mask_gt and mask agree. A
            bigger number is better, where 1.0 indicates a perfect segmentation.
    """
    accuracy = None
    ### YOUR CODE HERE
    accuracy = np.mean(mask_gt == mask)
    ### END YOUR CODE
    return accuracy
mask_gt = np.zeros((100, 100))
mask = np.zeros((100, 100))

# Test compute_accracy function
mask_gt[20:50, 30:60] = 1
mask[30:50, 30:60] = 1

accuracy = compute_accuracy(mask_gt, mask)

print('Accuracy: %0.2f' % (accuracy))
if accuracy != 0.97:
    print('Check your implementation!')

plt.subplot(121)
plt.imshow(mask_gt)
plt.title('Ground Truth')
plt.axis('off')

plt.subplot(122)
plt.imshow(mask)
plt.title('Estimate')
plt.axis('off')
plt.show()
Accuracy: 0.97

segmentation parameters

You can use the script below to evaluate a segmentation method’s ability to separate foreground from background on the entire provided dataset. Use this script as a starting point to evaluate a variety of segmentation parameters.
下のスクリプトを使って分割メソッドの全ての提供データセットに対する背景と前景を区別する能力を評価できる。このスクリプトをスターティング・ポイントとして使用して、さまざまなセグメンテーションパラメータの組み合わせを評価する。

def evaluate_segmentation(mask_gt, segments):
    """ Compare the estimated segmentation with the ground truth.
    Note that 'mask_gt' is a binary mask, while 'segments' contain k segments. 
    This function compares each segment in 'segments' with the ground truth and
    outputs the accuracy of the best segment.
    Args:
        mask_gt - The ground truth foreground-background segmentation. A
            logical of size H x W where mask_gt[y, x] is 1 if and only if
            pixel (y, x) of the original image was part of the foreground.
        segments - An array of the same size as mask_gt. The value of a pixel
            indicates the segment it belongs.
    Returns:
        best_accuracy - Accuracy of the best performing segment.
            0 <= accuracy <= 1, where 1.0 indicates a perfect segmentation.
    """
    num_segments = np.max(segments) + 1
    best_accuracy = 0
    # Compare each segment in 'segments' with the ground truth
    for i in range(num_segments):
        mask = (segments == i).astype(int)
        accuracy = compute_accuracy(mask_gt, mask)
        best_accuracy = max(accuracy, best_accuracy)
    return best_accuracy
from utils import load_dataset, compute_segmentation

# Load a small segmentation dataset
imgs, gt_masks = load_dataset('./data')

# Set the parameters for segmentation.
num_segments = 3
clustering_fn = kmeans_fast
feature_fn = color_features
scale = 0.5
mean_accuracy = 0.0
segmentations = []

for i, (img, gt_mask) in enumerate(zip(imgs, gt_masks)):
    # Compute a segmentation for this image
    segments = compute_segmentation(img, num_segments,
                                    clustering_fn=clustering_fn,
                                    feature_fn=feature_fn,
                                    scale=scale)
    
    segmentations.append(segments)    
    # Evaluate segmentation
    accuracy = evaluate_segmentation(gt_mask, segments)    
    print('Accuracy for image %d: %0.4f' %(i, accuracy))
    mean_accuracy += accuracy
    
mean_accuracy = mean_accuracy / len(imgs)
print('Mean accuracy: %0.4f' % mean_accuracy)
/root/.pyenv/versions/3.7.0/envs/py37/lib/python3.7/site-packages/skimage/transform/_warps.py:24: UserWarning: The default multichannel argument (None) is deprecated.  Please specify either True or False explicitly.  multichannel will default to False starting with release 0.16.
  warn('The default multichannel argument (None) is deprecated.  Please '
/root/.pyenv/versions/3.7.0/envs/py37/lib/python3.7/site-packages/skimage/transform/_warps.py:105: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
  warn("The default mode, 'constant', will be changed to 'reflect' in "
/root/.pyenv/versions/3.7.0/envs/py37/lib/python3.7/site-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
  warn("Anti-aliasing will be enabled by default in skimage 0.15 to "
Accuracy for image 0: 0.8004
Accuracy for image 1: 0.9556
Accuracy for image 2: 0.9639
Accuracy for image 3: 0.8791
Accuracy for image 4: 0.9775
Accuracy for image 5: 0.6813
Accuracy for image 6: 0.6663
Accuracy for image 7: 0.6688
Accuracy for image 8: 0.8411
Accuracy for image 9: 0.9544
Accuracy for image 10: 0.8723
Accuracy for image 11: 0.8129
Accuracy for image 12: 0.7385
Accuracy for image 13: 0.6655
Accuracy for image 14: 0.7487
Accuracy for image 15: 0.5043
Mean accuracy: 0.7957
# Visualize segmentation results

N = len(imgs)
plt.figure(figsize=(15,60))
for i in range(N):

    plt.subplot(N, 3, (i * 3) + 1)
    plt.imshow(imgs[i])
    plt.axis('off')

    plt.subplot(N, 3, (i * 3) + 2)
    plt.imshow(gt_masks[i])
    plt.axis('off')

    plt.subplot(N, 3, (i * 3) + 3)
    plt.imshow(segmentations[i], cmap='viridis')
    plt.axis('off')

plt.show()

Include a detailed evaluation of the effect of varying segmentation parameters (feature transform, clustering method, number of clusters, resize) on the mean accuracy of foreground-background segmentations on the provided dataset. You should test a minimum of 10 combinations of parameters. To present your results, add rows to the table below (you may delete the first row).
与えられたデータセットにおける前景-背景分割平均精度に対する、種々のセグメンテーションパラメータ(特徴変換、クラスタ化手法、クラスタ数、リサイズ)の効果の詳細評価を列挙する。最低10個のパラメータ組み合わせをテストする必要がある。結果のプレゼンに、下のテーブルに行を加えていく(先頭行は消しても構わない)。

Feature Transform Clustering Method Number of segments Scale Mean Accuracy
Color K-Means 3 0.5 0.7957
参考サイトhttps://github.com/