# スタンフォード/CS131/宿題5-3 定量的評価

スポンサーリンク

## Quantitative Evaluation¶

Looking at images is a good way to get an idea for how well an algorithm is working, but the best way to evaluate an algorithm is to have some quantitative measure of its performance.

For this project we have supplied a small dataset of cat images and ground truth segmentations of these images into foreground (cats) and background (everything else). We will quantitatively evaluate different segmentation methods (features and clustering methods) on this dataset.

We can cast the segmentation task into a binary classification problem, where we need to classify each pixel in an image into either foreground (positive) or background (negative). Given the ground-truth labels, the accuracy of a segmentation is $(TP+TN)/(P+N)$.

Implement compute_accuracy in segmentation.py.
compute_accuracy関数を実装せよ。

from time import time
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rc
from skimage import io,color
import random
from segmentation import hierarchical_clustering,kmeans_fast,color_features
from scipy.spatial.distance import squareform, pdist
from skimage.util import img_as_float
from __future__ import print_function

%matplotlib inline
plt.rcParams['figure.figsize'] = (15.0, 10.0) # set default size of plots
plt.rcParams["font.size"] = "17"
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

def compute_accuracy(mask_gt, mask):
""" Compute the pixel-wise accuracy of a foreground-background segmentation
given a ground truth segmentation.
Args:
mask_gt - The ground truth foreground-background segmentation. A
logical of size H x W where mask_gt[y, x] is 1 if and only if
pixel (y, x) of the original image was part of the foreground.
mask - The estimated foreground-background segmentation. A logical
array of the same size and format as mask_gt.
Returns:
accuracy - The fraction of pixels where mask_gt and mask agree. A
bigger number is better, where 1.0 indicates a perfect segmentation.
"""
accuracy = None
return accuracy

mask_gt = np.zeros((100, 100))

# Test compute_accracy function

print('Accuracy: %0.2f' % (accuracy))
if accuracy != 0.97:

plt.subplot(121)
plt.title('Ground Truth')
plt.axis('off')

plt.subplot(122)
plt.title('Estimate')
plt.axis('off')
plt.show()

Accuracy: 0.97

スポンサーリンク

## segmentation parameters¶

You can use the script below to evaluate a segmentation method’s ability to separate foreground from background on the entire provided dataset. Use this script as a starting point to evaluate a variety of segmentation parameters.

def evaluate_segmentation(mask_gt, segments):
""" Compare the estimated segmentation with the ground truth.
Note that 'mask_gt' is a binary mask, while 'segments' contain k segments.
This function compares each segment in 'segments' with the ground truth and
outputs the accuracy of the best segment.
Args:
mask_gt - The ground truth foreground-background segmentation. A
logical of size H x W where mask_gt[y, x] is 1 if and only if
pixel (y, x) of the original image was part of the foreground.
segments - An array of the same size as mask_gt. The value of a pixel
indicates the segment it belongs.
Returns:
best_accuracy - Accuracy of the best performing segment.
0 <= accuracy <= 1, where 1.0 indicates a perfect segmentation.
"""
num_segments = np.max(segments) + 1
best_accuracy = 0
# Compare each segment in 'segments' with the ground truth
for i in range(num_segments):
best_accuracy = max(accuracy, best_accuracy)
return best_accuracy

from utils import load_dataset, compute_segmentation

# Load a small segmentation dataset

# Set the parameters for segmentation.
num_segments = 3
clustering_fn = kmeans_fast
feature_fn = color_features
scale = 0.5
mean_accuracy = 0.0
segmentations = []

# Compute a segmentation for this image
segments = compute_segmentation(img, num_segments,
clustering_fn=clustering_fn,
feature_fn=feature_fn,
scale=scale)

segmentations.append(segments)
# Evaluate segmentation
print('Accuracy for image %d: %0.4f' %(i, accuracy))
mean_accuracy += accuracy

mean_accuracy = mean_accuracy / len(imgs)
print('Mean accuracy: %0.4f' % mean_accuracy)

/root/.pyenv/versions/3.7.0/envs/py37/lib/python3.7/site-packages/skimage/transform/_warps.py:24: UserWarning: The default multichannel argument (None) is deprecated.  Please specify either True or False explicitly.  multichannel will default to False starting with release 0.16.
warn('The default multichannel argument (None) is deprecated.  Please '
/root/.pyenv/versions/3.7.0/envs/py37/lib/python3.7/site-packages/skimage/transform/_warps.py:105: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
warn("The default mode, 'constant', will be changed to 'reflect' in "
/root/.pyenv/versions/3.7.0/envs/py37/lib/python3.7/site-packages/skimage/transform/_warps.py:110: UserWarning: Anti-aliasing will be enabled by default in skimage 0.15 to avoid aliasing artifacts when down-sampling images.
warn("Anti-aliasing will be enabled by default in skimage 0.15 to "

Accuracy for image 0: 0.8004
Accuracy for image 1: 0.9556
Accuracy for image 2: 0.9639
Accuracy for image 3: 0.8791
Accuracy for image 4: 0.9775
Accuracy for image 5: 0.6813
Accuracy for image 6: 0.6663
Accuracy for image 7: 0.6688
Accuracy for image 8: 0.8411
Accuracy for image 9: 0.9544
Accuracy for image 10: 0.8723
Accuracy for image 11: 0.8129
Accuracy for image 12: 0.7385
Accuracy for image 13: 0.6655
Accuracy for image 14: 0.7487
Accuracy for image 15: 0.5043
Mean accuracy: 0.7957

# Visualize segmentation results

N = len(imgs)
plt.figure(figsize=(15,60))
for i in range(N):

plt.subplot(N, 3, (i * 3) + 1)
plt.imshow(imgs[i])
plt.axis('off')

plt.subplot(N, 3, (i * 3) + 2)
plt.axis('off')

plt.subplot(N, 3, (i * 3) + 3)
plt.imshow(segmentations[i], cmap='viridis')
plt.axis('off')

plt.show()


Include a detailed evaluation of the effect of varying segmentation parameters (feature transform, clustering method, number of clusters, resize) on the mean accuracy of foreground-background segmentations on the provided dataset. You should test a minimum of 10 combinations of parameters. To present your results, add rows to the table below (you may delete the first row).

Feature Transform Clustering Method Number of segments Scale Mean Accuracy
Color K-Means 3 0.5 0.7957

スポンサーリンク
スポンサーリンク

フォローする