From 73834f5ef1c7be73d054baf4cf5f14a39fef17dc Mon Sep 17 00:00:00 2001
From: Jon Whiteaker <jbw@berkeley.edu>
Date: Mon, 5 Mar 2012 11:12:04 -0800
Subject: jon's final pass part one

---
 experimental.tex | 121 ++++++++++++++++++++++++++++++-------------------------
 1 file changed, 67 insertions(+), 54 deletions(-)

(limited to 'experimental.tex')

diff --git a/experimental.tex b/experimental.tex
index f513b92..dee0626 100644
--- a/experimental.tex
+++ b/experimental.tex
@@ -1,40 +1,51 @@
 \section{Real-World Evaluation}
 \label{sec:experiment}
 
-We conduct a real-life uncontrolled experiment using the Kinect to test to the
-algorithm.  First we describe our approach to 
-data collection.  Second we describe how the data is processed and classified.
-Finally, we discuss the results.
+We conduct a real-life uncontrolled experiment using the Kinect to test our
+algorithms.  First we describe our approach to data collection.  Second we
+describe how the data is processed and classified.  Finally, we discuss the
+results.
 
 \subsection{Dataset} 
+\label{sec:experiment:dataset}
 
 The Kinect outputs three primary signals in real-time: a color image stream, a
 depth image stream, and microphone output (\fref{fig:hallway}).  For our
 purposes, we focus on the depth image stream.  As the Kinect was designed to
 interface directly with the Xbox 360, the tools to interact with it on a PC are
-limited.  The OpenKinect project released libfreenect~\cite{libfreenect}, a
-reverse engineered driver which gives access to the raw depth images of the
-Kinect.  This raw data could be used to implement skeleton fitting algorithms,
-\eg those of Plagemann~\etal{}~\cite{plagemann:icra10}.  Alternatively,
+limited.  The OpenKinect project released
+\textsf{libfreenect}~\cite{libfreenect}, a reverse engineered driver which
+gives access to the raw depth images of the Kinect.  This raw data could be
+used to implement skeleton fitting algorithms, \eg those of
+Plagemann~\etal{}~\cite{plagemann:icra10}.  Alternatively,
 OpenNI~\cite{openni}, an open framework led by PrimeSense, the company behind
-the technology of the Kinect, offers figure detection and skeleton fitting
+the technology of the Kinect, offers figure tracking and skeleton fitting
 algorithms on top of raw access to the data streams.  More recently, the Kinect
-for Windows SDK~\cite{kinect-sdk} was released, and its skeleton fitting
-algorithm operates in real-time without calibration.  
-
-Prior to the release of the Kinect SDK, we experimented with using OpenNI for
-skeleton recognition with positive results.  Unfortunately, the skeleton
-fitting algorithm of OpenNI requires each individual to strike a specific pose
-for calibration, making it more difficult to collect a lot of data.  Upon the
-release of the Kinect SDK, we selected it to perform our data collection, given
-that it is the state-of-the-art and does not require calibration.
+for Windows SDK~\cite{kinect-sdk} was released, also with figure tracking
+and skeleton fitting algorithms.
+%and its skeleton fitting
+%algorithm operates in real-time without calibration.  
+
+We evaluated both OpenNI and the Kinect SDK for skeleton recognition.  The
+skeleton fitting algorithm of OpenNI requires each individual to strike a
+specific pose for calibration, making it more difficult to collect a lot of
+data.  We select the Kinect SDK to perform our data collection since it 
+operates in real-time without calibration.  
+
+%Prior to the release of the Kinect SDK, we experimented with using OpenNI for
+%skeleton recognition with positive results.  Unfortunately, the skeleton
+%fitting algorithm of OpenNI requires each individual to strike a specific pose
+%for calibration, making it more difficult to collect a lot of data.  Upon the
+%release of the Kinect SDK, we selected it to perform our data collection, given
+%that it is the state-of-the-art and does not require calibration.
 
 We collect data using the Kinect SDK over a period of a week in a research
 laboratory setting.  The Kinect is placed at the tee of a well traversed
 hallway.  The view of the Kinect is seen in \fref{fig:hallway}, showing the
 color image, the depth image, and the fitted skeleton of a person in a single
-frame.  For each frame where a person is detected and a skeleton is fitted we
-capture the 3D coordinates of 20 body joints, and the color image.
+frame.  Skeletons are fit from \~1-5 meters away from the Kinect.  For each
+frame where a person is detected and a skeleton is fit we capture the 3-D
+coordinates of 20 body joints, and the color image.
 
 \begin{figure}[t]
   \begin{center}
@@ -53,16 +64,10 @@ Kinect SDK. Inferred means that even though the joint is not visible in the
 frame, the skeleton-fitting algorithm attempts to guess the right location.
 
 
-Ground truth person identification is obtained by manually labelling each run
-based on the images captured by the RGB camera of the Kinect. For ease of
-labelling, only the runs with people walking toward the camera are kept. These
-are the runs where the average distance from the skeleton joints to the camera
-is increasing.
-
 \subsection{Experiment design}
 \label{sec:experiment-design}
 
-We preprocess the data set to extract \emph{features}
+We preprocess the dataset to extract \emph{features}
 from the raw data.  First, the lengths of 15 body parts are computed from the
 joint coordinates. These are distances between two contiguous joints in the
 human body. If one of the two joints of a body part is not present or inferred
@@ -94,11 +99,19 @@ ShoulderCenter-Spine &\\
 \end{table}
 \vspace{-2.5\baselineskip}
 
-Each detected skeleton also has an ID number which identifies the figure it
-maps to from the figure detection stage. When there are consecutive frames with
-the same ID, it means that the skeleton-fitting algorithm was able to detect
-the skeleton in a contiguous way. This allows us to define the concept of a
-\emph{run}: a sequence of frames with the same skeleton ID.
+Each detected skeleton also has an ID number obtained from the figure detection
+stage. When there are consecutive frames with the same ID, it means that figure
+detection was able to track the figure in a contiguous way. This allows us to
+define the concept of a \emph{run}: a sequence of frames with the same skeleton
+ID. Because of errors in the depth image when a figure enters or exits the
+range of the camera, we only keep the frames of a run that are 2-3 meters away
+from the Kinect.
+
+Ground truth person identification is obtained by manually labelling each run
+based on the images captured by the color camera of the Kinect. For ease of
+labelling, only the runs with people walking toward the camera are kept. These
+are the runs where the average distance from the skeleton joints to the camera
+is increasing.
 
 We perform five experiments.  First, we test the performance of
 skeleton recognition using traditional 10-fold cross validation, to
@@ -133,23 +146,23 @@ happens if the noise from the Kinect is reduced.
   \end{center}
   \vspace{-1.5\baselineskip}
   \caption{Distribution of the frequency of each individual in the
-  data set}
+  dataset}
   \label{fig:frames}
 \end{figure}
 
 \subsection{Offline learning setting}
 \label{sec:experiment:offline}
 
-In the first experiment, we study the accuracy of skeleton recognition
-using 10-fold cross validation.  The data set is partitioned into 10
-continuous time sequences of equal size. For a given recall threshold,
-the algorithm is trained on 9 sequences and tested on the last
-one. This is repeated for all 10 possible testing sequences. Averaging
-the prediction rate over these 10 training-testing experiments yields
-the prediction rate for the chosen threshold. We test the mixture of
-Gaussians (MoG) and sequential hypothesis testing (SHT) models, and
-find that SHT generally performs better than MoG, and that accuracy
-increases as group size decreases.
+In the first experiment, we study the accuracy of skeleton recognition using
+10-fold cross validation.  The dataset is partitioned into 10 continuous time
+sequences of equal size. For a given recall threshold, the algorithm is trained
+on 9 sequences and tested on the last one. This is repeated for all 10 possible
+testing sequences. Averaging the prediction rate over these 10 training-testing
+experiments yields the prediction rate for the chosen threshold. We test the
+mixture of Gaussians (MoG) and sequential hypothesis testing (SHT) models, with
+varying group size $n_p = \{3,5,10,25\}$.
+%and find that SHT generally performs better than MoG, and that accuracy
+%increases as group size decreases.
 
 
 \fref{fig:offline} shows the precision-recall plot as the threshold varies.
@@ -158,7 +171,7 @@ Both algrithms perform three times better than the majority class baseline of
 different group sizes: people are ordered based on their frequency of
 appearance (\fref{fig:frames}), and all the frames belonging to people beyond a
 given rank in this ordering are removed.  The decrease of performance when
-increasing the number of people in the data set can be explained by the
+increasing the number of people in the dataset can be explained by the
 overlaps between skeleton profiles due to the noise, as discussed in
 Section~\ref{sec:uniqueness}, but also by the very few number of runs available
 for the least present people, as seen in \fref{fig:frames}, which does not
@@ -197,8 +210,8 @@ setting.  Even though the previous evaluation is standard, it does not properly
 reflect reality. A real-life setting could be as follows. The camera is placed
 at the entrance of a building. When a person enters the building, his identity
 is detected based on the electronic key system and a new labeled run is added
-to the data set. The identification algorithm is then retrained on the
-augmented data set, and the newly obtained classifier can be deployed in the
+to the dataset. The identification algorithm is then retrained on the
+augmented dataset, and the newly obtained classifier can be deployed in the
 building. 
 
 In this setting, the sequential hypothesis testing (SHT) algorithm is more
@@ -248,15 +261,15 @@ experiments.
 In the third experiment, we compare the performance of skeleton recognition
 with the performance of face recognition as given by \textsf{face.com}.  At the
 time of writing, this is the best performing face recognition algorithm on the
-LFW data set~\footnote{\url{http://vis-www.cs.umass.edu/lfw/results.html}}.
+LFW dataset~\footnote{\url{http://vis-www.cs.umass.edu/lfw/results.html}}.
 The results show that face recognition has better accuracy than skeleton
 recognition, but not by a large margin.
 
 We use the publicly available REST API of \textsf{face.com} to do face
-recognition on our data set.  Due to the restrictions of the API, for this
+recognition on our dataset.  Due to the restrictions of the API, for this
 experiment we train on one half of the data and test on the remaining half. For
 comparison, MoG algorithm is run with the same training-testing partitioning of
-the data set. In this setting, SHT is not relevant for the comparison, because
+the dataset. In this setting, SHT is not relevant for the comparison, because
 \textsf{face.com} does not give the possibility to mark a sequence of frames as
 belonging to the same run. This additional information would be used by the SHT
 algorithm and would thus bias the results in favor of skeleton recognition.
@@ -299,7 +312,7 @@ obvious one is when people are walking away from the camera. Coming back to the
 raw data collected during the experiment design, we manually label the runs of
 people walking away from the camera. In this case, it is harder to get the
 ground truth classification and some of runs are dropped because it is not
-possible to recognize the person. Apart from that, the data set reduction is
+possible to recognize the person. Apart from that, the dataset reduction is
 performed exactly as explained in Section~\ref{sec:experiment-design}. Our
 results show that we can identify people walking away from the camera just as
 well as when they are walking towards the camera.
@@ -316,9 +329,9 @@ well as when they are walking towards the camera.
 
 \fref{fig:back} compares the curve obtained in \xref{sec:experiment:offline}
 with people walking toward the camera, with the curve obtained by running the
-same experiment on the data set of runs of people walking away from the camera.
+same experiment on the dataset of runs of people walking away from the camera.
 The two curves are sensibly the same. However, one could argue that as the two
-data sets are completely disjoint, the SHT algorithm is not learning the same
+datasets are completely disjoint, the SHT algorithm is not learning the same
 profile for a person walking toward the camera and for a person walking away
 from the camera. The third curve of \fref{fig:back} shows the precision-recall
 curve when training and testing on the combined dataset of runs toward and away
@@ -334,7 +347,7 @@ the Kinect.
 %perfectly distinguish two different skeletons at a low noise level. Therefore,
 %the only source of classification error in our algorithm is the dispersion of
 %the observed limbs' lengths away from the exact measurements.
-To simulate a reduction of the noise level, the data set is modified as
+To simulate a reduction of the noise level, the dataset is modified as
 follows: we compute the average profile of each person, and for each frame we
 divide the empirical variance from the average by 2. Formally, using
 the same notations as in Section~\ref{sec:mixture of Gaussians}, each
@@ -348,7 +361,7 @@ camera.
 
 \fref{fig:var} compares the Precision-recall curve of
 \fref{fig:offline:sht} to the curve of the same experiment run on
-the newly obtained data set.
+the newly obtained dataset.
 
 %\begin{figure}[t]
 %  \begin{center}
-- 
cgit v1.2.3-70-g09d2

