\section{Real-World Evaluation}
\label{sec:experiment}

We conduct a real-life uncontrolled experiment using the Kinect to test to the
algorithm.  First we present the manner and environment in which we perform
data collection.  Second we describe how the data is processed and classified.
Finally, we discuss the results.

\subsection{Dataset} 

The Kinect outputs three primary signals in real-time: a color image stream, a
depth image stream, and microphone output.  For our purposes, we focus on the
depth image stream.  As the Kinect was designed to interface directly with the
Xbox 360, the tools to interact with it on a PC are limited.
Libfreenect~\cite{libfreenect} is a reverse engineered driver which gives
access to the raw depth images from the Kinect.  This raw data could be used to
implement the algorithms \eg of Plagemann~\etal{}~\cite{plagemann:icra10}.
Alternatively, OpenNI~\cite{openni}, a framework sponsored by
PrimeSense~\cite{primesense}, the company behind the technology of the Kinect,
offers figure detection and skeleton fitting algorithms on top of raw access to
the data streams.  However, the skeleton fitting algorithm of OpenNI requires
each individual to strike a specific pose for calibration.  More recently, the
Kinect for Windows SDK~\cite{kinect-sdk} was released, and its skeleton fitting
algorithm operates in real-time without calibration.  Given that the Kinect for
Windows SDK is the state-of-the-art, we use it to perform our data collection.

We collect data using the Kinect SDK over a period of a week in a research
laboratory setting.  The Kinect is placed at the tee of a well traversed
hallway.  The view of the Kinect is seen in \fref{fig:hallway}, showing the
color image, the depth image, and the fitted skeleton of a person in a single
frame.  For each frame where a person is detected and a skeleton is fitted we
collect the 3D coordinates of 20 body joints, and the color image recorded by
the RGB camera.

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.99\textwidth]{graphics/hallway.png}
  \end{center}
  \caption{Experiment setting. Color image, depth image, and fitted
  skeleton as captured by the Kinect in a single frame}
  \label{fig:hallway}
\end{figure}

For some frames, one or several joints are out of the frame or are occluded by
another part of the body. In those cases, the coordinates of these joints are
either absent from the frame or present but tagged as \emph{Inferred} by the
Kinect SDK. Inferred means that even though the joint is not visible in the
frame, the skeleton-fitting algorithm attempts to guess the right location.


Ground truth person identification is obtained by manually labelling each run
based on the images captured by the RGB camera of the Kinect. For ease of
labelling, only the runs with people walking toward the camera are kept. These
are the runs where the average distance from the skeleton joints to the camera
is increasing.

\subsection{Experiment design}
\label{sec:experiment-design}

Several reductions are then applied to the data set to extract \emph{features}
from the raw data.  First, the lengths of 15 body parts are computed from the
joint coordinates. These are distances between two contiguous joints in the
human body. If one of the two joints of a body part is not present or inferred
in a frame, the corresponding body part is reported as absent for the frame.
Second, the number of features is reduced to 9 by using the vertical symmetry
of the human body: if two body parts are symmetric about the vertical axis, we
bundle them into one feature by averaging their lengths. If only one of them is
present, we take the value of its counterpart. If none of them are present, the
feature is reported as missing for the frame. The resulting nine features are:
Head-ShoulderCenter, ShoulderCenter-Shoulder, Shoulder-Elbow, Elbow-Wrist,
ShoulderCenter-Spine, Spine-HipCenter, HipCenter-HipSide, HipSide-Knee,
Knee-Ankle.  Finally, any frame with a missing feature is filtered out.

Each detected skeleton also has an ID number which identifies the figure it
maps to from the figure detection stage. When there are consecutive frames with
the same ID, it means that the skeleton-fitting algorithm was able to detect
the skeleton in a contiguous way. This allows us to define the concept of a
\emph{run}: a sequence of frames with the same skeleton ID.

\begin{table}
\begin{center}
\caption{Data set statistics. The right part of the table shows the
average numbers for different intervals of $k$, the rank of a person
in the ordering given by the number of frames}
\label{tab:dataset}
\begin{tabular}{|l|r||r|r|r|}
\hline
Number of people & 25 & $k\leq 5$ & $5\leq k\leq 20$ & $k\geq 20$\\
\hline
Number of frames & 15945 & 1211 & 561 & 291 \\
\hline
Number of runs & 244 & 18 & 8 & 4\\
\hline
\end{tabular}
\end{center}
\end{table}

\subsection{Results}

\paragraph{Offline setting.}

The mixture of Gaussians model is evaluated on the whole dataset by
doing 10-fold cross validation: the data set is partitioned into 10
subsamples of equal size. For a given recall threshold, the algorithm
is trained on 9 subsamples and trained on the last one. This is
repeated for the 10 possible testing subsample. Averaging the
prediction rate over these 10 training-testing experiments yields the
prediction rate for the chosen threshold.

Figure \ref{fig:mixture} shows the precision-recall plot as the
threshold varies. Several curves are obtained for different group
sizes: people are ordered based on their numbers of frames, and all
the frames belonging to someone beyond a given rank in this ordering
are removed from the data set. The decrease of performance when
increasing the number of people in the data set can be explained
by the overlaps between skeleton profiles due to the noise, as
discussed in Section~\ref{sec:uniqueness}, but also by the very few
number of runs available for the least present people, as seen in
Table~\ref{tab:dataset}, which does not permit a proper training of
the algorithm.

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.80\textwidth]{graphics/10fold-naive.pdf}
  \end{center}
  \caption{Precision-Recall curve for the mixture of Gaussians model
  with 10-fold cross validation. The data set is restricted to the top
  $n$ most present people}
  \label{fig:mixture}
\end{figure}

\paragraph{Online setting.}

Even though the previous evaluation is standard, it does not properly
reflect the reality. A real-life setting could be the following: the
camera is placed at the entrance of a building. When a person enters
the building, his identity is detected based on the electronic key
system and a new labeled run is added to the data set. The
identification algorithm is then retrained on the augmented data set,
and the newly obtained classifier can be deployed in the building.

In this setting, the Sequential Hypothesis Testing (SHT) algorithm is more
suitable than the algorithm used in the previous paragraph, because it
accounts for the fact that a person identity does not change across a
run. The analysis is therefore performed by partitioning the dataset
into 10 subsamples of equal size. For a given threshold, the algorithm
is trained and tested incrementally: trained on the first $k$
subsamples (in the chronological order) and tested on the $(k+1)$-th
subsample. Figure~\ref{fig:sequential} shows the prediction-recall
curve when averaging the prediction rate of the 10 incremental
experiments.

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.80\textwidth]{graphics/online-sht.pdf}
  \end{center}
  \caption{Precision-Recall curve for the sequential hypothesis
  testing algorithm in the online setting. $n$ is the size of the
  group as in Figure~\ref{fig:mixture}}
  \label{fig:sequential}
\end{figure}

\paragraph{Face recognition.}

We then compare the performance of skeleton recognition with the
performance of face recognition as given by \textsf{face.com}
\todo{REFERENCE NEEDED}. At the time of writing, this is the best
performing face recognition algorithm on the LFW data set
\cite{face-com}.

We use the publicly available REST API of \textsf{face.com} to do face
recognition on our data set: the training is done on half of the data
and the testing is done on the remaining half. For comparison, the
Gaussian mixture algorithm is run with the same training-testing
partitioning of the data set. In this setting, the Sequential
Hypothesis Testing algorithm is not relevant for the comparison,
because \textsf{face.com} does not give the possibility to mark a
sequence of frames as belonging to the same run. This additional
information would be used by the SHT algorithm and would thus bias the
results in favor of skeleton recognition.

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.80\textwidth]{graphics/face.pdf}
  \end{center}
  \caption{Precision-Recall curve for face recognition and skeleton recognition}
  \label{fig:face}
\end{figure}

\paragraph{People walking away from the camera.}

The performance of face recognition and skeleton recognition are
comparable in the previous setting \todo{is that really
true?}. However, there are many cases where only skeleton recognition
is possible. The most obvious one is when people are walking away from
the camera. Coming back to the raw data collected during the
experiment design, we manually label the runs of people walking away
from the camera. In this case, it is harder to get the ground truth
classification and some of runs are dropped because it is not possible
to recognize the person. Apart from that, the data set reduction is
performed exactly as explained in Section~\ref{sec:experiment-design}.

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.80\textwidth]{graphics/back.pdf}
  \end{center}
  \caption{Precision-Recall curve for the sequential hypothesis
  testing algorithm in the online setting with people walking away
  from and toward the camera. All the people are included}
  \label{fig:back}
\end{figure}

Figure~\ref{fig:back} compares the curve obtained in the online
setting with people walking toward the camera, with the curve obtained
by running the same experiment on the data set of runs of people
walking away from the camera. The two curves are sensibly the
same. However, one could argue that as the two data sets are
completely disjoint, the SHT algorithm is not learning the same
profile for a person walking toward the camera and for a person
walking away from the camera. Figure~\ref{fig:back2} shows the
Precision-Recall curve when training on runs toward the camera and
testing on runs away from the camera.

\todo{PLOT NEEDED}

\paragraph{Reducing the noise.} Predicting potential improvements of
the prediction rate of our algorithm is straightforward. The algorithm
relies on 9 features only. Section~\ref{sec:uniqueness} shows that
6 of these features alone are sufficient to perfectly distinguish two
different skeletons at a low noise level. Therefore, the only source
of classification error in our algorithm is the dispersion of the
observed limbs' lengths away from the exact measurements.

To simulate a possible reduction of the noise level, the data set is
modified as follows: all the observations for a given person are
homothetically contracted towards their average so as to divide their
empirical variance by 2. Formally, if $x$ is an observation in the
9-dimensional feature space for the person $i$, and if $\bar{x}$ is
the average of all the observations available for this person in the
data set, then $x$ is replaced by $x'$ defined by:
\begin{equation}
  x' = \bar{x} + \frac{x-\bar{x}}{\sqrt{2}}
\end{equation}
We believe that a reducing factor of 2 for the noise's variance is
realistic given the relative low resolution of the Kinect's infrared
camera. 

Figure~\ref{fig:var} compares the Precision-Recall curve of
Figure~\ref{fig:sequential} to the curve of the same experiment run on
the newly obtained data set.

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.80\textwidth]{graphics/var.pdf}
  \end{center}
  \caption{Precision-Recall curve for the sequential hypothesis
  testing algorithm in the online setting for all the people with and
  without halving the variance of the noise}
  \label{fig:var}
\end{figure}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "kinect"
%%% End: