\section{Real-World Evaluation}
\label{sec:experiment}

We conduct a real-life uncontrolled experiment using the Kinect to test our
algorithms.  First we describe our approach to data collection.  Second we
describe how the data is processed and classified.  Finally, we discuss the
results.

\subsection{Dataset} 
\label{sec:experiment:dataset}

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.49\textwidth]{graphics/frames.pdf}
  \end{center}
  \vspace{-1.5\baselineskip}
  \caption{Distribution of the frequency of each individual in the
  dataset}
  \label{fig:frames}
\end{figure}

\begin{figure*}[t]
\begin{center}
\subfloat[Mixture of Gaussians]{
    \includegraphics[width=0.49\textwidth]{graphics/offline-nb.pdf}
    \label{fig:offline:nb}
}
\subfloat[Sequential Hypothesis Testing]{
    \includegraphics[width=0.49\textwidth]{graphics/offline-sht.pdf}
    \label{fig:offline:sht}
}
  \caption{Results with 10-fold cross-validation for the top $n_p$ most present people}
\label{fig:offline}
\end{center}
\end{figure*}

The Kinect outputs three primary signals in real-time: a color image stream, a
depth image stream, and microphone output (\fref{fig:hallway}).  For our
purposes, we focus on the depth image stream.  
%As the Kinect was designed to
%interface directly with the Xbox 360, the tools to interact with it on a PC are
%limited.  The OpenKinect project released
%\textsf{libfreenect}~\cite{libfreenect}, a reverse engineered driver which
%gives access to the raw depth images of the Kinect.  This raw data could be
%used to implement skeleton fitting algorithms, \eg those of
%Plagemann~\etal{}~\cite{plagemann:icra10}.  Alternatively,
%OpenNI~\cite{openni}, an open framework led by PrimeSense, the company behind
%the technology of the Kinect, offers figure tracking and skeleton fitting
%algorithms on top of raw access to the data streams.  More recently, the Kinect
%for Windows SDK~\cite{kinect-sdk} was released, also with figure tracking
%and skeleton fitting algorithms.
%%and its skeleton fitting
%%algorithm operates in real-time without calibration.  
%
We evaluated both OpenNI and the Kinect SDK for skeleton recognition.  The
skeleton fitting algorithm of OpenNI requires each individual to strike a
specific pose for calibration, making it more difficult to collect a lot of
data.  We select the Kinect SDK to perform our data collection since it 
operates in real-time without calibration.  

%Prior to the release of the Kinect SDK, we experimented with using OpenNI for
%skeleton recognition with positive results.  Unfortunately, the skeleton
%fitting algorithm of OpenNI requires each individual to strike a specific pose
%for calibration, making it more difficult to collect a lot of data.  Upon the
%release of the Kinect SDK, we selected it to perform our data collection, given
%that it is the state-of-the-art and does not require calibration.

We collect data using the Kinect SDK over a period of a week in a research
laboratory setting.  The Kinect is placed at the tee of a frequently used
hallway.  For each frame, the Kinect SDK performs figure detection to identify
regions of interest.  Then, it fits a skeleton to the identified figures and
outputs a set of joints in real world coordinates.  The view from the Kinect
SDK is seen in \fref{fig:hallway}, showing the color image, the depth image
with detected figures, and the fitted skeleton of a person in a single frame.
Skeletons are fit from roughly 1-5 meters away from the Kinect.  For each frame
with a skeleton we record the color image and the positions of the joints.

For some frames, one or several joints are out of the frame or are occluded by
another part of the body. In those cases, the coordinates of these joints are
either absent from the frame or present but tagged as \emph{Inferred} by the
Kinect SDK. Inferred means that even though the joint is not visible in the
frame, the skeleton-fitting algorithm attempts to guess the right location.
Note that in the experiment design we exclude inferred data points.

\subsection{Experiment design}
\label{sec:experiment-design}

We preprocess the dataset to extract \emph{features}
from the raw data.  First, the lengths of 15 body parts are computed from the
joint coordinates. These are distances between two contiguous joints in the
human body. If one of the two joints of a body part is not present or inferred
in a frame, the corresponding body part is reported as absent for the frame.
Second, we reduce the number of features to nine by using the vertical symmetry
of the human body: if two body parts are symmetric about the vertical axis, we
bundle them into one feature by averaging their lengths. If only one of them is
present, we take its value. If neither of them is present, the feature is
reported as missing for the frame. Any frame with a missing feature is
filtered out.  The resulting nine features include the six arm, leg, and pelvis
measurements from \xref{sec:uniqueness}, and three additional measurements:
spine length, shoulder breadth, and head size.  Here we list the nine features as
pairs of joints:
%The resulting nine features are: Head-ShoulderCenter, ShoulderCenter-Shoulder,
%Shoulder-Elbow, Elbow-Wrist, ShoulderCenter-Spine, Spine-HipCenter,
%HipCenter-HipSide, HipSide-Knee, Knee-Ankle.  

\vspace{\baselineskip}
\begin{table}[!h]
\begin{center}
\vspace{-1.5\baselineskip}
\begin{tabular}{ll}
Head-ShoulderCenter & Elbow-Wrist\\ 
HipCenter-Hip & ShoulderCenter-Shoulder\\
ShoulderCenter-Spine & Hip-Knee\\
Shoulder-Elbow & Spine-HipCenter\\
Knee-Ankle& \\
\end{tabular}
\vspace{-2.5\baselineskip}
\end{center}
\vspace{1\baselineskip}
\end{table}

Each detected skeleton also has an ID number obtained from the figure detection
stage. When there are consecutive frames with the same ID, it means that figure
detection was able to track the figure in a contiguous way. This allows us to
define the concept of a \emph{run}: a sequence of frames with the same skeleton
ID. Because of errors in the depth image when a figure enters or exits the
range of the camera, we only keep the frames of a run that are 2-3 meters away
from the Kinect.

Ground truth person identification is obtained by manually labelling each run
based on the images captured from the color image stream of the Kinect. For ease of
labelling, only the runs with people walking toward the Kinect are kept. These
are the runs where the average distance from the skeleton joints to the Kinect
is increasing.

We perform five experiments.  First, we test the performance of
skeleton recognition using traditional 10-fold cross validation, to
represent an offline learning setting.  Second, we run our algorithms
in an online learning setting by training and testing the data over
time.  Third, we pit skeleton recognition against the state-of-the-art
in face recognition.  Next, we test how our solution performs when
people are walking away from the camera.  Finally, we study what
happens if the noise from the Kinect is reduced.

%\begin{table}
%\begin{center}
%\caption{Data set statistics. The right part of the table shows the
%average numbers for different intervals of $k$, the rank of a person
%in the ordering given by the number of frames}
%\label{tab:dataset}
%\begin{tabular}{|l|r||r|r|r|}
%\hline
%Number of people & 25 & $k\leq 5$ & $5\leq k\leq 20$ & $k\geq 20$\\
%\hline
%Number of frames & 15945 & 1211 & 561 & 291 \\
%\hline
%Number of runs & 244 & 18 & 8 & 4\\
%\hline
%\end{tabular}
%\end{center}
%\end{table}

\subsection{Offline learning setting}
\label{sec:experiment:offline}

In the first experiment, we study the accuracy of skeleton recognition using
10-fold cross validation.  The dataset is partitioned into 10 continuous time
sequences of equal size. For a given recall threshold, the algorithm is trained
on 9 sequences and tested on the last one. This is repeated for all 10 possible
testing sequences. Averaging the prediction rate over these 10 training-testing
experiments yields the prediction rate for the chosen threshold. We test the
mixture of Gaussians (MoG) and sequential hypothesis testing (SHT) models, with
varying group size $n_p = \{3,5,10,25\}$.
%and find that SHT generally performs better than MoG, and that accuracy
%increases as group size decreases.


\fref{fig:offline} shows the precision-recall plot as the threshold varies.
Both algrithms perform three times better than the majority class baseline of
15\% with a recall of 100\% on all people.  We make two main observations.
First, as expected, SHT performs better than MoG because of temporal smoothing.
Second, performance is inversely proportional to group size.  As we test
against more people, there are more overlaps between skeleton profiles due to
the noise, as discussed in Section~\ref{sec:uniqueness}.  Also, the least
present people have a small number of frames, as seen in \fref{fig:frames},
which may not permit a proper training of the algorithm.  For 3 and 5
people (typical family sizes), we see recognition rates mostly above 90\%, and
we reach 90\% accuracy at 60\% recall for a group size of 10 people.

%Several curves are obtained for
%different group sizes: people are ordered based on their frequency of
%appearance (\fref{fig:frames}), and all the frames belonging to people beyond a
%given rank in this ordering are removed.  The decrease of performance when
%increasing the number of people in the dataset can be explained by the
%overlaps between skeleton profiles due to the noise, as discussed in
%Section~\ref{sec:uniqueness}, but also by the very few number of runs available
%for the least present people, as seen in \fref{fig:frames}, which does not
%permit a proper training of the algorithm.

%\begin{figure}[t]
%  \begin{center}
%    \includegraphics[width=0.80\textwidth]{graphics/10fold-naive.pdf}
%  \end{center}
%  \caption{Precision-Recall curve for the mixture of Gaussians model
%  with 10-fold cross validation. The data set is restricted to the top
%  $n$ most present people}
%  \label{fig:mixture}
%\end{figure}

\subsection{Online learning setting}

In the second experiment, we evaluate skeleton recognition in an online
setting.  Even though the previous evaluation is standard, it does not properly
reflect reality. A real-world setting could be as follows. The camera is placed
at the entrance of a building. When a person enters the building, his identity
is detected based on the electronic key system and a new labeled run is added
to the dataset. The identification algorithm is then retrained on the
augmented dataset, and the newly obtained classifier can be deployed in the
building. 

%In this setting, the sequential hypothesis testing (SHT) algorithm is more
%suitable than the algorithm used in Section~\ref{sec:experiment:offline}, because it
%accounts for the fact that a person identity does not change across a
%run. 
We only evaluate SHT in this setting since it already takes consecutive frames
into account and because it performed better than MoG in the offline setting
(\xref{sec:experiment:offline}).  We partition the dataset into 10 time
sequences of equal size. For a given threshold, the algorithm is trained and
tested incrementally: train on the first $k$ sequences (in the chronological
order) and test on the $(k+1)$-th sequence. \fref{fig:online} shows the
prediction-recall curve when averaging the prediction rate over the 10
incremental experiments. Overall performance is worse than in
\fref{fig:offline:sht} since the system trains on less data than in
\xref{sec:experiment:offline} in all but the last step.  We still see
recognition rates mostly above 90\% for group sizes of 3 and 5.

\begin{figure}[t]
%\subfloat[Mixture of Gaussians]{
%    \includegraphics[width=0.49\textwidth]{graphics/online-nb.pdf}
%    \label{fig:online:nb}
%}
%\subfloat[Sequential hypothesis testing]{
\begin{center}
    \includegraphics[width=0.49\textwidth]{graphics/online-sht.pdf}
\end{center}
\vspace{-1.5\baselineskip}
    \caption{Results for the online setting, where $n_p$ is the size of
    the group as in Figure~\ref{fig:offline}}
    \label{fig:online}
\end{figure}

\begin{figure}[t!]
  \begin{center}
    \includegraphics[width=0.49\textwidth]{graphics/face.pdf}
  \end{center}
\vspace{-1.5\baselineskip}
  \caption{Results for face recognition versus skeleton recognition
  with $n_p=5$ people}
  \label{fig:face}
\end{figure}

\subsection{Face recognition}

In the third experiment, we compare the performance of skeleton recognition
with the performance of face recognition.  For this experiment we set $n_p = 5$
and train on one half of the data and test on the remaining half. For
comparison, the MoG algorithm is run with the same training-testing
partitioning of the dataset.  The results are shown in \fref{fig:face}.
Skeleton recognition performs within 10\% of face recognition at most
thresholds.

%In the third experiment, we compare the performance of skeleton recognition
%with the performance of face recognition as given by \textsf{face.com}.  At the
%time of writing, this is the best performing face recognition algorithm on the
%LFW dataset\footnote{\url{http://vis-www.cs.umass.edu/lfw/results.html}}.
%
%We use the REST API of \textsf{face.com} to do face recognition on our dataset.
%Due to the restrictions of the API, for this experiment we set $n_p = 5$ and
%train on one half of the data and test on the remaining half. For comparison,
%the MoG algorithm is run with the same training-testing partitioning of the
%dataset. In this setting, SHT is not relevant for the comparison, because
%\textsf{face.com} does not give the possibility to mark a sequence of frames as
%belonging to the same run. This additional information would be used by the SHT
%algorithm and would thus bias the experiment in favor of skeleton recognition.
%The results are shown in \fref{fig:face}.  Skeleton recognition performs
%within 10\% of face recognition at most thresholds.
%outperforms
%skeleton recognition, but by less than 10\% at most thresholds. 
%These results are promising, given that \textsf{face.com} is the
%state-of-the-art in face recognition.

%However, this result does not take into account the disparity in the number of
%runs which face recognition and skeleton recognition can classify frames,
%which we discuss in the next experiment.

\subsection{Walking away}

In the next experiment, we include the runs in which people are walking away
from the Kinect that we could positively identify.  While, face
recognition outperforms skeleton recognition in the previous setting,
there are many cases where only skeleton recognition is possible. For example,
when people are walking away from the Kinect. Coming back to the raw data
collected during the experiment design, we manually label the runs of people
walking away from the camera. In this case, it is harder to get the ground
truth classification and some of runs are dropped because it is not possible to
recognize the person. Apart from that, the dataset reduction is performed
exactly as explained in Section~\ref{sec:experiment-design}. Our results show
that we can identify people walking away from the camera comparably to when
they are walking towards the camera.

%\begin{figure}[t]
%  \begin{center}
%    \includegraphics[width=0.80\textwidth]{graphics/back.pdf}
%  \end{center}
%  \caption{Precision-Recall curve for the sequential hypothesis
%  testing algorithm in the online setting with people walking away
%  from and toward the camera. All the people are included}
%  \label{fig:back}
%\end{figure}

\fref{fig:back} compares the results obtained in \xref{sec:experiment:offline}
with people walking toward the camera, with the results of the
same experiment on the dataset of runs of people walking away from the camera.
The two results are similar. However, one could argue that as the two
datasets are completely disjoint, the SHT algorithm is not learning the same
profile for a person walking toward the camera and for a person walking away
from the camera. The third curve of \fref{fig:back} shows the precision-recall
curve when training and testing on the combined dataset of runs toward and away
from the camera with similar performance.  Note that while we could not obtain
enough labeled data for a full comparison when it is dark, manual experiments
show similar performance when there is no visible light.

\begin{figure}[t!]
  \centering
  \includegraphics[width=0.49\textwidth]{graphics/back.pdf}
\vspace{-1.5\baselineskip}
\caption{Results with people walking away from and toward the camera}
\label{fig:back}
\end{figure}

\subsection{Reducing the noise} 

For the final experiment, we explore the potential of skeleton recognition with
a higher resolution depth camera, as has been speculated for the Kinect
2~\footnote{\url{http://www.eurogamer.net/articles/2011-11-25-kinect-2-so-accurate-it}\\\url{-can-lip-read}}.
Since a higher resolution camera is not readily available, we simulate a higher
resolution by artificially reducing the noise from our Kinect dataset.
%Predicting potential improvements of the prediction rate of our
%algorithm is straightforward. The algorithm relies on 9 features only.
%\xref{sec:uniqueness} shows that 6 of these features alone are sufficient to
%perfectly distinguish two different skeletons at a low noise level. Therefore,
%the only source of classification error in our algorithm is the dispersion of
%the observed limbs' lengths away from the exact measurements.
To simulate a reduction of the noise level, the dataset is modified as follows:
we measure the average skeletal profile of each person across the entire
dataset, and for each frame we divide the empirical variance from the average
by 2. Formally, using the same notations as in Section~\ref{sec:mixture of
Gaussians}, each observation $\bx_i$ is replaced by $\bx_i'$ defined by:
\begin{equation}
  \bx_i' = \bar{\bx}_{y_i} + \frac{\bx_i-\bar{\bx}_{y_i}}{2}
\end{equation}
We believe that reducing the noise's variance by half is realistic
given the relatively low resolution of the Kinect's infrared camera. 

\fref{fig:var} compares the precision-recall curve of \fref{fig:offline:sht} to
the curve of the same experiment run on the newly obtained dataset. We observe
a roughly 20\% increase in performace across most thresholds.  We
believe these results would significantly outperform face recognition
in a similar setting.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.49\textwidth]{graphics/var.pdf}
\caption{Results with and without halving the variance of the noise}
\label{fig:var}
\end{figure}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "kinect"
%%% End: