1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
|
\documentclass[10pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[hmargin=1.2in, vmargin=1.2in]{geometry}
\usepackage{amsmath,amsfonts}
\title{\large Review of \emph{On Bayes and Nash experiment design for hypothesis testing problems}}
\date{}
\begin{document}
\maketitle
\paragraph{Summary.} This paper studies a standard ANOVA hypothesis testing
problem: given $K$ treatments, determine whether they all have the same average
effect, or whether there exists a pair of treatments whose average effects are
different. The authors consider the likelihood ratio test and the focus is on
finding and characterizing the optimal design. The contributions are as
follows:
\begin{itemize}
\item first, optimality is formulated as a maxi-min problem: find the most
powerful design for the worst possible value of the unknown treatment
effects in the alternative hypothesis. A previous paper of the authors
had established that the balanced design is optimal for this criterion.
\item next, a ``pseudo-Bayes'' approach is adopted: there is a prior on the
average treatment effects, and the goal is to find the design which is
most powerful on average over this prior. It is found that the balanced
design is optimal in that sense whenever the prior is
exchangeable.
\item next, the authors consider a game theoretic approach where the
optimal design problem is formulated as a two-player zero-sum game
between the ``max'' player attempting to find the optimal design and
the ``min'' player attempting to find the worst possible configuration
of average effects. The unique Nash equilibrium of the game is computed
for two different choices of strategy space for the max player. Again,
the balanced design is found to be the optimal strategy (assuming it is
contained in the strategy space of the max player).
\item finally, the authors consider a ``tree order'' setting in which the null
hypothesis is that all treatments have the same effects, and the
alternative is that all treatments $2\leq i\leq K$ have an effect
greater than treatment $1$. The Nash equilibrium of the two-player game
is computed.
\end{itemize}
\paragraph{Overall impression.} The problem considered in this paper is natural
and interesting but I found the results to be underwhelming. Specifically:
\begin{itemize}
\item the fact that the balanced design is optimal was already known for
some criteria (e.g. A-optimality and, by a previous of the authors, the
maxi-min criterion). As revealed by the proofs, all of these results
(including the new criteria considered in this work) express in
slightly different ways that the underlying ANOVA setting is symmetric.
\item I do not completely buy the game theoretic formulation. There is
nothing inherently strategic in the setting considered. The fact that
any maxi-min or mini-max problem can be \emph{interpreted} as
a two-player game is always true and it does not seem that the present
paper is saying more than that.
\item I am not convinced by the value of arbitrarily restricting the action
space of the max-player (the experiment designer) at the beginning of
Section 4 and of the resulting Theorem 4.1. Would it make more sense to
directly consider the general case where the max-player can choose
\emph{any} design, and in particular, where the balanced design is part
of the action space. In other words, the story leading to Theorem 4.3
seems unnecessary to me and I think this theorem could have been stated
first.
\end{itemize}
\paragraph{Minor comments.}
\begin{itemize}
\item In the definition of $\mathcal{M}_\delta$ (equation (7)), it would
be good to have a remark about how $\delta$ is chosen. For examples,
what are implications of choosing one value vs.\ another value, how to
choose it in practice, etc.
\item In section 4, I think it would be better to define the value of game
outside of Theorem 4.2. Defining it formally, explain that it is only
defined when $\min\max\dots = \max\min\dots$ and that it requires the
action space to be convex.
\item related to the previous point, the use of the word \emph{value} in
the proofs is inconsistent. It seems to be sometimes used to refer the
value of an arbitrary pair of strategies, whereas in game theory, it
usually only refers to the $\min\max$ value.
\item paragraph above Theorem 4.3: the phrasing ``Since any $\xi$ can be
written as mixture of other elements\dots'' is a bit vague. It would be
better to simply say: ``Since the action space $\Xi$ is convex''.
Relatedly, when introducing mixed strategies, I would emphasize the
importance of having a convex strategy space, which is the key property
needed to guarantee the existence of Nash equilibria and the reason to
consider mixed strategies in the first place, when the action space is
finite.
\end{itemize}
\paragraph{Conclusion.} The paper is well-written and constitutes a clean and
short exposition of simple results.
\end{document}
|
