From f3234dc87d46644234b6114b93fd1e92eddf289f Mon Sep 17 00:00:00 2001
From: Thibaut Horel <thibaut.horel@gmail.com>
Date: Wed, 15 Jul 2020 00:03:51 -0400
Subject: Add EJS review

---
 ejs-1910-038.tex | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
 create mode 100644 ejs-1910-038.tex

(limited to 'ejs-1910-038.tex')

diff --git a/ejs-1910-038.tex b/ejs-1910-038.tex
new file mode 100644
index 0000000..d904944
--- /dev/null
+++ b/ejs-1910-038.tex
@@ -0,0 +1,95 @@
+\documentclass[10pt]{article}
+\usepackage[T1]{fontenc}
+\usepackage[utf8]{inputenc}
+\usepackage[hmargin=1.2in, vmargin=1.2in]{geometry}
+\usepackage{amsmath,amsfonts}
+
+\title{\large Review of \emph{On Bayes and Nash experiment design for hypothesis testing problems}}
+\date{}
+
+\begin{document}
+
+\maketitle
+
+\paragraph{Summary.} This paper studies a standard ANOVA hypothesis testing
+problem: given $K$ treatments, determine whether they all have the same average
+effect, or whether there exists a pair of treatments whose average effects are
+different. The authors consider the likelihood ratio test and the focus is on
+finding and characterizing the optimal design.  The contributions are as
+follows:
+\begin{itemize}
+	\item first, optimality is formulated as a maxi-min problem: find the most
+		powerful design for the worst possible value of the unknown treatment
+		effects in the alternative hypothesis.  A previous paper of the authors
+		had established that the balanced design is optimal for this criterion.
+	\item next, a ``pseudo-Bayes'' approach is adopted: there is a prior on the
+		average treatment effects, and the goal is to find the design which is
+		most powerful on average over this prior. It is found that the balanced
+		design is optimal in that sense whenever the prior is
+		exchangeable.
+	\item next, the authors consider a game theoretic approach where the
+		optimal design problem is formulated as a two-player zero-sum game
+		between the ``max'' player attempting to find the optimal design and
+		the ``min'' player attempting to find the worst possible configuration
+		of average effects. The unique Nash equilibrium of the game is computed
+		for two different choices of strategy space for the max player. Again,
+		the balanced design is found to be the optimal strategy (assuming it is
+		contained in the strategy space of the max player).
+	\item finally, the authors consider a ``tree order'' setting in which the null
+		hypothesis is that all treatments have the same effects, and the
+		alternative is that all treatments $2\leq i\leq K$ have an effect
+		greater than treatment $1$. The Nash equilibrium of the two-player game
+		is computed.
+\end{itemize}
+
+\paragraph{Overall impression.} The problem considered in this paper is natural
+and interesting but I found the results to be underwhelming. Specifically:
+\begin{itemize}
+	\item the fact that the balanced design is optimal was already known for
+		some criteria (e.g. A-optimality and, by a previous of the authors, the
+		maxi-min criterion). As revealed by the proofs, all of these results
+		(including the new criteria considered in this work) express in
+		slightly different ways that the underlying ANOVA setting is symmetric.
+	\item I do not completely buy the game theoretic formulation. There is
+		nothing inherently strategic in the setting considered.  The fact that
+		any maxi-min or mini-max problem can be \emph{interpreted} as
+		a two-player game is always true and it does not seem that the present
+		paper is saying more than that.
+	\item I am not convinced by the value of arbitrarily restricting the action
+		space of the max-player (the experiment designer) at the beginning of
+		Section 4 and of the resulting Theorem 4.1. Would it make more sense to
+		directly consider the general case where the max-player can choose
+		\emph{any} design, and in particular, where the balanced design is part
+		of the action space.  In other words, the story leading to Theorem 4.3
+		seems unnecessary to me and I think this theorem could have been stated
+		first.
+\end{itemize}
+
+\paragraph{Minor comments.}
+\begin{itemize}
+	\item  In the definition of $\mathcal{M}_\delta$ (equation (7)), it would
+		be good to have a remark about how $\delta$ is chosen. For examples,
+		what are implications of choosing one value vs.\ another value, how to
+		choose it in practice, etc.
+	\item In section 4, I think it would be better to define the value of game
+		outside of Theorem 4.2. Defining it formally, explain that it is only
+		defined when $\min\max\dots = \max\min\dots$  and that it requires the
+		action space to be convex.
+	\item related to the previous point, the use of the word \emph{value} in
+		the proofs is inconsistent. It seems to be sometimes used to refer the
+		value of an arbitrary pair of strategies, whereas in game theory, it
+		usually only refers to the $\min\max$ value.
+	\item paragraph above Theorem 4.3: the phrasing ``Since any $\xi$ can be
+		written as mixture of other elements\dots'' is a bit vague. It would be
+		better to simply say: ``Since the action space $\Xi$ is convex''.
+		Relatedly, when introducing mixed strategies, I would emphasize the
+		importance of having a convex strategy space, which is the key property
+		needed to guarantee the existence of Nash equilibria and the reason to
+		consider mixed strategies in the first place, when the action space is
+		finite.
+\end{itemize}
+
+\paragraph{Conclusion.} The paper is well-written and constitutes a clean and
+short exposition of simple results. 
+
+\end{document}
-- 
cgit v1.2.3-70-g09d2

