diff options
| -rw-r--r-- | ejs-1910-038.tex | 95 |
1 files changed, 95 insertions, 0 deletions
diff --git a/ejs-1910-038.tex b/ejs-1910-038.tex new file mode 100644 index 0000000..d904944 --- /dev/null +++ b/ejs-1910-038.tex @@ -0,0 +1,95 @@ +\documentclass[10pt]{article} +\usepackage[T1]{fontenc} +\usepackage[utf8]{inputenc} +\usepackage[hmargin=1.2in, vmargin=1.2in]{geometry} +\usepackage{amsmath,amsfonts} + +\title{\large Review of \emph{On Bayes and Nash experiment design for hypothesis testing problems}} +\date{} + +\begin{document} + +\maketitle + +\paragraph{Summary.} This paper studies a standard ANOVA hypothesis testing +problem: given $K$ treatments, determine whether they all have the same average +effect, or whether there exists a pair of treatments whose average effects are +different. The authors consider the likelihood ratio test and the focus is on +finding and characterizing the optimal design. The contributions are as +follows: +\begin{itemize} + \item first, optimality is formulated as a maxi-min problem: find the most + powerful design for the worst possible value of the unknown treatment + effects in the alternative hypothesis. A previous paper of the authors + had established that the balanced design is optimal for this criterion. + \item next, a ``pseudo-Bayes'' approach is adopted: there is a prior on the + average treatment effects, and the goal is to find the design which is + most powerful on average over this prior. It is found that the balanced + design is optimal in that sense whenever the prior is + exchangeable. + \item next, the authors consider a game theoretic approach where the + optimal design problem is formulated as a two-player zero-sum game + between the ``max'' player attempting to find the optimal design and + the ``min'' player attempting to find the worst possible configuration + of average effects. The unique Nash equilibrium of the game is computed + for two different choices of strategy space for the max player. Again, + the balanced design is found to be the optimal strategy (assuming it is + contained in the strategy space of the max player). + \item finally, the authors consider a ``tree order'' setting in which the null + hypothesis is that all treatments have the same effects, and the + alternative is that all treatments $2\leq i\leq K$ have an effect + greater than treatment $1$. The Nash equilibrium of the two-player game + is computed. +\end{itemize} + +\paragraph{Overall impression.} The problem considered in this paper is natural +and interesting but I found the results to be underwhelming. Specifically: +\begin{itemize} + \item the fact that the balanced design is optimal was already known for + some criteria (e.g. A-optimality and, by a previous of the authors, the + maxi-min criterion). As revealed by the proofs, all of these results + (including the new criteria considered in this work) express in + slightly different ways that the underlying ANOVA setting is symmetric. + \item I do not completely buy the game theoretic formulation. There is + nothing inherently strategic in the setting considered. The fact that + any maxi-min or mini-max problem can be \emph{interpreted} as + a two-player game is always true and it does not seem that the present + paper is saying more than that. + \item I am not convinced by the value of arbitrarily restricting the action + space of the max-player (the experiment designer) at the beginning of + Section 4 and of the resulting Theorem 4.1. Would it make more sense to + directly consider the general case where the max-player can choose + \emph{any} design, and in particular, where the balanced design is part + of the action space. In other words, the story leading to Theorem 4.3 + seems unnecessary to me and I think this theorem could have been stated + first. +\end{itemize} + +\paragraph{Minor comments.} +\begin{itemize} + \item In the definition of $\mathcal{M}_\delta$ (equation (7)), it would + be good to have a remark about how $\delta$ is chosen. For examples, + what are implications of choosing one value vs.\ another value, how to + choose it in practice, etc. + \item In section 4, I think it would be better to define the value of game + outside of Theorem 4.2. Defining it formally, explain that it is only + defined when $\min\max\dots = \max\min\dots$ and that it requires the + action space to be convex. + \item related to the previous point, the use of the word \emph{value} in + the proofs is inconsistent. It seems to be sometimes used to refer the + value of an arbitrary pair of strategies, whereas in game theory, it + usually only refers to the $\min\max$ value. + \item paragraph above Theorem 4.3: the phrasing ``Since any $\xi$ can be + written as mixture of other elements\dots'' is a bit vague. It would be + better to simply say: ``Since the action space $\Xi$ is convex''. + Relatedly, when introducing mixed strategies, I would emphasize the + importance of having a convex strategy space, which is the key property + needed to guarantee the existence of Nash equilibria and the reason to + consider mixed strategies in the first place, when the action space is + finite. +\end{itemize} + +\paragraph{Conclusion.} The paper is well-written and constitutes a clean and +short exposition of simple results. + +\end{document} |