diff options
Diffstat (limited to 'aoas-2013-005.tex')
| -rw-r--r-- | aoas-2013-005.tex | 158 |
1 files changed, 158 insertions, 0 deletions
diff --git a/aoas-2013-005.tex b/aoas-2013-005.tex new file mode 100644 index 0000000..e9c3922 --- /dev/null +++ b/aoas-2013-005.tex @@ -0,0 +1,158 @@ +\documentclass[10pt]{article} +\usepackage[T1]{fontenc} +\usepackage[utf8]{inputenc} +\usepackage[hmargin=1in, vmargin=1in]{geometry} +\usepackage{amsmath,amsfonts} + + +\title{\vspace{-2em}\large Review: \emph{A Multi-Agent Reinforcement Learning Framework +for Treatment Effects Evaluation in Two-Sided Markets}} +\author{Submission 2013-005 to the \emph{Journal of Applied Statistics}} +\date{} + +\begin{document} + +\maketitle + +\paragraph{Summary.} +\looseness=-1 +This paper considers the problem of \emph{off-policy} learning in multi-agent +reinforcement learning. The model (Section 2) considers $N$ agents/units +evolving according to a Markov decision process: at each time step $t$, each +agent is assigned a treatment/action in $\{0,1\}$, resulting in a vector of +rewards $R_t\in\mathbb{R}^N$ (one for each agent). An underlying state in state +space $\mathbb{S}$ evolves according to a Markov transition kernel +$\mathcal{P}: \mathbb{S}\times\{0,1\}^N \to \Delta(\mathbb{S})$: given the +state $s_t$ at time step $t$ and vector of treatments $a_t\in\{0,1\}^N$, +$\mathcal{P}(s_t, a_t)$ specifies the probability distribution of the state at +time step $t+1$. Finally, a stationary policy $\pi:\mathbb{S}\to\{0,1\}^N$ +chooses a vector of treatments given an observed state. + +The goal of this paper is to design estimators for the expected average reward +when choosing treatments according to a given policy $\pi$ over $T$ time steps; +the difficulty being that in the observed data, the treatments might differ +from the ones that would have been chosen under the policy $\pi$. The authors +start from a simple importance-sampling based estimator which suffers from +prohibitively large variance due to the exponential size of the action space +$\{0,1\}^N$. To address this problem, they introduce a mean-field approximation +in which the dependency of the reward $r_i$ of agent $i$ on the treatments and +states of the other agents is reduced to a scalar summary, thus significantly +reducing the dimensionality of the problem and resulting in their first +estimator $\hat{V}^{\rm IS}$ (Section 3.1 and 3.3). This estimator is then +combined with a standard $Q$-learning based estimator in a manner known as +\emph{doubly robust} estimation, resulting in their final estimator +$\hat{V}^{\rm DR}$. This way of combining estimators in the context of RL is +sometimes known as double reinforcement learning. The estimator is stated to be +consistent and approximately normal in the appendix, with proofs supplied in +the supplementary material. Finally the estimator is evaluated experimentally +in the context of ride-sharing platforms, first on synthetic data in Section +4 and then on real data in Section 5. + +\paragraph{Scope and contributions.} A major concern I have regarding this paper +is with the way it is currently framed, making it particularly difficult to +appreciate its contributions. Specifically: +\begin{enumerate} + \item The title and abstract mention \emph{two-sided markets}, but nothing + in the formulation is specific to two-sided markets, since the problem + is modeled at the level of a spatial unit (a geographical region in the + example of ride-sharing platforms) in which a single state variable + abstracts away all the details of both sides of the market. When + I first read the paper, I was initially expecting to see both sides of + the market—consumers and service providers—being modeled separately as + two coupled Markov decision processes. Instead, this paper deals with + a generic multi-agent reinforcement learning problem and ride-sharing + platforms only appear in the evaluation (4 out of 28 pages in total). + \item The title and language in sections 1 and 2 use terms from the causal + inference literature, such as treatment effect and potential outcomes. + But once the ATE is defined as the difference of the value of two + policies, it becomes clear that the problem is exactly the one of + \emph{off-policy evaluation} in reinforcement learning. Hence, the + paper has little to do with causal inference and is firmly anchored in + the reinforcement learning literature by building from recent results + in this area. + \item The model is described as a multi-agent one, but it could be + equivalently described with a single agent whose action space is + $\{0,1\}^N$ and reward is the sum of the rewards of the agents. Hence + the problem is not as much about multi-agent as about dealing with an + exponentially large action space: \emph{this should be mentioned + prominently}. It is however true that the main assumption driving all + the results and methods, the \emph{mean-field approximation}, is more + naturally stated using the perspective of multiple agents whose rewards + only depend on a scalar summary of the actions and states of their + neighbors. +\end{enumerate} + +Following the above observations, I believe a much more accurate title for the +paper would be: \emph{Off-policy valuation estimation for multi-agent +reinforcement learning in the mean-field regime}. It also becomes easier to +appreciate the main contribution of this paper: the introduction of +a mean-field approximation to circumvent the high-dimensionality of the action +space. + +\paragraph{Major comments.} The understanding of the paper's scope coming from +the previous paragraph raises the following concerns: +\begin{itemize} + \looseness=-1 + \item given the importance of the mean-field approximation in this paper, + it is surprising that it is not discussed more. Is it possible to test + from data the extent to which it holds? If so, how? Can experimental + evidence for its validity be provided in the data used in the + evaluation sections (4 and 5)? + \item related to the previous point: I didn't find any discussion of how to + choose the mean-field functions $m_i$ in practice. The evaluation + sections do not seem to mention how these functions where chosen. + \item once the mean-field approximation is introduced, the problem is + effectively reduced to a low-dimensional reinforcement learning problem + and the methodological contribution (and theoretical analysis) seem to + follow from an almost routine adaptation of previous papers. If it is + not the case, the paper should do a better job at describing what is + novel in the adaption of these previous methods. + \item the evaluation section mentions that a comparison is made with the + DR-NM method, but it does not appear anywhere on the plots reporting + the MSE (only the DR-NS method appears). + \item given that the $\hat V^{\rm DR}$ estimator crucially uses the + regularized policy iteration estimator from Farahmand et al. (2016) and + Lioa et al. (2020) (by combining it with the $\hat{V}^{\rm IS}$ + estimator), I believe this estimator \emph{by itself} should also be + used as a baseline in the evaluation. + \item the code and synthetic data used for the evaluation should be + provided. +\end{itemize} + +\paragraph{Other comments.}\begin{itemize} + \item in the proof of Theorem 3 in the supplementary material, last line of + page 5, the union bound guarantees that the last inequality holds with + probability at least $1-O(N^{-1} T^{-2})$ and not $1-O(N^{-1}T^{-1})$, + if I am not mistaken. This does not change the conclusion of the + theorem. + \item can the (CMIA) assumption on 6 be thought of as a kind of Markovian + assumption? If I am not mistaken it is weaker than saying that + $R_{i,t}$ is independent of $(A_j, R_j, S_j)_{0\leq j<t}$ conditioned + on $A_t, S_t$. Maybe this should be stated to provide more intuition. + \item notations $r_i^*$ and $Q_i^*$ in the mean-field approximation are + somewhat confusing since they conflict with the “starred” notation for + potential outcomes. +\end{itemize} + +\paragraph{Typos.} +\begin{itemize} + \item page 7, third paragraph: \emph{important ratio} $\to$ + \emph{importance ratio}. + \item page 10, Algorithm 1: \emph{Initial} $\to$ \emph{Initialize}. + \item the references need to be carefully checked for formatting. In + particular, Markov and Fisher are consistently spelled with + a lowercase initial letter. +\end{itemize} + +\paragraph{Recommendation.} +I believe that the problems about the framing of the paper described in +\emph{Scope and contributions} warrant at least a major revision. Furthermore, +the concerns raised in \emph{Major comments} suggest that both the +methodological and experimental contributions are somewhat limited and might +justify a rejection. It could however be that the whole is more than the sum of +its parts, given the relevance and timeliness of the application. + +%\bibliographystyle{plain} +%\bibliography{main} + +\end{document} |