1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
|
\documentclass[10pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[hmargin=1in, vmargin=1in]{geometry}
\usepackage{amsmath,amsfonts}
\title{\vspace{-2em}\large Review: \emph{A Multi-Agent Reinforcement Learning Framework
for Treatment Effects Evaluation in Two-Sided Markets}}
\author{Submission 2013-005 to the \emph{Journal of Applied Statistics}}
\date{}
\begin{document}
\maketitle
\paragraph{Summary.}
\looseness=-1
This paper considers the problem of \emph{off-policy} learning in multi-agent
reinforcement learning. The model (Section 2) considers $N$ agents/units
evolving according to a Markov decision process: at each time step $t$, each
agent is assigned a treatment/action in $\{0,1\}$, resulting in a vector of
rewards $R_t\in\mathbb{R}^N$ (one for each agent). An underlying state in state
space $\mathbb{S}$ evolves according to a Markov transition kernel
$\mathcal{P}: \mathbb{S}\times\{0,1\}^N \to \Delta(\mathbb{S})$: given the
state $s_t$ at time step $t$ and vector of treatments $a_t\in\{0,1\}^N$,
$\mathcal{P}(s_t, a_t)$ specifies the probability distribution of the state at
time step $t+1$. Finally, a stationary policy $\pi:\mathbb{S}\to\{0,1\}^N$
chooses a vector of treatments given an observed state.
The goal of this paper is to design estimators for the expected average reward
when choosing treatments according to a given policy $\pi$ over $T$ time steps;
the difficulty being that in the observed data, the treatments might differ
from the ones that would have been chosen under the policy $\pi$. The authors
start from a simple importance-sampling based estimator which suffers from
prohibitively large variance due to the exponential size of the action space
$\{0,1\}^N$. To address this problem, they introduce a mean-field approximation
in which the dependency of the reward $r_i$ of agent $i$ on the treatments and
states of the other agents is reduced to a scalar summary, thus significantly
reducing the dimensionality of the problem and resulting in their first
estimator $\hat{V}^{\rm IS}$ (Section 3.1 and 3.3). This estimator is then
combined with a standard $Q$-learning based estimator in a manner known as
\emph{doubly robust} estimation, resulting in their final estimator
$\hat{V}^{\rm DR}$. This way of combining estimators in the context of RL is
sometimes known as double reinforcement learning. The estimator is stated to be
consistent and approximately normal in the appendix, with proofs supplied in
the supplementary material. Finally the estimator is evaluated experimentally
in the context of ride-sharing platforms, first on synthetic data in Section
4 and then on real data in Section 5.
\paragraph{Scope and contributions.} A major concern I have regarding this paper
is with the way it is currently framed, making it particularly difficult to
appreciate its contributions. Specifically:
\begin{enumerate}
\item The title and abstract mention \emph{two-sided markets}, but nothing
in the formulation is specific to two-sided markets, since the problem
is modeled at the level of a spatial unit (a geographical region in the
example of ride-sharing platforms) in which a single state variable
abstracts away all the details of both sides of the market. When
I first read the paper, I was initially expecting to see both sides of
the market—consumers and service providers—being modeled separately as
two coupled Markov decision processes. Instead, this paper deals with
a generic multi-agent reinforcement learning problem and ride-sharing
platforms only appear in the evaluation (4 out of 28 pages in total).
\item The title and language in sections 1 and 2 use terms from the causal
inference literature, such as treatment effect and potential outcomes.
But once the ATE is defined as the difference of the value of two
policies, it becomes clear that the problem is exactly the one of
\emph{off-policy evaluation} in reinforcement learning. Hence, the
paper has little to do with causal inference and is firmly anchored in
the reinforcement learning literature by building from recent results
in this area.
\item The model is described as a multi-agent one, but it could be
equivalently described with a single agent whose action space is
$\{0,1\}^N$ and reward is the sum of the rewards of the agents. Hence
the problem is not as much about multi-agent as about dealing with an
exponentially large action space: \emph{this should be mentioned
prominently}. It is however true that the main assumption driving all
the results and methods, the \emph{mean-field approximation}, is more
naturally stated using the perspective of multiple agents whose rewards
only depend on a scalar summary of the actions and states of their
neighbors.
\end{enumerate}
Following the above observations, I believe a much more accurate title for the
paper would be: \emph{Off-policy valuation estimation for multi-agent
reinforcement learning in the mean-field regime}. It also becomes easier to
appreciate the main contribution of this paper: the introduction of
a mean-field approximation to circumvent the high-dimensionality of the action
space.
\paragraph{Major comments.} The understanding of the paper's scope coming from
the previous paragraph raises the following concerns:
\begin{itemize}
\looseness=-1
\item given the importance of the mean-field approximation in this paper,
it is surprising that it is not discussed more. Is it possible to test
from data the extent to which it holds? If so, how? Can experimental
evidence for its validity be provided in the data used in the
evaluation sections (4 and 5)?
\item related to the previous point: I didn't find any discussion of how to
choose the mean-field functions $m_i$ in practice. The evaluation
sections do not seem to mention how these functions where chosen.
\item once the mean-field approximation is introduced, the problem is
effectively reduced to a low-dimensional reinforcement learning problem
and the methodological contribution (and theoretical analysis) seem to
follow from an almost routine adaptation of previous papers. If it is
not the case, the paper should do a better job at describing what is
novel in the adaption of these previous methods.
\item the evaluation section mentions that a comparison is made with the
DR-NM method, but it does not appear anywhere on the plots reporting
the MSE (only the DR-NS method appears).
\item given that the $\hat V^{\rm DR}$ estimator crucially uses the
regularized policy iteration estimator from Farahmand et al. (2016) and
Lioa et al. (2020) (by combining it with the $\hat{V}^{\rm IS}$
estimator), I believe this estimator \emph{by itself} should also be
used as a baseline in the evaluation.
\item the code and synthetic data used for the evaluation should be
provided.
\end{itemize}
\paragraph{Other comments.}\begin{itemize}
\item in the proof of Theorem 3 in the supplementary material, last line of
page 5, the union bound guarantees that the last inequality holds with
probability at least $1-O(N^{-1} T^{-2})$ and not $1-O(N^{-1}T^{-1})$,
if I am not mistaken. This does not change the conclusion of the
theorem.
\item can the (CMIA) assumption on 6 be thought of as a kind of Markovian
assumption? If I am not mistaken it is weaker than saying that
$R_{i,t}$ is independent of $(A_j, R_j, S_j)_{0\leq j<t}$ conditioned
on $A_t, S_t$. Maybe this should be stated to provide more intuition.
\item notations $r_i^*$ and $Q_i^*$ in the mean-field approximation are
somewhat confusing since they conflict with the “starred” notation for
potential outcomes.
\end{itemize}
\paragraph{Typos.}
\begin{itemize}
\item page 7, third paragraph: \emph{important ratio} $\to$
\emph{importance ratio}.
\item page 10, Algorithm 1: \emph{Initial} $\to$ \emph{Initialize}.
\item the references need to be carefully checked for formatting. In
particular, Markov and Fisher are consistently spelled with
a lowercase initial letter.
\end{itemize}
\paragraph{Recommendation.}
I believe that the problems about the framing of the paper described in
\emph{Scope and contributions} warrant at least a major revision. Furthermore,
the concerns raised in \emph{Major comments} suggest that both the
methodological and experimental contributions are somewhat limited and might
justify a rejection. It could however be that the whole is more than the sum of
its parts, given the relevance and timeliness of the application.
%\bibliographystyle{plain}
%\bibliography{main}
\end{document}
|
