Reinforcement Learning exercise 3.13

Exercise 3.13 Give an equation for $q_{\pi }$ in terms of $v_{\pi }$ and the four-argument $p$.

First, we need to derive a formula from multiplication formula of probability theory: $$\begin{array}{rl}p(x|y)& =\frac{p(x,y)}{p(y)}\\ & =\frac{{\sum }_{z}p(x,y,z)}{p(y)}\\ & =\frac{{\sum }_z[p(x|y,z)\cdot p(z|y)\cdot p(y)]}{p(y)}\\ & =\sum _z[p(x|y,z)\cdot p(z|y)](1)\end{array}$$ With formula (1), we can calculate $q_{\pi }(s,a)$ as below: $$\begin{array}{rl}{q}_{\pi }(s,a)& ={\mathbb{E}}_{\pi }(G_t|S_t=s,A_t=a)\\ & ={\mathbb{E}}_{\pi }(R_{t+1}+\gamma G_{t+1}|S_t=s,A_t=a)\\ & ={\mathbb{E}}_{\pi }(R_{t+1}|S_t=s,A_t=a)+\gamma {\mathbb{E}}_{\pi }(G_{t+1}|S_t=s,A_t=a)\\ & =\sum _{r}r\cdot Pr(R_{t+1}=r|S_t=s,A_t=a)\\ & +\gamma \sum _{g_{t+1}}{g}_{t+1}\cdot Pr(G_{t+1}=g_{t+1}|S_t=s,A_t=a)\end{array}$$ Here, according to definition, $g_{t+1}={\sum }_{k=0}^{\infty }{\gamma }^k\cdot r_{t+2+k}$. And use formula (1), we can derive: $$\begin{array}{rl}{q}_{\pi }(s,a)& =\sum _{r}r\cdot Pr(R_{t+1}=r|S_t=s,A_t=a)\\ & +\gamma \sum _{g_{t+1}}{g}_{t+1}\cdot Pr(G_{t+1}=\sum _{k=0}^{\infty }{\gamma }^k\cdot r_{t+2+k}|S_t=s,A_t=a)\\ & =\sum _{r}r\cdot \sum _{s^{\prime }}Pr(R_{t+1}=r|S_t=s,A_t=a,S_{t+1}=s^{\prime })\cdot Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a)\\ & +\gamma \sum _{g_{t+1}}{g}_{t+1}\cdot \sum _{s^{\prime }}Pr(G_{t+1}=\sum _{k=0}^{\infty }{\gamma }^k\cdot r_{t+2+k}|S_t=s,A_t=a,S_{t+1}=s^{\prime })\cdot Pr(S_{t+1}=s^{\prime }|S_t=s,a_t=a)\end{array}$$ Because in Markov Process, $G_{t+1}$ is the reward of status $S_{t+1}=s^{\prime }$, the information of $S_t=s$ and $A_t=a$ are no effect on $G_{t+1}$. So: $$\begin{array}{rl}{q}_{\pi }(s,a)& =\sum _{r}r\cdot \sum _{s^{\prime }}Pr(R_{t+1}=r|S_t=s,A_t=a,S_{t+1}=s^{\prime })\cdot Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a)\\ & +\gamma \sum _{g_{t+1}}{g}_{t+1}\cdot \sum _{s^{\prime }}Pr(G_{t+1}=\sum _{k=0}^{\infty }{\gamma }^k\cdot r_{t+2+k}|S_{t+1}=s^{\prime })\cdot Pr(S_{t+1}=s^{\prime }|S_t=s,a_t=a)\\ & =\sum _{s^{\prime }}\{Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a)\cdot [\sum _{r}r\cdot Pr(R_{t+1}=r|S_t=s,A_t=a,S_{t+1}=s^{\prime })\\ & +\gamma \sum _{g_{t+1}}{g}_{t+1}\cdot Pr(G_{t+1}=\sum _{k=0}^{\infty }{\gamma }^k\cdot r_{t+2+k}|S_{t+1}=s^{\prime })]\}\\ & =\sum _{s^{\prime }}\{p(s^{\prime }|s,a)\cdot [{\mathbb{E}}_{\pi }(r|s,a,s^{\prime })+\gamma \cdot {\mathbb{E}}_{\pi }(G_{t+1}|S_{t+1}=s^{\prime })]\}\\ & =\sum _{s^{\prime }}\{p(s^{\prime }|s,a)\cdot [{\mathbb{E}}_{\pi }(r|s,a,s^{\prime })+\gamma \cdot v_{\pi }(s^{\prime })\left]\right\}\end{array}$$ Denote $p(s^{\prime }|a,s)=P_{s,s^{\prime }}^a$ and ${\mathbb{E}}_{\pi }(r|s,a,s^{\prime })=R_{s,s^{\prime }}^a$, then $$q_{\pi }(s,a)=\sum _{s^{\prime }}\{P_{s,s^{\prime }}^a\cdot [R_{s,s^{\prime }}^a+\gamma \cdot v_{\pi }(s^{\prime })\left]\right\}(2)$$ Here, the equation (2) is the result.