Reinforcement Learning Exercise 3.15

Exercise 3.15 In the gridworld example, rewards are positive for goals, negative for running into the edge of the world, and zero the rest of the time. Are the signs of these rewards important, or only the intervals between them? Prove, using (3.8), that adding a constant $c$ to all the rewards adds a constant, $v_c$, to the values of all states, and thus does not affect the relative values of any states under any policies. What is $v_c$ in terms of $c$ and $\gamma$?

First, for $v_{\pi }$, according to definition: $$\begin{array}{rl}{v}_{\pi }(s)& ={\mathbb{E}}_{\pi }(G_t|S_t=s)\\ & ={\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k\cdot R_{t+k+1}|S_t=s)\end{array}$$ Denote $\hat{R}=R+c$, for $\hat{R}$, there is: $$\begin{array}{rl}{\hat{v}}_{\pi }(s)& ={\mathbb{E}}_{\pi }({\hat{G}}_t|S_t=s)\\ & ={\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k\cdot {\hat{R}}_{t+k+1}|S_t=s)\\ & ={\mathbb{E}}_{\pi }[\sum _{k=0}^{\infty }{\gamma }^k\cdot (R_{t+k+1}+c)|S_t=s]\\ & ={\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k\cdot R_{t+k+1}|S_t=s)+{\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k\cdot c|S_t=s)\\ & ={\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k\cdot R_{t+k+1}|S_t=s)+\sum _{k=0}^{\infty }{\gamma }^k\cdot c\\ & =v_{\pi }(s)+\frac{c}{1-\gamma }\end{array}$$ $$\therefore v_c=\frac{c}{1-\gamma }$$