Dice dilemma: get paid the sum or go home

Question

Question:

Players $A$ and $B$ each play a die-rolling game. They have one $10$-sided fair die in front of them labelled $\displaystyle\{0, 1, 2, 3, 4, 5, 6, 7, 8, 9\}$.

Player $A$ goes first and starts rolling the die, and keeps track of their cumulative sum. They have the option to stop whenever they want to. If they choose to stop, they receive their current sum in dollars from a third party, and Player $B$ ends up with nothing, and the game ends. However, if Player A ever rolls a $0$ before stopping, their current sum is reset to $0$, and the die is passed onto Player $B$, and now they play with the same rules as above. If Player $B$ chooses to stop before they roll a $0$, they get paid their current sum, and likewise Player $A$ ends up with nothing; otherwise, the die is again passed back to Player $A$.

The problem is: Find the optimal stopping strategies for both players, in short, Nash Equilibrium(s), and how much they expect to make under such a play.

Assume that the rolling of the die is being done independently without showing it to the other player.

---

My Approach so far:

I have solved the version of this problem where the loser pays the winner their current cumulative sum, if they choose to stop. For that problem, it turns out that the optimal strategy is to keep rolling until the cumulative sum is at least $34$. The result was derived by writing out the DP equation. That stopping strategy is the same for both players A and B (since the situations are symmetric), and playing this, they expect to make around $\ 11.14478881$.

For our version, if we consider a one-player case where they win their sum if they stop before rolling a $0$ or go home bust, we see that the strategy to stop is "keep rolling until the cumulative sum is at least $45$."

For the two-player case, if Player $B$ always stops on their first roll, Player $A$ can have the same one-player version threshold (stop iff $\ge 45$). If Player $B$ increases their threshold, then Player $A$ might want to increase it further. I believe that we have to consider two separate thresholds for each player and then try to optimize them.

Answer 1

EDIT: Eric's comment is right. There was an error in the formula of the expected gain for the game, oversimplifying the solution. Here is a corrected version.

---

Assume each player fixes probabilities $r_k$ to roll gain the dice when their cumulated sum is $k>0$, inducing a probability $q$ of hitting zero. We show first that for a given $q$, the optimal expected value of the cumulated sum ($E$) is obtained by fixing a threshold $s$ such that $r_k=1$ for all $k<s$, and $r_k=0$ for all $k>s$. Then $E$ appears as a continuous, piecewise linear and concave function of $q$. Finally, the equilibrium conditions imply that $q_A=q_B$ and a short algorithm gives the unique numerical solution.

Range of $q$

If all probabilities $r_k$ are zero, there is only one roll and the expected gain is $\$4.5$ with $q=0.1$.

At the other end, if for all $k>0$ we have $r_k=1$, then the sequence of rolls ends a.s. with a zero, and $q=1$.

In any other case, the probability of hitting $0$ is strictly between $0.1$ and $1$.

Optimal strategy for a sequence of rolls with fixed $q$

Note $a_k$ the probability that the value $k\ge 1$ occurs in the sequence of cumulated sums. If $a_k=0$, then we can assume wlog that $r_k=0$. By convention, we fix $r_0=a_0=1$ and for all $k<0$, $a_k=0$. Then we get the following relation for all $k\ge 1$: $$a_k=\frac{1}{10}\sum_{j=k-9}^{k-1}a_jr_j$$

The expected gain of the sequence is $$E=\sum_{k=1}^{+\infty}k\,a_k(1-r_k)$$ and the probability of hitting $0$ is $$q=\frac{1}{10}\sum_{k=0}^{\infty}a_kr_k$$

We can show by induction on $k>j$ that all coefficients $a_k$ depend linearly on $r_j$, with nonnegative slope, so $q$ increases continuously wrt $r_j$.

If there exists $j<k$ such that $r_j<1$ with $a_j>0$ and $r_k>0$ then we can slightly increase $r_j$ and decrease $r_k$ so that $q$ is kept constant, until the former is $1$ or the latter is $0$. But that lowers the probability $a_jr_j$ to stop on $j$ and raise other probabilities to stop on higher values, so $E$ increases.

Therefore any optimal strategy on one sequence must be founded on a threshold $s$ as stated above, with a probability $r$ to roll again if you the cumulated sum attained exactly that value. And each such strategy is completely determined by $q$.

Conversely, any value $q\in(0.1,1)$ is attained by intermediate value theorem.

Optimal expected gain of a sequence as a function of $q$

For a fixed threshold $s$, the equations above reduce to $$E=sa_s(1-r)+\sum_{k=s+1}^{\infty}ka_k$$ $$q=\frac{\displaystyle \left(\sum_{k=0}^{s-1}a_k\right)+a_sr}{10}$$ but for all $k>s+9$ we have $a_k=0$. Then the derivatives are $$\frac{\partial E}{\partial r}=-sa_s+\sum_{k=s+1}^{s+9}k\frac{\partial a_k}{\partial r}$$ $$\frac{\partial q}{\partial r}=\frac{a_s}{10}$$ with for all $s+1\le k\le s+9$, $$\frac{\partial a_k}{\partial r}=\frac{a_s}{10}$$ so we get $$\frac{\partial E}{\partial q}=-10s+\sum_{k=s+1}^{s+9}k=45-s$$

Optimal choice of $q$ in the iterated game

Since the expected gain for A in the whole game is $\frac{E_A}{1-q_Aq_B}$, we fix $q_B$ and look for an optimal $q_A$. In a diagram with variables $q_A$ and $y$, the line $y=1-q_Aq_B$ crosses the horizontal axis in $q_A=\frac{1}{q_B}$, and the ratio $\frac{E_A}{1-q_Aq_B}$ is maximal for $q_A$ such that the tangent to the curve $y=E_A$ crosses the horizontal axis on the same point. By concavity of $E$ wrt $q$, there is only one optimal value of $q_A$ (or maybe an interval between two consecutive thresholds).

But we deduce that $q_A$ and $q_B$ satisfy and non-decreasing relation, and conversely with the same relation, so any Nash equilibrium must satisfy $q_A=q_B$.

Numerical application

We calculate for all $s<1000$ the values of $a_s,\dots a_{s+9}$, $q$ and $E$ to study the sign of $$\frac{\partial}{\partial q_A}\frac{E_A}{1-q_Aq_B}=\frac{(45-s)(1-q_Aq_B)+q_BE_A}{(1-q_Aq_B)^2}$$ in the special case $q_A=q_B$.

The numerator becomes negative when $s>86$, with $q\approx 0.85$ and $E\approx 14$ so the expected gain for A is around $\$49$, and for B it is $\$42$. To obtain a better precision, we should calculate the optimal $r$.

Answer 2

$ \def\ed{\stackrel{\text{def}}{=}} \def\op{\big(} \def\cp{\big)} $ Here's a proof that both players' choosing a stopping threshold of $\ 87\ $ any time they receive the dice is a Nash equilibrium, as conjectured by user3257842 in a comment. The equilibrium expected payoffs, rounded correctly to eight decimal places are $\ 48.57477898\ $ for player $\ A\ $ and $\ 41.04042506\ $ for player $\ B\ .$ The exact expected payoffs are actually rational numbers, but with numerators and denominators of $175$ decimal digits and $173$ decimal digits, respectively.

To prove the result, it's sufficient to show that if player $\ B\ $ always chooses a stopping threshold of $\ 87\ ,$ then player $\ A$'s best strategy is to do the same. It immediately follows from this that if player $\ A\ $ always chooses a stopping threshold of $\ 87\ $ then player $\ B$'s best strategy is also to do the same, because as soon as player $\ B\ $ receives the dice, he or she will be in exactly the same position as player $\ A\ $ was at the start of the game.

Let $\ \sigma_s\ $ be the probability either player will successfully reach a stopping threshold of $\ S\ $ in a single series of rolls given that his or her currently accumulated sum is $\ s\ .$ Then $\ \sigma_s\ $ satisfies the following recursion $$ \sigma_{s-1}=\frac{1}{10}\sum_{i=0}^8\sigma_{s+i} $$ and boundary condition $$ \sigma_s=1 $$ for $\ s\in\{S,S+1,\dots,S+8\,\}\ ,$ and $\ \sigma_0\ $ is the probability that the player will reach the threshold from the start of the series of rolls. If $$ C\ed\pmatrix{\frac{1}{10}&\frac{1}{10}&\dots&\frac{1}{10}&\frac{1}{10}\\ 1&0&\dots&0&0\\ 0&1&\dots&0&0\\ \vdots&&\ddots&\vdots&\vdots\\ 0&0&\dots&1&0} $$ is the companion matrix of the recursion, then $$ \pmatrix{\sigma_{s-1}\\\sigma_s\\\vdots\\\sigma_{s+7}}=C\pmatrix{\sigma_s\\\sigma_{s+1}\\\vdots\\\sigma_{s+8}}\ , $$ and $$ \sigma_0=\boldsymbol{\epsilon}_1^TC^S\mathbf{1}\ , $$ where $\ \boldsymbol{\epsilon}_1\ $ is the first column of the $\ 9\times9\ $ identity matrix and $\ \mathbf{1}\ $ the $\ 9\times1\ $ matrix all of whose entries are $1.$

Let \begin{align} \sigma_*&\ed\boldsymbol{\epsilon}_1^TC^{87}\mathbf{1}\\ &\approx0.1551083521 \end{align} be the probability of a player's reaching a stopping threshold of $87$ in a single series of rolls.

If player $\ B\ $ always chooses a stopping threshold of $\ 87\ ,$ then one can treat player $\ A$'s problem as effectively playing a single-player game which terminates with probability $\ \sigma_*\ $ whenever he or she rolls a zero, or restarts from scratch with probability $\ 1-\sigma_*\ $ and with player $\ A$'s cumulative sum being reset to zero. In this latter case, player $\ A\ $ will be in exactly the same position as at the start of the game, so the optimal stopping threshold $\ S_a\ $ and optimal expected value $\ v_a\ ,$ say, will always be the same whenever that position is reached. Let $\ \alpha_s\ $ be player $\ A$'s expected payoff given that he or she has currently accumulated a sum of $\ s\ $ in a single series of rolls in this single player game. Then $\ \alpha_s\ $ satisfies the following recursion: $$ \alpha_{s-1}=\frac{(1-\sigma_*)v_a+\sum_\limits{i=0}^8\alpha_{s+i}}{10}\label{1}\tag{1} $$ and boundary conditions $$ \alpha_s=s $$ for $\ s\in\{S_a,S_a+1,\dots,S_a+8\,\}\ ,$ and $\ v_a=\alpha_0\ .$ Subtracting $\ (1-\sigma_*)v_a\ $ from both sides of identity \eqref{1} gives $$ \alpha_{s-1}-(1-\sigma_*)v_a=\frac{\sum_\limits{i=0}^8\op\alpha_{s+i}-(1-\sigma_*)v_a\cp}{10}\ , $$ so $\ \alpha_{s-1}-(1-\sigma_*)v_a\ $ satisfies the same recursion as $\ \sigma_s\ ,$ but with different boundary conditions. Therefore \begin{align} v_a&=\alpha_0\\ &=\boldsymbol{\epsilon}_1^TC^{S_a}\op\op S_a-(1-\sigma_*)v_a\cp\mathbf{1}+\boldsymbol{k}\cp+(1-\sigma_*)v_a\\ &=\op S_a-(1-\sigma_*)v_a\cp\boldsymbol{\epsilon}_1^TC^{S_a}\mathbf{1}+\boldsymbol{\epsilon}_1^TC^{S_a}\boldsymbol{k}+(1-\sigma_*)v_a\ \end{align} where $$ \boldsymbol{k}\ed\pmatrix{0\\1\\\vdots\\8}\ . $$ Solving this linear equation for $\ v_a\ $ gives $$ v_a=\frac{S_a\boldsymbol{\epsilon}_1^TC^{S_a}\mathbf{1}+\boldsymbol{\epsilon}_1^TC^{S_a}\boldsymbol{k}}{\sigma_*+(1-\sigma_*)\boldsymbol{\epsilon}_1^TC^{S_a}\mathbf{1}}\ .\tag{2}\label{2} $$ The following Magma script, which can be run by copying and pasting it into the online Magma calculator and clicking the Submit button, evaluates the expression on the right side of the above equation in exact rational arithmetic for every value of $\ S_a\ $ from $\ S_a=1\ $ to $\ S_a=150\ ,$ and verifies that its maximum value for $\ S_a\ $ in that range occurs at $\ S_a=87\ ,$ and, rounded to eight decimal places, is $\ 48.57477898\ .$

C:=Matrix(Rationals(),9,9, 
[1/10,1/10,1/10,1/10,1/10,1/10,1/10,1/10,1/10,         
 1,0,0,0,0,0,0,0,0,
 0,1,0,0,0,0,0,0,0,
 0,0,1,0,0,0,0,0,0,
 0,0,0,1,0,0,0,0,0,
 0,0,0,0,1,0,0,0,0,
 0,0,0,0,0,1,0,0,0,
 0,0,0,0,0,0,1,0,0,
 0,0,0,0,0,0,0,1,0]);
ones:=Matrix(Rationals(),9,1, [1,1,1,1,1,1,1,1,1]);
Sb:=87;
sigvec:=C^Sb*ones;
sigma:=sigvec[1,1];
maxva:=0/1;
for S in [1..150] do
  d:=Matrix(Rationals(),9,1,[S,S+1,S+2,S+3,S+4,S+5,S+6,S+7,S+8]);
  e:=C^S*d;
  pim:=C^S*ones;
  va:=e[1,1]/(sigma+(1-sigma)*pim[1,1]);
  if va gt maxva then
     maxva:=va;
     argmaxva:=S;
   end if;
 end for;
 print(argmaxva);
 print(RealField(10)!maxva);
 print(RealField(10)!((1-sigma)*maxva));
 print(RealField(10)!sigma);

This doesn't quite prove the result, because there remains a possibility that a value of $\ S_a\ $ greater than $\ 150\ $ might result in a larger value for the expression on the right side of equation \eqref{2}. This can be ruled out, however, by expressing $\ v_a\ $ in terms of the eigenvalues and eigenvectors of $\ C\ $ and appealing to the dominance of its Perron-Frobenius eigenvalue for sufficiently large values of $\ S_a\ .$ The matrix $\ C\ $ is a primitive matrix with non-negative entries, one real (Perron-Frobenius) eigenvalue $\ \lambda_1\approx$$\,0.9794303016 \ ,$ and four distinct complex conjugate pairs $\ \lambda_2,$$\,\lambda_3=\overline{\lambda_2},$$\,\lambda_4,$$\,\lambda_5=\overline{\lambda_4},$$\,\lambda_6,$$\,\lambda_7=\overline{\lambda_6},$$\,\lambda_8\ $ and $\ \lambda_9=\overline{\lambda_8}\ $ of eigenvalues, all of whose moduli do not exceed $\ 0.781\ .$ The matrix $\ C^S\ $ can be expanded as \begin{align} C^S&=\sum_{i=1}^9\lambda_i^S\boldsymbol{r}_i\boldsymbol{\ell}_i^T\\ &=\lambda_1^S\left(\boldsymbol{r}_1\boldsymbol{\ell}_1^T+\sum_{i=2}^9\left(\frac{\lambda_i}{\lambda_ 1}\right)^S\boldsymbol{r}_i\boldsymbol{\ell}_i^T \right)\ , \end{align} where $\ \boldsymbol{r}_i\ $ and $\ \boldsymbol{\ell}_i^T\ $ are unit right and left eigenvectors, respectively, of $\ C\ $ corresponding to the eigenvalue $\ \lambda_i\ ,$ with $\ \boldsymbol{r}_1\ge\mathbf{0}\ $ and $\ \boldsymbol{\ell}_1\ge\mathbf{0}\ .$ For sufficiently large values of $\ S\ ,$ the sum $\ \sum_\limits{i=2}^9\left(\frac{\lambda_i}{\lambda_ 1}\right)^S\boldsymbol{r}_i\boldsymbol{\ell}_i^T \ $ becomes utterly negligible with respect to $\ \boldsymbol{r}_1\boldsymbol{\ell}_1^T\ ,^\color{red}{\dagger}\ $ so, to a very good approximation, $$ C^S\approx\lambda_1^S\boldsymbol{r}_1\boldsymbol{\ell}_1^T $$ and \begin{align} v_a&\approx \frac{\boldsymbol{\epsilon}_1^T\boldsymbol{r}_1\lambda_1^{S_a}\op\boldsymbol{\ell}_1^T\mathbf{1}S_a+\boldsymbol{\ell}_1^T\boldsymbol{k}\cp}{\sigma_*+(1-\sigma_*)\boldsymbol{\epsilon}_1^TC^{S_a}\mathbf{1}}\\ &<\sigma_*^{-1}\boldsymbol{\epsilon}_1^T\boldsymbol{r}_1\lambda_1^{S_a}\op\boldsymbol{\ell}_1^T\mathbf{1}S_a+\boldsymbol{\ell}_1^T\boldsymbol{k}\cp\\ \end{align} The derivative of the function $\ \lambda_1^{S_a}\op\boldsymbol{\ell}_1^T\mathbf{1}S_a+\boldsymbol{\ell}_1^T\boldsymbol{k}\cp\ $ is positive for $\ S_a<-\frac{1}{\ln\lambda_1}-\frac{ \boldsymbol{\ell}_1^T\boldsymbol{k}}{\boldsymbol{\ell}_1^T\mathbf{1}}\approx45.5\ ,$ zero at $\ S_a=-\frac{1}{\ln\lambda_1}-\frac{ \boldsymbol{\ell}_1^T\boldsymbol{k}}{\boldsymbol{\ell}_1^T\mathbf{1}}\ $ and negative for $\ S_a>-\frac{1}{\ln\lambda_1}-\frac{ \boldsymbol{\ell}_1^T\boldsymbol{k}}{\boldsymbol{\ell}_1^T\mathbf{1}}\ ,$ so the function itself is certainly decreasing for $\ S_a\ge150\ .$ For such vales of $\ S_a\ ,$ therefore, \begin{align} v_a&<\sigma_*^{-1}\boldsymbol{\epsilon}_1^T\boldsymbol{r}_1\lambda_1^{150}\op150\,\boldsymbol{\ell}_1^T\mathbf{1}+\boldsymbol{\ell}_1^T\boldsymbol{k}\cp\\ &\approx47.7\ . \end{align} Since this is less than the value of $\ v_a\ $ at $\ S_a=87\ ,$ it follows that the latter is the maximum over all positive integral values of $\ S_a\ .$

Player $\ A\ $ will succeed in reaching the stopping threshold of $\ 87\ $ with probability $\ \sigma_*\ ,$ and fail to do so with probability $\ 1-\sigma_*\ .$ In the later case, player $\ B\ $ will be in the same situation that player $\ A\ $ was at the beginning of the game, so his or her expected payoff, given that that situation has occurred, will be $\ v_a\ .$ His or her expected payoff, $\ v_b\ ,$ for the game itself is therefore given by \begin{align} v_b&=\sigma_*0+\op1-\sigma_*\cp v_a\\ &\approx41.04042506\ . \end{align}

For the sake of completeness, I have below included the Magma script I used to calculate the eigenvalues and eigenvectors of $\ C\ $ and the bound on $\ v_a\ $ for $\ S_a\ge150\ .$ Note that the Perron-Frobenius eigenvalue is the eighth in the list returned by Magma in this code, not the first.

$\color{red}{\dagger}$ For $\ S\ge100\ ,$ for instance, $\ \left|\frac{\lambda_i}{\lambda_ 1}\right|^S<1.5\times10^{-10}\ .$

CC:=Matrix(ComplexField(20),9,9, 
[1/10,1/10,1/10,1/10,1/10,1/10,1/10,1/10,1/10,
 1,0,0,0,0,0,0,0,0,
 0,1,0,0,0,0,0,0,0,
 0,0,1,0,0,0,0,0,0,
 0,0,0,1,0,0,0,0,0,
 0,0,0,0,1,0,0,0,0,
 0,0,0,0,0,1,0,0,0,
 0,0,0,0,0,0,1,0,0,
 0,0,0,0,0,0,0,1,0]);
e9:=Matrix(ComplexField(20),9,1,[0,0,0,0,0,0,0,0,1]);
ones:=Matrix(ComplexField(20),9,1,[1,1,1,1,1,1,1,1,1]);
k:=Matrix(ComplexField(20),9,1, [0,1,2,3,4,5,6,7,8]);
ident:=ScalarMatrix(ComplexField(20),9,1.0);
U:=ZeroMatrix(ComplexField(20),9,9);
evs:=Eigenvalues(CC);
for i in [1..9] do
  ev:=evs[i,1];
  m:=CC-ev*ident;
  for j in [1..9] do
    m[9,j]:=1.0;
  end for;
  evec:=m^(-1)*e9;
  norm:=0.0;
  for j in [1..9] do
    norm:=norm+Modulus(evec[j,1])^2;
  end for;
  norm:=Sqrt(norm);
  for j in [1..9] do
    U[j,i]:=(1/norm)*evec[j,1];
  end for;
end for;
r1:=ColumnSubmatrix(U,8,1);
ell1:=ColumnSubmatrix(Transpose(U^(-1)),8,1);
eps1r1:=Modulus(r1[8,1]);
ell1t1:=Transpose(ell1)*ones;
ell1t1:=Modulus(ell1t1[1,1]);
ell1tk:=Transpose(ell1)*k;
ell1tk:=Modulus(ell1tk[1,1]);
lam1:=Modulus(evs[8,1]);
sigma:=CC^87*ones;
sigma:=Modulus(sigma[1,1]);
print(-1/Log(lam1)-ell1tk/ell1t1);
print(sigma^(-1)*eps1r1*lam1^150*(ell1t1*150+ell1tk));

Answer 3

We are looking for a symmetric equilibrium parametrized by player's target cumulative sum $t$. For a given $t$, it is in principle possible to obtain (numerically via dynamic programming):

• the success probability $p_t$ of reaching the target without encountering a $0$.
• the expected final sum $v_t$ conditional on hitting target.

Assuming both players have the same target, the value for the player with the die $E_t$ satisfies: $$E_t = p_t v_t + (1-p_t)(1-p_t) E_t\implies E_t = \dfrac{p_tv_t}{1-q_t^2} = \dfrac{v_t}{2-p_t}$$ (either we succeed at reaching the target, gaining $v_t$, or we fail and the other player proceeds to fail, resetting the game).

• approximate win probability: $t/5$ expected moves to reach target, so $p_t \approx 0.9^{t/5}$
• approximate terminal value after meeting target $v_t \approx t+4$.

Now consider indifference to single-time deviation: At optimal $t$, whenever we have exactly $t$ on our hands, we must be indifferent between ending the game with $t$, and risking another roll: $$t = 0.9 (t+5) + 0.1 (1-p_t) \cdot \dfrac{v_t}{2-p_t}$$

• RHS is the extra roll EV: either we win the roll, gaining $\text{avg}(1,\dots,9)=5$. Or with probability 0.1 we lost the roll, and now need the other player to fail his round, in which case we by definition expect $E_t$.

plugging in our approximate $p_t$ and $v_t$ and solving produces $t=84$.