13

In the paper On the Convergence of Stochastic Iterative Dynamic Programming Algorithms (Jaakkola et al. 1994), the authors claim that a statement is "easy to show". I am starting to suspect that it isn't easy and they might have just wanted to avoid having to show it.

Lemma 3 A stochastic process $X_{n+1}(x)=(1-\alpha(x))X_n(x)+\gamma\beta_n(x)\vert\vert X_n\vert\vert$ converges to zero w.p.l provided

  1. $x\in S$, where $S$ is a finite set.

  2. $\sum_n\alpha_n(x)=\infty, \sum_n\alpha_n^2(x)<\infty, \sum_n\beta_n(x)=\infty, \sum_n\beta_n^2(x)<\infty$ and $E\{\beta_n(x)\vert P_n\},$ $E\{\alpha_n(x)\vert P_n\}$ uniformly over $x$ w.p.l.

where $$P_n=\{X_n, X_{n-1},\cdots,\alpha_{n-1},\alpha_{n-1},\cdots,\beta_{n-1},\beta_{n-2},\cdots\}$$ and $\alpha_n(x),\beta_n(x)$ are assumed to be non-negative and mutually independent given $P_n$.

Proof Essentially, the proof is an application of Lemma $2$. To this end, assume that we keep $\vert\vert X_n\vert\vert\leq C_1$ by scaling, which allows the iterative process to be bounded by $$\vert X_{n+1}(x)\vert\leq (1-\alpha_n(x))\vert X_n(x)\vert+\gamma\beta_n(x)C_1.$$ This is linear in $\vert x_n(x)\vert $ and can be easily shown to converge w.p.l to some $X^*(x)$, where $\vert\vert X^*\vert\vert \leq \gamma C_1$. Hence, for small enough $\epsilon$, there exists $M_1(\epsilon)$ such that $\vert\vert X_n\vert\vert\leq \cdots$

The question is:

Is the statement that is described as "easily shown" in the proof of Lemma 3 of Jaakkola's paper actually provable? I.e. do we have $|X_n(x)| \to X^*(x)$ with probability one?

What I have so far:

  1. Since $X_n$ is bounded in a compact interval, it certainly has convergent subsequences
  2. $|X_{n+1}-X_n|\le (\alpha_n+\gamma \beta_n)C_1$, which implies that it converges to zero, but that isn't enough for it to be a Cauchy sequence even with the first statement
  3. $\lim\inf X_n\ge 0$ $$ \begin{align} X_{n+1}(x)&=(1-\alpha_n(x))X_n(x) + \gamma\beta_n(x)\|X_n\| \\ &\ge (1-\alpha_n(x))X_n(x)\\ &=\prod^n_{k=0}(1-\alpha_k(x))X_0 \to 0 \end{align}$$ since $\sum \alpha_n=\infty$ (c.f. Infinite Product).
  4. Because of $\alpha_n(x) \to 0$ we know that $(1-\alpha_n(x))\ge 0$ for almost all $n$, and if $X_n(x)\ge 0$ then $$ X_{n+1}(x) = \underbrace{(1-\alpha_n(x))}_{\ge 0} \underbrace{X_n(x)}_{\ge 0} +\underbrace{\gamma\beta_n(x)\|X_n\|}_{\ge 0} $$ Thus, if one element of the sequence is positive, all following elements will be positive too. The sequences that stay negative converge to zero ($\lim\inf X_n\ge 0$). The other sequences will be positive for almost all n.
  5. For $\|X_n\|$ not to converge $\|X_n\|=\max_x X_n(x)$ for an infinite amount of n. If it were equal to the maximum of the negative sequences for almost all $n$ it would converge. $$\|X_n\|=\max_x -X_n(x) \le \max_x - \prod_{k=0}^n (1-\alpha_k) X_0 \to 0 $$
  6. If we set $\beta_n=0$ we have $$X_m=\prod_{k=n}^{m-1} (1-\alpha_k)X_n \to 0$$ So my intuition is: since $\beta_n$ is smaller than $\alpha_n$ (on average) replacing $\beta_n$ with $\alpha_n$ should probably be fine, since you introduce a larger difference to zero. So I think going in the direction $$X_{n+1}\sim (1-\alpha_n)X_n +\gamma \alpha_n X_n = (1-(1-\gamma)\alpha_n)X_n$$ Which is fine since $\sum(1-\gamma)\alpha_n =\infty$ for $\gamma\in(0,1)$

But I still need to formalize replacing $\beta_n$ with $\alpha_n$, which only works if I take the expected value. And I don't know if the expected value leaves the infinite sums intact. I also have to justify replacing the norm with just one element. I think I can assume that the norm is the max norm without disrupting later proofs. And since $\lim\inf X_n\ge 0$, $|X_n|$ is basically equal to $X_n$.

I am also a bit confused since the approach I am currently following would show that it converges to 0 instantly, while the proof wants me to show that it converges to some $X^*$ and then continues with arguments on how to show that it converges to $0$ from there. Which makes me think that I am not on the "intended proof path". So maybe I am missing something obvious that could save me a lot of trouble. Especially since they claim it should be easy.

EDIT (2025): This question arose during the creation of my Bachelor's thesis. The fact that this question garnered 13 upvotes without any suggestion for a solution makes me highly suspect this is a circular argument, but I have moved on from this topic since then.

Bowei Tang
  • 5,158
Felix Benning
  • 3,187
  • You say you haven't used $\sum_n \alpha_n^2 < \infty$. How did you deduce $\alpha_n \to 0$? Also, should the $\alpha$ in the first equation be an $\alpha_n$? Also, why can you scale the $X_n$'s to be uniformly bounded by some $C$? How do you know they can't blow up? – mathworker21 Mar 15 '19 at 17:05
  • @mathworker21 1. Good point (forgot) 2. Yes (I assume - wouldn't make sense otherwise) 3. Yeah, that is noted weirdly since they write $X_{n+1}=\dots$ when it should be $X_{n+1}= \dots $ if it is then smaller than $C_1$ otherwise scale it by multiplying with some $\lambda^k$ (where $\lambda<1$). They use that in a different lemma to bootstrap convergence: Since for $X_{n+1}=G(X_n)$ you can push the scaling through G. So scaling at some later point is equivalent to scaling the starting point. And they argue if it converges then you only scale for a finite number of times -> it converges anyway – Felix Benning Mar 15 '19 at 18:30
  • I'm still confused about 3. You don't know it converges and thus you don't know that scaling finitely many times is equivalent to scaling an arbitrary number of times. Consider the example $x_{n+1} = 2x_n$. Scaling by $1/3$ (say) an infinite number of times makes all the $x_n$'s equal to $0$, which makes $(x_n)_n$ converge. However, if $x_n = 2^n$, then clearly it does not converge. – mathworker21 Mar 15 '19 at 18:34
  • @mathworker21 Sorry for being unclear: so first you assume that you just scale as much as neccessary to keep it within the bound. Then you show it converges to 0 if you do that. Okay but if it converges to zero, then there exists some N after which you do not need to scale anymore as it stays smaller than C at that point. (Since you only ever scale as much as neccessary, the scaling would not be enough to keep it in an epsilon environment around 0). This means you only scale a finite amount. – Felix Benning Mar 15 '19 at 18:43
  • ... So in your case you are scaling too much if you start with $x_1=1$ you would scale with $1/3$ only every other time when the series passes a threshold let's say 3. And only as much as needed. So 1->2->4/3->8/3->16/9 – Felix Benning Mar 15 '19 at 18:46
  • 2
    What is the meaning of $|.|$? – user159517 Mar 15 '19 at 18:47
  • @FelixB. I think our disagreement is stemming from the definition of scaling. I thought you had to scale each $X_n$ so that $X_{n+1} = (1-\alpha_n)X_n+\gamma \beta_n ||X_n||$ still holds for each $n$. In my $x_{n+1} = 2x_n$ example, you scale each $x_n$ by $1/3$ every time you scale, so all the $x_n$'s would end up at 0 – mathworker21 Mar 15 '19 at 18:48
  • @user159517 the paper he linked to defined $||X_n|| = \max_x |X_n(x)|$. – mathworker21 Mar 15 '19 at 18:49
  • @user159517 it is a norm - while they haven't specified which one I think it is safe to assume it is the maximum norm – Felix Benning Mar 15 '19 at 18:49
  • @FelixB. If you scale your way (i.e. only a specific $X_n$), then you can't use the equation $X_{n+1} = (1-\alpha_n)X_n+\gamma \beta_n ||X_n||$ anymore – mathworker21 Mar 15 '19 at 18:50
  • @mathworker21 As I said they were a bit sloppy, but you can still use it as an inequality, and if you stay with a certain $\omega$ in the 1-set for which the $\alpha_n, \beta_n$ have their specified properties, then you can assume that you only ever scale $X_0$ and that the equation holds. – Felix Benning Mar 15 '19 at 18:52
  • How is $\gamma$ defined? Why do you think that $\beta_n$ is smaller than $\alpha_n$ on average? – user159517 Mar 15 '19 at 19:03
  • @user159517 It is not defined in the theorem by itself but it is a discount factor so assuming $\gamma\in(0,1)$ is appropriate. And regarding the second question: because the second assumption in the lemma states that the expected value is smaller. – Felix Benning Mar 15 '19 at 19:07
  • What is the set $S$? In particular, is $S$ (1) the domain of $X_n$, (2) a set which the theorem guarantees to exists, or (3) any arbitrary finite set? –  Mar 15 '19 at 19:16
  • @Strants it is the domain of $X_n$ – Felix Benning Mar 15 '19 at 19:17
  • What does "$E[\beta_n|P_n]$ uniformly" mean? – Michael Mar 21 '19 at 10:51
  • The sums coverge uniformly for all x, although reading dvoretzky's proof it probably needs to be uniformly over every possible history – Felix Benning Mar 21 '19 at 17:57
  • The role of $x$ is unclear to me. It looks like an equivalent problem removes that variable, ie, assume S had only one element. The assumption on conditional expectation in bullet 2 also looks unclear, possibly having some typo. – Michael Jul 28 '25 at 17:11

0 Answers0