1. Introduction

1.1. Overview

The aim of this lecture note is to give a very brief introduction to the theory of large deviations. In probability theory, a recurrent theme is the convergence of random variables, and the law of large numbers (LLN) is among the fundamental results concerning their “typical” behavior. But beyond this scope, mathematicians yearn for a more refined picture of the “atypical configurations” or “rare events” that deviate from the expected outcome either by a small or a large amount. In this pursuit, the former is addressed in the central limit theorem (CLT) while the latter gives rise to the large deviation principle (LDP). To demonstrate these, we would like to begin with the following example.

Example 1.1 (Bernoulli trial). A Bernoulli trial is a series of identical and independent experiments with exactly two possible outcomes, say $0$ and $1$ , which may be modeled as a sequence of i.i.d. random variables $(𝑋_{𝑖})_{𝑖 = 1}^{\infty}$ with the Bernoulli distribution:

ℙ (𝑋_{𝑖} = 1) = 𝑝 and ℙ (𝑋_{𝑖} = 0) = 1 - 𝑝 (𝑝 \in (0, 1)) .

The core questions regarding these trials revolve around the number of 1′s in an $𝑛$ -trial, which is mathematically phrased as a random variable $𝑆_{𝑛} = \sum_{𝑖 = 1}^{𝑛} 𝑋_{𝑖}$ . It is not hard to determine the distribution of $𝑆_{𝑛}$ :

ℙ (𝑆_{𝑛} = ⌊ 𝑛 𝑥 ⌋) = (\binom{𝑛}{⌊ 𝑛 𝑥 ⌋}) 𝑝^{⌊ 𝑛 𝑥 ⌋} {(1 - 𝑝)}^{𝑛 - ⌊ 𝑛 𝑥 ⌋} if 𝑥 \in [0, 1],

for which, here and throughout our study, we intend to examine the logarithm of the probability of the associated events. Specifically, by Stirling’s approximation ( $log 𝑛! = 𝑛 log 𝑛 - 𝑛 + 𝑂 (log 𝑛)$ ),

\begin{aligned} 𝑛^{- 1} log ℙ (𝑆_{𝑛} = ⌊ 𝑛 𝑥 ⌋) \\ = & \frac{⌊ 𝑛 𝑥 ⌋}{𝑛} log (\frac{𝑝}{\frac{⌊ 𝑛 𝑥 ⌋}{𝑛}}) + (1 - \frac{⌊ 𝑛 𝑥 ⌋}{𝑛}) log (\frac{1 - 𝑝}{1 - \frac{⌊ 𝑛 𝑥 ⌋}{𝑛}}) + 𝑂 (\frac{log 𝑛}{𝑛}) . \end{aligned}

Asymptotically, given that $𝑥 \mapsto - 𝑥 log 𝑥$ is uniformly continuous on $[0, 1]$ ,

lim_{𝑛 \to \infty} 𝑛^{- 1} log ℙ (𝑆_{𝑛} = ⌊ 𝑛 𝑥 ⌋) = - 𝐼 (𝑥),

where

𝐼 (𝑥) = 𝑥 log (\frac{𝑥}{𝑝}) + (1 - 𝑥) log (\frac{1 - 𝑥}{1 - 𝑝})

is plotted in Figure 1. In the language of large deviation theory, the sequence $𝑛^{- 1} 𝑆_{𝑛}$ is said to satisfy the large deviation principle (which will be defined rigorously later) with rate function $𝐼$ .

Some comments are in order.

$𝐼 (𝑥)$ is a convex function with
$𝐼' (𝑥) = log (\frac{𝑥}{1 - 𝑥}) - log (\frac{𝑝}{1 - 𝑝}) and 𝐼 ″ (𝑥) = \frac{1}{𝑥} + \frac{1}{1 - 𝑥} .$ In particular, its minimum is attained by a unique point $𝑥 = 𝑝 = 𝔼 (𝑋_{1})$ . This implies the law of large numbers by the Borel-Cantelli lemma.
Observe that the second-order Taylor expansion
$𝐼 (𝑥) = \frac{{(𝑥 - 𝑝)}^{2}}{2 𝑝 (1 - 𝑝)} + 𝑂 (| 𝑥 - 𝑝 |^{2}) .$
When considering a small deviation $𝑥 = 𝑝 + 𝑦 / \begin{matrix} \end{matrix} \sqrt{𝑛}$ , this is “roughly” consistent with the central limit theorem:
$ℙ (\frac{\sqrt{𝑛} (𝑛^{- 1} 𝑆_{𝑛} - 𝑝)}{\sqrt{𝑝 (1 - 𝑝)}} = \frac{𝑦}{\sqrt{𝑝 (1 - 𝑝)}}) \approx \frac{1}{\sqrt{2 𝜋 𝑝 (1 - 𝑝)}} 𝑒^{\frac{𝑦^{2}}{2 𝑝 (1 - 𝑝)}} .$

rate function 𝐼(𝑥) — **Figure 1**: rate function $𝐼 (𝑥)$

1.2. General settings

Throughout the note, we let ${𝜇_{𝜀}}_{𝜀}$ be a collection of probability measures on a common measurable space $(𝒳︀, ℬ︀)$ . The family is indexed by a set of non-negative real numbers $𝜀 > 0$ with an accumulation point at $0$ , as we are primarily interested in the limiting behavior of these measures as $𝜀 \to 0$ . Within this framework, we assume that $𝒳︀$ is a Hausdorff topological space and that $ℬ︀$ contains the Borel $𝜎$ -algebra ${ℬ︀}_{𝒳︀}$ .

Definition 1.2 (rate function). Let $𝐼 : 𝒳︀ \to [0, \infty]$ be a function with sublevel sets $Ψ_{𝐼} (𝛼) ≔ {𝑥 : 𝐼 (𝑥) \leq 𝛼}$ and effective domain ${𝒟︀}_{𝐼} ≔ {𝑥 : 𝐼 (𝑥) < \infty}$ .

$𝐼$ is called a rate function if it is lower semicontinuous.
A rate function $𝐼$ is said to be good if $Ψ_{𝐼} (𝛼)$ is compact for all $𝛼 < \infty$ .

Definition 1.3 (large deviation principle). The family ${𝜇_{𝜀}}_{𝜀}$ is said to satisfy the large deviation principle with rate $𝐼 : 𝒳︀ \to [0, \infty]$ if for every $Γ \in ℬ︀$ ,

- inf_{𝑥 \in \overset{˚}{Γ}} 𝐼 (𝑥) \leq \underset{𝜀 \to 0}{lim inf} 𝜀 log 𝜇_{𝜀} (Γ) \leq \underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (Γ) \leq - inf_{𝑥 \in \overset{─}{Γ}} 𝐼 (𝑥) .

This definition of large deviation principle provides the most general framework for our discussion. In particular, one can further adapt the definition for any sequence of random variables as follows: The sequence $(𝑋_{𝑛})_{𝑛 \in ℕ}$ is said to satisfy the large deviation principle if the associated family of probability laws ${𝜇_{𝑛^{- 1}}}_{𝑛 \in ℕ}$ satisfies the large deviation principle. This convention is applied to all statements concerning random variables.

The rationale behind our specific settings is tied directly to the following characterizations of the large deviation principle. By assuming ${ℬ︀}_{𝒳︀} \subseteq ℬ︀$ , we immediately have the following equivalent statement.

Remark 1.4. Suppose ${ℬ︀}_{𝒳︀} \subseteq ℬ︀$ . Then, ${𝜇_{𝜀}}_{𝜀}$ satisfies the large deviation principle with rate $𝐼$ if and only if the following hold.

(upper bound) For any closed set $𝐹 \subset 𝒳︀$ ,
$\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐹) \leq - inf_{𝑥 \in 𝐹} 𝐼 (𝑥) .$
(lower bound) For any open set $𝐺 \subset 𝒳︀$ ,
$\underset{𝜀 \to 0}{lim inf} 𝜀 log 𝜇_{𝜀} (𝐺) \geq - inf_{𝑥 \in 𝐺} 𝐼 (𝑥) .$

As for the Hausdorff assumption, it ensures that compact sets are well-behaved, particularly concerning the notion of exponential tightness.

Definition 1.5. Suppose $ℬ︀$ contains all compact subsets of $𝒳︀$ . The family ${𝜇_{𝜀}}$ is exponentially tight if for every $𝛼 < \infty$ , there exists a compact set $𝐾$ such that

\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐾^{𝑐}) < - 𝛼 .

As every compact set is closed in a Hausdorff space, we can relax the LDP conditions under the assumption of exponential tightness as follows:

Proposition 1.6. Suppose ${ℬ︀}_{𝒳︀} \subset ℬ︀$ . If ${𝜇_{𝜀}}$ is exponentially tight, then the following hold.

(upper bound) If (1.2) holds for every compact set, so does it for every closed set.
(lower bound) If (1.3) holds for every open set, then $𝐼$ is good.

Notably, any ${𝜇_{𝜀}}$ admitting a rate function that satisfies the relaxed conditions is said to satisfy the weak LDP.

Proof. Suppose ${𝜇_{𝜀}}$ is exponentially tight. If $𝐹$ is a closed set, then for any closed set $𝐹$ ,

\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐹) \leq - inf_{𝑥 \in 𝐹} 𝐼 (𝑥)

. For all $\infty > 𝛼 > 0$ , there exists a compact set $𝐾_{𝛼}$ such that

\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐾_{𝛼}^{𝑐}) < - 𝛼 .

Hence,

\begin{aligned} \underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐹) & \leq max {\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐹 \cap 𝐾_{𝛼}), \underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐾_{𝛼}^{𝑐})} \\ = lim_{𝑎 \to \infty} \underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐹 \cap 𝐾_{𝛼}) \leq - inf_{𝑥 \in 𝐹 \cap 𝐾_{𝛼}} 𝐼 (𝑥), \end{aligned}

where the last inequality holds as $𝐹 \cap 𝐾_{𝛼}$ is compact due to the Hausdorff assumption.

With exponential tightness and the lower bound for open sets, we can find for every $\infty > 𝛼 > 0$ a compact set $𝐾_{𝛼}$ such that

- 𝛼 > \underset{𝜀 \to 0}{lim inf} 𝜀 log 𝜇_{𝜀} (𝐾_{𝛼}^{𝑐}) \geq - inf_{𝑥 \in 𝐾_{𝛼}^{𝑐}} 𝐼 (𝑥) .

This naturally implies

Ψ_{𝐼} (𝛼) \subset 𝐾_{𝛼},

which is a closed subset due to lower semicontinuity and therefore compact due to the Hausdorff assumption. $□$

2. Cramér’s theorem in $ℝ$

In this section, we establish the Large Deviation Principle for the empirical mean of i.i.d. random variables. We begin by introducing the primary tools of our analysis: the logarithmic moment generating function and its Fenchel-Legendre transform. Let $(𝑋_{𝑖})_{𝑖 = 1}^{\infty}$ be i.i.d. random variables with law $𝜇 \in 𝑀_{1} (ℝ)$ , and let $𝜇_{𝑛}$ be the law of the empirical mean ${\hat{𝑆}}_{𝑛} = \frac{1}{𝑛} \sum_{𝑖 = 1}^{𝑛} 𝑋_{𝑖}$ . We denote the expectation of $𝑋_{1}$ by $\overset{─}{𝑥} ≔ 𝔼 𝑋_{1}$ , whenever it is well-defined.

Definition 2.1. The logarithmic moment generating function (log MGF) associated with the law $𝜇$ is defined as

Λ (𝜆) ≔ log 𝔼 (𝑒^{⟨ 𝜆, 𝑋_{1} ⟩}) .

Definition 2.2. The Fenchel-Legendre transform of $Λ (𝜆)$ is

Λ^{*} (𝑥) ≔ sup_{𝜆 \in ℝ} [⟨ 𝜆, 𝑥 ⟩ - Λ (𝜆)] .

Example 2.3. There is a geometric interpretation of the Fenchel-Legendre transform of $Λ (𝜆)$ ; that is, $Λ^{*} (𝑥)$ is the supremum of the $𝑦$ -intercepts of all lines with slope $𝑥$ that lie below $Λ$ .

rate function 𝐼(𝑠) — **Figure 2**: rate function $𝐼 (𝑠)$

For the empirical means, the large deviation principle takes the following form:

Theorem 2.4 (Cramér). Let $𝑋_{𝑖}$ , $𝜇_{𝑛}$ , $Λ$ , and $Λ^{*}$ be as defined on $ℝ$ . Then, ${𝜇_{𝑛}}$ satisfies the LDP with the convex rate function $Λ^{*}$ , namely,

For any closed set $𝐹$ , ${lim sup}_{𝑛 \to \infty} \frac{1}{𝑛} log 𝜇_{𝑛} (𝐹) \leq - {inf}_{𝑥 \in 𝐹} Λ^{*} (𝑥)$ .
For any open set $𝐺$ , ${lim sup}_{𝑛 \to \infty} \frac{1}{𝑛} log 𝜇_{𝑛} (𝐺) \geq - {inf}_{𝑥 \in 𝐺} Λ^{*} (𝑥)$ .

Furthermore,

If $0 \in \overset{˚}{{𝒟︀}_{Λ}}$ , then $Λ^{*}$ is good.

To motivate the proof, we first examine the role of the log MGF. For real-valued random variables, Markov’s inequality provides the key estimate for the LDP upper bound ((1.2)). Specifically, for any $𝜆 \geq 0$ :

\begin{aligned} 𝜇_{𝑛} [𝑥, \infty) \leq 𝑒^{- 𝜆 𝑥} 𝔼 (𝑒^{𝜆 {\hat{𝑆}}_{𝑛}}) = 𝑒^{- 𝜆 𝑥} {(𝔼 (𝑒^{𝜆 𝑋_{1}}))}^{𝑛} \\ \Rightarrow & \underset{𝑛 \to \infty}{lim sup} \frac{1}{𝑛} log 𝜇_{𝑛} [𝑥, \infty) \leq inf_{𝜆 \geq 0} - 𝜆 𝑥 + Λ (𝜆) . \end{aligned}

Similarly,

\underset{𝑛 \to \infty}{lim sup} \frac{1}{𝑛} log 𝜇_{𝑛} (- \infty, 𝑥] \leq inf_{𝜆 \leq 0} - 𝜆 𝑥 + Λ (𝜆) .

These observations suggest that $Λ^{*}$ is the natural candidate for the rate function. The following lemma summarizes the key properties of $Λ$ and $Λ^{*}$ .

Lemma 2.5. Let $Λ$ and $Λ^{*}$ be as defined. Then,

$Λ$ is a convex function and $Λ^{*}$ is a convex rate function.
Either of following holds.
- If $Λ (𝜆) < \infty$ only when $𝜆 = 0$ , then $Λ^{*}$ is identically zero.
- If $Λ (𝜆) < \infty$ for some $𝜆 > 0$ (respectively, $𝜆 < 0$ ), then $\overset{─}{𝑥} < \infty$ (respectively, $\overset{─}{𝑥} > - \infty$ ) is well-defined. Under the circumstances,
  
  $Λ^{*} (𝑥) = {\begin{cases} {sup}_{𝜆 \geq 0} [𝜆 𝑥 - Λ (𝜆)] & if 𝑥 \geq \overset{─}{𝑥}, \\ {sup}_{𝜆 \leq 0} [𝜆 𝑥 - Λ (𝜆)] & if 𝑥 \leq \overset{─}{𝑥}, \end{cases}$
  
  (2.3)
  
  which satisfies
  - $Λ^{*}$ is decreasing on $(- \infty, \overset{─}{𝑥}]$ and increasing on $[\overset{─}{𝑥}, \infty)$ , and
  - ${inf}_{𝑥 \in ℝ} Λ^{*} (𝑥) = 0$ and if $\overset{─}{𝑥} \in ℝ$ , then $Λ^{*} (\overset{─}{𝑥}) = 0$ .
$Λ$ is differentiable in ${\overset{˚}{𝒟︀}}_{Λ}$ with $Λ' (𝜆) = {(𝔼 (𝑒^{𝜆 𝑋_{1}}))}^{- 1} 𝔼 (𝑋_{1} 𝑒^{𝜆 𝑋_{1}})$ and
$Λ' (𝜆) = 𝑥 \Rightarrow Λ^{*} (𝑥) = 𝜆 𝑥 - Λ (𝜆) .$
If $0 \in {\overset{˚}{𝒟︀}}_{Λ}$ , then $Λ^{*}$ is a good rate function. Moreover, if ${𝒟︀}_{Λ} = ℝ$ , then ${lim}_{| 𝑥 \mapsto \infty} \frac{Λ^{*} (𝑥)}{| 𝑥 |} = \infty$ .

Proof. (1) By Hölder’s inequality, given any $𝜆, 𝜆' \in ℝ$ and any $𝑡, 𝑡' \in [0, 1]$ satisfying $𝑡 + 𝑡' = 1$ ,

log 𝔼 (𝑒^{⟨ 𝑡 𝜆 + 𝑡' 𝜆', 𝑋_{1} ⟩}) \leq 𝑡 log 𝔼 (𝑒^{⟨ 𝜆, 𝑋_{1} ⟩}) + 𝑡' log 𝔼 (𝑒^{⟨ 𝜆', 𝑋_{1} ⟩}),

proving the convexity of $Λ$ . The convexity of $Λ^{*}$ follows from definition:

sup_{𝜆 \in ℝ} [⟨ 𝜆, 𝑡 𝑥 + 𝑡' 𝑥 ⟩ - Λ (𝜆)] \leq 𝑡 sup_{𝜆 \in ℝ} [⟨ 𝜆, 𝑥 ⟩ - Λ (𝜆)] + 𝑡' sup_{𝜆 \in ℝ} [⟨ 𝜆, 𝑥' ⟩ - Λ (𝜆)] .

To prove the lower semicontinuity, observe that for any $𝜆 \in ℝ$ and any $𝑥 \in ℝ$ ,

\underset{𝑦 \to 𝑥}{lim inf} ⟨ 𝜆, 𝑦 ⟩ - Λ (𝜆) = ⟨ 𝜆, 𝑥 ⟩ - Λ (𝜆),

implying ${lim inf}_{𝑦 \to 𝑥} Λ^{*} (𝑦) \geq Λ^{*} (𝑥)$ . The non-negativity of $Λ^{*}$ follows from the fact $Λ^{*} (𝑥) \geq ⟨ 0, 𝑥 ⟩ - Λ (0) = 0$ .

(2) The case ${𝒟︀}_{Λ} = {0}$ is automatic. If $Λ (𝜆) < \infty$ for some $𝜆 > 0$ , then, due to the fact that $𝑒^{𝑥} \geq 𝑥$ for all $𝑥 \geq 0$ ,

𝔼 [𝑋_{1} 𝟙_{{𝑋_{1} \geq 0}}] \leq 𝜆^{- 1} 𝔼 [𝑒^{⟨ 𝜆, 𝑋_{1} ⟩} 𝟙_{{𝑋_{1} \geq 0}}] \leq 𝜆^{- 1} exp (Λ (𝜆)) < \infty,

meaning both $𝔼 (⟨ 𝜆, 𝑋_{1} ⟩)$ and $Λ (𝜆)$ are well-defined. Hence, by Jensen’s inequality,

\overset{─}{𝑥} = 𝜆^{- 1} log \circ exp (𝔼 (⟨ 𝜆, 𝑋_{1} ⟩)) \leq 𝜆^{- 1} Λ (𝜆) < \infty .

The argument proceeds similarly for $Λ (𝜆) > - \infty$ whenever $𝜆 < 0$ .

We then proceed to prove (2.3) and its related properties. Since now $\overset{─}{𝑥}$ is well-defined, by Jensen’s inequality, for all $𝜂 < 0$ and $𝑥 \geq \overset{─}{𝑥}$ (similar for $𝜂 > 0$ and $𝑥 \leq \overset{─}{𝑥}$ ),

⟨ 𝜂, 𝑥 ⟩ - Λ (𝜂) = log 𝔼 (𝑒^{⟨ 𝜂, 𝑥 - 𝑋_{1} ⟩}) \leq ⟨ 𝜂, 𝑥 - \overset{─}{𝑥} ⟩ \leq 0 = ⟨ 0, 𝑥 ⟩ - Λ (0),

from which (2.3) follows. Moreover, (2.3) implies the monotonicity on $(\overset{─}{𝑥}, \infty)$ and $(\infty, \overset{─}{𝑥})$ . Finally, if $\overset{─}{𝑥} \in ℝ$ , then by (2.4), ${inf}_{𝑥 \in ℝ} Λ^{*} (𝑥) = Λ^{*} (\overset{─}{𝑥}) = 0$ . If $\overset{─}{𝑥} = - \infty$ (similar for $\overset{─}{𝑥} = - \infty$ ), we deduce from Markov’s inequality that for all $𝜆 \geq 0$

log 𝜇 [𝑥, \infty) \leq log 𝑒^{- ⟨ 𝜆, 𝑥 ⟩} 𝔼 (𝑒^{⟨ 𝜆, 𝑋_{1} ⟩}),

from which it follows that $0 \leq {inf}_{𝑥 \in ℝ} Λ^{*} (𝑥) \leq {lim}_{𝑥 \to \overset{─}{𝑥}} Λ^{*} (𝑥) \leq 0$ , as desired.

(3) The differentiability follows from the dominated convergence theorem. Let $𝑓_{𝜀} (𝑥) = \frac{𝑒^{(𝜆 + 𝜀) 𝑥} - 𝑒^{𝜆 𝑥}}{𝜀}$ so that $𝑓_{𝜀} (𝑥) \to 𝑥 𝑒^{𝜆 𝑥}$ as $𝜀 \to 0$ and $| 𝑓_{𝜀} (𝑥) | \leq \frac{𝑒^{𝜆 𝑥} (𝑒^{𝛿 | 𝑥 |} - 1)}{𝛿} ≕ ℎ_{𝛿} (𝑥)$ whenever $| 𝜀 | < 𝛿$ . Since $𝔼 (ℎ_{𝛿} (𝑋_{1})) < \infty$ for all sufficiently small $𝛿$ , we may apply the dominated convergence theorem to derive the derivative of $Λ$ . Finally, the function $𝑔 (𝜂) ≔ ⟨ 𝜂, 𝑥 ⟩ - Λ (𝜂)$ is concave and thus $𝑔' (𝜆) = 0$ implies $𝑔 (𝜆) = {sup}_{𝜂 \in ℝ} 𝑔 (𝜂)$ , proving the proposed property.

(4) Suppose $[𝜆_{-}, 𝜆_{+}] \subset {𝒟︀}_{Λ}$ is a non-degenerate interval containing $0$ . Then, for $𝜆 \in [𝜆_{-}, 𝜆_{+}]$

\underset{| 𝑥 \mapsto \infty}{lim inf} \frac{Λ^{*} (𝑥)}{| 𝑥 |} \geq \underset{| 𝑥 \mapsto \infty}{lim inf} [𝜆 sign (𝑥) - \frac{Λ (𝜆)}{| 𝑥 |}] \geq min {- 𝜆_{-}, 𝜆_{+}} > 0,

implying $Λ^{*} (𝑥) \to \infty$ as $| 𝑥 | \to \infty$ . Hence, the sublevel set $Ψ_{Λ^{*}} (𝛼)$ is closed and bounded for all $𝛼 < \infty$ , and $Λ^{*}$ is good. When ${𝒟︀}_{Λ} = ℝ$ , then the result follows by letting $𝜆_{+} = - 𝜆_{-} \to \infty$ . $□$

Proof of Theorem 2.4. (1) For brevity, let $𝐼_{𝐹} = {inf}_{𝑥 \in 𝐹} ⟨ 𝜆, 𝑥 ⟩ - Λ (𝜆)$ . If $𝐼_{𝐹} = 0$ , then the inequality is trivial. Hence, we assume $𝐼_{𝐹} > 0$ . Under the circumstances, the numbers $𝑥_{+} = inf [\overset{─}{𝑥}, \infty) \cap 𝐹$ and $𝑥_{-} = sup (- \infty, \overset{─}{𝑥}] \cap 𝐹$ are different from $\overset{─}{𝑥}$ . By Markov’s inequality, for all $𝜆 > 0$ and $𝜆' < 0$ ,

𝜇_{𝑛} (𝐹) \leq 𝜇_{𝑛} [𝑥_{+}, \infty) + 𝜇_{𝑛} (- \infty, 𝑥_{-}] \leq 𝑒^{- 𝑛 (⟨ 𝜆, 𝑥_{+} ⟩ - Λ (𝜆))} + 𝑒^{- 𝑛 (⟨ 𝜆', 𝑥_{-} ⟩ - Λ (𝜆'))} .

By Lemma 2.5,

𝜇_{𝑛} (𝐹) \leq 𝑒^{- 𝑛 Λ^{*} (𝑥_{+})} + 𝑒^{- 𝑛 Λ^{*} (𝑥_{-})} \leq 2 𝑒^{𝐼_{𝐹}} .

Taking the normalized logarithm and letting $𝑛 \to \infty$ yields the desired inequality.

(2) It suffices to show that for all measures $𝜇$ and all $𝛿 > 0$ ,

\underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log 𝜇_{𝑛} (- 𝛿, 𝛿) \geq inf_{𝜆 \in ℝ} Λ (𝜆) = - Λ^{*} (0),

for once this is proved, one may simply consider the translation $𝑌 = 𝑋 - 𝑥$ to deduce that the log MGF $Λ_{𝑌} (𝜆) = Λ (𝜆) - ⟨ 𝜆, 𝑥 ⟩$ and that $Λ_{𝑌}^{*} (𝑧) = Λ^{*} (𝑧 + 𝑥)$ , which in turn imply

\underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log 𝜇_{𝑛} (𝑥 - 𝛿, 𝑥 + 𝛿) \geq - Λ^{*} (𝑥) .

This proves the desired lower bound.

To prove (2.5), assume first that (a) $𝜇 (- \infty, 0) > 0$ , (b) $𝜇 (0, \infty) > 0$ , and (c) $𝜇$ is boundedly supported. Under the circumstances, (a) and (b) imply $Λ (𝜆) \to \infty$ as $| 𝜆 | \to \infty$ , and (c) implies $Λ$ is finite on $ℝ$ . Consequently, by Lemma 2.5, there exists $𝜂 \in ℝ$ such that

Λ' (𝜆) = 0 and Λ (𝜆) = inf_{𝜂 \in ℝ} Λ (𝜂) .

Define a probability measure $\tilde{𝜇}$ by

\frac{𝑑 \tilde{𝜇}}{𝑑 𝜇} (𝑥) = 𝑒^{⟨ 𝜆, 𝑥 ⟩ - Λ (𝜆)} .

Observe that

\begin{aligned} 𝜇_{𝑛} (- 𝜀, 𝜀) & = \int_{| \sum_{𝑖 = 1}^{𝑛} 𝑥_{𝑖} | < 𝑛 𝜀} 𝑑 𝜇 (𝑥_{1}) \dots 𝑑 𝜇 (𝑥_{𝑛}) \\ = \int_{| \sum_{𝑖 = 1}^{𝑛} 𝑥_{𝑖} | < 𝑛 𝜀} 𝑒^{\sum_{𝑖 = 1}^{𝑛} [- ⟨ 𝜆, 𝑥_{𝑖} ⟩ + Λ (𝜆)]} 𝑑 \tilde{𝜇} (𝑥_{1}) \dots 𝑑 \tilde{𝜇} (𝑥_{𝑛}) . \\ \geq 𝑒^{𝑛 Λ (𝜆) - 𝑛 𝜀 | 𝜆 |} {\tilde{𝜇}}_{𝑛} (- 𝜀, 𝜀) . \end{aligned}

By Lemma 2.5,

𝔼_{\tilde{𝑋} \sim \tilde{𝜇}} (\tilde{𝑋}) = \int 𝑥 𝑒^{⟨ 𝜆, 𝑥 ⟩ - Λ (𝜆)} 𝑑 𝜇 (𝑥) = {(𝔼 (𝑒^{⟨ 𝜆, 𝑥 ⟩}))}^{- 1} 𝔼 (𝑋_{1} 𝑒^{⟨ 𝜆, 𝑋_{1} ⟩}) = Λ' (𝜆) = 0 .

Hence, by the law of large numbers,

lim_{𝑛 \to \infty} {\tilde{𝜇}}_{𝑛} (- 𝜀, 𝜀) = 1 .

Hence, for all $0 < 𝜀 < 𝛿$ ,

\underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log 𝜇_{𝑛} (- 𝛿, 𝛿) \geq \underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log 𝜇_{𝑛} (- 𝜀, 𝜀) \geq Λ (𝜆) - 𝜀 | 𝜆 |,

proving (2.5) by letting $𝜀 \to 0$ .

Now, if the support of $𝜇$ is not bounded, fix $𝑀 > 0$ such that (a) $𝜇 [- 𝑀, 0) > 0$ , (b) $𝜇 (0, 𝑀] > 0$ . Hence, by letting $𝜈$ be the normalized law of $𝑋_{1}$ on ${| 𝑋_{1} | \leq 𝑀}$ , we have that

\begin{aligned} 𝜇_{𝑛} (- 𝛿, 𝛿) & \geq \int_{\frac{1}{𝑛} \sum_{𝑖 = 1}^{𝑛} 𝑥_{𝑖} \in (- 𝛿, 𝛿)} \prod_{𝑖 = 1}^{𝑛} 𝟙_{[- 𝑀, 𝑀]} (𝑥_{𝑖}) 𝑑 𝜇 (𝑥_{1}) \dots 𝑑 𝜇 (𝑥_{𝑛}) \\ = 𝜈_{𝑛} (- 𝛿, 𝛿) \cdot {𝜇 [- 𝑀, 𝑀]}^{𝑛} \end{aligned}

and that the log MGF associated with $𝜈$ is

log \int_{- 𝑀}^{𝑀} 𝑒^{⟨ 𝜆, 𝑥 ⟩} 𝑑 𝜇 (𝑥) - log 𝜇 [- 𝑀, 𝑀] ≕ Λ_{𝑀} (𝜆) - log 𝜇 [- 𝑀, 𝑀] .

Hence, by the case of bounded support,

\underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log 𝜇_{𝑛} (- 𝜀, 𝜀) \geq log 𝜇 [- 𝑀, 𝑀] + \underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log 𝜈_{𝑛} (- 𝜀, 𝜀) \geq inf_{𝜆 \in ℝ} Λ_{𝑀} (𝜆),

and therefore, by writing $𝐼_{𝑀} = - {inf}_{𝜆 \in ℝ} Λ_{𝑀} (𝜆)$ and $𝐼^{*} = {lim sup}_{𝑀 \to \infty} 𝐼_{𝑀}$ ,

\underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log 𝜇_{𝑛} (- 𝜀, 𝜀) \geq - 𝐼^{*} .

Since $Λ_{𝑀}$ is increasing in $𝑀$ , so is $- 𝐼_{𝑀}$ . Therefore, $𝐼^{*} > = - \infty$ and the level sets ${𝜆 : Λ_{𝑀} (𝜆) \leq - 𝐼^{*}}$ are decreasing compacted sets, admitting some $𝜆_{0}$ in their intersection. By monotone convergence theorem,

Λ (𝜆_{0}) = lim_{𝑀 \to \infty} Λ_{𝑀} (𝜆_{0}) \leq - 𝐼^{*} .

Combining (2.6) and (2.7) proves (2.5).

Finally, if $𝜇 (- \infty, 0) = 0$ or $𝜇 (0, \infty) = 0$ , then $Λ$ is monotone and ${inf}_{𝜆 \in ℝ} Λ (𝜆) = log 𝜇 ({0})$ . The bound $𝜇_{𝑛} (- 𝛿, 𝛿) \geq 𝜇_{𝑛 ({0})} = {𝜇 ({0})}^{𝑛}$ then yields (2.5).

(3) It follows from Lemma 2.5(4). $□$

3. Gärtner-Ellis theorem.

Let $Λ_{𝜀}$ be the log MGF associated with $𝑑$ -dimensional real random variables $𝑍_{𝜀}$ , which can be defined as

Λ_{𝜇_{𝜀}} (𝜆) ≔ log 𝔼 [𝑒^{⟨ 𝜆, 𝑍_{𝜀} ⟩}] .

The Gärtner-Ellis theorem states the following.

Definition 3.1. A convex function $Λ : ℝ^{𝑑} \to (- \infty, \infty]$ is essentially smooth if:

$\overset{˚}{{𝒟︀}_{Λ}}$ is nonempty.
$Λ$ is differentiable throughout $\overset{˚}{{𝒟︀}_{Λ}}$ .
$Λ$ is steep, namely, ${lim}_{𝑛 \to \infty} | \nabla Λ (𝜆_{𝑛}) | = \infty$ whenever $(𝜆_{𝑛})_{𝑛 \in ℕ}$ is a sequence in $\overset{˚}{{𝒟︀}_{Λ}}$ converging to a boundary point of $\overset{˚}{{𝒟︀}_{Λ}}$ .

Theorem 3.2 (Gärtner-Ellis). Suppose that $Λ (𝜆) ≔ {lim}_{𝜀 \to 0} 𝜀 Λ_{𝜀} (𝜀 𝜆)$ exists for all $𝜆 \in ℝ^{𝑑}$ as extended real numbers and $0 \in \overset{˚}{{𝒟︀}_{Λ}}$ .

For any closed set $𝐹$ ,
$\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (𝐹) \leq - inf_{𝑥 \in 𝐹} Λ^{*} (𝑥) .$
For any open set $𝐺$ ,
$\underset{𝜀 \to 0}{lim inf} 𝜀 log 𝜇_{𝜀} (𝐺) \geq - inf_{𝑥 \in 𝐺 \cap ℱ︀} Λ^{*} (𝑥),$ where $ℱ︀$ is the set of exposed points of $Λ^{*}$ admitting an exposing hyperplane belonging to $\overset{˚}{{𝒟︀}_{Λ}}$ .
If $Λ$ is an essentially smooth, lower semicontinuous function, then the LDP holds with the good rate function $Λ^{*}$ .

The proof of the third statement relies on several results from convex analysis; for clarity, we state these results here and defer their proofs.

Lemma 3.3. Under the same assumption as in Theorem 3.2, the following hold.

$Λ (𝜆)$ is a convex function, $Λ (𝜆) > - \infty$ everywhere.
$Λ^{*} (𝑥)$ is a good convex rate function.
Suppose that $𝑦 = \nabla Λ (𝜂)$ for some $𝜂 \in \overset{˚}{{𝒟︀}_{Λ}}$ . Then $Λ^{*} (𝑦) = ⟨ 𝜂, 𝑦 ⟩ - Λ (𝜂)$ . Moreover $𝑦 \in ℱ︀$ , with $𝜂$ being the exposing hyperplane for $𝑦$ .

Lemma 3.4 (Rockafellar). If

Λ : ℝ^{𝑑} \to (- \infty, \infty]

is an essentially smooth, lower semicontinuous, convex function, then

ri {𝒟︀}_{Λ^{*}} \subseteq ℱ︀

With these tools, we can now proceed with the proof of the Gärtner-Ellis theorem. For convenience, we first introduce the following auxiliary function:

Definition 3.5. The $𝛿$ -rate function associated with a rate function $𝐼$ is a function defined as

𝐼^{𝛿} (𝑥) = min {𝐼 (𝑥) - 𝛿, \frac{1}{𝛿}} .

Proof of Theorem 3.2. (1) It suffices to prove the inequality for all compact sets and that ${𝜇_{𝜀}}$ is exponentially tight.

The upper bound for compact sets follows essentially from Markov’s inequality. Choose for each $𝑥 \in Γ$ a $𝜆_{𝑥} \in ℝ^{𝑑}$ and an open neighborhood $𝐴_{𝑥}$ of $𝑥$ such that

\begin{matrix} ⟨ 𝜆_{𝑥}, 𝑥 ⟩ - Λ (𝑥) \geq 𝐼^{𝛿} (𝑥), \\ inf_{𝑦 \in 𝐴_{𝑥}} [⟨ 𝜆_{𝑥}, 𝑦 ⟩ - ⟨ 𝜆_{𝑥}, 𝑥 ⟩] \geq - 𝛿, \end{matrix}

where $𝐼^{𝛿}$ is the $𝛿$ -rate function associated with $Λ^{*}$ as defined in (3.1). Applying Markov’s inequality yields

𝜇_{𝜀} (𝐴_{𝑥}) \leq 𝔼 (𝑒^{⟨ \frac{𝜆_{𝑥}}{𝜀}, 𝑍_{𝜀} ⟩ - ⟨ \frac{𝜆_{𝑥}}{𝜀}, 𝑥 ⟩}) exp (- inf_{𝑦 \in 𝐴_{𝑥}} ⟨ \frac{𝜆_{𝑥}}{𝜀}, 𝑦 ⟩ - ⟨ \frac{𝜆_{𝑥}}{𝜀}, 𝑥 ⟩),

which in turn implies

𝜀 log 𝜇_{𝜀} (𝐴_{𝑥}) \leq 𝛿 - [⟨ 𝜆_{𝑥}, 𝑥 ⟩ - 𝜀 Λ_{𝜇_{𝜀}} (\frac{𝜆_{𝑥}}{𝜀})] .

Now that $Γ$ is a compact set, one can find a finite cover ${𝐴_{𝑥_{𝑖}}}_{𝑖 = 1}^{𝑁}$ with $𝑥_{𝑖} \in Γ$ such that

\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀} (Γ) \leq 𝛿 - min_{1 \leq 𝑖 \leq 𝑁} 𝐼^{𝛿} (𝑥_{𝑖}) \leq 𝛿 - inf_{𝑥 \in Γ} 𝐼^{𝛿} (𝑥),

proving the theorem by letting $𝛿 \to 0$ .

It is left to demonstrate the exponential tightness of $𝜇_{𝜀}$ , which is equivalent to that for all marginals $𝜇^{𝑗}$ of $𝜇$ on $𝑗$ -th coordinate and follows essentially from Lemma 2.5 (4). Let $𝛽 > 0$ be sufficiently small so that ${\overset{─}{𝐵}}_{𝜌} (0) \subset {\overset{˚}{𝒟︀}}_{Λ^{𝑗}}$ . By Markov’s inequality, we have that for all $𝜌 \in ℝ$ ,

\underset{𝜀 \to 0}{lim sup} 𝜀 log 𝜇_{𝜀}^{𝑗} [𝜌, \infty) \leq \underset{𝜀 \to 0}{lim sup} - 𝛽 𝜌 + 𝜀 Λ_{𝜇_{𝜀}} (𝜀^{- 1} 𝛽 𝐞_{𝑗}) \leq - 𝛽 𝜌 + Λ^{*} (𝛽 𝐞_{𝑗}),

where $𝐞_{𝑗}$ is the $𝑗$ -th vector in the standard basis and the right-hand side converges to $- \infty$ as $𝜌 \to \infty$ . The same arguments applies to ${lim sup}_{\infty} 𝜀 log 𝜇_{𝜀}^{𝑗} (- \infty, 𝜌]$ . The two estimates combined prove the exponential tightness.

(2) The case $Λ (𝜆) = - \infty$ for some $𝜆 \in {𝒳︀}^{*}$ is trivial, for $Λ^{*} (\cdot) = \infty$ everywhere. Assuming $Λ (𝜆) < \infty$ for all $𝜆 \in {𝒳︀}^{*}$ , one may adopt the change of measure argument as before in the following manner. Let $𝐺$ be any fixed open set, $𝑦 \in 𝐺 \cap ℱ︀$ , $𝛿 > 0$ , and $𝜂 \in {\overset{˚}{𝒟︀}}_{Λ}$ be an exposing hyperplane for $Λ^{*}$ . By continuity of $𝜂$ , we choose an open neighborhood $𝐵_{𝛿} \subset 𝐺$ of $𝑦$ such that

sup_{𝑧 \in 𝐵_{𝛿}} ⟨ 𝜂, 𝑧 - 𝑦 ⟩ < 𝛿 .

Now that $Λ_{𝜇_{𝜀}} (\frac{𝜂}{𝜀})$ is well-defined for every sufficiently small $𝜀 > 0$ , we define a probability measure ${\tilde{𝜇}}_{𝜀}$ equivalent to $𝜇_{𝜀}$ :

\frac{𝑑 {\tilde{𝜇}}_{𝜀}}{𝑑 𝜇_{𝜀}} (𝑧) = exp [⟨ \frac{𝜂}{𝜀}, 𝑧 ⟩ - Λ_{𝜇_{𝜀}} (\frac{𝜂}{𝜀})]

so that

\begin{aligned} 𝜀 log 𝜇_{𝜀} (𝐵_{𝛿}) & = 𝜀 log \int_{𝐵_{𝛿}} exp [- ⟨ \frac{𝜂}{𝜀}, 𝑧 ⟩ + Λ_{𝜇_{𝜀}} (\frac{𝜂}{𝜀})] 𝑑 {\tilde{𝜇}}_{𝜀} (𝑧) \\ = 𝜀 Λ_{𝜇_{𝜀}} (\frac{𝜂}{𝜀}) - ⟨ 𝜂, 𝑦 ⟩ + 𝜀 log \int_{𝐵_{𝛿}} exp [⟨ \frac{𝜂}{𝜀}, 𝑦 - 𝑧 ⟩] 𝑑 {\tilde{𝜇}}_{𝜀} (𝑧), \end{aligned}

yielding

\begin{aligned} \underset{𝜀 \to 0}{lim inf} 𝜀 log 𝜇_{𝜀} (𝐵_{𝛿}) & \geq Λ (𝜂) - ⟨ 𝜂, 𝑦 ⟩ + lim_{𝛿 \to 0} \underset{𝜀 \to 0}{lim inf} 𝜀 log {\tilde{𝜇}}_{𝜀} (𝐵_{𝛿}), \\ \geq - Λ^{*} (𝑦) + lim_{𝛿 \to 0} \underset{𝜀 \to 0}{lim inf} 𝜀 log {\tilde{𝜇}}_{𝜀} (𝐵_{𝛿}) . \end{aligned}

It remains to show that ${lim}_{𝛿 \to 0} {lim inf}_{𝜀 \to 0} 𝜀 log {\tilde{𝜇}}_{𝜀} (𝐵_{𝛿}) = 0$ . To this end, we claim that

\underset{𝜀 \to 0}{lim sup} 𝜀 log {\tilde{𝜇}}_{𝜀} (𝐵_{𝛿}^{𝑐} \cap {[- 𝜌, 𝜌]}^{𝑑}) < 0 for all 𝛿 > 0, 𝜌 > 0,

\underset{𝜀 \to 0}{lim sup} 𝜀 log {\tilde{𝜇}}_{𝜀} ({({[- 𝜌, 𝜌]}^{𝑑})}^{𝑐}) < 0 for all large 𝜌 > 0,

which together implies the desired estimate. To prove (3.3), define the function

\tilde{Λ} (\cdot) ≔ \underset{𝜀 \to 0}{lim sup} 𝜀 Λ_{{\tilde{𝜇}}_{𝜀}} (\frac{\cdot}{𝜀})

and observe that

\begin{matrix} \tilde{Λ} (\cdot) = Λ (\cdot + 𝜂) - Λ (𝜂), \\ {\tilde{Λ}}^{*} (\cdot) = Λ^{*} (\cdot) + Λ (𝜂) - ⟨ 𝜂, \cdot ⟩ . \end{matrix}

One deduces from the Markov’s inequality that

\underset{𝜀 \to 0}{lim sup} 𝜀 log {\tilde{𝜇}}_{𝜀} (𝐵_{𝛿}^{𝑐} \cap 𝐾_{𝛼}) \leq - inf_{𝑧 \in 𝐵_{𝛿}^{𝑐} \cap 𝐾_{𝛼}} {\tilde{Λ}}^{*} (𝑧) < 0,

since ${\tilde{Λ}}^{*}$ is lower semicontinuous and $𝜂$ is an exposing hyperplane. For (3.4), observe that $0 \in {\overset{˚}{𝒟︀}}_{\tilde{Λ}}$ , so the inequality follows from (3.2).

(3) With Lemma 3.3 and Lemma 3.4, we have that for ever open set $𝐺$ ,

\underset{𝜀 \to 0}{lim inf} 𝜀 log 𝜇_{𝜀} (𝐺) \geq - inf_{𝑥 \in 𝐺 \cap ℱ︀} Λ^{*} (𝑥) = - inf_{𝑥 \in 𝐺} Λ^{*} (𝑥)

$□$

We now prove the lemmas.

Proof of Lemma 3.3. (1) The convexity follows from Hölder’s inequality as in Lemma 2.5. Precisely, given any $𝑡, 𝑡' \in [0, 1]$ satisfying $𝑡 + 𝑡' = 1$ ,

log 𝔼 (𝑒^{⟨ 𝑡 𝜆 + 𝑡' 𝜆', 𝑍_{𝜀} ⟩}) \leq 𝑡 log 𝔼 (𝑒^{⟨ 𝜆, 𝑍_{𝜀} ⟩}) + 𝑡' log 𝔼 (𝑒^{⟨ 𝜆', 𝑍_{𝜀} ⟩}),

proving the convexity of $Λ_{𝜇_{𝜀}}$ and hence $\overset{─}{Λ}$ .

If $Λ (𝜆) = - \infty$ for some $𝜆 \in ℝ^{𝑑}$ , then by convexity $Λ (𝛼 𝜆) = - \infty$ for all $𝛼 \in (0, 1]$ . Since $Λ (0) = 0$ , it follows by convexity that $Λ (- 𝛼 𝜆) = \infty$ for all $𝛼 \in (0, 1]$ , contradicting the assumption that $0 \in \overset{˚}{𝒟︀}$ . Thus, $Λ > - \infty$ everywhere.

(2) Since $0 \in \overset{˚}{{𝒟︀}_{Λ}}$ , it follows that $𝐵_{𝛿} (0) \subset \overset{˚}{{𝒟︀}_{Λ}}$ for some $𝛿 > 0$ , and $𝑐 = {sup}_{𝜆 \in 𝐵_{𝛿} (0)} Λ (𝜆) < \infty$ since the convex function $Λ$ is continuous in $\overset{˚}{{𝒟︀}_{Λ}}$ . Therefore,

Λ^{*} (𝑥) \geq sup_{𝜆 \in {\overset{─}{𝐵}}_{𝛿} (0)} [⟨ 𝜆, 𝑥 ⟩ - Λ (𝜆)] \geq | 𝛿 | | 𝑥 | - sup_{𝜆 \in {\overset{─}{𝐵}}_{𝛿} (0)} Λ (𝑥),

implying $Ψ_{Λ^{*}} (𝛼)$ is bounded for every $𝛼 < \infty$ . The function $Λ^{*}$ is convex and lower semicontinuous by a routine check as conducted in Lemma 2.5. Combining these implies that $Λ^{*}$ is a good convex rate function.

(3) Suppose now that for some $𝑥 \in ℝ^{𝑑}$ ,

Λ (𝜂) = ⟨ 𝜂, 𝑦 ⟩ - Λ^{*} (𝑦) \leq ⟨ 𝜂, 𝑥 ⟩ - Λ^{*} (𝑥) .

Then, for every $𝜃 \in ℝ^{𝑑}$ ,

⟨ 𝜃, 𝑥 ⟩ \leq Λ (𝜂 + 𝜃) - Λ (𝜂) .

In particular,

⟨ 𝜃, 𝑥 ⟩ \leq lim_{𝜀 \to 0} \frac{1}{𝜀} [Λ (𝜂 + 𝜀 𝜃) - Λ (𝜂)] = ⟨ 𝜃, \nabla Λ (𝜂) ⟩ .

Since the inequality holds for all $𝜃 \in ℝ^{𝑑}$ , $𝑥 = \nabla Λ (𝜂) = 𝑦$ . Hence, $𝑦$ is an exposed point of $Λ^{*}$ with exposing hyperplane $𝜂 \in \overset{˚}{{𝒟︀}_{Λ}}$ . $□$

Before proving Rockafellar’s lemma, let us recall the definition of relative interior of a convex set.

Definition 3.6. The relative interior of a nonempty convex set $𝐶$ is defined as

ri 𝐶 = {𝑦 \in 𝐶 : for all 𝑥 \in 𝐶, 𝑦 - 𝜀 (𝑥 - 𝑦) \in 𝐶 for some 𝜀 > 0} .

Proof of Lemma 3.4. Assume without loss of generality that ${𝒟︀}_{Λ^{*}} \neq \emptyset$ for the lemma is vacuous otherwise. Under the circumstances, fix henceforth $𝑥 \in ri {𝒟︀}_{Λ^{*}}$ and define a function

𝑓 (𝜆) ≔ Λ (𝜆) - ⟨ 𝜆, 𝑥 ⟩ + Λ^{*} (𝑥) .

Observe that

𝑓 : ℝ^{𝑑} \to [0, \infty]

is convex, lower semincontinuous, and

{inf}_{𝜆 \in ℝ^{𝑑}} 𝑓 (𝜆) = 0

. Moreover,

𝑓^{*} (\cdot) = Λ^{*} (\cdot + 𝑥) - Λ^{*} (𝑥)

. Therefore, from

𝑥 \in ri {𝒟︀}_{Λ^{*}}

it follows that

0 \in ri {𝒟︀}_{𝑓^{*}}

. By Lemma C.2, there exists

𝜂 \in {𝒟︀}_{Λ}

such that

𝑓 (𝜂) = 0

. Let

\tilde{Λ} (\cdot) = Λ (\cdot + 𝜂) - Λ (𝜂)

, so that

\tilde{Λ}

is an essentially smooth, convex function and

\tilde{Λ} (0) = 0

. Consequently, by Lemma C.3,

\tilde{Λ}

is finite in a neighborhood of the origin and thus

𝜂 \in \overset{˚}{{𝒟︀}_{Λ}}

, at which

𝑓

is differentiable by the assumption. Hence,

𝑓 (𝜂) = {inf}_{𝜆 \in ℝ^{𝑑}} 𝑓 (𝜆)

, implying that

\nabla 𝑓 (𝜂) = 0

, i.e.,

𝑥 = \nabla Λ (𝜂)

. It now follows from Lemma 3.3(2) that

𝑥 \in ℱ︀

. Since

𝑥 \in ri {𝒟︀}_{Λ^{*}}

is arbitrary, the proof is complete.

□

The following theorem is a generalization of the Gärtner-Ellis theorem (Theorem 3.2), which essentially follows from the same proof with marginal distributions replaced by projection measures in every direction. Its proof can be found in [1].

Theorem 3.7 (Baldi). Suppose ${𝜇_{𝜀}}$ is an exponentially tight family of probability measures on $𝒳︀$ .

For every closed set $𝐹 \subset 𝒳︀$ , ${lim sup}_{𝜀 \to 0} 𝜀 log 𝜇_{𝜀} (𝐹) \leq - {inf}_{𝑥 \in 𝐹} {\overset{─}{Λ}}^{*} (𝑥)$ .
Let $ℱ︀$ be the set of exposed points of ${\overset{─}{Λ}}^{*}$ with an exposing hyperplane $𝜆$ for which
$Λ (𝜆) = lim_{𝜀 \to 0} 𝜀 Λ_{𝜇_{𝜀}} (\frac{𝜆}{𝜀}) exists and \overset{─}{Λ} (𝛾 𝜆) < \infty for some 𝛾 > 1 .$
Then, for every open set $𝐺 \subset 𝒳︀$ ,
$\underset{𝜀 \to 0}{lim inf} 𝜀 log 𝜇_{𝜀} (𝐺) \geq - inf_{𝑥 \in 𝐺 \cap ℱ︀} {\overset{─}{Λ}}^{*} (𝑥) .$
If for every open set $𝐺 \subset 𝒳︀$ , ${inf}_{𝑥 \in 𝐺 \cap ℱ︀} {\overset{─}{Λ}}^{*} (𝑥) = {inf}_{𝑥 \in 𝐺} {\overset{─}{Λ}}^{*} (𝑥)$ , then ${𝜇_{𝜀}}$ satisfies the LDP with the good rate function ${\overset{─}{Λ}}^{*}$ .

Proof. See [1, Theorem 4.5.20].

□

The theorem is accompanied by a generalized criterion of differentiability of $Λ$ in the following sense.

Definition 3.8 (Gateaux differentiable). Let

𝒳︀

be a topological vector space. A function

𝑓 : {𝒳︀}^{*} \to ℝ

is said to be Gateaux differentiable if for all

𝜆, 𝜃 \in {𝒳︀}^{*}

, the function

𝑡 \in ℝ \mapsto 𝑓 (𝜆 + 𝑡 𝜃)

is differentiable at

𝑡 = 0

Corollary 3.9. Let

{𝜇_{𝜀}}

be exponentially tight probability measures on the Banach space

𝒳︀

. Suppose that

Λ (\cdot) = {lim}_{𝜀 \to 0} Λ_{𝜇_{𝜀}} (\frac{\cdot}{𝜀})

is finite-valued, Gateaux differentiable, and lower semicontinuous in

{𝒳︀}^{*}

with respect to the weak* topology. Then

{𝜇_{𝜀}}

satisfies the LDP with the good rate function

Λ^{*}

Proof. See [1, Corollary 4.5.27].

□

4. Applications

4.1. Cramér’s theorem in $ℝ^{𝑑}$

Combining the Gärtner-Ellis theorem with our previous results, we can establish the LDP for $𝑑$ -dimensional i.i.d. random variables ${(𝑋_{𝑖})}_{𝑖}$ . As before, let $𝜇_{𝑛}$ be the law of the empirical mean, and let $Λ$ and $Λ^{*}$ be defined accordingly.

Theorem 4.1 (Cramér). Suppose

{𝒟︀}_{Λ} = ℝ^{𝑑}

, then the family

{𝜇_{𝑛}}

satisfies the LDP with the convex good rate function

Λ^{*}

While powerful, this theorem does not encompass all scenarios where an LDP holds; a refined version requires only that $0 \in \overset{˚}{{𝒟︀}_{Λ}}$ for the result to remain valid.

Example 4.2. Let $𝜇$ be a Borel probability measure on $ℝ$ with a density $𝑓 (𝑥) = 𝐶_{𝑑} \cdot 𝑒^{- ‖ 𝑥 ‖} / (1 + ‖ 𝑥 ‖^{𝑑 + 2})$ and $Λ$ be its log MGF, where $𝐶_{𝑑}$ is the normalizing constant. We will show in the following that $0 \in {\overset{˚}{𝒟︀}}_{Λ} = {𝜆 \in ℝ^{𝑑} : ‖ 𝜆 ‖ \leq 1}$ , so the large deviation principle holds with a good rate function; however, $Λ$ is not steep and hence the Gärtner-Ellis theorem does not apply.

On one hand, with the aid of Cauchy-Schwarz inequality $| ⟨ 𝜆, 𝑥 ⟩ | \leq ‖ 𝜆 ‖ ‖ 𝑥 ‖$ , we realize that the estimate

𝑒^{⟨ 𝜆, 𝑥 ⟩} 𝑓 (𝑥) \leq \frac{𝐶_{𝑑} 𝑒^{‖ 𝑥 ‖ (‖ 𝜆 ‖ - 1)}}{1 + ‖ 𝑥 ‖^{𝑑 + 2}} (\forall 𝜆 \in ℝ^{𝑑})

holds and therefore ${𝒟︀}_{Λ} \supseteq {𝜆 \in ℝ^{𝑑} : ‖ 𝜆 ‖ \leq 1}$ . On the other hand, given any $𝜆 \neq 0$ , we have that along direction $𝜆$ ,

𝑒^{⟨ 𝜆, 𝑥 ⟩} 𝑓 (𝑥) = \frac{𝐶_{𝑑} 𝑒^{𝑡 ‖ 𝜆 ‖ (‖ 𝜆 ‖ - 1)}}{1 + ‖ 𝑥 ‖^{𝑑 + 2}} (𝑥 = 𝑡 𝜆, \forall 𝑡 \in ℝ),

is unbounded as $𝑡 \to \infty$ if $‖ 𝜆 ‖ > 1$ , implying ${𝒟︀}_{Λ} = {𝜆 \in ℝ^{𝑑} : ‖ 𝜆 ‖ \leq 1}$ . Finally, we have uniform bound

\frac{𝐶_{𝑑} 𝑒^{- 2 ‖ 𝑥 ‖}}{1 + ‖ 𝑥 ‖^{𝑑 + 2}} \leq 𝑒^{⟨ 𝜆, 𝑥 ⟩} 𝑓 (𝑥) \leq \frac{𝐶_{𝑑}}{1 + ‖ 𝑥 ‖^{𝑑 + 2}} (𝜆 \in {𝒟︀}_{Λ}),

Therefore, within the interior of ${𝒟︀}_{Λ}$ ,

sup_{𝜆 \in {\overset{˚}{𝒟︀}}_{Λ}} | \nabla Λ (𝜆) | \leq {(\int \frac{𝑒^{- 2 ‖ 𝑥 ‖}}{1 + ‖ 𝑥 ‖^{𝑑 + 2}} 𝑑 𝑥)}^{- 1} \int \frac{‖ 𝑥 ‖ 𝑒^{- 2 ‖ 𝑥 ‖}}{1 + ‖ 𝑥 ‖^{𝑑 + 2}} 𝑑 𝑥 < \infty .

This proves that $Λ$ is not steep.

4.2. Large deviations for finite state Markov chains

In this section, we study the large deviations of Markov chains on a finite alphabet $Σ$ . Let $𝚷 = {𝜋 (𝑖, 𝑗)}_{𝑖, 𝑗 = 1}^{| Σ |}$ be a stochastic matrix, and consider a Markov chain ${(𝑌_{𝑘})}_{𝑘 \geq 0}$ with transition probability $𝚷$ . We denote by $ℙ_{𝜎}^{𝜋}$ the probability law of the chain starting at state $𝜎 \in Σ$ :

ℙ_{𝜎}^{𝜋} (𝑌_{1} = 𝑦_{1}, \dots, 𝑌_{𝑛} = 𝑦_{𝑛}) = 𝜋 (𝜎, 𝑦_{1}) \prod_{𝑖 = 1}^{𝑛 - 1} 𝜋 (𝑦_{𝑖}, 𝑦_{𝑖 + 1}) .

We are interested in the empirical mean

𝑍_{𝑛} = \sum_{𝑘 = 1}^{𝑛} 𝑋_{𝑘},

where $𝑋_{𝑘} = 𝑓 (𝑌_{𝑘})$ for a given function $𝑓 : Σ \to ℝ^{𝑑}$ . The log MGF can be analyzed using the following family of non-negative matrices:

𝜋_{𝜆} (𝑖, 𝑗) = 𝜋 (𝑖, 𝑗) 𝑒^{⟨ 𝜆, 𝑓 (𝑗) ⟩} (𝑖, 𝑗 \in Σ) .

Theorem 4.3 (Perron-Frobenius). Let $𝐁 = {𝐵 (𝑖, 𝑗)}_{𝑖, 𝑗 = 1}^{| Σ |}$ be an irreducible matrix. Then $𝐵$ possesses an eigenvalue $𝜌$ (called the Perron–Frobenius eigenvalue) such that:

$𝜌 > 0$ is real.
For any eigenvalue $𝜆$ of $𝐵$ , $| 𝜆 | \leq 𝜌$ .
There exist left and right eigenvectors corresponding to the eigenvalue $𝜌$ that have strictly positive coordinates.
The left and right eigenvectors $𝜇$ , $𝜃$ corresponding to the eigenvalue $𝜌$ are unique up to a constant multiple.
For every $𝑖 \in Σ$ and every $𝜑 = (𝜑_{1}, \dots, 𝜑_{| Σ |})$ such that $𝜑_{𝑗} > 0$ for all $𝑗$ ,
$lim_{𝑛 \to \infty} \frac{1}{𝑛} log [\sum_{𝑗 = 1}^{| Σ |} 𝐵^{𝑛} (𝑖, 𝑗) 𝜑_{𝑗}] = lim_{𝑛 \to \infty} \frac{1}{𝑛} log [\sum_{𝑗 = 1}^{| Σ |} 𝜑_{𝑗} 𝐵^{𝑛} (𝑗, 𝑖)] = log 𝜌 .$

Proof. See, for example, Wikipedia.

□

Theorem 4.4. Let ${𝑌_{𝑘}}$ be a finite state Markov chain possessing an irreducible transition matrix $𝚷$ . For every $𝑧 \in ℝ^{𝑑}$ , define

𝐼 (𝑧) = sup_{𝜆 \in ℝ^{𝑑}} {⟨ 𝜆, 𝑧 ⟩ - log 𝜌 (𝚷_{𝜆})}

Then the empirical mean $𝑍_{𝑛}$ satisfies the LDP with the convex, good rate function $𝐼$ . Explicitly, for any set $Γ \subseteq ℝ^{𝑑}$ , and any initial state $𝜎 \in Σ$ ,

- inf_{𝑧 \in \overset{˚}{Γ}} 𝐼 (𝑧) \leq \underset{𝑛 \to \infty}{lim inf} \frac{1}{𝑛} log ℙ_{𝜎}^{𝜋} (𝑍_{𝑛} \in Γ) \leq \underset{𝑛 \to \infty}{lim sup} \frac{1}{𝑛} log ℙ_{𝜎}^{𝜋} (𝑍_{𝑛} \in Γ) \leq - inf_{𝑧 \in \overset{─}{Γ}} 𝐼 (𝑧) .

Proof. Consider the logarithmic generating function

Λ_{𝑛} (𝜆) ≔ log 𝔼_{𝜎}^{𝜋} [𝑒^{⟨ 𝜆, 𝑍_{𝑛} ⟩}] .

By Gärtner–Ellis theorem (Theorem 3.2), it is enough to check that the limit

Λ (𝜆) ≔ lim_{𝑛 \to \infty} \frac{1}{𝑛} Λ_{𝑛} (𝑛 𝜆)

exists, is finite and differentiable everywhere in $ℝ^{𝑑}$ , and satisfies $Λ (𝜆) = log 𝜌 (𝚷_{𝜆})$ . To begin, note that

Λ_{𝑛} (𝜆) = log \sum_{𝑦_{1}, \dots, 𝑦_{𝑛}} ℙ_{𝜎}^{𝜋} (𝑌_{1} = 𝑦_{1}, \dots, 𝑌_{𝑛} = 𝑦_{𝑛}) \prod_{𝑖 = 1}^{𝑛} 𝑒^{⟨ 𝜆, 𝑓 (𝑦_{𝑘}) ⟩} = log \sum_{𝑦_{𝑛}} {(𝚷_{𝜆})}^{𝑛} (𝜎, 𝑦_{𝑛}) .

Since $𝚷_{𝜆}$ is irreducible, we have

Λ (𝜆) = log 𝜌 (𝚷_{𝜆}) .

To show that it is differentiable, we apply the implicit function theorem. Explicitly, consider the functions $𝐹_{𝑦} : ℝ \times ℝ^{| Σ |} \times ℝ \to ℝ \times ℝ^{| Σ |}$ , parametrized by $𝑦 \in ℝ^{| Σ |}$ , defined by

𝐹_{𝑦} (𝜌, 𝑥, 𝜆) = (⟨ 𝑦, 𝑥 ⟩ - 1, 𝚷_{𝜆} 𝑥 - 𝜌 𝑥) .

Clearly, $𝐹_{𝑦}$ is continuously differentiable. Hence, it suffices to show that for every $𝜆_{0}$ with the associated Perron-Frobenius eigenvalue $𝜌_{0}$ and left and right eigenvectors $𝑦_{0}$ and $𝑥_{0}$ of $𝚷_{𝜆_{0}}$ satisfying $⟨ 𝑦_{0}, 𝑥_{0} ⟩ = 1$ , we have (1) $𝐹_{𝑦_{0}} (𝜌_{0}, 𝑥_{0}, 𝜆_{0}) = 0$ and (2) the Jacobian matrix

𝜕_{𝜌, 𝑥} 𝐹_{𝑦_{0}} (𝜌_{0}, 𝑥_{0}, 𝜆_{0}) = (\begin{matrix} 0 & 𝑦_{0}^{𝑇} \\ - 𝑥_{0} & 𝚷_{𝜆_{0}} - 𝜌_{0} 𝐼 . \end{matrix})

is invertible. Indeed, (1) is clear, and (2) holds because if it did not, there exists $(𝑢, 𝑣) \in ℝ \times ℝ^{| Σ |} ∖ {(0, 0)}$ such that

{\begin{cases} ⟨ 𝑦_{0}, 𝑣 ⟩ = 0, \\ - 𝑢 𝑥_{0} + 𝚷_{𝜆_{0}} 𝑣 - 𝜌_{0} 𝑣 = 0, \end{cases}

which is impossible. This implies, by the implicit function theorem, that there exists a continuously differentiable function

𝜆 \mapsto (𝜌 (𝜆), 𝑥 (𝜆))

on a neighborhood of

𝜆_{0}

such that

𝐹_{𝑦_{0}} (𝜌 (𝜆), 𝑥 (𝜆), 𝜆) = 0

. In particular,

𝜆 \mapsto log 𝜌 (𝚷_{𝜆}) = log 𝜌 (𝜆)

is continuously differentiable on a neighborhood of

𝜆_{0}

. The proof is finished by noting that

𝜆_{0}

is arbitrary.

□

As a corollary, we have the following.

Corollary 4.5. The empirical averages $𝐿_{𝑛}^{𝐘} (𝑖) = \frac{1}{𝑛} \sum_{𝑘 = 1}^{𝑛} 𝟙_{𝑖} (𝑌_{𝑘})$ , with $𝑖 \in Σ$ , satisfy the LDP with the good rate function

𝐼 (𝑞) = sup_{𝜆 \in ℝ^{𝑑}} {⟨ 𝜆, 𝑞 ⟩ - log 𝜌 (𝚷_{𝜆})} = {\begin{cases} {sup}_{𝑢 > 0} \sum_{𝑗 \in Σ} 𝑞_{𝑗} log [\frac{𝑢_{𝑗}}{{(𝐮 𝚷)}_{𝑗}}] & 𝑞 \in 𝑀_{1} (Σ), \\ \infty & 𝑞 \notin 𝑀_{1} (Σ) \end{cases}

where

𝜋_{𝜆} (𝑖, 𝑗) = 𝜋 (𝑖, 𝑗) 𝑒^{𝜆_{𝑗}}

and the inequality between vectors is compared entrywise.

Proof. The first equality follows immediately from Theorem 4.4 by taking

𝑓 = (𝟙_{1}, 𝟙_{2}, \dots, 𝟙_{| Σ |}) .

To prove the second inequality, we first note that one inequality is more obvious than the other: By taking $𝐮$ to be the left probability eigenvector of $𝚷_{𝜆}$ , we see the inequality “ $\leq$ ”. To prove the other inequality, assume $𝑞 \in 𝑀_{1} (Σ)$ and choose $𝜆_{𝑗} = log [𝑢_{𝑗} / {(𝐮 𝚷)}_{𝑗}]$ so that $𝐮 𝚷_{𝜆} = 𝐮$ and that $𝜌 (𝚷_{𝜆}) = 1$ . Therefore, by definition,

𝐼 (𝑞) \geq \sum_{𝑗 = 1}^{| Σ |} 𝑞_{𝑗} log [\frac{𝑢_{𝑗}}{{(𝐮 𝚷)}_{𝑗}}],

finishing the proof.

□

We also consider a derived process ${(𝑌_{𝑘} 𝑌_{𝑘 + 1})}_{𝑘 \geq 0}$ of consecutive pairs to obtain Sanov’s theorem. Such a process has the transition matrix $𝚷^{(2)}$ defined by

𝜋^{(2)} ((𝑘 \times ℓ, 𝑖 \times 𝑗)) = 𝟙_{ℓ} (𝑖) 𝜋 (𝑖, 𝑗) .

As is discussed in the following, one can determine the large deviations of the pair empirical measure

𝐿_{𝑛, 2}^{𝐘} (𝑦) = \frac{1}{𝑛} \sum_{𝑖 = 1}^{𝑛} 𝟙_{𝑦} (𝑌_{𝑖 - 1} 𝑌_{𝑖})

For $𝑞 \in 𝑀_{1} (Σ^{2})$ , we write

𝑞_{1} (𝑖) = \sum_{𝑗 = 1}^{| Σ |} 𝑞 (𝑖, 𝑗) and 𝑞_{2} (𝑖) = \sum_{𝑗 = 1}^{| Σ |} 𝑞 (𝑗, 𝑖),

and call $𝑞$ shift-invariant if $𝑞_{1} = 𝑞_{2}$ .

Theorem 4.6. Assume that $𝚷$ is irreducible. Then for every probability measure $𝑞 \in 𝑀_{1} ({(𝑖, 𝑗) : 𝜋 (𝑖, 𝑗) > 0})$ ,

𝐼_{2} (𝑞) = {\begin{cases} \sum_{𝑖} 𝑞_{1} (𝑖) 𝐻 (𝑞 (\cdot | 𝑖) | 𝜋 (𝑖, \cdot)) & if 𝑞 is shift-invariant, \\ \infty & otherwise. \end{cases}

Proof. By Corollary 4.5,

𝐼_{2} (𝑞) = sup_{𝑢 > 0} \sum_{𝑖, 𝑗 \in Σ} 𝑞 (𝑖, 𝑗) log [\frac{𝑢_{𝑖, 𝑗}}{{(𝐮 𝚷^{(2)})}_{𝑖, 𝑗}}] = sup_{𝑢 > 0} \sum_{𝑖, 𝑗 \in Σ} 𝑞_{𝑖, 𝑗} log [\frac{𝑢_{𝑖, 𝑗}}{\sum_{𝑘} 𝑢_{𝑘, 𝑖} 𝜋_{𝑖, 𝑗}}] .

If $𝑞$ is not invariant, then $𝑞_{1} (𝑗_{0}) < 𝑞_{2} (𝑗_{0})$ for some $𝑗_{0}$ . For $𝐮$ such that $𝑢 (\cdot, 𝑗) = 1$ when $𝑗 \neq 𝑗_{0}$ and $𝑢 (\cdot, 𝑗_{0}) = 𝑒^{𝛼}$ , we have $𝐼_{2} (𝑞) = \infty$ if we let $𝛼 \to \infty$ .

If $𝑞$ is invariant,

\sum_{𝑖, 𝑗 \in Σ} 𝑞 (𝑖, 𝑗) log [\frac{\sum_{𝑘} 𝑢_{𝑘, 𝑖} 𝑞_{2} (𝑗)}{\sum_{𝑘} 𝑢_{𝑘, 𝑗} 𝑞_{1} (𝑖)}] = 0 .

Hence,

\begin{aligned} 𝐼_{2} (𝑞) - \sum_{𝑖} 𝑞_{1} (𝑖) 𝐻 (𝑞 (\cdot | 𝑖) | 𝜋 (𝑖 | \cdot)) = sup_{𝑢 > 0} \sum_{𝑖, 𝑗 \in Σ} 𝑞 (𝑖, 𝑗) log [\frac{𝑢_{𝑖, 𝑗} 𝑞_{1} (𝑖)}{\sum_{𝑘} 𝑢_{𝑘, 𝑖} 𝑞 (𝑖, 𝑗)}] \\ = & sup_{𝑢 > 0} {- \sum_{𝑗} 𝑞_{2} (𝑗) 𝐻 (𝑞' (\cdot | 𝑗) | 𝑢' (\cdot | 𝑗))}, \end{aligned}

where $𝑢' (\cdot | 𝑗) = \frac{𝑢 (\cdot, 𝑗)}{\sum_{𝑖} 𝑢 (𝑖, 𝑗)}$ and $𝑞' (\cdot | 𝑗) = \frac{𝑞 (\cdot, 𝑗)}{\sum_{𝑖} 𝑞 (𝑖, 𝑗)}$ . This implies

𝐼_{2} (𝑞) \leq \sum_{𝑖 \in Σ} 𝑞_{1} (𝑖) 𝐻 (𝑞 (\cdot | 𝑖) | 𝜋 (𝑖, \cdot)) .

Taking $𝐮 > 0$ approaching $𝑞$ proves the theorem. $□$

A. Probability theory

A.1. Basic inequalities

Proposition A.1 (Borel-Cantelli). Let ${(𝑆_{𝑛})}_{𝑛 = 1}^{\infty}$ be a sequence of events in a probability space $(𝒳︀, ℬ︀, 𝜇)$ . Then,

𝜇 (\underset{𝑛 \to \infty}{lim sup} 𝑆_{𝑛}) \leq lim_{𝑁 \to \infty} \sum_{𝑛 = 𝑁}^{\infty} 𝜇 (𝑆_{𝑛}) .

Proposition A.2 (Markov). Let $𝑋$ be a nonnegative random variable. Then, for every $𝛼 > 0$ ,

𝜇 (𝑋 \geq 𝛼) \leq 𝛼^{- 1} \cdot 𝔼 [𝑋] .

Proposition A.3 (Hölder). Let $𝑋, 𝑌$ be non-negative random variables. Then, for $𝑝, 𝑞 \in [1, \infty]$ satisfying $\frac{1}{𝑝} + \frac{1}{𝑞} = 1$ ,

{𝔼 (𝑋^{𝑝})}^{1 / 𝑝} {𝔼 (𝑌^{𝑞})}^{1 / 𝑞} \geq 𝔼 (𝑋 𝑌),

where the equality holds if and only if

𝑋 = 𝑐 𝑌

for some

𝑐 > 0

Proof. Without loss of generality, assume $𝔼 (𝑋^{𝑝}) = 𝔼 (𝑌^{𝑞}) = 1$ to paraphrase the proposition: $1 \geq 𝔼 (𝑋 𝑌)$ with equality holding if and only if $𝑋 = 𝑐 𝑌$ for some $𝑐 > 0$ . Under the circumstances, it is not hard to verify that Young’s inequality

𝑋 𝑌 \leq \frac{𝑋^{𝑝}}{𝑝} + \frac{𝑌^{𝑞}}{𝑞}

holds if and only if

𝑋^{𝑝} = 𝑌^{𝑞}

. The paraphrased proposition then follows by integrating both sides.

□

Proposition A.4 (Jensen). Let

𝑋

be a real-valued random variable and

𝜑 : ℝ \to ℝ

is convex. If

𝔼 (𝑋)

and

𝔼 (𝜑 (𝑋))

are defined, then

𝔼 (𝜑 (𝑋)) \geq 𝜑 (𝔼 (𝑋))

with the convention

𝜑 (+ \infty) = {lim}_{𝑥 \to \infty} 𝜑 (𝑥)

and

𝜑 (- \infty) = {lim}_{𝑥 \to - \infty} 𝜑 (𝑥)

Proof. We make use of the property of convex functions that

𝜑 (𝑥) = sup {𝑎 𝑥 + 𝑏 : 𝜑 (𝑥) \geq 𝑎 𝑥 + 𝑏 for all 𝑥 \in ℝ} .

The left-hand side is by definition larger than the other, and thus it remains to show the remaining. To this end, it suffices to show that for all $(𝑥_{0}, 𝑦_{0})$ satisfying $𝜑 (𝑥_{0}) \geq 𝑦_{0}$ , there exists a linear function $𝜓 (𝑥) = 𝑎 (𝑥 - 𝑥_{0}) + 𝑦_{0}$ such that $𝜑 (𝑥) \leq 𝜓 (𝑥)$ for all $𝑥$ . Essentially, this is achieved by choosing

\begin{aligned} 𝑎 \in & [sup_{𝑥 < 𝑥_{0}} \frac{𝜑 (𝑥) - 𝜑 (𝑥_{0})}{𝑥 - 𝑥_{0}}, inf_{𝑥 > 𝑥_{0}} \frac{𝜑 (𝑥) - 𝜑 (𝑥_{0})}{𝑥 - 𝑥_{0}}] \\ = & [\underset{𝑥 \to 𝑥_{0}}{lim inf} \frac{𝜑 (𝑥) - 𝜑 (𝑥_{0})}{𝑥 - 𝑥_{0}}, \underset{𝑥 \to 𝑥_{0}}{lim sup} \frac{𝜑 (𝑥) - 𝜑 (𝑥_{0})}{𝑥 - 𝑥_{0}}] . \end{aligned}

Now that (A.1) coincides with

𝜑 (𝑥) = sup {𝑎 𝑥 + 𝑏 : 𝜑 (𝑥) \geq 𝑎 𝑥 + 𝑏 for all 𝑥 \in ℝ, 𝑎, 𝑏 \in ℚ},

one can enumerate the linear functions on the right-hand side by $(𝜓_{𝑖})_{𝑖 = 1}^{\infty}$ and obtain

𝔼 (𝜑 (𝑋)) \geq 𝔼 (max_{1 \leq 𝑖 \leq 𝑛} 𝜓_{𝑖} (𝑋)) \geq max_{1 \leq 𝑖 \leq 𝑛} 𝜓_{𝑖} (𝔼 (𝑋)) .

𝔼 (𝑋) \in ℝ

, then the proposition holds by letting

𝑛 \to \infty

. If

𝔼 (𝑋) = \infty

(similar for

𝔼 (𝑋) = - \infty

) and

𝜑 (\infty) > - \infty

, then right-hand side of the above still converges to

𝜑 (\infty)

, while the proposition is trivial when

𝔼 (𝑋) = \infty

and

𝜑 (\infty) = - \infty

□

A.2. Radon-Nikodym theorem

Definition A.5 (absolute continuity). Let

𝜇

and

𝜈

be defined on a common measurable space

(𝒳︀, ℬ︀)

. We say

𝜈

is absolutely continuous with respect to

𝜇

, denoted by

𝜈 ≪ 𝜇

, if

𝜈 (𝐴) = 0

for every

𝐴 \in ℬ︀

satisfying

𝜇 (𝐴) = 0

Theorem A.6 (Radon-Nikodym). Suppose $𝜈, 𝜇$ are two $𝜎$ -finite measures on a common measurable space $(𝒳︀, ℬ︀)$ and $𝜈 ≪ 𝜇$ , then there exists a $ℬ︀$ -measurable function $𝑓 : 𝒳︀ \to [0, \infty)$ such that for every $𝐴 \in ℬ︀$ ,

𝜈 (𝐴) = \int_{𝐴} 𝑓 𝑑 𝜇 .

A.3. Laws of large numbers

Theorem A.7 (Weak Law of Large Numbers). Suppose

(𝑋_{𝑖})_{𝑖 = 1}^{\infty}

are i.i.d. and

𝔼 (𝑋_{1}) = 0

. Then,

𝑛^{- 1} \sum_{𝑖 = 1}^{𝑛} 𝑋_{𝑖} \to 0

in probability.

Proof. Equivalently, we can assume $𝑋_{𝑖}$ is nonnegative and $𝔼 (𝑋_{1}) < \infty$ and prove $𝑛^{- 1} \sum_{𝑖 = 1}^{𝑛} 𝑋_{𝑖} \to 𝔼 (𝑋_{1})$ . Let $𝑀 > 0$ and $𝑋_{𝑖} = 𝑌_{𝑖, 1} + 𝑌_{𝑖, 2}$ with $𝑌_{𝑖, 1} = max {𝑋_{𝑖}, 𝑀}$ . Immediately,

ℙ (\frac{1}{𝑛} \sum_{𝑖 = 1}^{𝑛} 𝑌_{𝑖, 1} > 𝜀) \leq \frac{𝑛 𝑀}{𝑛^{2} 𝜀^{2}} \to 0 as 𝑛 \to \infty .

On the other hand,

ℙ (\frac{1}{𝑛} \sum_{𝑖 = 1}^{𝑛} 𝑌_{𝑖, 2} > 𝜀) \leq \frac{𝑛 𝔼 (𝑋_{1} 𝟙_{𝑋_{1} \geq 𝑀})}{𝑛 𝜀} \to 0 as 𝑀 \to \infty .

The theorem is proved by combining the above.

□

Lemma A.8 (Kronecker). Suppose

𝑎_{𝑛} > 0

and

𝑎_{𝑛} ↗︎ \infty

. Then

\sum_{𝑖 = 1}^{\infty} 𝑎_{𝑛}^{- 1} 𝑥_{𝑛} < \infty

implies

𝑎_{𝑛}^{- 1} \sum_{𝑖 = 1}^{𝑛} 𝑥_{𝑖} \to 0

Proof. Writing $𝑏_{𝑛} = (𝑎_{𝑛}^{- 1} - 𝑎_{𝑛 - 1}^{- 1})$ with $𝑎_{0}^{- 1} ≔ 0$ , one may use summation by parts to deduce

𝑎_{𝑛}^{- 1} \sum_{𝑖 = 1}^{𝑛} 𝑥_{𝑖} = \sum_{𝑖 = 1}^{𝑛} 𝑎_{𝑖} (𝑎_{𝑖}^{- 1} 𝑥_{𝑖}) = \sum_{𝑖 = 1}^{𝑛} 𝑎_{𝑖}^{- 1} 𝑥_{𝑖} + \sum_{𝑖 = 1}^{𝑛 - 1} \frac{𝑎_{𝑖} - 𝑎_{𝑖 + 1}}{𝑎_{𝑛}} \sum_{𝑗 = 1}^{𝑖} 𝑎_{𝑗}^{- 1} 𝑥_{𝑗} .

The last expression converges to

0

𝑛 \to \infty

□

Proposition A.9 (Kolmogorov's Criterion of SLLN). Suppose

(𝑋_{𝑖})_{𝑖 = 1}^{\infty}

are independent such that

𝔼 (𝑋_{𝑛}) = 0

and

\sum_{𝑖 = 1}^{\infty} 𝑖^{- 2} Var (𝑋_{𝑖}) < \infty

. Then,

𝑛^{- 1} \sum_{𝑖 = 1}^{𝑛} 𝑋_{𝑖} \to 0

a.s.

Proof. By virtue of the independence, one observes that

𝔼 ({| \sum_{𝑖 = 1}^{\infty} \frac{𝑋_{𝑖}}{𝑖} |}^{2}) \leq \sum_{𝑖 = 1}^{\infty} \frac{Var (𝑋_{𝑖})}{𝑖^{2}} < \infty,

which, in particular, implies

\sum_{𝑖 = 1}^{\infty} \frac{𝑋_{𝑖}}{𝑖}

is finite almost surely, which in turn implies the conclusion by Lemma A.8.

□

Proposition A.10 (Strong Law of Large Numbers). Suppose

(𝑋_{𝑖})_{𝑖 = 1}^{\infty}

are i.i.d. and

𝔼 (𝑋_{1}) = 0

. Then,

𝑛^{- 1} \sum_{𝑖 = 1}^{𝑛} 𝑋_{𝑖} \to 𝔼 (𝑋_{1})

a.s.

Proof. It suffices to prove the case $𝔼 (| 𝑋_{1} |) < \infty$ , for if otherwise, one may still apply the result to $𝑌_{𝑛} = min {𝑀, 𝑋_{𝑛}}$ for every $𝑀 > 0$ if $𝔼 (𝑋_{1}^{+}) < \infty$ (resp., $𝑌_{𝑛} = max {- 𝑀, 𝑋_{𝑛}} 𝑋_{𝑛}$ if $𝔼 (𝑋_{1}^{-}) > - \infty$ ) so that $𝔼 (| 𝑌_{𝑛} |) < \infty$ and that $𝑌_{𝑛} \leq 𝑋_{𝑛}$ (resp., $𝑌_{𝑛} \geq 𝑋_{𝑛}$ ), leading to the conclusion by letting $𝑀 \to \infty$ .

To begin, truncate $𝑋_{𝑛}$ to define

𝑌_{𝑛} = 𝟙_{{| 𝑋_{𝑛} | \leq 𝑛}} 𝑋_{𝑛} - 𝔼 (𝑋_{𝑛} 𝟙_{{| 𝑋_{𝑛} | \leq 𝑛}}) .

Then, verify the assumption of Proposition A.9

\begin{aligned} \sum_{𝑖 = 1}^{\infty} \frac{Var (𝑌_{𝑖})}{𝑖^{2}} & \leq \sum_{𝑖 = 1}^{\infty} \frac{𝔼 (| 𝑋_{1} |^{2} 𝟙_{{| 𝑋_{1} | \leq 𝑖}})}{𝑖^{2}} = (𝔼 (| 𝑋_{1} |^{2} \sum_{𝑖 = max {1, | 𝑋_{1} |}}^{𝑛} \frac{1}{𝑖^{2}})) \\ \leq ℙ (| 𝑋_{1} | < 1) + 𝔼 (| 𝑋_{1} |^{2} \sum_{𝑖 = | 𝑋_{1} |}^{𝑛} \frac{2}{𝑖 (𝑖 + 1)}) \\ \leq ℙ (| 𝑋_{1} | < 1) + 2 𝔼 (| 𝑋_{1} |) . \end{aligned}

to deduce

\frac{1}{𝑛} \sum_{𝑖 = 1}^{\infty} 𝑌_{𝑖} \to 0 .

On the other hand, by the Borel-Cantelli lemma,

ℙ (| 𝑋_{𝑛} | > 𝑛 infinitely often) \leq \underset{𝑛 \to \infty}{lim sup} \sum_{𝑖 = 𝑛}^{\infty} ℙ (| 𝑋_{𝑖} | \geq 𝑖) \leq \underset{𝑛 \to \infty}{lim sup} 𝔼 (| 𝑋_{1} |) .

Combining (A.3) and (A.4) gives that

lim_{𝑛 \to \infty} 𝑛^{- 1} \sum_{𝑖 = 1}^{𝑛} 𝑋_{𝑖} = lim_{𝑛 \to \infty} 𝑛^{- 1} \sum_{𝑖 = 1}^{𝑛} (𝑌_{𝑖} + 𝔼 (| 𝑋_{1} | 𝟙_{{| 𝑋_{1} | \leq 𝑖}})) = 𝔼 (𝑋_{1}) .

$□$

B. Functional analysis

Theorem B.1 (Hahn-Banach Extension Theorem). Let $𝒳︀$ be a real vector space. Suppose that

$𝒴︀$ is a linear subspace of $𝒳︀$ ,
$𝑝 : 𝒳︀ \to ℝ$ is a convex functional,
$𝑓 : 𝒴︀ \to ℝ$ is linear and $𝑓 \leq 𝑝 |_{𝒴︀}$ .

Then, there exists a linear functional $𝑔 : 𝒳︀ \to ℝ$ such that $𝑔 |_{𝒴︀} = 𝑓 and 𝑔 \leq 𝑝$ .

Proof. The theorem is proved by transfinite induction.

If $𝒴︀ \neq 𝒳︀$ , choose $𝑦 \in 𝒴︀ ∖ 𝒳︀$ , $𝒴︀' = span ({𝑦} \cup 𝒳︀)$ and extend $𝑓$ to $𝑓' : 𝒴︀' \to ℝ$ by

𝑓' (𝑥 + 𝑡 𝑦) = 𝑓 (𝑥) + 𝑡 𝛼 for all 𝑥 \in 𝒴︀, 𝑡 \in ℝ,

where one may choose, if possible,

𝛼 (= 𝑓' (𝑥')) \in [sup_{𝑡 < 0} sup_{𝑥 \in 𝒴︀} \frac{𝑝 (𝑥 + 𝑡 𝑥') - 𝑓 (𝑥)}{𝑡}, inf_{𝑡 > 0} inf_{𝑥 \in 𝒴︀} \frac{𝑝 (𝑥 + 𝑡 𝑥') - 𝑓 (𝑥)}{𝑡}]

so that $𝑓' (𝑥) \leq 𝑝 (𝑥)$ on $𝒴︀'$ . To see the above interval is indeed nonempty, note that the convexity of $𝑝$ together with $𝑝 \geq 𝑓$ implies that for each $𝑡 > 0$ ,

\frac{𝑝 (𝑧 + 𝑡 𝑥') - 𝑓 (𝑧)}{𝑡} \geq \frac{𝑝 (𝑧' - 𝑡 𝑥') - 𝑓 (𝑧')}{- 𝑡} for all 𝑧, 𝑧' \in 𝒴︀, 𝑡 > 0,

and hence the non-emptiness follows from the finite intersection property.

To conclude the proof, consider the set $𝒞︀$ of all pairs $(𝒴︀, 𝑓)$ such that $𝒴︀$ is a subspace of $𝒳︀$ and that $𝑓 : 𝒴︀ \to ℝ$ is linear on $𝒴︀$ with $𝑓 \leq 𝑝 |_{𝒴︀}$ . Define further a partial order $\leq$ by

(𝒴︀, 𝑓) \leq (𝒴︀', 𝑓') if and only if 𝒴︀ \subseteq 𝒴︀' and 𝑓' |_{𝒴︀} = 𝑓 .

Hence, for any chain ${({𝒴︀}_{𝑖}, 𝑓_{𝑖})} \subset 𝒞︀$ , the space $𝒴︀ = \cup_{𝑖} {𝒴︀}_{𝑖}$ is a vector space and the function $𝑓 (𝑥) ≔ 𝑓_{𝑖} (𝑥)$ whenever $𝑥 \in {𝒴︀}_{𝑖}$ , which forms a maximal element $(𝒴︀, 𝑓) \in 𝒞︀$ of the chain. Hence, by Zorn’s lemma, there exists a maximal element $(𝒴︀, 𝑓) \in 𝒞︀$ , and $𝒴︀ = 𝒳︀$ by the first part of the proof. $□$

In the context of topological vector spaces, an equivalent of Theorem B.1 is phrased in terms of separation, as follows. Recall that a core point $𝑥$ of a subset $𝐴$ of a vector space $𝒳︀$ is the point satisfying that for every $𝑦 \subset 𝒳︀$ there exists $𝜀 = 𝜀 (𝑦)$ such that $𝑦 + 𝛿 𝑦 \in 𝐴$ for all $| 𝛿 | \leq 𝜀$ , for which we denote $cor 𝐴 = {𝑥 \in 𝐴 : 𝑥 is a core point}$ .

Theorem B.2 (Hahn-Banach Separation Theorem). Let

𝐴

and

𝐵

be disjoint, nonempty, convex subsets of a real topological vector space

𝒳︀

. Assume that

cor 𝐴 \neq \emptyset

. Then there is a linear functional

𝑓 : 𝒳︀ \to ℝ

satisfying

{sup}_{𝑎 \in 𝐴} 𝑓 (𝑎) \leq {inf}_{𝑏 \in 𝐵} 𝑓 (𝑏)

and

𝐴 \cup 𝐵 ⊄ {𝑓 = 𝛼}

for all

𝛼

. Moreover, if in addition

int 𝐴 \neq \emptyset

, then

𝑓

can be chosen so that

𝑓 \in {𝒳︀}^{*}

and

𝑓 (𝑎) < 𝑓 (𝑏)

for all

𝑎 \in 𝐴

and

𝑏 \in 𝐵

Proof of Theorem B.2 by Theorem B.1. We first prove the general case, after which we point out the necessary modifications for the case where $int 𝐴 \neq \emptyset$ .

Let $𝐶 = 𝐴 - 𝐵$ and $𝑧 \in cor (𝐴 - 𝐵)$ . Define $𝐶 = 𝐴 - 𝐵 - 𝑧$ so that $- 𝑧 \notin 𝐶$ and its associated gauge

𝑝_{𝐶} (𝑥) = inf {𝑡 > 0 : 𝑥 \in 𝑡 𝐶} .

Then, $𝑝_{𝐶} : 𝒳︀ \to ℝ$ is a sublinear functional. Indeed, since $0 \in cor 𝐶$ , $𝑝_{𝑐}$ is real-valued. Moreover, $𝑝_{𝐶} (𝑡 𝑥) = 𝑡 𝑝_{𝐶} (𝑥)$ for all $𝑡 > 0$ by definition and $𝑝_{𝐶} (𝑥 + 𝑦) \leq 𝑝_{𝐶} (𝑥) + 𝑝_{𝐶} (𝑦)$ by convexity of $𝐶$ . Consider the subspace $𝒴︀ = span {- 𝑧}$ and define the linear functional $𝑔 (- 𝑡 𝑧) = 𝑡$ . To apply the extension theorem, observe that $𝑝_{𝐶} (- 𝑧) \geq 1$ and thus $𝑔 \leq 𝑝 |_{𝒴︀}$ , yielding a linear extension $𝑓 : 𝒳︀ \to ℝ$ of $𝑔$ satisfying $𝑓 \leq 𝑝_{𝐶}$ by Theorem B.1. To see $𝑓$ separates $𝐴$ and $𝐵$ , observe that for any $𝑎 \in 𝐴$ and $𝑏 \in 𝐵$ ,

\begin{aligned} 𝑓 (𝑎) = 𝑓 (𝑎 - 𝑏 - 𝑧) + 𝑓 (𝑧) + 𝑓 (𝑏) \\ \leq & 𝑝_{𝐶} (𝑎 - 𝑏 - 𝑧) + 𝑓 (𝑧) + 𝑓 (𝑏) \leq 1 - 1 + 𝑓 (𝑏) = 𝑓 (𝑏) . \end{aligned}

The number $𝛼$ in question can be arbitrary in $[{sup}_{𝑎 \in 𝐴} 𝑓 (𝑎), {inf}_{𝑏 \in 𝐵} 𝑓 (𝑏)]$ . To see that that $𝐴 \cup 𝐵 ⊄ {𝑓 = 𝑎}$ , note that $𝑧 = 𝑎 - 𝑏$ for some $𝑎 \in 𝐴$ and $𝑏 \in 𝐵$ and $𝑓 (𝑧) = - 1 = 𝑓 (𝑎) - 𝑓 (𝑏)$ as desired.

Now if $int 𝐴 \neq \emptyset$ , pick $𝑧 \in int (𝐴 - 𝐵)$ and proceed as before. Under the circumstances, $𝐶$ is a neighborhood of $0$ . Hence, $| 𝑓 (𝑥) | \leq max {| 𝑝_{𝐶} (𝑥) |, | 𝑝_{𝐶} (- 𝑥) |} \leq 1$ for all $𝑥$ in the neighborhood $𝐶 \cap - 𝐶$ of $0$ , proving the continuity. Furthermore, $𝑝_{𝐶} (𝑎 - 𝑏 - 𝑧) < 1$ due to the fact $𝑧 \in int (𝐴 - 𝐵)$ , rendering (B.1) a strict inequality. $□$

Proof of Theorem B.1 by Theorem B.2. Let

𝐴 = {(𝑥, 𝛼) \in 𝒳︀ \times ℝ : 𝑝 (𝑥) < 𝛼} and 𝐵 = {(𝑦, 𝛽) \in 𝒴︀ \times ℝ : 𝑓 (𝑦) \geq 𝛽} .

Then, $𝐴$ and $𝐵$ are convex and every point in $𝐴$ is a core point. Thus, by Theorem B.2, there exists a linear functional $ℎ : 𝒳︀ \to ℝ$ and numbers $𝜌 \in ℝ$ such that

sup_{(𝑥, 𝛼) \in 𝐴} ℎ (𝑥) + 𝜌 𝛼 \leq 𝛾 ≔ inf_{(𝑦, 𝛽) \in 𝐵} ℎ (𝑦) + 𝜌 𝛽 (𝛾 \in ℝ) .

Under the circumstances, we observe, by letting $𝛼 \to \infty$ , that $𝜌 \leq 0$ . It is clear that $𝜌 \neq 0$ , for if otherwise, the inequality (B.1) above holds if and only if $ℎ (𝑥) = 0$ for all $𝑥 \in 𝒳︀$ , contradicting Theorem B.2 as $ℎ (𝐴) = ℎ (𝐵) = {0}$ . Now that $𝜌 < 0$ , the inequality (B.2) implies that

\begin{matrix} 𝜌^{- 1} ℎ (𝑦) + 𝑓 (𝑦) \leq 𝜌^{- 1} 𝛾 for all 𝑦 \in 𝒴︀ \Rightarrow 𝑓 = 𝜌^{- 1} (𝛾 - ℎ |_{𝒴︀}), \\ 𝜌^{- 1} 𝑔 (𝑥) + 𝑝 (𝑥) \geq 𝜌^{- 1} 𝛾 for all 𝑥 \in 𝒳︀ \Rightarrow 𝑝 \geq 𝜌^{- 1} (𝛾 - ℎ) . \end{matrix}

The proof is finished by choosing

𝑔 = 𝜌^{- 1} (𝛾 - ℎ)

□

Theorem B.3. Suppose

𝒳︀

is a locally convex real topological vector space. Suppose

𝐴

and

𝐵

are two disjoint, nonempty, convex sets in the locally convex topological vector space

𝒳︀

. If

𝐴

is compact and

𝐵

is closed, then there exists an

𝑓 \in {𝒳︀}^{*}

such that

{sup}_{𝑎 \in 𝐴} 𝑓 (𝑎) < {inf}_{𝑏 \in 𝐵} 𝑓 (𝑏)

Proof. Suppose

𝑉

is a convex neighborhood of

0

such that

(𝐴 + 𝑉) \cap 𝐵 = \emptyset

. Apply Theorem B.2 to

𝐴 + 𝑉

and

𝐵

to obtain

𝑓 \in {𝒳︀}^{*}

such that

{sup}_{𝑎 \in 𝐴} 𝑓 (𝑎) < {sup}_{𝑎 \in 𝐴 + 𝑉} 𝑓 (𝑎)

since

𝑓 (𝐴)

is compact and

𝑓 (𝐴 + 𝑉)

is open.

□

C. Basics in convex analysis

Lemma C.1 (duality). Let

𝒳︀

be a locally convex topological vector space. If

𝑓 : 𝒳︀ \to (- \infty, \infty]

is convex and lower semicontinuous, then

𝑓^{* *} (𝑥) = 𝑓 (𝑥)

Proof. We should assume, without loss of generality, that $𝑓$ is not identically $\infty$ , for the lemma holds obviously otherwise. Define

\begin{matrix} ℰ︀ = {(𝑥, 𝛼) \in 𝒳︀ \times ℝ : 𝑓 (𝑥) \leq 𝛼}, \\ {ℰ︀}^{*} = {(𝜆, 𝛼) \in {𝒳︀}^{*} \times ℝ : 𝑓^{*} (𝜆) \leq 𝛼}, \end{matrix}

which are convex subsets in the locally convex topological vector spaces $𝒳︀ \times ℝ$ and ${𝒳︀}^{*} \times ℝ$ , respectively.

It is not hard to verify by definition that $𝑓^{* *} \leq 𝑓$ . Hence, it suffices to prove the other inequality. Equivalently, it asserts that if $𝑓 (𝑥) > 𝛼$ , then there is $(𝜆, 𝛽) \in {𝒳︀}^{*}$ such that $⟨ 𝜆, 𝑥 ⟩ - 𝛽 > 𝛼$ . Then, observe that $𝐴 = {(𝑥, 𝛼)}$ is compact and $𝐵 = ℰ︀$ is closed by lower semicontinuity of $𝑓$ and nonempty by the fact $𝑓$ is not identically $\infty$ . Moreover, under the assumption $𝑓 (𝑥) > 𝛼$ , $𝐴 \cap 𝐵 = \emptyset$ . These altogether allow us to apply the Hahn-Banach separation theorem (specifically, Theorem B.3) to yield some $𝜂 \in {𝒳︀}^{*}$ and $𝜌 \in ℝ$ satisfying

𝛾 ≔ sup_{(𝑦, 𝛽) \in ℰ︀} ⟨ 𝜂, 𝑦 ⟩ - 𝜌 𝛽 < ⟨ 𝜂, 𝑥 ⟩ - 𝜌 𝛼

where $𝜌 \geq 0$ must hold as there exists $𝑦 \in 𝒳︀$ with $𝑓 (𝑦) < \infty$ .

If $𝜌 > 0$ , then (C.1) implies that $𝛼 < ⟨ 𝜌^{- 1} 𝜂, 𝑥 ⟩ - 𝜌^{- 1} 𝛾$ . Moreover, $(𝜌^{- 1} 𝜂, 𝜌^{- 1} 𝛾) \in {ℰ︀}^{*}$ as desired, for if otherwise, $𝑓^{*} (𝜌^{- 1} 𝜂) > 𝜌^{- 1} 𝛾$ contradicting the definition of $𝛾$ .

If $𝜌 = 0$ , then for all $(𝜇, 𝛽) \in {ℰ︀}^{*}$ , define $(𝜇_{𝛿}, 𝛽_{𝛿}) ≔ (\frac{𝜂}{𝛿} + 𝜇, \frac{𝛾}{𝛿} + 𝛽)$ for $𝛿 > 0$ . In fact, $(𝜇_{𝛿}, 𝛽_{𝛿}) \in {ℰ︀}^{*}$ since

\begin{aligned} 𝑓^{*} (𝜇_{𝛿}) - 𝛽_{𝛿} \leq sup_{𝑦 \in 𝒳︀} [⟨ 𝜇_{𝛿}, 𝑦 ⟩ - 𝑓 (𝑦) - 𝛽_{𝛿}] \\ \leq & sup_{𝑦 \in 𝒳︀} [\frac{1}{𝛿} (⟨ 𝜂, 𝑦 ⟩ - 𝛾) + (⟨ 𝜇, 𝑦 ⟩ - 𝑓^{*} (𝜇)) - 𝑓 (𝑦)] \leq 0 . \end{aligned}

Moreover,

[⟨ 𝜂_{𝛿}, 𝑥 ⟩ - 𝑓^{*} (𝜂_{𝛿})] \geq lim_{𝛿 \to 0} [\frac{1}{𝛿} (⟨ 𝜂, 𝑥 ⟩ - 𝛾) + (⟨ 𝜇, 𝑥 ⟩ - 𝑓^{*} (𝜇))] = \infty,

yielding the desired $𝜆 = 𝜂_{𝛿}$ when $𝛿$ is sufficiently small. $□$

Lemma C.2. Suppose

𝑓 : ℝ^{𝑑} \to [0, \infty]

is a convex, lower semicontinuous function with

{inf}_{𝜆 \in ℝ^{𝑑}} 𝑓 (𝜆) = 0

and

0 \in ri {𝒟︀}_{𝑓^{*}}

, then

𝑓 (𝜂) = 0

for some

𝜂 \in ℝ^{𝑑}

Proof. Note that $𝑓^{*} (0) = 0$ , and hence the lemma, by duality (Lemma C.1), is equivalent to the existence of $𝜂 \in ℝ^{𝑑}$ such that $⟨ 𝜂, 𝑥 ⟩ \leq 𝑓^{*} (𝑥)$ for all $𝑥 \in ℝ^{𝑑}$ . This latter claim, by convexity of $𝑓^{*}$ , is equivalent to

⟨ 𝜂, 𝑥 ⟩ \leq lim_{𝛿 \to 0 +} \frac{𝑓^{*} (𝛿 𝑥) - 𝑓^{*} (0)}{𝛿} = inf_{𝛿 > 0} \frac{𝑓^{*} (𝛿 𝑥) - 𝑓^{*} (0)}{𝛿} ≕ 𝑔 (𝑥) for all 𝑥 \in ℝ^{𝑑} .

To prove the claim, define

𝐴 = \overset{─}{{(𝑥, 𝛼) \in ℝ^{𝑑} \times ℝ : 𝑔 (𝑥) \leq 𝛼}} and 𝐵 = {(0, - 1)},

Obviously, $𝐵$ is convex, compact, and nonempty. On the other hand, $𝐴$ is nonempty, convex, and $𝐴 \cap 𝐵 = \emptyset$ . Obviously, $(0, 0) \in 𝐴$ . To prove convexity, note that $𝑔$ is a convex function and closure of a convex set is convex. Indeed,

\begin{aligned} 𝑔 (𝑡 𝑥 + (1 - 𝑡) 𝑦) = lim_{𝛿 \to 0 +} \frac{𝑓^{*} (𝑡 𝛿 𝑥 + (1 - 𝑡) 𝛿 𝑦)}{𝛿} \\ \leq & lim_{𝛿 \to 0 +} 𝑡 \cdot \frac{𝑓^{*} (𝛿 𝑥)}{𝛿} + (1 - 𝑡) \cdot \frac{𝑓^{*} (𝛿 𝑦)}{𝛿} = 𝑡 𝑔 (𝑥) + (1 - 𝑡) 𝑔 (𝑦) . \end{aligned}

Finally, to prove $𝐴 \cap 𝐵 = \emptyset$ , we first show that $0 \in ri {𝒟︀}_{𝑔} = {𝒟︀}_{𝑔}$ . Under the circumstances, we deduce that $𝑔 |_{{𝒟︀}_{𝑔}}$ is continuous at $0 \in ri {𝒟︀}_{𝑔} = {𝒟︀}_{𝑔}$ and thus $𝐴 \cap 𝐵 = \emptyset$ follows naturally. To this end, note that $0 \in ri {𝒟︀}_{𝑓^{*}}$ implies $0 \in ri {𝒟︀}_{𝑔}$ . Indeed, $𝑔 (0) = 0$ and if $𝑥 \in {𝒟︀}_{𝑔}$ , then $𝑓^{*} (𝑥) < \infty$ and for all small $𝜀 > 0$ ,

𝑔 (- 𝜀 𝑥) = inf_{𝛿 > 0} \frac{𝑓^{*} (𝛿 (- 𝜀) 𝑥)}{𝛿} \leq 𝑓^{*} (- 𝜀 𝑥) < \infty .

In particular, the above implies $0 \in ri {𝒟︀}_{𝑔}$ . Moreover, $0 \in ri {𝒟︀}_{𝑔}$ implies also $ri {𝒟︀}_{𝑔} = {𝒟︀}_{𝑔}$ since $𝑥 \in {𝒟︀}_{𝑔}$ if and only if $𝑡 𝑥 \in {𝒟︀}_{𝑔}$ for all $𝑡 > 0$ .

We now may apply Theorem B.3 to $𝐴$ and $𝐵$ to yield some $𝜇 \in 𝒳︀$ such that

⟨ 𝜇, 0 ⟩ + 𝜌 = 𝜌 > sup_{(𝑥, 𝛼) \in 𝐴} ⟨ 𝜇, 𝑥 ⟩ - 𝜌 𝛼,

where $𝜌 > 0$ since $(0, 0) \in 𝐴$ . Hence, for every $𝑥 \in 𝒳︀$ with $𝑓^{*} (𝑥) < \infty$ , we have that

⟨ 𝜌^{- 1} 𝜇, 𝜀 𝑥 ⟩ \leq 𝑔 (𝜀 𝑥) + 1 for all 𝜀 > 0 \Rightarrow ⟨ 𝜌^{- 1} 𝜇, 𝑥 ⟩ \leq 𝑔 (𝑥) .

The proof is concluded by choosing $𝜂 = 𝜌^{- 1} 𝜇$ . $□$

Lemma C.3. Let

𝑓

be an essentially smooth, convex function. If

𝑓 (0) = 0

and

𝑓^{*} (𝑥) = 0

for some

𝑥 \in ℝ^{𝑑}

, then

0 \in \overset{˚}{{𝒟︀}_{𝑓}}

Proof. Since $𝑓 (0) = 0$ , it follows by convexity of $𝑓$ that

𝑓 (𝑡 𝜆) \leq 𝑡 𝑓 (𝜆), for all 𝑡 \in [0, 1], 𝜆 \in ℝ^{𝑑} .

Moreover, since $𝑓^{*} (𝑥) = 0$ ,

𝑓 (𝑡 𝜆) \geq ⟨ 𝑡 𝜆, 𝑥 ⟩ - 𝑓^{*} (𝑥) \geq - 𝑡 | 𝜆 | | 𝑥 | .

Because $𝑓$ is essentially smooth, there exists a closed ball ${\overset{─}{𝐵}}_{𝑟} (𝑧) \subset \overset{˚}{{𝒟︀}_{𝑓}}$ in which $𝑓$ is differentiable. Hence,

𝑀 = sup_{𝜆 \in {\overset{─}{𝐵}}_{𝑟} (𝑧)} {𝑓 (𝜆) \lor | 𝜆 | | 𝑥 |} < \infty

Hence, for any $𝑡 \in (0, 1]$ and any $𝜃 \in {\overset{─}{𝐵}}_{𝑡 𝑟} (𝑡 𝑧)$ different from $𝑡 𝑧$ , by the convexity of $𝑓$ ,

𝑓 (𝜃) - 𝑓 (𝑡 𝑧) \leq \frac{| 𝜃 - 𝑡 𝑧 |}{𝑡 𝑟} (𝑓 (𝑦) - 𝑓 (𝑡 𝑧)) \leq \frac{2 𝑡 𝑀}{𝑡 𝑟} | 𝜃 - 𝑡 𝑧 |,

where $𝑦 = 𝑡 𝑧 + \frac{𝑡 𝑟}{| 𝜃 - 𝑡 𝑧 |} (𝜃 - 𝑡 𝑧) \in {\overset{─}{𝐵}}_{𝑡 𝑟} (𝑡 𝑧)$ . Similarly, $𝑓 (𝜃) - 𝑓 (𝑡 𝑧) \geq - \frac{2 𝑡 𝑀}{𝑡 𝑟} | 𝜃 - 𝑡 𝑧 |$ . Hence,

| 𝑓 (𝜃) - 𝑓 (𝑡 𝑧) | \leq \frac{2 𝑡 𝑀}{𝑡 𝑟} | 𝜃 - 𝑡 𝑧 | for all 𝜃 \in {\overset{─}{𝐵}}_{𝑡 𝑟} (𝑡 𝑧) .

Observe that

𝑡 𝑧 \in \overset{˚}{{𝒟︀}_{𝑓}}

because of the convexity of

{𝒟︀}_{𝑓}

. Hence, by assumption,

\nabla 𝑓 (𝑡 𝑧)

exists, and by the preceding inequality,

| \nabla 𝑓 (𝑡 𝑧) | \leq 2 \frac{𝑀}{𝑟}

. Since

𝑓

is steep, by considering

𝑡 \to 0

, in which case

𝑡 𝑧 \to 0

, we conclude that

0 \in \overset{˚}{{𝒟︀}_{𝑓}}

□

1. Introduction
1.1. Overview
1.2. General settings
2. Cramér’s theorem in $ℝ$
3. Gärtner-Ellis theorem.
4. Applications
4.1. Cramér’s theorem in $ℝ^{𝑑}$
4.2. Large deviations for finite state Markov chains
A. Probability theory
A.1. Basic inequalities
A.2. Radon-Nikodym theorem
A.3. Laws of large numbers
B. Functional analysis
C. Basics in convex analysis
References

References

[1] Dembo, Amir, and Zeitouni, Ofer, Large Deviations Techniques and Applications, Springer Berlin Heidelberg, 2010.↩

An introduction to large deviation theory

1. Introduction

1.1. Overview

1.2. General settings

2. Cramér’s theorem in ℝ

3. Gärtner-Ellis theorem.

4. Applications

4.1. Cramér’s theorem in ℝ𝑑

4.2. Large deviations for finite state Markov chains

A. Probability theory

A.1. Basic inequalities

A.2. Radon-Nikodym theorem

A.3. Laws of large numbers

B. Functional analysis

C. Basics in convex analysis

References

2. Cramér’s theorem in $ℝ$

4.1. Cramér’s theorem in $ℝ^{𝑑}$