Brownian Motion and SDE Foundations

1. Why an SDE Appears

An optimizer is a discrete random recurrence: one parameter update after another. An SDE is a possible small-step, many-update approximation to that recurrence. It is not a law of nature that we assume at the beginning.

2. The Probability Objects

Formally, all randomness lives on a probability space (Omega, F, P). Read Omega as every possible sequence of random minibatches, shuffles, augmentations, and other draws. A random variable maps an outcome omega in Omega to a number or vector.

A stochastic process is a random function of time. Thus W_t abbreviates W(t, omega). Fixing omega gives one sample path; probability statements range over all such paths.

3. Independent Noise and Variance

The elementary result beneath Brownian motion is that independent centered fluctuations add in variance, not in standard deviation. If Z_1, ..., Z_n are independent and centered, then

Expand the square. Cross terms vanish because for i != j, E[Z_i Z_j] = E[Z_i] E[Z_j] = 0:

With unit-variance terms and unit coefficients, the sum has variance n and standard deviation sqrt(n). This is the precise sense in which centered independent noise accumulates like sqrt(n).

4. The Scaled Random Walk

Let xi_1, xi_2, ... be independent standardized random variables:

For a small time mesh h > 0, define the step process

The superscript (h) labels the approximation that takes one random kick every h units of time; it is not an exponent. The floor counts how many kicks have occurred by time t.

Why is a kick order sqrt(h)? There are about t / h kicks by time t. If each has standard deviation a(h), variance addition gives

A finite nonzero limit proportional to t requires a(h)^2 / h to approach a constant, hence a(h) must be proportional to sqrt(h).

The displayed process is a step function. Linearly interpolate between values at 0, h, 2h, ... to obtain continuous paths. Donsker's invariance principle says these interpolated paths converge in distribution, on each finite interval, to Brownian motion. This is convergence of probability laws on whole functions, not pointwise convergence of one chosen path.

5. Definition of Brownian Motion

It follows that

For s <= t, write W_t = W_s + (W_t - W_s). The later increment is independent of W_s, hence

In multiple dimensions, W_t is a vector of independent scalar Brownian motions, so Cov(W_t - W_s) = (t - s) I.

6. Quadratic Variation and Ito Integration

Quadratic variation

On a partition 0 = t_0 < ... < t_n = T, let Delta W_i = W_{t_{i+1}} - W_{t_i}. Then

This has a short mean-square proof. Since Delta W_i ~ N(0, Delta t_i),

The increments are independent, so the sum has expectation T and variance

Therefore it converges to T in mean square, hence in probability. The mnemonic (dW)^2 = dt refers to this limit; it is not literal infinitesimal arithmetic.

The Ito integral

For an adapted process H_t, define the stochastic integral as the limit of left-endpoint sums:

The left endpoint prevents H_{t_i} from looking at the fresh increment Delta W_i. It is the mathematical form of not peeking at a future minibatch before taking an update. Its key lemma is the Ito isometry:

For stepwise H, the proof is the same cross-term cancellation as the variance lemma. The isometry then defines the integral as an L^2 limit.

Why Ito's rule has an extra term

Taylor expansion has a quadratic term. For Brownian motion it survives after summing because squared increments accumulate to time:

For f(x) = x^2, this gives d(W_t^2) = 2 W_t dW_t + dt. Ordinary calculus would incorrectly omit the final term.

7. What an SDE Means

Let parameters be a vector theta_t in R^p. The notation

describes a short random increment. It is not an ordinary equation involving a derivative dW_t / dt. Start with the earlier definition of Brownian motion and build the increment explicitly.

From W to dW

Standard scalar Brownian motion satisfies W_(t+h) - W_t ~ N(0, h). A q-dimensional Brownian motion contains q independent copies, hence

This is the precise origin of the informal differential:

The first line is shorthand for the Gaussian-increment law, not equality between ordinary infinitesimals. Brownian motion is a random path; it has no finite ordinary velocity.

Why W has q noise coordinates, not p parameter coordinates

theta has p coordinates because the model has p trainable scalar degrees of freedom. W has q coordinates because it is a standardized coordinate system for independent random sources. It is not a second parameter vector and does not need to have the same dimension.

G(theta, t) has shape p x q. Its j-th column is the parameter-space direction and size of the j-th independent unit noise source:

The actual intrinsic noise object is the parameter covariance rate

D is positive semidefinite. Terminology varies: some texts call G the diffusion matrix, while others call D the diffusion matrix. Here G is the diffusion coefficient and D is the diffusion covariance.

The factorization is not unique. Rotating the Brownian coordinates and counter-rotating G leaves D unchanged. One can take q = p and a matrix square root of D, or q = rank(D) to omit redundant noise coordinates. Only D is a property of the parameter dynamics.

Derive the operational short-time meaning

Condition on F_t, the complete past. Then theta_t, and therefore b(theta_t, t) and G(theta_t, t), are known. The next Brownian increment is fresh:

Take conditional expectation. The fresh noise has mean zero:

Then compute covariance. Deterministic terms disappear, and a matrix transforms covariance on the left and right:

Replacing h by dt gives the operational statement used in the guide:

This is where drift and diffusion come from in an optimizer application: compute the conditional first and second moments of one update. The SDE does not supply them; it summarizes the matched limiting process.

A complete two-parameter example

Let theta = (theta_1, theta_2), choose two independent Brownian sources, and set

The noise part of the increment is then

The first parameter receives only source one. The second receives both sources. Thus the parameter noises are correlated even though dW_1 and dW_2 are independent:

The off-diagonal 2 comes from the shared source W_1. This is why a single scalar noise_scale is insufficient for a general optimizer: noise can have different sizes and correlations in different parameter directions.

Integral and Euler-Maruyama forms

The SDE notation means the integral equation

Its standard small-step simulation is exactly the short-time construction above:

The next section matches this form to minibatch SGD. It explains why the main guide asks which quantities make deterministic increments O(h) and random increments O(sqrt(h)).

8. Matching Minibatch SGD to an SDE

Let X_i(theta) be the gradient from one randomly selected training unit. At fixed parameters, define

Sigma(theta) is a p x p matrix: the covariance of one randomly drawn unit gradient around the population gradient. It is not the cross-covariance of two different units.

The old label xi_B meant “noise for a batch of size B,” not “the i-th summand.” This page now uses xi_batch to make that distinction explicit. Under conditionally iid sampling, Section 9 derives

Conditioning on F_t fixes the complete past and current parameters. The equality is exact for sampling with replacement under the stated definition of Sigma; the approximation used in practice recognizes that training units are often correlated.

Use the covariance in SGD

With gradient-flow time t_k = k * eta, one update has

Section 7 says that an SDE noise term G dW_t contributes covariance G G^T dt. We therefore need a matrix G whose product with its transpose is (eta / B) * Sigma. Take the positive-semidefinite matrix square root Sigma^(1/2), defined by Sigma^(1/2) Sigma^(1/2) = Sigma, and set

Then the SDE noise increment has exactly the required covariance:

The scalar square root appears because covariance is quadratic in the noise coefficient. The matrix square root chooses directions and correlations whose covariance is Sigma. Replacing the discrete state theta_k by its continuous approximation theta_t, the local-moment-matched SDE is therefore

A formal diffusion limit needs small updates, regular state dependence, and a martingale or weak-dependence central-limit argument. It is normally a local approximation, not an exact finite-LR identity.

9. What Batch Averaging Actually Establishes

High alignment does not contradict conditional independence

Write one example gradient as

Under conditional iid sampling, centered fluctuations from distinct draws have zero cross-covariance. But the raw gradients can still be highly aligned because they share a nonzero mean:

Thus high cosine similarity or a common descent direction is not evidence against the assumption. The 1 / B calculation concerns only fluctuations around the common mean.

The iid calculation

A “diagonal term” means i = j in this batch-index double sum, not a diagonal entry of Sigma. Each diagonal term is the full matrix

Conditional independence zeros the i != j terms. There are B diagonal terms:

What fails in practice, and what is load-bearing

The iid condition is sufficient, not the actual load-bearing claim. The batch rule needs aggregate batch-noise covariance to be approximately proportional to 1 / B over the transferred range. In general, for i != j, let C_ij = Cov(X_i, X_j | F_t). Then

If all within-unit covariances equal Sigma and all cross-covariances are one matrix C, then

Persistent positive cross-covariance prevents noise from falling as 1 / B. This arises, for example, when tokens from one document are treated as independent samples, packed or duplicated sequences are correlated, dropout/augmentation randomness is shared, or batches are deliberately non-iid. For a finite dataset of size N sampled uniformly without replacement,

That is close to Sigma / B only when B is much smaller than N. In general correlation can differ by parameter direction, so there need not be one scalar effective batch size.

This covariance law alone still does not derive AdamW's square-root rule; the main guide additionally matches the adaptive-moment recurrences and their SDE coefficients.

10. Limits of the Approximation

This is the bridge back to the main guide: use this page for definitions and lemmas; require the main page to show the optimizer-specific matching calculation before accepting a transfer rule.

← Return to the main guide's SDE section