Foreword: why these exist
Narrative
These are a collection of artifacts from my long (sparse effort over years) journey to understand muP, which, as I learned more about it, is not a single solved problem, but a whole field of research.
It begins with "A Spectral Condition for Feature Learning" which was the only way, other than the EleutherAI blog post, I was going to have a chance of understanding any of this in a reasonable timeframe (I, probably like most people, was not going to go read the tensor programs papers, and if I'd known the final level of knowledge I'd be left with, I wouldn't have even considered opening the first and reading into it).
I found the blog significantly lacking in rigor, which has at least in the past been how I approach math (I have always come at math from deep understanding and find a lack of ability to trace the full set of assumptions, arguments, and step-by-step algebra difficult to handle).
The paper is significantly nicer and more rigorous, but I actually run into the same issue because it leaves a lot of steps up to you and I was constantly asking how we were going to get from these arguments to realistic training, and it doesn't really (tbf, no math you read in papers does, and it is a skill issue to some extent, see there's more to math than rigor and proofs and compilation errors in mathematical reading and how to resolve them, and to some extent an expected level of familiarity, authors not wanting to restate known math in their paper, and not having space even to show steps).
In the paper, there are multiple gaps left open, and handwavy arguments such as non-cancellation that I wished had formal and well-illustrated exposition.
As I went through it on alphaXiv, I left annotations so that as I came back to this across months, I could pick up my understanding where I left off. These annotations are the most written by me things here, though I let Astra make some edits before uploading them. Past this artifact, we get into entirely LM generated guides, though they come from significant discussion with me.
It took me roughly a year to feel I had digested this paper, over the course of maybe 1-2 weeks in actual hours. Partway through and still afterwards, I was feeling let down in how far you could take this, I hadn't known that this paper wasn't going to prove all of hyperparameter transfer to me (or even explain how to get to Adam instead of SGD, making it, as far as I could verify, entirely inapplicable cause who uses SGD).
So I wanted to extend the argument to AdamW without relying on tensor programs or math that I was very unfamiliar with. My ability to verify LM generated math was insufficient, as exposed by later LMs auditing the initial versions, and given I have not even had these subjected to Lean, the current version has a reasonable chance to be wrong as well, though it is significantly more convincing at least.
The argument for this is actually quite nice to follow, I think mainly by leaning on significant assumptions, but ones I can understand and can see are measurable.
Believing at least partially that you could do hyperparameter transfer across width, I wanted to also work through Complete(d) P, which claimed to extend hyperparameter transfer to the most important other axes of training (depth, batch, duration, using AdamW) (though even this, for instance, would need adjustments to apply to MoE models, or mHC, ... every time you do something new, you have to revisit your math, oof).
But I was missing a lot of background needed to understand it, e.g. wtf is Brownian motion actually. So I had discussions with Codex to build a guide and to explain background I was missing in SDE and Brownian motion. I made it through these and came out the other end with a pretty solid understanding and set of mental visuals/maps to intuit the ideas, which I enjoy a bit.
I intended to eventually clean these up and maybe extend them to be more complete coverage of the paper, but I have been way too busy (single-handedly pulling off the training of 30B long-context VLMs, writing their papers, and developing what has become the most efficient LoRA RL training infra in open source), and at this point my thoughts on math have pivoted significantly (I'll explain more, but I see no reason for me or those with similar goals to understand this math any more).
Reconsidering and looking forward
But as I was going past this toward my underlying goal of doing some scaling law experiments for low rank models (i.e. just a LoRA in place of each full weight matrix) (to do scaling law research, you need to minimize the noise caused by poor hyperparameters, because you can't tune at every scale), I, with LM assistance, decided the theory for hyperparam transfer did not apply to models of this shape, and so tried to develop it. This immediately went past me at a pace I could not match and as I reviewed the questions Codex posed to me and its explanations of its steps, I took a step back to analyze my frustrations with math and rethink my approach:
- It typically feels like a black hole of time, because what do I come out with afterwards: I am only naturally a tiny bit better at reading math; I don't do it enough to gain much familiarity in the next thing I might be interested in and it is hard to come back to it and trust or understand anything again due to spending time only occasionally on math.
- I spend it to gain some measure of confidence in its correctness which has been in the past overly high; I can easily miss subtle errors and those can invalidate everything. This is to me the same as trying to debug a complex program without running it. And in general, I don't think anything which isn't thoroughly measured/debugged against reality has significantly greater probability than 0 of being correct.
- I don't have much predictive power, during or afterwards, which I'd say would come from genuinely deep understanding and would require significantly more time and effort in a given area than I have available. So, I can't really do much with math afterwards. Thus, I'm there only to choose whether I believe it and to try to grasp some measure of how someone argued it.
Some of the thoughts I came away with
Verifying step-level math is not worthwhile: I have non-significant chances of catching errors and I will not remember, be able to quickly recreate, or trust my own verification for a significant period of time (despite it taking most of the time in my reading), and I seem to have acquired only marginal gains in ability to do this verification over time meaning I don't expect any significant acceleration in future reading.
The most cogent thing I ever come away with is visual maps/intuitions. These are a bit durable across time (entirely unlike verification of algebraic steps), give me some level of comfort in understanding how something was argued and more predictive power than any step-level verification.
However, in general I categorize the uses for math in two buckets: art/appreciation and real application. My utility is not in the first, I appreciate the art of RSI and ASI far more than clever mathematical ideas (thus math is not where I want to spend my time) and the second bucket is a capability which AI is far superior in (I don't believe much human lead exists at this point or will in the future, given the fast verification loop that math has).
If you fall into the second category, why bother with it at all? Generalizing from my experience and experimentation with heavily AI-driven development, where my understanding of training infra has been important to getting as far and as fast as I have with ART, my capabilities impact the final outcomes strongly. At least with ~GPT 5.2-5.6, though I believe this is still the case with Astra, Opus 5.5 and Fable 5.2.
If the same is true for math, and it does appear to be (e.g. https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed56) for now, you will probably have some edge in capabilities if you understand math very well (also see Sébastien Bubeck/the team behind OpenAI's Navier-Stokes, I suspect at least their general understanding of how to do effective mathematical research contributed to the result when compared to someone with no ideas other than "solve navier-stokes"). So maybe there is some benefit still to my understanding and capabilities in math.
Again, however, getting optimal outcomes does seem significantly easier now than with GPT 5.2. When agents are the ones who will do everything and their ideas are so often good, your aim is most often to get them to thoroughly consider everything, and not forget the goals. I think if they have a good understanding of the goals and you can keep them in context as pressures throughout an analysis, agents can do a pretty solid job of designing solutions (~90% of the time). Thus, it is more about asking the right questions, or just knowing that you need to ask some questions (and getting those questions from an LM) because the LM hasn't thought through everything yet, or simply inducing the right long-term pressures such that what you get is likely to be what you want (e.g. build e2e performance tests -> you get fast code, the rest of your time is just about getting fast code sooner).
I'd say atm this points to understanding agent behavior being the most powerful current tool, and even that we'd like to go away, since we're aiming for pretty much just having to explain the goals. As we advance ASI, I am unsure of any particular need to understand things deeply.
And thus while the first idea would likely have been useful years earlier, I have no drive to study math anymore. I feel I am surpassed and we are on track to all be far surpassed here. The same is probably true of coding.
As a measurable outcome, people will be capable of switching technical professions of increasingly wide gaps (e.g. me -> mathematics research or mathematician -> LLM inference). This is awesome, we are getting much closer to only being limited in what we can accomplish by our resources and the goals we aim for, and many technical barriers you would've previously had to choose to devote years of time to (e.g. writing kernels) are practically gone. I place high probabilities on technical studies being ephemeral in utility.
Hopefully we can extend our own intelligence via these same systems though (I mean BCI), which would give us a chance to follow, contribute and to properly appreciate capabilities and achievements as we go forwards.
Again, I have not taken the time to clean any of these up, and I see little value in working through them, except as an intellectual challenge for people around the same mathematical level as me. But I figured maybe they'd be interesting and maybe you'd enjoy my thoughts on math and my take that these are relics (at least for my utility).
NN Parameterization
I take the practical motivations behind parameterization research to be the following: in training, we have a set of choices to make,
c = (compute, width, depth, batch, tokens, optimizer family, data, algorithms, learning rate, ...)
and a performance function J(c) that costs compute to evaluate, say a training run + evals. Write c = (x, u), where u is the subset of choices we want to determine and x contains the remaining choices. If we can solve for near-enough-optimal u(x) with less compute than we would spend evaluating J(x, u) at candidate values of u (~none to apply a typical parameterization rule such as muP), then we iterate faster, or say, we reallocate compute and research effort to the other parameters. Perhaps we do this with some math by studying learning dynamics, perhaps we fit models for them.
I think that wide of a lens makes parameterization only one technique for making the iteration loop of RSI faster/cheaper, and treating that as the research question is an exciting direction.