Feature learning and neural scaling laws
In a controlled model family, moving out of the lazy regime improves the observed scaling exponent. Depth strengthens the effect, while a fully connected control stays flat.
The central result is simple: in diagonal linear networks, stronger feature learning can produce a better data-scaling exponent than the lazy, kernel-like regime. The effect is largest in an intermediate range and becomes more pronounced with depth.
- The lazy-regime exponent follows ξ = min(b - 1, 2a) in this setup.
- Moving to lower initialization scale improves the exponent most around the middle of the tested feature-learning range.
- The gain is roughly +0.30 at a = 1.5 and +0.15 at a = 2.0, then vanishes by a = 2.5.
- Increasing depth from 2 to 4 roughly doubles the gain at a = 1.5.
- A fully connected control does not show the same improvement.
The question
Neural scaling laws describe how loss changes as data, compute, or model size grows. The familiar empirical curve is useful, but it does not explain which learning mechanism sets the exponent. We wanted a setting small enough to analyze and controlled enough to isolate feature learning.
Diagonal linear networks provide that setting. They retain a meaningful lazy-to-rich transition without the many confounds of a large nonlinear model. In the lazy regime, the learned representation barely moves. In the richer regime, the model adapts features as it learns.
Here ξ is the measured data-scaling exponent and the study varies the task and initialization parameters represented by a and b. The formula above describes the lazy baseline used for comparison.
What we changed
We trained diagonal linear networks across data scales, initialization scales, and depths. For each configuration, we fit the loss curve and compared its exponent with the lazy prediction. The important comparison is not a single low loss value. It is the slope: how quickly the system improves as more data becomes available.
| Condition | Observed change from lazy baseline | Reading |
|---|---|---|
| a = 1.5 | About +0.30 | Largest tested gain |
| a = 2.0 | About +0.15 | Smaller but visible gain |
| a = 2.5 | No clear gain | Effect has disappeared |
| Fully connected control | Flat | No corresponding rich-regime improvement |
The non-monotonic shape matters. Richer is not automatically better without limit. The strongest change appears between the extremes, where features move enough to alter the learning curve without making optimization dominate the result.
Depth amplifies the mechanism
At a = 1.5, increasing network depth from 2 to 4 produces about twice the exponent gain. This supports a mechanistic reading: depth can strengthen the feature-learning contribution instead of merely adding parameters to the same lazy process.
The fully connected control is equally important. If the gain came from our fitting procedure or from a generic training artifact, we would expect it to appear there too. It does not. That does not prove a universal law, but it narrows the plausible explanation inside this controlled family.
Where the result stops
This is a finite-dimensional, linear model family. Deep, rich configurations can be harder to optimize, and optimization failures can contaminate the exponent estimate. The rich-regime exponent is an empirical observation in the present experiments, not a closed-form theorem.
The next phase is to separate optimization limits from the underlying scaling behavior, repeat the measurement across wider dimensional ranges, and test which parts survive in nonlinear systems. A useful result should become more precise as it travels outward, not more dramatic.
That is also why this page is dated. The measurements above describe the current Phase 2 draft. We will revise the record when the evidence changes.