Learn2Splat: Extending the Horizon of
Learned 3DGS Optimization

A meta-learned optimizer for 3D Gaussian Splatting that reaches higher quality than Adam at equal wall-clock time and keeps improving over long optimization horizons, with one model for both sparse- and dense-view settings.

Naama Pearl*1 Stefano Esposito*1 Haofei Xu1,3 Amit Peleg1 Patricia Gschossmann1
Lorenzo Porzi2 Peter Kontschieder2 Gerard Pons-Moll1 Andreas Geiger1
1University of Tübingen, Tübingen AI Center 2Meta Reality Labs 3ETH Zurich
Code (Will be released) Paper arXiv
Setting
Dataset
Scene
Iterations (log)
t = 0

A single Learn2Splat checkpoint serves both settings. We compare it to Adam (standard 3DGS optimization),
G3R⋆ (our implementation of the G3R meta-training scheme) and, in the sparse setting, ReSplat.
Ctrl + scroll on an image to zoom in
Reference
Legend

Learn2Splat is a learned optimizer for 3DGS that stays above Adam throughout optimization and keeps improving over long horizons, where prior learned optimizers degrade. Prior learned optimizers rely on LR schedules or time encodings tied to a predefined iteration count. Learn2Splat maintains performance via a meta-training scheme (checkpoint buffer and optimizer rollout) and architectural modifications. A single model, meta-trained jointly on sparse-view (ReSplat init) and dense-view (SfM init) low-resolution DL3DV scenes, generalizes zero-shot to other datasets, resolutions and view counts. Each cell shows one target view at iteration t, labelled with that method's PSNR on the scene; the chart shows the PSNR averaged over all scenes of the dataset.

Abstract
3D Gaussian Splatting (3DGS) optimization is most commonly performed using general-purpose first-order optimizers such as Adam or SGD. Although robust across scenes, they update each parameter independently without exploiting the structural and spatial relationships among Gaussians, which slows convergence. Recent works introduced learned optimizers that predict correlated updates informed by inter-parameter and inter-Gaussian dependencies. However, those are trained for a fixed number of optimization iterations and rely on manually scheduled learning rates to avoid degradation.

In this paper, we introduce Learn2Splat, a learned optimizer for 3DGS that avoids degradation over extended optimization horizons without auxiliary mechanisms. To enable this, we propose a meta-learning scheme that extends the optimization horizon via a checkpoint buffer and an optimizer rollout strategy, combined with an architecture that encodes gradient scale information in its latent states. Results show higher novel view synthesis quality at equal wall-clock time, while remaining stable over long horizons, with zero-shot generalization to unseen datasets and settings. To support our findings, we build a unified framework to train and evaluate learned and standard optimizers across sparse and dense view settings. Code and models will be released publicly.
Method

Meta-training scheme & architecture

Learn2Splat replaces Adam with a meta-learned network predicting per-Gaussian updates from Adam-normalized gradients and maintained latent states. The core architecture module is a kNN-based Point Transformer that captures spatial Gaussian relationships.

3DGS optimization paradigms
3DGS Optimization Paradigms.
  • Per-scene optimization: iterative updates via loss evaluation, backpropagation, and standard optimizer rules.
  • Feed-forward networks (FFN): scene representation predicted in a single forward pass with a frozen pre-trained model.
  • Learned optimizers: iterative updates using a frozen meta-trained model that predicts steps from signals such as image-space errors or loss gradients.
Learn2Splat meta-training and architecture
Meta-training and Architecture.
  • Meta iteration initialization. At each meta-iteration, a 3D scene is sampled and its Gaussians are initialized either from (1) SfM points or feed-forward (FFN) predictions at t = 0, or (2) an intermediate optimization state retrieved from a Checkpoint Buffer. The buffer stores partially optimized scenes together with their optimizer states, exposing the learned optimizer to both early- and late-stage optimization regimes. After the meta-iteration, the scene is further optimized using several frozen rollout steps before being pushed back into the buffer.
  • Meta iteration. Starting from the sampled scene state, the inner loop rolls out the learned optimizer for τ iterations. At each step, gradients of a reconstruction loss with respect to the Gaussian parameters are computed and passed to the learned optimizer to predict parameter updates. The outer meta-loop evaluates the updates quality and updates the learned optimizer parameters through meta-gradients.
  • Model architecture. The optimizer takes as input the current Gaussian parameters, Adam-normalized gradients, and per-Gaussian latent states. A kNN-based Point Transformer propagates information across neighboring Gaussians and predicts updated latent states, while a parallel State Scale MLP predicts state-scaling coefficients that preserve gradient magnitude information. The scaled latent states are then passed to an Update MLP, which predicts the final Gaussian parameter updates. Dashed lines denote concatenation and ⊙ indicates element-wise scaling.
Checkpoint Buffer

Diverse optimization states

Stores intermediate scene states from previous meta-iterations. Exposes the optimizer to states from large early gradients through fine late-stage refinements without extending the computational graph.

Optimizer Rollout

Learning from rollouts

Scenes are further optimized with a frozen snapshot before buffering. Rollout horizon grows from 1→50 steps over the first 10k meta-iterations, teaching the optimizer to recover from its own mistakes.

State Scale MLP

Gradient scale encoding

Predicts per-Gaussian scaling coefficients from Adam-normalized gradients, restoring magnitude information suppressed by transformer normalization so updates decay naturally as the loss decreases.

Zero-shot Generalization

Unseen datasets & resolutions

Trained on low-resolution DL3DV scenes only, a single Learn2Splat model generalizes zero-shot to new datasets, resolutions and view counts: in the sparse setting to high-resolution and 32-view DL3DV and to RealEstate10K, and in the dense setting to DTU, LLFF and Mip-NeRF360.

One Model, Two Training Configurations

A single model is meta-trained jointly on two configurations of DL3DV, alternating between them across meta-iterations. Both use low-resolution images (256×448), with 8 context views at each inner iteration and a fixed set of 6 target views to evaluate the update; training runs for 150k meta-iterations on 4 NVIDIA A100 GPUs (about 3 days). We compare it to dedicated per-configuration checkpoints in the analysis.

Sparse configuration

ReSplat initialization

A sparse-view, forward-facing setup with ReSplat feed-forward initialization. The same 8 context views, taken from a short window of the scene video, are used at every inner step, and latent states are initialized from the feed-forward features.

  • Init: ReSplat FFN (57K primitives at train res.)
  • Views: 8 fixed context + 6 target views
  • Resolution: 256×448
  • Zero-shot to: RealEstate10K, higher resolutions
Dense configuration

SfM initialization

A dense-view, large-baseline setup with SfM point cloud initialization and random latent states. Each scene draws a pool of 64 views by farthest-point sampling, and every inner step samples 8 context views from it. Augmented by keeping 10–100% of the SfM points, with COLMAP's black points removed.

  • Init: SfM points (random latent states)
  • Views: 8 sampled via furthest-point + 6 target views
  • Resolution: 256×448
  • Zero-shot to: DTU, LLFF, Mip-NeRF360, higher resolutions
Results

Higher quality at equal wall-clock time

A single Learn2Splat checkpoint is evaluated on eight benchmarks: four dense-view datasets from an SfM initialization and four sparse-view configurations from a ReSplat initialization. Every optimizer gets the same runtime, Adam's wall-clock time for 4000 iterations, and runs as many of its own iterations as fit within it. All optimizers use spherical harmonics of degree 3 from the start, and adaptive density control is disabled to compare them at a fixed capacity.

Optimization trajectories on Mip-NeRF360

Legend: L2S, G3R*, Adam, Adam-tuned, Adam peak PSNR vs. wall-clock time and vs. iteration on Mip-NeRF360
Learn2Splat stays above Adam and remains stable over long horizons with higher final PSNR (Mip-NeRF360, dense views). Left: PSNR vs. wall-clock time, over the runtime of the table above. Right: PSNR vs. optimization iteration. Dots and labels mark where each run first reaches Adam's peak PSNR; the left panel lists each method's best PSNR. Although each Learn2Splat iteration is ~2× slower than an Adam step, it reaches every quality level of Adam in less wall-clock time.
Dense views · SfM init

DL3DV, DTU, LLFF, Mip-NeRF360

On 140 DL3DV scenes (in-domain) and 15 DTU, 8 LLFF and 9 Mip-NeRF360 scenes (zero-shot), Learn2Splat improves over Adam by 0.1–1.7 dB within the fixed runtime. G3R⋆ peaks within 100 steps, as it is tuned for early performance, then degrades, ending 1.8–3.2 dB below Learn2Splat and diverging on DTU.

Sparse views · ReSplat init

DL3DV (8 / 32 views), RealEstate10K

On 140 low- and high-resolution DL3DV scenes and 72 RealEstate10K scenes, Learn2Splat stays above Adam throughout and improves by close to 0.5 dB, while Adam drops 0.1–0.3 dB below its own peak. G3R⋆ ends 2.6–3.5 dB below on DL3DV and diverges on RealEstate10K; ReSplat improves only for the first ~10 iterations.

More Results

Second-order optimization and densification

Analysis
Video

Optimization videos

Learn2Splat reaches higher PSNR early and keeps improving to the end, while G3R⋆ and ReSplat degrade.

Interactive teaser

Drag the split lines to compare the methods in one view; click to freeze them.

*ReSplat is absent from the dense scenes: it relies on pixel-aligned Gaussians from the input views, so it cannot start from SfM points.

Citation

BibTeX

If you find this work useful, please cite:

@article{pearl2026learn2splat,
  title   = {Learn2Splat: Extending the Horizon of Learned 3DGS Optimization},
  author  = {Pearl, Naama and Esposito, Stefano and Xu, Haofei and Peleg, Amit and Gschossmann, Patricia and
             Porzi, Lorenzo and Kontschieder, Peter and Pons-Moll, Gerard and Geiger, Andreas},
  journal = {arXiv preprint arXiv:2605.15760},
  year    = {2026}
}