Gradient Accumulation
A training technique that aggregates gradients across multiple micro-batches before performing an optimization step.
Gradient Accumulation simulates large batch sizes on resource-limited hardware. It computes gradients for small micro-batches over multiple steps, summing them before updating model weights, maintaining training stability.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.