[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-the-learning-rate-shortcut-for-llm-training-has-a-flaw":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},2662,"the-learning-rate-shortcut-for-llm-training-has-a-flaw","The Learning Rate Shortcut for LLM Training Has a Flaw","Researchers found that a common trick for estimating optimal learning rates breaks down at scale, requiring a smarter approach.","A standard assumption used to cut LLM training costs turns out to be wrong at larger scales.\n\nTraining a large language model requires tuning the learning rate — the step size the optimizer takes when adjusting model weights. To avoid running expensive sweeps at full scale, researchers and practitioners extrapolate from smaller runs, assuming the optimal learning rate follows a predictable log-linear pattern as model size and data volume grow. New research from arXiv challenges that assumption directly. Experiments on GPT-2-style models ranging from 22 million to 707 million parameters, trained on 5 billion to 100 billion tokens, show that the optimal learning rate curves upward at larger scales — making straight-line extrapolation inaccurate.\n\nThe finding matters because learning rate transfer is a real cost-saving tool in an industry where training runs cost millions of dollars. If the scaling law is wrong, practitioners may be starting full-scale training with a miscalibrated optimizer — wasting compute before a single useful token is produced. The researchers identify a fix: substitute \"effective learning rate\" (the step size in normalized weight space) for the raw learning rate, and extrapolate from data scale rather than model size.\n\nThe paper also offers a mechanistic explanation: when the optimal learning rate is small, weight norms take longer to reach equilibrium, and a larger effective step size is needed to clear that transient phase — which is what introduces the curvature. Tests with the AdamH optimizer, which controls effective learning rate directly, back this up. It is a narrow but consequential correction to a shortcut the field has quietly relied on.","[\"ai\",\"machine-learning\",\"llm\",\"training\"]","2026-06-30T04:00:00.000Z","2026-06-30T10:27:51.673Z","2026-06-30T10:27:54.628Z","published",null,[],"ai",[24,26,27,28],"machine-learning","llm","training",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.29158",0,{"sections":35},[36,40,45,50,55,60,65,70,75,80,85,89,94,99],{"name":37,"slug":24,"count":38,"latest_published_at":39},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":41,"slug":42,"count":43,"latest_published_at":44},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":46,"slug":47,"count":48,"latest_published_at":49},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":79},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":81,"slug":82,"count":83,"latest_published_at":84},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":86,"slug":87,"count":83,"latest_published_at":88},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":100,"slug":101,"count":102,"latest_published_at":103},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]