[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-study-explains-why-adams-two-momentum-knobs-can-be-tied":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},6961,"study-explains-why-adams-two-momentum-knobs-can-be-tied","Study Explains Why Adam's Two Momentum Knobs Can Be Tied","Setting Adam's two momentum parameters equal removes a lag term that causes erratic updates, yielding smoother training across six tasks.","A new paper explains why Adam's two momentum settings work just as well when forced to match.\n\nResearchers studied Adam's two momentum parameters, beta1 and beta2, which control how long the optimizer remembers past gradients. Earlier work found that tying the two values together still preserves Adam's strong performance, even though it collapses two separate memory scales into one. The new analysis pins down why: in a continuous-time model of training, each update splits into a sign component plus a magnitude-lag term driven by mismatch between the two memory times, and that lag term vanishes exactly when the parameters are equal. Tests on real training gradients across six vision and language tasks confirmed it, showing tied updates dominated by sign rather than magnitude swings, with smoother update-size trajectories.\n\nAdam has powered most large-model training for nearly a decade, but practitioners have mostly treated the beta1\u002Fbeta2 split as folklore rather than an understood design choice. This work gives a concrete, testable reason why collapsing to one memory scale doesn't hurt performance, and it flags mismatched settings as a specific, quantifiable source of instability during training. That matters for anyone tuning optimizers on large runs, where every wasted restart costs real compute.\n\nIt won't change how you call your optimizer tomorrow, but it's a rare case of theory catching up with a trick engineers already used by habit.","[\"ai\",\"machine-learning\",\"optimization\",\"dev-tools\"]","2026-09-18T04:00:00.000Z","2026-09-19T00:38:13.413Z","2026-09-19T00:38:25.314Z","published",null,[],"ai",[24,26,27,28],"machine-learning","optimization","dev-tools",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2601.21739",0,{"sections":35},[36,39,43,48,53,57,61,66,69,74,79,84,89,94],{"name":37,"slug":24,"count":38,"latest_published_at":18},"AI",4082,{"name":40,"slug":41,"count":42,"latest_published_at":18},"Security","security",661,{"name":44,"slug":45,"count":46,"latest_published_at":47},"Policy","policy",339,"2026-09-17T12:00:00.000Z",{"name":49,"slug":50,"count":51,"latest_published_at":52},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":54,"slug":55,"count":56,"latest_published_at":18},"Hardware","hardware",155,{"name":58,"slug":59,"count":60,"latest_published_at":18},"Science","science",125,{"name":62,"slug":63,"count":64,"latest_published_at":65},"Consumer Tech","consumer-tech",99,"2026-09-09T17:27:33.000Z",{"name":67,"slug":28,"count":68,"latest_published_at":18},"Dev Tools",78,{"name":70,"slug":71,"count":72,"latest_published_at":73},"Software","software",75,"2026-09-10T20:41:21.000Z",{"name":75,"slug":76,"count":77,"latest_published_at":78},"Startups","startups",55,"2026-09-09T23:14:29.000Z",{"name":80,"slug":81,"count":82,"latest_published_at":83},"Gaming","gaming",43,"2026-09-10T12:18:06.000Z",{"name":85,"slug":86,"count":87,"latest_published_at":88},"General","general",41,"2026-09-08T01:57:23.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]