[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-smartmixed-lets-neurons-pick-their-own-activation-functions":10,"sections":35},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":30,"feedback":34,"feedback_at":22,"cost_usd":34,"total_tokens":34},4424,"smartmixed-lets-neurons-pick-their-own-activation-functions","SmartMixed Lets Neurons Pick Their Own Activation Functions","A two-phase training method called SmartMixed lets individual neurons choose from six activation functions, then locks in those choices for efficient inference.","A new training strategy lets neural networks assign each neuron its own activation function instead of applying one blanket choice to the whole model.\n\nSmartMixed works in two phases. In the first, neurons sample from a pool of six candidates — ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, and SELU — using a differentiable mechanism that keeps the selection process trainable. In the second phase, those choices are frozen, converting the network into something that runs with standard vectorized operations and no extra runtime overhead. The researchers tested the approach on MNIST using feedforward networks of varying sizes and found that neurons in different layers tend to gravitate toward different activation functions, suggesting the network is learning something real about what each layer needs.\n\nActivation functions are one of those foundational choices in deep learning that practitioners typically make once and forget. The default is usually ReLU, and for good reason — it is simple, fast, and works well enough. SmartMixed challenges that convention by treating the activation function as a learnable parameter rather than a fixed design decision, which could matter most in domains where squeezing accuracy from a fixed architecture matters more than keeping the training setup simple.\n\nThe honest caveat: the evaluation stops at MNIST, a dataset so well-trodden it barely registers as a benchmark anymore. Whether per-neuron activation learning holds up on harder vision tasks, language models, or anything involving transformers remains an open question the paper does not answer.","[\"machine learning\",\"neural networks\",\"deep learning\",\"research\"]","2026-07-08T04:00:00.000Z","2026-07-08T09:01:22.766Z","2026-07-08T09:01:25.688Z","published",null,[],"ai",[26,27,28,29],"machine learning","neural networks","deep learning","research",[31],{"name":32,"url":33},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2510.22450",0,{"sections":36},[37,41,46,51,56,61,66,71,76,81,86,90,95,100],{"name":38,"slug":24,"count":39,"latest_published_at":40},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":42,"slug":43,"count":44,"latest_published_at":45},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":47,"slug":48,"count":49,"latest_published_at":50},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":52,"slug":53,"count":54,"latest_published_at":55},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":57,"slug":58,"count":59,"latest_published_at":60},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":62,"slug":63,"count":64,"latest_published_at":65},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":67,"slug":68,"count":69,"latest_published_at":70},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":72,"slug":73,"count":74,"latest_published_at":75},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":77,"slug":78,"count":79,"latest_published_at":80},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":82,"slug":83,"count":84,"latest_published_at":85},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":87,"slug":88,"count":84,"latest_published_at":89},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":91,"slug":92,"count":93,"latest_published_at":94},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]