[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-new-method-scores-ai-security-learning-tools-without-labels":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},5009,"new-method-scores-ai-security-learning-tools-without-labels","New Method Scores AI Security Learning Tools Without Labels","A new benchmark-free scoring method judges AI 'learning harnesses' by how closely they converge toward a stronger teacher model over time.","A new evaluation method scores AI systems that learn from feedback in the field, without needing a labeled test set to check their work.\n\nThe approach targets \"continual learning harnesses\" - setups that pair a large language model with retrieval or memory so it improves at cybersecurity tasks over time, without retraining. Researchers found that labeled benchmarks for these harnesses are scarce, outdated, or unrepresentative of real deployments, so teams often cannot tell if a harness helps at all. Their fix: have a stronger \"teacher\" model give sparse, high-precision corrections to a smaller \"student\" running the harness, then measure how much the student's answers converge toward the teacher's over time. Tested across multiple security tasks, model families, and harness designs, that convergence score tracked closely with improvement measured against a real, held-out gold standard.\n\nThis matters because security teams evaluating AI tooling are usually flying blind - there is no updated answer key for a brute-force detection harness or a new phishing-triage agent. A proxy metric that does not require labels means practitioners can actually compare two harnesses on their own data instead of trusting vendor claims.\n\nThe paper also notes that using one AI model to judge another only works when the judge is meaningfully stronger - a reminder that \"AI reviewing AI\" is not the free lunch some vendors pitch it as.","[\"ai\",\"security\",\"llm-evaluation\",\"machine-learning\"]","2026-08-17T04:00:00.000Z","2026-08-17T05:13:33.065Z","2026-08-17T05:13:44.883Z","published",null,[],"ai",[24,26,27,28],"security","llm-evaluation","machine-learning",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.13608",0,{"sections":35},[36,40,43,48,53,58,63,68,73,78,83,88,93,98],{"name":37,"slug":24,"count":38,"latest_published_at":39},"AI",3293,"2026-08-20T04:00:00.000Z",{"name":41,"slug":26,"count":42,"latest_published_at":39},"Security",435,{"name":44,"slug":45,"count":46,"latest_published_at":47},"Policy","policy",210,"2026-08-19T09:32:27.000Z",{"name":49,"slug":50,"count":51,"latest_published_at":52},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":54,"slug":55,"count":56,"latest_published_at":57},"Hardware","hardware",140,"2026-08-19T18:25:42.000Z",{"name":59,"slug":60,"count":61,"latest_published_at":62},"Consumer Tech","consumer-tech",95,"2026-08-18T16:05:00.000Z",{"name":64,"slug":65,"count":66,"latest_published_at":67},"Science","science",90,"2026-08-19T18:41:02.000Z",{"name":69,"slug":70,"count":71,"latest_published_at":72},"Software","software",73,"2026-08-18T07:51:50.000Z",{"name":74,"slug":75,"count":76,"latest_published_at":77},"Dev Tools","dev-tools",69,"2026-08-18T04:00:00.000Z",{"name":79,"slug":80,"count":81,"latest_published_at":82},"Startups","startups",47,"2026-08-19T19:13:46.000Z",{"name":84,"slug":85,"count":86,"latest_published_at":87},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":89,"slug":90,"count":91,"latest_published_at":92},"General","general",33,"2026-08-18T22:18:13.000Z",{"name":94,"slug":95,"count":96,"latest_published_at":97},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":99,"slug":100,"count":101,"latest_published_at":102},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]