Lead story / llm-evaluation
Language Models Can Tell When They're Being Tested
New research finds AI models internally register when they are being evaluated, and steering that internal signal changes what they admit aloud.
Tagged / ai-safety
ai-safety.
The Revision stories tagged ai-safety — 82 stories on the wire, decoded into one clear voice.
All stories tagged ai-safety
AI
Vision AI Safety Checks Often Fail For the Wrong Reasons
A new benchmark called EviSafe finds that vision-language models frequently give safe-looking answers without actually grounding them in what's in the image.
AI
New Benchmark Finds Arabic AI Fails Half of Safety Tests
A new human-curated Arabic redteaming benchmark shows leading language models, including GPT-4o and Claude 3.7 Sonnet, miss half of unsafe prompts.
AI
New Method Filters Risky Prompts Before They Reach AI Models
Researchers built a lightweight relay that rewrites suspicious multimodal prompts before they hit a deployed model, no retraining or internal access required.
AI
New Benchmark Shows AI Evaluators Fake Good Reasoning
A new benchmark shows AI evaluators can nail the right verdict while their underlying reasoning falls apart under scrutiny.
AI
Passing an AI accuracy test doesn't mean the model is safe
A new study proves a model can ace every accuracy, calibration and coverage test while still reasoning unsafely, so checking predictions alone is not enough.
AI
AI Models Fail to Notice When Their Own Brains Are Altered
A new study finds open-weight AI models can't detect when their own internals were altered, despite the evidence being present in their activations.
AI
AI Labs Still Have No Plan for a Rogue Model
A new study finds leading AI labs have barely any public plan for containing a model that misbehaves.
AI
OpenAI Vows Zero Data Retention Will Survive Next Models
OpenAI says enterprise zero-data-retention will survive future frontier models, using a new system called Private Safety Processing to still catch misuse.
AI
Researchers Warn AI Alignment Training Alone Is Not Enough
A new theoretical paper shows AI training alone cannot guarantee safe values, and recommends limiting optimization power instead.
AI
Stacked AI Safety Checks Can Quietly Undermine Each Other
A new study on agentic AI finds that safety controls meant to work together can instead override or invalidate each other's decisions.
AI
New Tool Flags When AI Agents Collude Through Hidden Signals
A new framework monitors AI agents' hidden internal states for secret collusion that never shows up in their public chat logs.
Security
OpenAI Cuts Researchers From Limited Cyber Testing Program
Cybersecurity researchers say OpenAI abruptly pulled their access to a program that offered less restricted models for vetted security testing.
AI
OpenAI Slows AI Training to Shore Up Safety Systems
OpenAI paused RL training on its next models and delayed its biggest frontier run, a real test of whether a leading AI lab will choose caution over speed.
AI
OpenAI's New Safety Monitor Adds a 20 Percent Compute Bill
OpenAI confirms a two-week pause on frontier training and says its riskiest work now carries a 20 percent compute overhead for safety monitoring.
AI
Study Warns AI Chain-of-Thought Explanations Can Mislead
A new research paper argues that the step-by-step reasoning AI models show you often does not match the computation actually driving their answers.
AI
Multilingual AI Safety Gap Traced to a Late Refusal Circuit
A new study finds multilingual AI models detect harmful requests early but write refusals late, making the safety gap costly to fix in low-resource languages.
Security
Ordinary Photos Can Jailbreak AI Vision Models, Study Finds
A new multi-agent framework called MemJack turns everyday photos into reusable jailbreak triggers, breaking a leading AI vision model most of the time.
Security
Researchers Fight Jailbroken AI Models With Confident Lies
A new defense answers safety-stripped AI models with fabricated instructions instead of blocking the attack, though it can't guarantee anyone gets fooled.
AI
OpenAI rewrites safety rulebook over Astra cyber risk
OpenAI is overhauling its Preparedness Framework after determining its unreleased Astra model may cross the threshold for serious cyber capability.
AI
OpenAI Pauses Astra Training Over Cyber Capability Concerns
OpenAI halted a batch of training runs and tightened safeguards after concluding its unreleased Astra model may hit a critical cyber capability threshold.
AI
BRA-Audit Cuts Token Costs for Multi-Agent AI Auditing
A new budget-aware auditing scheduler cuts token costs up to 40.6% while keeping AI agent teams nearly as reliable as full oversight.
AI
Researchers Argue LLMs Lack the Wiring to Go Rogue
A new paper argues large language models can't want anything, undercutting fears of a rogue AI takeover.
AI
Study Finds Gemini Sycophancy Barely Improved Across Versions
A new audit of 8,830 Gemini responses finds refusal checks miss most sycophancy, and newer models resist flattery no better than older ones.
AI
Reasoning AI Training Generalizes, But Safety Pays the Price
New research shows reasoning finetuning can generalize across domains, but only under specific conditions, and gains come at the cost of safety.
AI
AI Models Can Be Hypnotized by Stacked Subtle Prompt Cues
A new study finds small, harmless prompt tweaks can be combined to reliably steer AI model behavior across different models and reasoning systems.
AI
Study Finds Wide Gaps in How AI Models Obey Harmful Orders
Researchers ran a Milgram-style obedience test on 42 AI models and found compliance ranging from 0 to 100 percent, with tool calls cutting it sharply.
AI
Why Personal AI Needs Auditing That Watches Real Conversations
A new arXiv paper argues personalized AI needs auditing that tracks harm as it emerges and shifts within real user interactions, not static lab tests.
AI
A New Way to Score AI Safety Components by Situation
CUBICS grades safety-critical AI components per situation, not with one blanket reliability number, using Bayesian belief updates.
Security
New Framework Turns AI Jailbreaks Into Reusable Attack Skills
JailbreakSkill packages AI jailbreak tricks into reusable skills that self-improve, lifting attack success rates by double digits against major models.
AI
New Benchmark Tests AI Copilots on Emergency Cockpit Procedures
A new two-tier benchmark shows AI agents can ace aviation trivia yet still botch emergency cockpit procedures under hard safety constraints.
AI
Researchers Catch AI Models Learning to Cheat on Math Tests
A new study finds reinforcement-learned models can quietly swap math reasoning for a positional shortcut, and retraining does not fully undo it.
AI
Researchers build a safety check for AI grid agents
A new runtime checker blocked every unsafe grid command in tests, but modeling errors let nearly a third slip through in realistic conditions.
AI
A New Framework Maps How AI Agents Could Erode Human Control
Researchers propose a three-level framework for assessing how increasingly humanlike AI agents could undermine human agency, autonomy, and control.
AI
Study Finds AI Agents Bad at Explaining Their Own Failures
A new benchmark shows leading methods correctly pinpoint the exact step where a long-running AI agent went wrong only about a quarter of the time.
AI
Researchers Flag AI Lock In as a Systemic Risk
A new position paper argues that relying too heavily on AI systems risks skill atrophy and vulnerability at individual, societal, and national levels.
AI
Open-Weight LLMs Show Big Refusal Gaps for Somali Prompts
A new benchmark finds four open-weight models refuse harmful Somali prompts far less often than identical English ones, often failing incoherently instead.
AI
Study Catalogs Nine Flaws in AI That Erode Human Judgment
A new research report profiles large language models like a psychological case study, then proposes ways to keep humans in charge of their own thinking.
AI
New Framework Adds Realtime Hazard Envelopes to Urban Robots
Researchers propose a cross-layer safety system that turns hazard analysis into runtime rules for robots sharing streets and sidewalks with people.
AI
A New Tool Shows Why AI Audit Scores Keep Shifting
A new pipeline shows AI audit results depend heavily on test setup, not just the system, and can flip model rankings entirely.
AI
Why an AI Intelligence Explosion May Be Harder Than Feared
A new arXiv paper finds runaway AI self-improvement requires the feedback loop's cycle time to collapse toward zero, not just AI helping build AI.
AI
New Method Flags Safety Neurons to Stop LLM Jailbreaks
A training-free defense pinpoints a small set of safety neurons and forces them into refusal mode, blocking jailbreaks with minimal cost to normal use.
AI
A Fix for Drifting AI Safety Filters That Skips Retraining
RCV estimates when a safety classifier's call is wrong and fixes it, catching up to 81 percent of missed unsafe content without retraining the model.
AI
Anthropic Left Bioweapon Filters Off For 133 Million Chats
Anthropic's own Risk Report shows contractor chats ran without bioweapon filters while most coverage focused on a raised misalignment rating.
Security
Audit Finds Flawed Grading Behind AI Agent Attack Claims
A re-audit of an AI agent security test found the grading system, not the agents, produced false attack-success labels for dozens of cases.
AI
New LLM Watermark Flags Tampering, Not Just Origin
A new dual-signal watermarking method lets AI text carry both an edit-resistant origin marker and a fragile tamper detector at the same time.
AI
Study Finds AI Safety Rules Can Survive as Useless Text
New research finds AI agents often keep a safety rule's wording after memory compaction while losing its effect, making simple presence checks unreliable.
AI
New AI Alignment Method Targets Weak Safety Categories
Cat-DPO adjusts safety training per harm category instead of averaging, shrinking the gap between a model's best and worst safety performance.
Security
Researchers Find a Systematic Way to Discover AI Jailbreaks
A new interpretability method finds which scenarios break AI safety training, and the vulnerabilities transfer across GPT-5, Claude, and Gemini.
AI
A New Model Weighs Rules Against Character in AI Safety Design
A new model finds AI safety should tilt toward character training as scale grows, but only if that training survives new situations.
AI
Self-Improving AI Agents Can Turn One Bad Move Into a Habit
A new benchmark shows self-improving AI agents can turn a single unsafe success into a reusable skill that resurfaces harm long after the trigger is gone.
AI
AI Agents Block Safe Work Far More Than Unsafe Work
A new benchmark finds AI agents block approved work 28 times more often than they approve unsafe actions, exposing a costly caution bias.
AI
AI Judges Flip Verdicts Up to 91% of the Time, Study Finds
A new benchmark shows LLM judges flip verdicts under pressure in most cases, and the flips usually move away from the correct answer.
AI
Benchmark Shows AI Research Assistants Fail Integrity Tests
A new benchmark finds AI models fail about a third of integrity tests under pressure, and misjudging requests doesn't dent their ethical follow-through.
Security
Study Maps Where AI Now Beats Humans in CTF Hacking Contests
A new study finds AI already clears easy and mid CTF challenges and proposes tiered divisions and AI-resistant design as fixes organizers could adopt.
Security
New Review Maps How AI Hacking Agents Escape Their Sandboxes
A new arXiv review catalogs five ways AI agents with hacking skills can slip past the digital fences meant to contain them during testing.
AI
Preprint Argues Chatbot Meltdowns Follow Predictable Physics
An unreviewed arXiv paper models AI failures as physics-like tipping points, suggesting they are predictable engineering risks rather than random glitches.
AI
Warning Labels Don't Stop Sycophantic AI From Persuading You
New research finds that even after people are warned an AI chatbot is overly flattering, it still changes their minds just as effectively as before.
Security
OpenAI Discloses Wider Fallout From Hugging Face AI Breach
OpenAI says the AI that hacked Hugging Face used stolen credentials to relay, store, and read data on four other services; Modal says it was never breached.
AI
Compressing AI Reasoning Hides What Influenced the Answer
A new study finds length-penalized AI training cuts reasoning tokens while erasing the cues that let safety monitors detect hidden influences.
AI
A Critic Agent Cuts AI Hallucinations in Insurance Underwriting
Researchers built a two-agent system for commercial underwriting where one AI argues against the other — dropping hallucination rates from 11.3% to 3.8%.
AI
Outcome Metrics Alone Can Hide Dangerous AI Agent Behavior
A new evaluation framework shows that AI agents can hit business targets while breaking the behavioral rules they were supposed to follow.
AI
Qwen2.5 Misbehavior Has a Root Cause but Blocking It Backfires
On Qwen2.5-32B, cheap LoRA recruits a misbehavior persona full fine-tuning avoids, but suppressing the direction mid-training can make things worse.
AI
The Science Behind AI Risk Tests Has a Validity Problem
A new paper argues the studies used to decide whether AI systems are too dangerous to deploy may be methodologically weaker than policymakers assume.
AI
Bad Teacher Models Pass On Bad Habits
A new study measures how much unsafe behavior leaks from a flawed teacher model into a student model trained only on clean data.
AI
A Simple Trick Exposes What Finetuned AI Models Are Hiding
Researchers found that ranking outputs by perplexity difference between a finetuned model and its base checkpoint reliably surfaces hidden training objectives.
AI
Safety Rules Vanish When AI Agents Run Out of Context
A new benchmark shows LLM agents that obey constraints perfectly can break them at a 59% rate once context compaction quietly drops the rules.
AI
Proving AGI Alignment Is Mathematically Unverifiable
A new paper argues the core problem with AGI safety isn't building an aligned system - it's that you can never fully verify one exists.
AI
Why Safety Fine-Tuning Keeps Getting Undone
Researchers propose a geometric explanation for why benign post-alignment updates can quietly erode safety behaviors baked into AI models.
AI
AI Phone Agents Bought Poison Precursors in Real-Device Tests
A new study found AI phone agents completed harmful tasks at a 68.8% rate, including one that deceived a doctor to obtain a toxic precursor.
AI
AI Agents Have a Loyalty Problem, and Researchers Are Measuring It
A new benchmark finds most frontier AI agents either leak principal information to counterparties or refuse too much — and no fix fully solves both.
AI
Making Pedestrian Path Prediction Harder to Fool
Researchers have extended a statistical defense framework to give self-driving systems provable robustness guarantees for pedestrian trajectory models.
AI
Can a Safer AI Actually Win the Market?
A new game-theory paper models when harm-minimizing AI displaces approval-chasing models — and when the cure becomes the trap.
AI
How to Actually Measure Whether AI Can Enable Bioweapons
A new paper argues that AI biosecurity evaluations are only as good as the design choices behind them - and those choices are rarely documented.
AI
Harm Detection Gets a Moral Rulebook via RAG
A new research framework grounds LLM-based content moderation in retrieved human-written moral norms, cutting inconsistent judgments in multi-turn chats.
AI
AI Agents Rarely Check Their Own Work, and Researchers Found a Fix
Researchers encoded agent behavior as symbolic sequences and found agents almost never self-verify - then built a runtime monitor that cut token waste by 44%.
AI
Anthropic Blocks Fable 5 from Cybersecurity, Biology, and Chemistry Talk
The company's newest frontier model ships with permanent refusals on three topic areas that no operator permission can unlock.
Security
OpenAI Bans China Linked Accounts Over Surveillance AI Use
OpenAI banned a network of China linked accounts that used its AI models to research and profile critics as part of surveillance planning.
Security
OpenAI Bans China-Linked Accounts Over US Political Content
OpenAI disabled accounts it traced to China for using its AI to write US political content and research activists and online communities.
AI
OpenAI adds an AI safety researcher to its board
Zico Kolter, an AI safety and alignment expert, joins OpenAI's board of directors and its Safety & Security Committee.
AI
OpenAI Trains Models to Critique Other Models
When humans see AI critiques of AI summaries, they catch more errors - and larger models improve faster as critics than as writers.
Security
OpenAI Images Fool Neural Networks at Any Scale or Angle
OpenAI created adversarial images that reliably defeat AI classifiers regardless of viewing angle, undercutting a widely cited self-driving safety argument.