Are We Benchmarking AI Intelligence More Than AI Security?

AI models are routinely compared on reasoning, coding, mathematics, and overall performance. But how consistently are we measuring their security?

We have established security benchmarks such as RobustBench, HarmBench, JailbreakBench, CyberSecEval, and AgentDojo, each evaluating different aspects of AI security, from adversarial robustness to jailbreak resistance and prompt injection.

Yet there is no universally accepted benchmark that determines whether an AI model—or an autonomous AI agent—is secure.

A model might perform exceptionally well on capability benchmarks while remaining vulnerable to adversarial attacks, data leakage, prompt injection, or unauthorized tool execution.

And agentic AI makes benchmarking even more complicated. Security is no longer just about what a model says, but also what it can actually do.

My question for the Lounge.ai community:
Should AI developers and vendors be expected to publish standardized security benchmark results alongside their model capability scores?

And if so, which security metrics should matter most?

• Attack Success Rate (ASR)
• Adversarial robustness
• Prompt injection and jailbreak resistance
• Data leakage and privacy protection
• Unauthorized agent actions

I'm particularly interested in whether we need an industry-wide AI security benchmark comparable to the standardized performance benchmarks used to evaluate today's leading models.

What would you include in an AI security scorecard?