Security protocols verification relies on formal tools such as ProVerif and OFMC. This study
evaluates whether large language models (LLMs) can perform comparable analysis. We
test GPT and DeepSeek in chat and reasoning modes over three runs on 130 obfuscated
AnB/AnBx protocols covering 388 security goals, scored against ProVerif and OFMC. Each
provider uses a single model in both modes, switching reasoning on and off, so both
contrasts isolate reasoning itself. Chat models achieve 72.7% recall at 27.3% precision for
GPT and 69.3% recall at 27.2% precision for DeepSeek. Reasoning models reverse this
trade-off, reaching 66.5% precision and 54.5% recall for GPT and 45.4% precision and
57.3% recall for DeepSeek. Enabling reasoning lifts precision from 27.3% to 64.8% for
GPT and from 27.2% to 44.4% for DeepSeek on the consolidated verdict. The goal set
is imbalanced, with 89 vulnerable goals against 299 secure ones; a trivial always-secure
predictor scores 77.1% accuracy, which only GPT reasoning exceeds. All models perform
worst on authentication goals: reasoning models detect well under half of injective and
non-injective agreement attacks, whereas chat models over-flag them at low precision.
Confidentiality is the exception, with F1 up to 95.7% in reasoning mode. Verdicts are
unstable across runs: identical on 89.7% of goals for GPT reasoning, 74.0% for DeepSeek
reasoning, 70.1% for GPT chat, and 61.6% for DeepSeek chat. Self-reported confidence
is uniformly high yet shows no meaningful correlation with correctness. All results rest
on a single zero-shot prompt and two model providers, which limits generalisability.
On this benchmark, LLMs do not match formal verification, but may serve, at best, as
pre-screening filters.
Full article
