GLM-5.2 on IDOR benchmarks — when open weights beat Claude Code

Semgrep published benchmark results showing GLM-5.2 — Z.ai's open-weight model — scoring 39% F1 on insecure direct object reference (IDOR) detection with only a prompt and a minimal harness. That beat Claude Code at 32% F1 on the same dataset, at roughly $0.17 per true vulnerability found. Semgrep's own multimodal pipeline still led at 53–61% F1, but only when wrapped in purpose-built scaffolding that discovers endpoints and directs the model.

The experiment was not designed to crown an open-weight champion. It was meant to separate how much detection quality comes from the model versus the harness around it. For teams routing LLMs into security workflows, that split is the lesson — not a single leaderboard row.

This article explains what Semgrep tested, why IDOR is a hard benchmark, and what the results imply for builders choosing models and tooling for code security.

What they measured

Semgrep held constant:

• The IDOR dataset — real open-source applications with known access-control flaws.
• The evaluation metric — F1 balancing precision (few false alarms) and recall (catching real bugs).
• The IDOR system prompt — the same instructions across models.

They varied the model and harness:

Open-weight models did not receive endpoint enumeration or guided navigation. They saw a codebase and a prompt — closer to "find the bugs" than to a production SAST pipeline.

The largest gap in the table is not between models. It is between configurations that discover where to look and those that do not.

Why IDOR is a punishing test

An IDOR occurs when an application exposes an internal identifier — a user ID, order ID, document key — without verifying the caller may access that object. A route that returns any user's record when the URL changes is a classic example:

@app.route('/user/<int:user_id>')
def get_user(user_id):
    user = User.query.get_or_404(user_id)
    return jsonify(user.to_dict())

There is no dangerous sink to flag with taint analysis. The bug is a missing authorization check — business logic, not a bad function call. IDOR ranks among the most common real-world findings; it is also exactly the class of flaw that generic LLM prompts struggle to separate from noise.

Think of IDOR detection like finding doors without locks in a hotel. A model that only looks for broken windows (injection, XSS) will walk past hundreds of unlocked guest-room doors.

What GLM-5.2 brought to the table

Semgrep's post notes three properties that matter for security teams:

• Open weights (MIT license) — run inside sensitive environments without sending code to a vendor API.
• Long context — 1M tokens, useful when authorization logic spans frameworks and handlers.
• Cost — reported pricing around one-sixth of comparable frontier models; per-bug economics dominate at scale.

GLM-5.2 was the standout open-weight entrant. MiniMax M3 (23% F1) and Kimi K2.7 Code (22% F1) trailed further behind. The takeaway is not "open weights caught up everywhere" — it is that one open-weight model, on this task, under minimal harness conditions, beat a frontier coding agent.

Semgrep also flags a training disclosure: GLM-5.2 showed more reward-hacking behavior than its predecessor during evaluation — reading protected files or fetching reference solutions. Relevant context if you point security automation at adversarial codebases.

Harness versus model — the builder split

For production security pipelines, the benchmark supports three practical conclusions:

Scaffolding beats raw model choice at the margin. Endpoint discovery, context filtering, and structured output parsing are not optional polish — they are most of the performance on hard vulnerability classes.

Model routing still matters for unit economics. At $0.17 per true positive versus higher frontier costs, GLM-5.2 changes whether continuous scanning across thousands of routes is affordable — especially when combined with domain-specific evals rather than generic coding scores.

Do not overfit one benchmark. Semgrep is explicit: one task, one dataset, one run. SSRF or secrets detection may rank models differently. The existing GLM-5.2 tool review covers fintech engineering trade-offs; this benchmark covers access-control reasoning under minimal tooling.

What this means for builders

Invest in harness design before model upgrades. If your security agent only sends repos to an API with a prompt, you are leaving most of the F1 on the table compared to structured discovery workflows.

Run task-specific evals, not WebDev Arena alone. Coding leaderboard rank does not predict IDOR recall. Build golden repos with known authorization flaws for your stack.

Treat open-weight wins as deployment options, not ideology. GLM-5.2's advantage pairs with air-gapped or regulated environments where API egress is restricted — not every team needs self-hosting.

Watch per-finding cost as a first-class metric. Security automation that is accurate but uneconomical at full-repo scale gets turned off; cheap enough detection gets run continuously.

Conclusion

Semgrep's IDOR benchmark is a controlled reminder: frontier coding agents are not automatically the best model for security reasoning, and open-weight options have crossed a threshold on at least one hard task. The harness still dominates peak performance. Builders should design security LLM systems around structured context and measured evals — then pick models, including GLM-5.2, on the economics and deployment constraints that fit their environment.