Evaluating AI beyond correct-looking answers
Study design for a SANS-published comparison of two AI models on an expert task, migrating a 4,934-line firewall configuration to another vendor's platform, scored on whether the output preserved intent rather than whether it looked valid.
At a glance
- Role
- Study design and research protocols, with prompt-engineering support, literature review and manuscript development. Omar Zaman ran the migration, scored the results and wrote the paper.
- Published
- SANS, May 12, 2026, as master's research. The numbers here are the paper's reported results; they describe the models, not my contribution.
- Methods
- Comparative bench study, task decomposition, element-level evaluation, error and severity classification
- Outcome
- A reusable way to evaluate AI on expert work: slice the task, score preserved intent, separate how often errors happen from how much they matter, and count the correction
Summary
The question behind every AI adoption is how much checking, interpreting and correcting is still left to the expert. This study put two models, labeled in the paper as Claude 4.5-Extended and ChatGPT 5.2-Thinking, on the same expert task and scored them element by element on whether the result did what the source configuration did. One model succeeded on the first pass for 90% of 305 elements, the other for 68%. The more useful finding was that the model with fewer visible errors made the more consequential ones. Counting failures alone hides the ones that matter most.
What the study produced
- An evaluation rubric built on preserved intent
- An element counts as a first-pass success only when it commits without errors and preserves the source element's functional intent. Omissions count against accuracy.
- An error taxonomy with severity
- Syntax, reference, semantic and behavioral errors, each weighted by operational consequence rather than by how easy it is to notice.
- A task decomposition
- Three slices of the migration, so a single overall score couldn't hide weakness in the parts with the most consequence.
- A published paper
- Omar Zaman, Leveraging Large Language Models for Cross-Vendor Firewall Configuration Migration: A Comparative Case Study of Claude and ChatGPT. SANS, 2026. Read the paper.
The work
The problem: valid output isn't the same as useful output
Firewall migration means translating both commands and operational intent. Two platforms can implement routing, security policies and VPN behavior differently, so a command the new device accepts can still produce behavior that differs from the original.
The study took one 4,934-line Zyxel ZyWALL 110 configuration and migrated it to a Palo Alto PA-440 with two models. The research question was which model produced functionally equivalent configurations. The question behind it, for anyone adopting AI at work, is how much checking is still left to the expert.
My role
I owned the study design and helped set the research protocols. I also helped develop the initial idea, contributed to the literature review, supported prompt engineering when migration problems came up, and helped complete the manuscript. Omar conducted the migration and is the paper's author. The paper describes a single operator doing the prompting, validation and scoring; my prompt-engineering help supported that work, and I was not a second operator or scorer.
Break the workflow into comparable parts
| Slice | Scope |
|---|---|
| Core objects and security policies | Address and service objects, groups, policies and zones |
| NAT and internet-facing services | Address translation and published-service mappings |
| VPN and segmentation | Site-to-site VPN, remote access, policy-based forwarding and inter-zone routing |
A single overall score would have hidden weaknesses in the tasks with the most consequence. Slicing the work showed where performance differed.
Define success as preserved intent
The unit of measurement was the individual configuration element. An element counted as a first-pass success only when it committed without errors and preserved the source element's functional intent. Omissions counted against accuracy.
Separate how often errors happen from how much they matter
Errors were classed as syntax, reference, semantic or behavioral, and each got a severity based on its operational consequence. A syntax error that blocks a commit is easy to spot. A plausible configuration that commits and quietly changes network behavior needs an expert to catch it.
Treat correction as part of the work
The procedure covered analysis, phased generation, device validation and iterative correction, with errors fed back to the models. That made the expert's effort visible alongside the models' output.
What the study found
| Measure | Claude | ChatGPT |
|---|---|---|
| Security-policy first-pass success | 43 / 49 | 22 / 49 |
| NAT-rule first-pass success | 28 / 34 | 17 / 34 |
| Reported semantic errors | 2 | 12 |
| Approximate total iterations | ~50 | 63 |
Fewer visible errors did not mean better output. The paper reported more syntax errors for Claude and more semantic errors for ChatGPT: the model whose mistakes were easier to see made fewer of the ones that matter. One example was a VPN translation rule that used dynamic address-and-port translation where the intended behavior required static translation. Counting failures alone hides the ones that matter most.
Keeping the source's structure made the work traceable. Claude more often kept source names and rule structure. ChatGPT sometimes renamed zones or consolidated rules. Consolidating wasn't automatically wrong, but it made source-to-target comparison harder and could obscure intent. People who maintain a system need AI changes they can inspect.
Working after correction is not ready on the first pass. Both models eventually produced working configurations after correction. A good final result can hide a lot of work by the person checking it.
Why this matters outside the lab
The same question comes up in every team adopting AI: how much checking does the person still have to do, and would they catch the error that matters? It is why the agents I build start by reading and flagging, earn the right to act at gates, and log what they changed, and why every evaluation I design asks what a person must check.
Limits
This was a bench study of one configuration, one vendor pair, two model versions and one operator. It was not a multi-participant usability study or a general benchmark. There was no second scorer, no repeated runs and no manual-only baseline for measuring productivity. A follow-up could recruit several practitioners, blind the output evaluation, and measure verification time and error detection alongside task success.