Verified Exposure: Measuring What Security Scanners Miss in AI-Generated Cloud Infrastructure
Abstract
Cloud infrastructure is now written as code, and increasingly that code is written by language models. Studies that evaluate whether such generated infrastructure is secure judge it by running static scanners over the configuration text. A static scanner predicts rather than observes: it reports that a storage bucket appears publicly readable, but it never confirms that an outsider can reach it, and it holds no record of what access the engineer intended.
The accuracy of these scanners against deployed infrastructure has never been measured, so the security scores reported throughout this literature rest on an instrument nobody has calibrated. Six recent studies each recorded that the reference point they needed did not exist and substituted something weaker — a plan with fabricated credentials, a panel of language models, a unit test suite. Every one of them gave cost or speed as the reason. None argued that verified exposure would be uninformative.
This project measures that accuracy directly, and reports false negative and false positive rates with interval estimates, alongside agreement between two independent verification signals and an effective sample size that accounts for near-identical generations.
Research questions
- RQ1
How often do static scanners report no security violation on AI-generated Terraform that is verified as externally reachable after deployment, and is that silence a missed rule or an absent one?
- RQ2
How often do static scanners report a violation on resources that are not externally reachable, or that were intended to be public, and do those errors cluster in identifiable rule classes?
- RQ3
To what extent does stating the access boundary explicitly in the prompt reduce verified external exposure, compared with prompts that leave it unspecified?
- RQ4
How many of the configurations generated across repeated prompts are genuinely distinct, and what is the effective sample size behind the counts reported in this area?
Approach
A benchmark that records intent
Fifty AWS scenarios — thirty covering S3 bucket policies, access controls and public access blocks, twenty covering IAM roles, trust policies and permission boundaries. Each pairs a natural-language prompt with a machine-readable specification of intended access, stating per resource who should be able to reach it. A bucket open to the internet is correct for website images and wrong for backups; no existing benchmark can tell those apart.
Paired prompts
Every scenario is written twice: once leaving the access boundary unstated, once stating it explicitly. The two forms differ only in that sentence, so the effect of the instruction can be separated from the behaviour of the model.
Generation and scanning
Terraform is generated by three models at fixed temperature, with version strings and timestamps captured on every call, then scanned with Checkov and Trivy at pinned container digests. Most scanner rules concern encryption, logging or versioning and make no claim about reachability at all, so a mapping of which rules do is built and published before any measurement.
Deployment and dual verification
Configurations are applied into an isolated AWS account whose deployment role is bounded to the permitted resource types. Reachability is then established two independent ways: automated policy reasoning through IAM Access Analyzer, and direct requests issued from outside the account. Where the two disagree it is recorded as a result rather than resolved quietly, because it documents how layered cloud defences behave.
Calibration before use
The verification tier is run first against hand-written configurations of known exposure, arranged in matched pairs differing only in the setting that produces the leak. Sensitivity and specificity are reported per exposure class, and any class that fails calibration is excluded from the main study rather than quietly included.
Teardown and honest sample sizes
Teardown runs whether verification succeeded or failed, with an independent sweeper confirming nothing was left behind. Generations are canonicalised and clustered so that an effective sample size is reported beside the raw count — an adjustment no published study in this area currently makes.
Contribution
- The first deployment-verified measurement of static scanner accuracy for AI-generated cloud infrastructure, reported as a false negative rate with a confidence interval rather than a count of alerts.
- A benchmark that pairs each scenario with a machine-readable specification of intended access, separating a model that followed instructions from one that leaked data.
- A calibration study of the verification tier itself, so the instrument is shown to be correct before it is trusted.
- The first comparison between AWS IAM Access Analyzer and direct external probing, establishing whether the two agree and where they do not.
- An open-source verification harness that deploys a candidate configuration, checks what is reachable and destroys it again — for under USD 100, contradicting the cost assumption that has kept this step out of the literature.
Topics
- Infrastructure as Code
- Terraform
- Static analysis
- Cloud misconfiguration
- Large language models
- Automated policy reasoning