Skip to content
Article · Remediation

What evidence is enough to close a security fix?

Before closing a high-impact security finding, confirm the fix covers the underlying rule, preserves legitimate use, and is running everywhere it needs to be.

By Hexkey · Published Mar 18, 2025 · 4 sources

A passing retest can establish that the original report was addressed under specific conditions. Closure also depends on whether the security rule holds across the affected surface and whether the right change is running in production.

A fictional distributor removes a former employee’s access to a warehouse. The main dashboard stops showing that warehouse’s orders, and engineering marks the security ticket fixed. The business rule reaches further than the dashboard. Does the change also cover the mobile app, exports already scheduled under that user, and the session active when access was removed? Can staff still see the orders for warehouses they do manage? The release owner needs evidence for those questions before calling the fix complete.

What a follow-up fix can tell us

The Apache Log4j project published version 2.15.0 in December 2021 to address the vulnerability widely known as Log4Shell. Apache later documented that this initial fix was incomplete in certain non-default logging configurations and tracked the remaining issue separately. For teams using Java 8 or later at the time, 2.16.0 addressed that follow-up issue. The version numbers tell one part of the story. The remaining exposure depended on configuration, which is exactly what a verification plan must account for. Apache Log4j security advisory for the follow-up issue · Log4j 2.16.0 release notes

Frank Li and Vern Paxson examined security fixes for 3,094 public vulnerabilities across 682 open-source Git repositories in a study published at ACM CCS in 2017. To assess reliability, they manually reviewed 50 randomly sampled vulnerabilities from a subset whose fixes involved multiple commits beyond an initial 24-hour period. They identified 26 incomplete fixes and 17 that introduced regressions, with some overlap. Extrapolating from that subset, they estimated that about 7 percent of fixes in their dataset were incomplete and about 5 percent introduced regressions. Those estimates describe a historical sample built from public vulnerability records and linked Git commits. They give us examples of two failure modes to look for when deciding how much retest effort a critical change warrants. Li and Paxson’s study, sections 3 and 5.1.3

Write down the rule before testing the change

For the warehouse application, the rule could be that revoked access takes effect across warehouse data and actions at a defined point after removal, including for active sessions. Product and engineering need to agree on the point and how it applies to work already in progress. The expected behavior for an export that has been queued, an export already downloaded, and a new request can differ. Writing down those expectations makes it possible to verify the real fix and spot continuity problems before customers report them.

Start by preserving the original report and its evidence. Repeat the original check in the approved environment to confirm that the behavior changed for the reason intended. Include a comparison that should succeed. A dashboard that shows an error for every employee has stopped displaying unauthorized orders, and it has also stopped serving the warehouse. For an authorization fix, a legitimate action should still work under the appropriate role and scope.

Then follow the shared control in the code. If the dashboard and mobile app use the same permission decision, test enough of each integration to establish that the decision applies. If the background worker makes its own decision, give it its own case. In the warehouse example, include a session that was already active at the time of revocation. Prioritize variants that exercise a different path to the same business consequence. Put the durable checks where the team can run them when touching that code again. OWASP’s verification standard provides requirements against which teams can specify and review application controls, including authorization. OWASP Application Security Verification Standard

If a test unexpectedly passes on both versions, check the setup before drawing a conclusion. The account might already lack the relevant association, or the test record may have been cleaned up between runs. Where feasible, run the new regression against the prior build in an isolated environment with synthetic data and confirm that it detects the original failure. For an urgent fix, this can be a focused check against the affected behavior. The aim is to give engineering a test that would catch the problem if a later permission change reintroduced it.

Verify the running deployment

A successful build and a passing suite tell you about the artifact you tested. Establish which artifact is actually serving traffic or processing jobs. A blue-green deployment can leave old capacity receiving traffic during rollout. A separate worker or customer-managed installation may update on a different schedule. Record the release identifier and which instances or supported versions it covers. NIST’s enterprise patch guide treats verifying that a patch took effect and monitoring that it stays deployed as lifecycle steps. It specifically calls out old versions restored from backups as a reason to monitor. NIST’s patch management guide, sections 2.3.3 and 2.3.4

Choose evidence in proportion to the finding. A high-impact change might justify a controlled production check with a synthetic record alongside deployment telemetry. For an operation with material side effects, verify behavior in a representative environment and use production evidence to confirm the relevant version and configuration. Spell out which conclusion each artifact supports. If a temporary restriction protects users while a worker upgrade is pending, record the restriction, who verified it, and what still needs to ship.

When the evidence is mixed, keep the record specific. A change verified in the browser with worker verification still pending gives the incident owner a different decision from a successful deployment across both. The same applies when a rollback restores the vulnerable release. Assign follow-up to the person who owns the remaining path and keep the original evidence attached so they can pick up the work.

Hexkey’s view is that the person accepting closure should be able to read one short record and see the original behavior, the security rule and legitimate workflow tested, the deployment covered, and any remaining exposure. The application team can supply most of that from its existing tests and release system. For a finding that could have exposed sensitive data or granted broad privilege, a second practitioner can review that evidence and confirm the result. Budget for this work when you assign the fix. The ticket closes when the evidence supports the outcome you told the business to expect.

Sources and further reading

Want a second look at your own applications?

We find and verify what’s exposed, fix what matters, and modernize what can’t be secured piece by piece.