Fixing is not the same as securing
Instead of simply checking whether the fixed code compiled or passed automated tests, 1Password said it reviewed every generated fix for complete elimination of the vulnerability, preservation of application behavior, and avoidance of new security risks.
While only 26% of the patches successfully fixed the vulnerability without introducing application changes, 49.3% failed to remove at least one exploitable attack path, 2.3% fixed the original vulnerability but introduced a new one, and 2.2% both failed to remediate the issue and created an additional security weakness.
The researchers also found that passing pre-defined tests can create deeper problems. More than one-third of the patches that initially appeared successful were classified as “fragile” because they simply blocked the proof-of-concept (POC) exploit used during testing instead of addressing the underlying root cause.
