Is PR Scanning Enough? The Growing Threat of Security Debt in AppSec

In the world of application security (AppSec), developers and security teams continuously strive to balance agility and safety. A common practice is placing security gates on pull requests (PRs) to check for vulnerabilities when adding new dependencies. While this is an important first step, it only addresses part of the problem. PR gates catch vulnerabilities present at the time of addition, but they can’t account for those that emerge later. Our research highlights a significant blind spot: newly added packages are usually clean, meaning PR gates succeed initially, but this does not protect the application’s security over time. As new vulnerabilities surface, every application accrues security debt that must be managed effectively.

Hypothesis

When developers add new packages to their app manifest, they often don’t specify a fixed version, opting instead to use the latest available version. This practice seems logical, as the latest versions of actively maintained packages are typically clean, thanks to code scanning and dependency hygiene measures adopted by popular projects.

However, this exposes a critical inefficiency in relying solely on PR security gates. If the latest versions are already clean, PR gates serve primarily as a verification step rather than a robust defense mechanism. Over time, new CVEs (Common Vulnerabilities and Exposures) are discovered, affecting both the packages and their transitive dependencies, rendering the original PR gates increasingly irrelevant.

Result: The reliance on PR gates does little to address the evolving vulnerability posture of the application. As a result, every application inevitably accumulates security debt, which requires consistent tracking and proactive mitigation.

npm Research Findings

To validate our hypothesis, we conducted an extensive analysis of popular npm packages with over 1,000,000 yearly downloads. The goal was to assess the vulnerability profile of these widely used packages and their dependencies. Here's a breakdown of our findings:

Dataset Overview

• Total Packages Analyzed: 4,317
• Scope: We focused on the latest stable versions, explicitly excluding alpha, beta, or release candidate (RC) versions to maintain data accuracy and relevance.

Direct Vulnerabilities

Direct vulnerabilities refer to issues found within the package itself:

• Packages with Known Vulnerabilities: Only 3 out of 4,317 packages showed any known vulnerabilities.
• Packages with Critical/High (C/H) Vulnerabilities: Only 2 packages fell into this category, reinforcing the observation that actively maintained packages tend to be clean.

Transitive Vulnerabilities

Transitive vulnerabilities occur in dependencies used by a package. 

• Packages with C/H Vulnerabilities in Their Dependencies: 686 packages, representing 15.8% of the dataset.
• Aging Vulnerabilities: Among these packages with vulnerable dependencies, 40% had dependencies with vulnerabilities older than 90 days, suggesting a significant lag in updates or fixes.

Historical Vulnerabilities

While we cannot predict the future vulnerability profile of these packages, our hypothesis suggests that new vulnerabilities will likely be discovered over time. To validate this, we examined whether the packages had historical versions with known vulnerabilities:

• Vulnerable Historical Versions: Over 800 of the packages with a clean latest version in our dataset, had older versions with known vulnerabilities.
• Vulnerable Historical Timeframe: On average, we found that the time between the last vulnerable version and the latest secure version is approximately one year. This suggests that within just one year, a clean version of a package can become vulnerable, emphasizing the dynamic and evolving nature of dependency security risks.

This retrospective analysis supports the idea that even "clean" packages today can accumulate vulnerabilities over time, reinforcing the need for proactive security management.

Key Insights

1. The very small number of direct vulnerabilities in actively maintained npm packages underscores the effectiveness of PR gates in catching issues at the moment they are introduced.
2. The percentage of packages with transitive vulnerabilities (15.8%) suggests that critical/high vulnerabilities are unlikely to appear during the initial PR process and, therefore, won’t be caught by PR gates.
3. Historical vulnerabilities and update lags show that security debt is inevitable over time, even for applications that initially rely on clean packages.
4. This data reinforces that relying solely on PR scanning is insufficient for ensuring long-term application security and managing evolving vulnerabilities.

PyPI Research Findings

To broaden our analysis, we examined Python's PyPI repository, focusing on the top 1,000 packages by annual downloads. These packages are widely used in Python ecosystems, making their security profile critical for AppSec teams. Here’s what we discovered:

Direct Vulnerabilities

Direct vulnerabilities refer to issues identified within the packages themselves:

• Only 3 packages had known critical or high (C/H) vulnerabilities.
• Notably, all 3 packages with C/H vulnerabilities were more than 3 years old, suggesting that older, less actively maintained packages are more likely to harbor unresolved security issues.

Overall Vulnerability Profile

To get a complete picture, we included all severities (critical, high, medium, and low) and considered vulnerabilities in both direct and transitive dependencies:

• A total of 13 packages were found to have vulnerabilities across all severities and dependencies.
• This relatively low number indicates that, similar to npm, most actively maintained Python packages are clean at the time of their latest release.

These findings demonstrate that while PyPI packages currently have a low rate of direct and transitive vulnerabilities, over time, new vulnerabilities are likely to be discovered in these packages. This underscores the importance of implementing proactive security practices and conducting regular vulnerability assessments to complement and go beyond PR gates.

‍

Key Takeaways

• Latest Versions Are Relatively Clean: The latest versions of packages are generally secure, thanks to active maintenance, the recency of their release, and the limited time available for vulnerabilities to be discovered by the community.
• Vulnerability Emergence Over Time: Packages that begin as clean can develop vulnerabilities as new issues are identified in their code or dependencies, emphasizing the need for ongoing monitoring and regular updates.
• Security Debt Is Inevitable for Packages: Over time, security debt accumulates as new vulnerabilities are discovered in dependencies. Managing this debt requires a focused strategy. While PR scans are effective for identifying and resolving code vulnerabilities, they are insufficient for addressing the long-term challenges posed by dependency security debt in packages.

Conclusion: Managing Security Debt Beyond PR Scanning

While PR scanning is an important component of application security, it is not sufficient for maintaining long-term security. A robust AppSec program should include the following strategies:

1. Don’t Over-Invest in PR Gates: PR gates are effective for catching critical issues at the time of addition, particularly in areas like Static Application Security Testing (SAST), where shift-left practices shine. However, they are less effective for Software Composition Analysis (SCA), where vulnerabilities often emerge over time. Design PR gates to address essential use cases without slowing down development, balancing security with the need for velocity.
2. Continuous Monitoring: Regularly reassess vulnerabilities in dependencies, including both direct and transitive ones, even long after their initial inclusion in the application. This ongoing effort ensures that emerging risks are detected and addressed promptly, complementing the limitations of PR gates.
3. Security Debt Management: Build robust processes and tools to track and mitigate security debt over time. Prioritize fixes for critical and high vulnerabilities that are reachable and pose actual risks to the application. Worth noting, shift-left practices are not effective for SCA and packages since security debt in dependencies accumulates post-deployment, requiring long-term management strategies beyond PR gates.

Side Note: When the latest version of a package has vulnerabilities with no fixable alternatives, development teams often bypass blocking rules to ship features. This increases security debt and underscores the need for a comprehensive approach.

Final Thought: Security is a journey, not a one-time gate. As new vulnerabilities emerge and dependencies evolve, AppSec teams must adopt a proactive mindset, embracing tools and methodologies that go beyond PR scanning to safeguard their applications in the long term.

‍

‍