Supply Chain Security Issue in Kong Ingress Controller v3.4
As a longtime user of Kong Ingress Controller in production, recent inccident involving the unauthorized image of Kong Ingress Controller v3.4 underscore the critical importance of securing the open-source supply chain.
The Incident
https://github.com/Kong/kubernetes-ingress-controller/issues/6907
On December 23, 2024, an unauthorized version of Kong Ingress Controller (v3.4.0) was uploaded to DockerHub. This image contained malicious code enabling cryptojacking by connecting to crypto mining pools like pool.supportxmr.com. The issue was identified after users observed unusually high CPU usage in their Kubernetes clusters.
By January 2, 2025, Kong released version 3.4.1, removing the malicious code and rotating access keys. The company issued a security advisory and provided detection tools (e.g., YARA rules) to mitigate the impact.
What Went Wrong?
The root cause was a misconfiguration in Kong’s public repository build pipeline, which allowed unauthorized actors to inject malicious code. This highlights a broader issue: open-source supply chain vulnerabilities.
Risks in Open-Source Supply Chains
Open-source software offers immense value, but it’s not without risks:
1. Unauthorized Code Injection: Malicious actors exploiting CI/CD pipelines or repositories.
2. Trust in Tags: Relying on mutable image tags (e.g., latest or 3.4) instead of immutable digests.
3. Delayed Detection: Lack of robust monitoring and auditing of published artifacts.
These risks can lead to:
• Operational Disruptions: Unexplained resource consumption or outages.
• Data Breaches: Exposure of sensitive information.
• Reputational Damage: Loss of trust in critical software components.
How to Address This Problem
To protect your infrastructure and mitigate similar risks, consider the following best practices:
1. Pin Versions and Use Immutable Digests
Avoid relying on mutable tags like latest or 3.4. Instead, pin images to immutable digests:
image: kong/kubernetes-ingress-controller@sha256:b358296fa6a1458c977c0513ff918e80b708fa9d7721f9d438f3dfce24f60f4f2. Enable ImagePullPolicy
Set ImagePullPolicy: Always to ensure the Kubernetes cluster always pulls the latest image for a given tag.
3. Monitor Resource Usage
Implement continuous monitoring of CPU and memory usage in your clusters. Unusual spikes may indicate unauthorized activity, such as cryptojacking.
4. Use Container Scanning Tools
Tools like Trivy, Clair, or Aqua can scan container images for vulnerabilities and detect unexpected changes.
5. Implement Supply Chain Security Tools
Adopt tools like Sigstore, TUF (The Update Framework) to secure CI/CD pipelines and ensure the integrity of published artifacts.
6. Regularly Audit Your Dependencies
Audit both direct and transitive dependencies for vulnerabilities using tools like Dependabot, Snyk.
Lessons Learned
This incident serves as a wake-up call for organizations relying on open-source software. Even trusted tools like Kong Ingress Controller are not immune to supply chain attacks. By adopting proactive measures, such as pinning versions, auditing dependencies, and securing CI/CD pipelines, we can better protect our systems from similar threats.
Conclusion
The open-source community must prioritize supply chain security to maintain trust. As users, we also bear responsibility to implement safeguards in our environments. The Kong team’s quick response to this incident is commendable, but it reminds us all of the shared effort required to secure the software we rely on.
Let’s stay vigilant and work together to ensure a more secure open-source ecosystem.
