The story of my first CVE, and questioning myself about ethical hacking.

This post is about how I discovered and obtained my very first CVE. I won’t explain how the vulnerability works (sorry!), so don’t worry if you’re not into technical stuff.

While this is not the first public vulnerability I discovered, this one is very important to me because I had to follow the responsible disclosure process for the very first time.

Writing this blog post is an opportunity to share my experience and thoughts about disclosing critical vulnerabilities in public. This might be an issue every security researcher encounters at some point.

I’m sorry in advance if I’m being unclear or imprecise in this post. I’m not ready to disclose details about the vulnerability just yet. I might write another blog post to complete this one once I feel it’s safe to do so.

Background story

Alongside my junior pentester job at Formind, I’m an active member of the cybersecurity branch at EFREI, the engineering school I graduated from. I was able to participate in FIC’s European Cyber Cup (EC2) as a member of the 0verfl0wls CTF Team.

This was both the first and last time I could participate in the team because I graduated. Most team members are also not students anymore, so for the year 2022, 0verfl0wl had to be rebuilt from scratch.

To recruit new members, we (leaving members) decided to organize a private CTF and select the best players to rebuild the team. This event, the CyberNight, was held in November 2021 in the school and lasted a whole night.

And this is how it started…

Discovery

I was responsible for creating most web-related challenges as well as building the underlying infrastructure (reverse proxies, load balancers, services…) to host the challenges. Without going into the many constraints design decisions of this infrastructure (which could be a whole post on its own), I ended up running a Docker Swarm cluster to deploy services easily.

To administrate the cluster with ease, I added Portainer, which is basically a web UI to manage Docker, Docker Swarm and Kubernetes clusters. I use it on my own servers, it’s free, open-source, a really good project overall.

Now, here is the part where you get disappointed.

CVE-2022–24961 is already publicly disclosed, but at the time of writing this, you will not find much information about it. While I explained the whole vulnerability to the MITRE, I explicitly asked them not to write a detailed description of the vulnerability to prevent exploitation, and it was also requested by the Portainer team. Thus, I won’t provide more details here either.

All you need to know is that I discovered the vulnerability while using Portainer as would any regular user do, looking at security-related features in particular, more precisely the Portainer Agent, which adds a way to manager remote docker and Kubernetes from a single Portainer instance.

To me, this is the easiest way to find unknown vulnerabilities : once you know how the software works, you can start asking yourself the right questions, and eventually find where something is wrong.

How bad is it ?

What makes CVE-2022–24961 a critical vulnerability is the ability to get remote command execution, most of the time as root, on the vulnerable host.

It’s obviously not as bad a Log4shell, but still easier to exploit. The main difference is the attack surface.

You probably already know about Shodan. If you don’t, see it as Google, but you don’t just look for websites, you look for IP addresses, ports, server technologies… anything you can find on the internet network as a whole, not just the wwweb.

I wanted to get a rough landscape of the exploitabiliy of the vulnerability. Shodan could only give me a list of servers running Portainer Agent, not if they were vulnerable. So I retrieved the first 1000 IP addresses reported by Shodan, and started scanning.

For the technical guys out here, I know a little trick to get the first 1000 IP addresses matching any given query without an account and without any API limitation, using a Facet Analysis search. You can then combine parameters and even search by IP. https://www.shodan.io/search/facet?query=http.title%3A%22Apache2%22&facet=ip

Results : 350 hosts out of 1000 are vulnerable.

While Shodan can only give you the first 1000 IP addresses for free, it also tells you how many IP addresses it has in its database matching the query. I ended up with about 6000 hosts reported by Shodan and Censys (another shodan-like tool).

With some very basic math : 350 / 1000, that’s a 35% probability of success. So 0,35 x 6000 gives 2100 hypothetical vulnerable servers.

I need to mention that, when running a cluster of servers, you don’t just run one physical or virtual server, but many of them. As stated before, Portainer can manage either single Docker hosts or whole Kubernetes clusters. Those 2100 vulnerable hosts may hide many more servers beneath them, which could be rooted just as easily. The resulting computing power could allow bad actors to run an efficient cryptomining network or execute denial of service attacks.

So yes, this is really, really bad !

But good news (kinda) : by design, exploiting this vulnerability would actually patch it, or at least prevent anyone from detecting and exploiting it. If I was able to detect all those vulnerable servers, that means no one has ever exploited it before, at least not at scale.

Disclosure, and why it’s not an easy decision

At this point, I felt like I had to keep a secret and no one should ever know about it. But at the same time, I felt responsible for the vulnerable servers, and as an ethical hacker, I could not let someone else find it and exploit it for bad purposes.

The responsible disclosure process or “coordinated disclosure” process as stated on Wikipedia, Bugcrowd, OWASP (…) helps disclosing a vulnerability in the safest manner, but it always mentions that a vulnerability disclosure should be delayed to let vendors patch the vulnerabiliy as well as give users some time to apply the fix and secure their systems.

Well, what about Log4Shell you may ask ? Log4Shell was privately disclosed on 24th November 2021 and publicly disclosed two weeks later, on 9th December 2021. Given how many systems and organizations were impacted, there was no better choice than disclose it as soon as possible. Log4J is a widely used piece of software, and while it’s indeed very hard to patch everything, at least everyone is aware of the issue and can work on it.

Here, my situation is a little bit different. We’re only talking about a few thousands servers. But here is the thing : the patch is released, but it will not be applied on vulnerable servers.

Servers will never be patched

After talking about the vulnerability to the Portainer team, they wondered why so many servers were vulnerable, because the problem only exists when some very specific conditions are met, and it should never happen when using Portainer… until it’s not being used anymore.

That’s it. Vulnerable hosts are not running Portainer anymore. It means that users who previously used Portainer probably don’t care about it now, and are not even aware that the vulnerability exists. Why would they update a software they think they’re not using ? How can they even be aware of the patch ?

The portainer cannot warn them directly either, besides writing a blog post that will never reach its intended audience.

Remediation

Several choices were available to patch the vulnerability :

  1. Remove the vulnerable components entirely, which also removed all associated functionality;
  2. Disable the vulnerable component if it’s still running and Portainer is not installed

Portainer has multiple ways of deploying it’s agent. Depending on which one you choose, you end up with more or less security features by default. Consider that if you chose the weakest one and you uninstalled Portainer, you’re vulnerable as long as the agent is running.

Disclosure

If you’re still with me (thanks a lot), you may understand that I’m now in a weird position. I hold a critical vulnerability that cannot be fixed until it is exploited by someone (in simple terms).

For some time, I wondered why I would even need to request a CVE for this vulnerability. Since the fix won’t be applied, I could just move on and ignore the problem. Because that’s not my problem, and the responsible disclosure process does not require a vulnerability to be disclosed publicly. It’s up to the vendor.

But greed caught up with me. I have never requested a CVE, and the idea of hitting a critical for my first… I cannot refuse it, it is a rare achievement to put on a junior pentester resume. Anyhow, it was only a matter of time before the vulnerability was discovered by someone else, so why wait ?

When Portainer 2.1.11 was released, I requested a CVE ID to MITRE. In theory, I should be assigned a CVE ID for my discovery, and the MITRE team should get back to me to ask for some details, until they actually publish the CVE. I was surprised it didn’t go as planned : I requested the CVE with all details, and it was published the same day without delay between request and publication. I guess they don’t have enough time to postpone publication.

You can find the CVE details below, but you probably already found it.

The vulnerability scores a 9.8 on NIST’s website.

Timeline of events

09/12/2021
Discovered the vulnerability and mapped vulnerable hosts.

16/12/2021
Sent an email to security@portainer.io. Portainer acknowledged the issue very quickly and will work on a (partial) fix for the next release.

18/01/2022
Workaround implemented, waiting for release.

10/02/2022
Portainer 2.1.11 released, fixes the issue.

11/02/2022
CVE ID Requested to MITRE and assigned / published the same day.
Email sent to the Portainer Team to let them know about it. No news since :(

18/02/2022
NIST updates CVSS score to 9.8 CRITICAL

05/04/2022
This medium post.

A question that shouldn’t be asked

Since I discovered this vulnerability, I keep asking myself the same question over and over.

Should I patch this vulnerability myself ?

Because it is so easy. Because no one else will do it. Because I could save thousands of servers and their data from black hats, in a matter of seconds.

But I shouldn’t and won’t do it. I should not even have considered patching in the first place, this isn’t how vulnerability management works. But it is so tempting ! This is a topic we briefly discussed about with the Portainer team. We can’t do anything considered illegal, even if the intent is good. It is what it is, we need to move on.

Portainer could do it only if they had some auto-update feature implemented, which would require users to sign an agreement allowing Portainer to do so… But anyway, that’s not the case. At best, I could contact vulnerable server owners directly and tell them about this story, although I don’t have any way of doing so right now, and chances that vulnerable servers are part of a bug bounty are close to zero.

If you know someone who installed Portainer but who’s not using it anymore, make sure they don’t have a zombie agent running somewhere !

Thanks for reading :)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store