Mini Heart Attack: Supermicro BMC Hacked (or not?)

December 09, 2023

The Incident

I was recently doing some remote server maintenance when I came across something online that prompted me to check on the baseboard management controller (BMC) via the IPMI/remote management interface. For those unfamiliar, it is a remote management interface that can be used to monitor and control low-level functionality on the server, providing capabilities like remotely rebooting the machine or accessing the BIOS.

As I already had a VPN connection and root SSH session open on the server, I quickly ran the following just to confirm if the BMC interface was still available (I had made some network changes recently):

root@pve-host~:# nc 192.168.10.199 80
GET / HTTP/1.1

However, while typing this command, netcat returned early, splitting the manual HTTP request so my console looked like the following:

root@pve-host~:# nc 192.168.10.199 80
GET / root@pve-host:~# HTTP/1.1

I hit enter anyway, and a moment or two later, got something like this dumped to the terminal:

<HTML>
<HEAD>
<TITLE>Directory /</TITLE>
<BASE HREF="file:/">
</HEAD>
<BODY>
<H1>Directory listing of /</H1>
<UL>
<LI><A HREF="./">./</A>
<LI><A HREF="../">../</A>
<LI><A HREF="bin/">bin/</A>
<LI><A HREF="boot/">boot/</A>
<LI><A HREF="dev/">dev/</A>
<LI><A HREF="etc/">etc/</A>
<LI><A HREF="home/">home/</A>
<LI><A HREF="lib/">lib/</A>
<LI><A HREF="lib32/">lib32/</A>
<LI><A HREF="lib64/">lib64/</A>
<LI><A HREF="libx32/">libx32/</A>
<LI><A HREF="media/">media/</A>
<LI><A HREF="mnt/">mnt/</A>
<LI><A HREF="opt/">opt/</A>
<LI><A HREF="proc/">proc/</A>
<LI><A HREF="root/">root/</A>
<LI><A HREF="rpool/">rpool/</A>
<LI><A HREF="run/">run/</A>
<LI><A HREF="sbin/">sbin/</A>
<LI><A HREF="srv/">srv/</A>
<LI><A HREF="sys/">sys/</A>
<LI><A HREF="tmp/">tmp/</A>
<LI><A HREF="usr/">usr/</A>
<LI><A HREF="var/">var/</A>
</UL>
</BODY>
</HTML>
<html><body>
...
( Some random e-commerce website template (?!?) - removed for brevity )
...
</body>
</html>

Oh &#*@

My heart dropped.

A root directory listing followed by some completely random e-commerce website template — I had been hacked.

Clearly, malicious code was injected into the web server portion of the BMC and was serving up the root filesystem for viewing, maybe to see /etc/passwd, /etc/shadow, /root or other sensitive locations that could help an attacker gain further access or implant malware. I follow good security practices, routinely audit my systems and networks, only install trusted software and keep things automatically updated, but it didn’t matter — somehow I had been compromised.

I quickly jumped to the Supermicro website to check the latest available BMC firmware and BIOS versions, along with the release notes. Both the BMC firmware and BIOS on my system weren’t the latest versions, but they also weren’t egregiously out of date. The latest firmware release notes mentioned a couple of CVEs that were addressed, but they didn’t seem to be applicable for my case as the BMC interface isn’t (or shouldn’t be!) exposed to the Internet. Supermicro also variously states that BMC firmware and the BIOS should not be updated unless there is a specific issue you are trying to solve, though the release notes list the updates as “Recommended,” so it’s not clear whether or not you should be applying updates as they roll out or only as needed to resolve specific issues.

Even worse, I didn’t have physical access to my server (and wouldn’t for several hours), so I couldn’t even intervene to unplug network cables or otherwise take manual actions on the hardware. The BMC/IPMI web interface wasn’t available (though the command-line IPMI tools seemed to be working) so at most I could shut down the machine remotely through the Proxmox interface, but that would remove my VPN access to the network as well as take down all the other services running on that machine.

After some more furious and panicked digging, I found that I wasn’t able to recreate the response I had gotten. Repeated calls to netcat (or ping, etc.) would just time out as host not available/responding. I figured that either the injected payload had gotten into a weird state with my manual requests, or even worse, the attacker had caught onto me and had started taking immediate actions.

Then, in a moment of curiousity, I tried running the same netcat commands on a different machine in a different network, just to see if I had found some strange quirk of netcat. Maybe it had a bug in it and was returning results from some random TCP buffer? Quite unlikely, but not impossible. I ran the commands again on a different machine in a different network, and to my surprise, received a very similar response.

What was going on?

Well, after some Googling and further experimentation, I learned of a GNU tool that has been around since before I had ever even touched a command prompt. Any guesses?

lwp-request, also usable through its aliases GET, POST and HEAD, is a simple command-line tool used to directly invoke simple web requests. I had no idea this tool existed nor that it has apparently been included in most Linux distributions since the mid 90’s.

You can use the GET command to make a simple web request like follows:

root@pve-host:~# GET google.ca
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-CA"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/logos/doodles/2023/seasonal-holidays-2023-6753651837110165-law.gif" itemprop="image"><meta content="Seasonal Holidays 2023" property="twitter:title">
...

What had happened was my initial netcat request did indeed time out as the BMC interface was not available on the network. The command prompt reappearing partway through typing the “GET / HTTP/1.1” request reflects that. The command prompt then interpreted that line as a separate command, invoking the lwp-request (or GET) tool on first “/”, returning a local directory listing per the documentation, and then a malformed web address, which resolved to some placeholder or parked website with a generic e-commerce template. Can you guess what the URL was? http.ca (yes, apparently that site exists).

Not helping things was the recent flurry of news regarding BMC attacks on Supermicro boards, including this recent Ars Technica article published in October of this year. It helped set the stage for the only possible explanation of the initial finding being compromise.

Aftermath

After calming my nerves from the ordeal, I actually am glad it happened. I learned of a new GNU tool and received a healthy dose of motivation to once again review my systems and network for best security practices. I will likely update the BMC firmware and BIOS anyway just for peace of mind. Thankfully, this time, the whole thing was a false alarm, but it also is a good reminder to prepare and practice a disaster recovery plan. You never know when you might need it.