What happened when I emailed 3000 Mastodon admins about a critical vulnerability
As the fediverse and Mastodon grow in users and activity, so does the attention and the risk. This week Mastodon developers released a critical upgrade to address a recent security audit.
As the word got out about the upcoming security release, the community rushed like never before to get everyone safe. Inspired by the awesome Claire, who contacted many instances and even patched custom codebases to keep users safe, I wondered how to help. I learnt that instances from the joinmastodon.org list had been contacted, and someone was pinging admins with a bot. I upgraded my own instance quite soon thanks to a message by Michael Downey, but I know I would have liked to receive an email about it, so I did just that: I started emailing the admins of outdated instances.
The response has been amazing: the fediverse is a great community! As we learn to work better together, we can build something great. I’ve been on the ~fediverse for almost 15 years, since the early days of identi.ca, so I know that in the future I’ll need some reminder of the good times.
What they said
I’ve only received positive comments, several dozens of them. Some made my day:
- «Thank you for alerting me to the vulnerability CVE-2023-36460 in my instance. Your valuable input led to its prompt upgrade. While such updates aren’t automated yet, your message highlighted their importance. Your assistance is deeply appreciated.»
- «I wanted to reach out to say thanks for the outreach. The open web doesn’t survive without all the people who put in the real work on a day-to-day basis.»
- «Hello, passing along my thanks for the note regarding the recent Mastodon security update! You are truly looking out for the community».
- «Yay! I am now updated. Funnily enough, just after finishing the update, I […] saw an email from @nemobis prompting me to update because of the vulnerability - thanks for your care for the community!»
- «I really appreciate the mail. Luckily I patched my server just 40 minutes after the patch was released, but I just wanted to reach out to say I really respect you (auto-)mailing all admins regarding this vulnerability. We need more people like you in this world.»
- «Anywho, thanks for being passionate enough and actually explaining what it was. It should be updated now! Thank you for saving my bacon. :D»
- «This is the nicest and most thorough patch advisory I’ve ever read, and I’ve been maintaining community sites for almost 25 years. […] Regardless, thanks for going above and beyond, fedi-friend.»
- «Thanks for letting me know. You’re a treasure to the community.»
- «Thanks for letting me know about this, I have subscribed to @mastodon@mastodon.social to be notified about future updates. I have donated 10 euros to FSFE to show my appreciation of this act of kindness 😊».
- «Thanks so much for letting me know about this. I have gone ahead and updated the version of Mastodon I was running. Thank you once again for your service to the fediverse with this email!»
- «Thank you for your email. We’ve promptly updated our Mastodon instance at […] to version 4.1.4, addressing the vulnerabilities you mentioned. Your input has been invaluable in ensuring our users’ safety. We’ll continue exploring the resources you’ve shared for additional support in managing our instance. Your continued advice and updates are always welcome.»
Not a bad return for a few hours of work over the weekend.
A couple replies were rightly confused/annoyed because I made a silly mistake in the secound round of emails and I forgot to exclude the versions 4.0.5 and 4.0.6 (including many 4.0.6+hometown-1.1.1), as I mixed them up with an earlier glitch-soc version (see explanation of the forks).
Overall I can’t measure what impact the emails had on the speed of the upgrades. In the 12 hours after my first email, outdated instances decreased by about 10 % overall and 15 % across those I had notified (the bigger ones, probably more alert than average). In 24 hours after my second batch of emails, over Sunday, they decreased by a further 20 %. I’m pretty sure the notifications didn’t hurt, though, based on the replies.
What I wrote
The email itself was cobbled together in perhaps half an hour, though I relied on extensive previous research. From previous experience doing this for MediaWiki, I knew it had to satisfy a few requirements:
- Be short, have a clear subject, minimise links.
- Be positive: the recipients are good people doing their best for the common good.
- Be clear about the urgency/risks, but avoid inducing panic: we can handle it.
- Be personal: include the recipient name, domain, specific details showing the direct relevance of the message to them. (In the end I didn’t do this because I just went for bcc emails, see below.)
- Provide resources so that people can find help without me: I’m not trying to sell support services and I have no idea whether I’ll manage to reply to any replies.
I also avoided directly recommending any proprietary service like GitHub, although that’s a bit tricky since that’s where Mastodon’s official repository is hosted. There are many forks and distributions, so it was impossible to link them all anyway.
Email text
Subject: Critical vulnerability in Mastodon: please upgrade to 3.5.9+, 4.0.5+, or 4.1.3+
Dear Mastodon admin, thank you for offering a space in the fediverse to your users!
I noticed that your instance states it’s running on an old version of Mastodon which is affected by some severe vulnerabilities, just patched in this week’s release. Please upgrade as soon as possible.
You can find instructions in the release notes for the new version of the installation method or fork you use. See also the documentation: https://docs.joinmastodon.org/user/run-your-own/
An explanation is also available from Bill Toulas: «The most severe of the vulnerabilities is tracked as CVE-2023-36460 and has been named TootRoot. It gives attackers a particularly easy way to compromise target servers.
CVE-2023-36460 is a problem in Mastodon’s media processing code that allows using media files on toots (the equivalent of tweets) to cause a range of problems, from denial of service (DoS) to arbitrary remote code execution.
Although Mastodon’s security bulletin is laconic, security researcher Kevin Beaumont highlighted the risks associated with TootRoot, saying that a toot can be used to plant backdoors on the servers that deliver content to Mastodon users.»
If running your instance has become overwhelming, I understand! You may be interested in some links on how to get social, financial and technical support (including managed hosting): https://codeberg.org/fediverse/fediparty/wiki/Collectively-owned-instances#resources-on-running-an-instance
You can also ask questions and discuss with other admins on the #mastodon IRC channel on LiberaChat or the Matrix channel #mastodon_moderation:matrix.org and a “mastodon-admin” mailing list is available.
Best regards,
Federico (@nemobis@mamot.fr) (purely personal capacity)
How I compiled the data
I’d normally write a little Python script, but I decided to build on a previous bash one-liner to minimise surprises. I wanted to:
- get the list of domains from instances.social, so people can ask to be removed there if they want;
- use the Mastodon instance API with a simple combination of xargs, curl and jq, so that it can finish in a few minutes even if some hosts take forever to respond;
- produce a CSV I can use in Thunderbird, so that I can manually edit the less plausible results (and the main obfuscated email addresses).
The main ingredient was just a bunch of curl and jq calls:
curl --header "Authorization: Bearer $token" 'https://instances.social/api/1.0/instances/list?count=0&min_users=1&max_users=1000000&min_active_users=1' | jq -r '.[] | .[] | .name' | xargs -P50 -I§ sh -c "curl -s --connect-timeout 3 https://§/api/v1/instance | jq -er '[.email, .uri, .stats.user_count, .version ] | @csv'" 2>/dev/null > 2023-07-08T23_instances_min1.csv
I then manually sorted the CSV in LibreOffice to remove up-to-date versions, unlikely admin names, placeholder addresses etc. With a mix of spreadsheet functions, grep, cut and comm, I’ve removed the instances from joinmastodon and those I had already emailed, then I manually corrected the more obvious email obfuscations. Finally, I checked for domains without MX records, which would obviously not work, and removed these too (there were over 100):
cat 2023-07-08T23_instances_min1_filtered_nojoinmastodon_emails_notsent_edited_domains.txt | xargs -P10 -I§ sh -c 'MX=`dig +noall +answer +short MX §`; if [ -z "$MX" ]; then echo § ; fi'
The existence of an MX record is no guarantee of delivery. One curious case is a domain which gets email to Microsoft even though Microsoft doesn’t recognise it as customer: fun times for GDPR lawyers?
Finally, before contacting random hosts on the internet it’s better to spread the load, so I used shuf to randomise the order of the email addresses.
A lot of the manual work could be avoided in the future if I had kept track of the individual corrections in a script, but that’s the catch: I do not plan to do this on a regular basis. We know there are better ways of handling such security announcements, though it takes work. In MediaWiki we’ve always had a mediawiki-announce mailing list, but outdated wikis in the wild remain legion; recently it has helped a lot to have Debian packages with Debian security advisories.
How I sent the email
The proper way would be to set up a mailing list, even with a simple local mailman (or PHPlist or Sympa?) installation, at least to handle bounces and ideally to allow updating the information on recipients. In the past I’ve also used Thunderbird Mail Merge. With Mail Merge it’s easy to keep a local copy of each outgoing email, thread the answers etc., but in this case I didn’t want to.
On Friday I decided to start from ~500 outdated instances with over 10 active users, because unlike single-person instances they may be run by people who update on weekdays. 500 recipients can fit a single email in Bcc, so I decided to use the fsfe.org email address and mailserver that comes as perk for supporting FSFE. The first round went well, with some 6 % connection errors and bounces, so I just repeated this simplistic approach in the later rounds.
In the end I sent 443 emails on Friday night (2023-07-07T23:04:00Z) and 2805 more about 24 hours later, to all known instances with at least 1 active user. Perhaps 6 % were false positives due to my mistakes. So far I got over 100 replies and sent over 70 follow-up emails, while ~30 addresses bounced and ~60 mailservers didn’t accept email. I don’t know how many messages were actually delivered and how many went to spam, but a less than 3 % bounce rate is better than I expected.
As far as I know, the FSFE mailserver wasn’t blocked (errors contained one mention of spamhaus, but the IP address was deemed clear by spamhaus). People appreciated the help and I know at least two donations were made to FSFE and one to Mastodon gGmbH.