The short answer is no, Mail Transfer Agent Strict Transport Security (MTA-STS) does not require a specific root certificate authority (CA). However, it absolutely requires that your TLS certificates are issued by a publicly trusted certificate authority. This is a crucial distinction that often causes confusion.
Essentially, the system is built on the same foundation of trust that underpins HTTPS on the web. A sending mail server needs to be able to verify your server's identity, and it does this by checking the certificate against its own list of trusted CAs. If your certificate is from an unknown or private authority, the check will fail.
A publicly trusted Certificate Authority is an organization that is recognized and accepted by major operating systems, browsers, and other software. When you visit a secure website with https://, your browser automatically trusts the site's certificate because it was signed by a CA in its pre-installed trust store. Companies like Let's Encrypt, DigiCert, and GlobalSign are all examples of publicly trusted CAs.
MTA-STS works the same way. The sending mail server acts like a web browser, checking to see if your mail server's certificate is valid and issued by a known, trustworthy source. This prevents attackers from inserting their own fraudulent certificate to intercept your email, which is a primary goal of MTA-STS.
Using a self-signed certificate or one from a private, internal CA is a common point of failure when implementing MTA-STS. While such a certificate might be trusted within your own organization's network, it is completely unknown to the rest of the world.
When an external mail server like Gmail or Outlook attempts to validate your MTA-STS policy, it will see your certificate and check its signature against its list of trusted root CAs. Since your private CA isn't on that list, the validation will fail. As one forum post about MTA-STS failures with Outlook.com noted, an "untrusted root" is a direct cause of these issues. The sender's system has no reason to trust you, so it will likely fall back to sending over an unencrypted connection, or if your policy is set to enforce, the email may not be delivered at all.
So, while you don't need a specific CA, you do need a certificate that meets several criteria. Any certificate from a major public CA will work, including free ones from services like Let's Encrypt.
In conclusion, the focus shouldn't be on finding one specific CA for MTA-STS. Instead, ensure that any certificate you use for your mail servers and your MTA-STS policy web host is obtained from any well-regarded, public Certificate Authority. This leverages the global trust system that already secures the web and is the key to a successful MTA-STS implementation.