The short answer is yes, MTA-STS absolutely uses a Public Key Infrastructure (PKI) for validation, but it's important to understand precisely how. It doesn't use a dedicated PKI with keys you publish in your DNS, like DKIM does. Instead, it cleverly leverages the existing and widely trusted Web PKI, the same system that secures your connection to websites with HTTPS.
At its core, Mail Transfer Agent Strict Transport Security (MTA-STS) is a standard designed to ensure that emails are sent over an encrypted connection. It helps prevent attackers from intercepting and reading or modifying emails in transit. It does this by allowing a domain to publish a policy that states it will only accept email over a secure Transport Layer Security (TLS) connection.
How MTA-STS leverages the web PKI
MTA-STS relies on the Web PKI in two critical ways. First, the MTA-STS policy itself isn't just a DNS record. It's a text file that you must host on a web server under a specific subdomain, like mta-sts.yourdomain.com. Crucially, this web server must be secured with a valid TLS certificate from a trusted public Certificate Authority (CA). When a sending mail server looks up your policy, it first validates the TLS certificate of your web server. This ensures that the policy it's retrieving is authentic and hasn't been tampered with by an attacker.
Second, and most importantly, the policy file contains a list of your mail server hostnames (your MX records). When a sending server initiates a connection to one of your mail servers, it expects that server to present a valid TLS certificate that matches the hostname. The sending server validates this certificate against the same public trust store used by web browsers. If the certificate is invalid, expired, or for the wrong domain, the connection is dropped, preventing the email from being sent to a potential imposter.
Is this different from DKIM's use of public keys?
Yes, this is fundamentally different from how DKIM works. DKIM (DomainKeys Identified Mail) uses a cryptographic key pair for a different purpose: message authentication. With DKIM, you use a private key to create a unique digital signature based on the content of the email, and this signature is added to the email's headers. The corresponding public key is published in a DNS TXT record.
A receiving server can then use the public key from DNS to verify the signature, proving that the email hasn't been altered in transit and that it genuinely originated from a server authorized to send for that domain. It's about validating the message itself, not the connection.
- MTA-STS: Secures the connection (the transport layer) and validates the identity of the mail server using Web PKI.
- DKIM: Signs the email message (the application layer) to verify its content and origin using its own key pair in DNS.
- Validation Target: MTA-STS validates the server's TLS certificate. DKIM validates the email's signature.
The practical implications for email security
The main benefit of this approach is that MTA-STS protects against SMTP downgrade attacks. Before MTA-STS, email encryption was often "opportunistic". Servers would try to use an encrypted connection (via the STARTTLS command), but an attacker could intercept the communication, strip out the STARTTLS command, and force the servers to communicate in plaintext. With an active MTA-STS policy, the sending server knows encryption is mandatory and will refuse to connect if it can't be established.
By relying on the Web PKI to validate certificates, MTA-STS also thwarts man-in-the-middle (MITM) attacks where an attacker reroutes your email to a malicious server. The attacker's server won't have a valid TLS certificate for your mail domain, so the sending server will detect the forgery and terminate the connection.
In conclusion, MTA-STS is fundamentally dependent on Public Key Infrastructure for its validation process. It doesn't require you to manage your own signing keys, but instead cleverly piggybacks on the robust and universally trusted PKI that powers the secure web. This makes it a powerful and accessible tool for hardening the security of email in transit.
