Summary
The practice of some email service providers (ESPs) requiring clients to include unnecessary SPF records for custom DKIM signing is a concerning trend in email deliverability. This often stems from outdated documentation or a fundamental misunderstanding of how SPF and DKIM authentication protocols function independently. While both are critical for email security and inbox placement, their primary roles differ: SPF (Sender Policy Framework) authenticates the sending server's IP address against the domain specified in the Return-Path header (RFC 5321.MailFrom), whereas DKIM (DomainKeys Identified Mail) cryptographically signs the email message, verifying the sender's domain and ensuring content integrity. Forcing an SPF include when DKIM is handling authentication for the From domain can lead to critical issues, particularly the SPF 10-DNS-lookup limit, which can cause legitimate emails to fail authentication and be rejected or sent to spam.
Key findings
- Technical Misalignment: SPF and DKIM operate on different aspects of email authentication. SPF validates the sending IP, while DKIM validates the message's origin and integrity via a cryptographic signature. They are independent mechanisms.
- DNS Lookup Limit: Adding unnecessary SPF include mechanisms contributes to the SPF 10-DNS-lookup limit, potentially causing a PermError and invalidating the entire SPF record. More on this issue can be found in our guide on broken SPF records.
- Outdated Practices: Many ESPs continue to recommend these practices due to outdated internal documentation or a lack of understanding of modern email authentication standards.
- Reporting Inaccuracies: Some ESPs may report emails as 'delivered' even when they are rejected by recipient servers, masking underlying deliverability issues caused by misconfigurations.
Key considerations
- Prioritize DKIM for From Domain Alignment: For emails sent through an ESP, DKIM alignment with the RFC 5322.From domain is typically sufficient and often preferred for authentication. SPF is typically used for the Return-Path.
- Manage SPF Lookups: Avoid adding unnecessary include mechanisms to your SPF record to stay within the 10-DNS-lookup limit. Exceeding this limit renders your SPF record invalid, leading to authentication failures. Learn more about ESPs and common SPF errors.
- Challenge ESP Requirements: If an ESP demands a questionable SPF configuration, seek clarification or alternative solutions that align with best practices.
- Implement DMARC: A robust DMARC policy, alongside properly configured SPF and DKIM, is essential for monitoring and enforcing email authentication, even if SPF checks are misconfigured by an ESP.
What email marketers say
Email marketers frequently encounter challenges with ESPs regarding SPF and DKIM setup, often feeling caught between vendor requirements and optimal deliverability practices. Many express frustration over ESPs that provide confusing or technically unsound advice, particularly when it comes to SPF records. This leads to concerns about hitting SPF lookup limits, misrepresenting delivery statuses, and the overall technical competence of their service providers.
Key opinions
- Frustration with ESP Competence: Many marketers find themselves dealing with ESPs that appear technically incompetent, leading to unnecessary SPF requirements and misleading delivery reports (e.g., marking rejected emails as delivered).
- Needless SPF Includes: There is widespread disbelief when ESPs demand an SPF include for DKIM to function, recognizing this as a technically incorrect requirement. For specific examples, refer to our page on avoiding certain ESP SPF recommendations.
- Broad SPF Directives: Marketers dislike ESPs instructing them to include broad domains (e.g., include:everything.com) instead of specific IP addresses or more precise mechanisms, contributing to SPF lookup issues.
- DMARC Reporting Concerns: ESPs demanding DMARC records (even p=none) and then collecting all reports without client notification or clear instructions on policy enforcement is a significant concern.
Key considerations
- Advocate for Correct Setup: Marketers should push their ESPs for technically accurate SPF and DKIM configurations, especially regarding unnecessary SPF records. Consider our resources on how email authentication affects deliverability.
- Demand Accurate Reporting: Insist on transparent and accurate reporting of email delivery statuses to properly diagnose and address issues.
- Monitor DMARC Reports: Even if an ESP collects DMARC reports, marketers should ensure they have access to and understand these reports to maintain control over their domain's sending reputation.
- Educate Your Team: Empower your marketing team with knowledge about email authentication to better navigate technical discussions with ESPs and ensure compliant practices.
Marketer view
Email marketer from Email Geeks suggests that having to tell a client their ESP is technically incompetent is frustrating, especially when the ESP reports emails as 'delivered' that were actually rejected for spam. This discrepancy highlights a major issue with the ESP's reliability and reporting accuracy.
Marketer view
Email marketer from Reddit observes that many ESPs provide confusing or outdated instructions regarding SPF and DKIM setup, leading to unnecessary DNS record modifications. This adds complexity for senders trying to maintain good deliverability.
What the experts say
Email deliverability experts consistently point out the technical inaccuracies and potential pitfalls of ESPs requiring unnecessary SPF includes for DKIM. They delve into the historical context, such as past behaviors of major mailbox providers, and expose underlying reasons like outdated internal training or even attempts to inflate market share statistics. Experts stress that such practices not only create unnecessary DNS complexity but also expose senders to the risk of authentication failures and reduced deliverability.
Key opinions
- Historical Context: Experts note that past behaviors, such as Microsoft's temporary SPF lookups on the RFC 5321.MailFrom domain, may have influenced outdated ESP recommendations, though these issues are no longer prevalent.
- Ignorance and Outdated Training: A significant reason for these incorrect requirements is often a fundamental misunderstanding of email authentication within the ESPs, compounded by wrong or outdated internal training materials.
- Market Share Misrepresentation: One surprising (and 'awful') reason suggested for persistent SPF include demands is for ESPs to appear in industry reports measuring market share by analyzing SPF records of top domains.
- DMARC Policy Mismanagement: Experts also criticize ESPs that demand DMARC records (especially p=none) and then absorb all DMARC reports without properly informing clients or helping them move to stricter policies.
Key considerations
- Focus on Core Protocols: Ensure that SPF aligns the Return-Path domain and DKIM aligns the From domain. This is the foundation for proper email authentication. A basic explanation of these protocols is available in our simple guide to DMARC, SPF, and DKIM.
- Monitor DNS Lookups: Regularly check your SPF record for the number of DNS lookups to avoid exceeding the limit, which can lead to a PermError. Consider alternative SPF mechanisms to reduce lookups when possible, as discussed in our article on hidden SPF DNS timeouts at Microsoft.
- Educate ESPs (When Possible): Where feasible, engage with ESPs to highlight their outdated or incorrect guidance, advocating for better, technically sound instructions for their clients.
- Verify DMARC Reporting: Ensure you retain control over your DMARC reports or that your ESP transparently shares them and provides clear guidance on interpreting and acting on the data. For more on this, check out this guide on email authentication requirements.
Expert view
Expert from Email Geeks observes that Microsoft did indeed stop breaking SPF only to move on to breaking DKIM by modifying the body content of emails. This highlights a continuous challenge with major mailbox providers' implementation of email authentication standards.
Expert view
Expert from SpamResource clarifies that SPF and DKIM serve distinct purposes in email authentication; SPF validates the mail server, while DKIM authenticates the email content and sender's domain with a digital signature. Misunderstanding these roles can lead to improper configuration advice.
What the documentation says
Official documentation and technical specifications for SPF and DKIM clearly define their respective roles and operational mechanisms, highlighting why requiring SPF includes for DKIM authentication is technically unnecessary. These standards emphasize the importance of distinct authentication processes to verify different aspects of an email's origin and integrity. Understanding these foundational documents is key to identifying and rectifying misconfigurations perpetuated by some ESPs.
Key findings
- SPF's Purpose: RFC 7208 defines SPF as a mechanism for domain owners to publish which hosts are authorized to send mail from their domain, primarily checking the RFC 5321.MailFrom (Return-Path) address.
- DKIM's Purpose: RFC 6376 specifies DKIM as a method for cryptographically signing email messages, allowing recipient mail servers to verify that the message's content has not been tampered with and that the sender is authorized to use the signing domain (usually aligned with the RFC 5322.From address).
- Independence of Protocols: The specifications do not mandate that SPF and DKIM be directly linked for the purpose of primary domain authentication (e.g., DKIM signing of the From domain doesn't rely on SPF for that same domain).
- SPF Lookup Limit: The SPF specification explicitly warns about the 10-DNS-lookup limit, beyond which an SPF record evaluation results in a 'PermError' which can be detrimental to deliverability. Our guide on formatting SPF TXT records covers this in detail.
Key considerations
- Adhere to RFCs: Always prioritize adherence to the official RFC specifications for SPF and DKIM. These are the authoritative sources for proper implementation. L-Soft provides a good overview of these authentication standards.
- Optimize SPF Records: Design your SPF records to be efficient and avoid unnecessary lookups. If multiple ESPs are used, consider strategies like SPF flattening or careful consolidation of includes.
- Leverage DMARC for Comprehensive Feedback: DMARC leverages both SPF and DKIM results to provide aggregate and forensic reports, offering crucial insights into authentication failures, regardless of underlying ESP configuration issues. Abusix details these protocols.
- Educate Yourself: Understanding the technical basis of email authentication protocols empowers senders to identify and challenge incorrect guidance from their ESPs, protecting their deliverability.
Technical article
RFC 7208 states that SPF is used to detect spoofing by verifying the IP address of the sender against a list of authorized sending hosts for a domain. It explicitly outlines how SPF authenticates the Return-Path domain.
Technical article
RFC 6376 (DKIM) outlines that DKIM allows an organization to associate a domain name with an email message by cryptographically signing it, ensuring content integrity and sender identity independent of the mail path. This mechanism focuses on the message content and header fields.
Related resources
6 resources
Related pages
How do broken SPF records, like those with too many DNS lookups or exceeding size limits, affect email deliverability and authentication?
How to format SPF TXT records, add domain includes, and avoid DNS size issues?
Why do some ESPs recommend SPF records when they are not needed?
What are best practices and costs for implementing DKIM, SPF, and DMARC?
Why you shouldn't add MailChimp to your SPF record
Why your emails fail at Microsoft: the hidden SPF DNS timeout
A simple guide to DMARC, SPF, and DKIM
How do SPF, DKIM, DMARC, and dedicated IPs affect email deliverability when using a third-party ESP?
Why is DKIM failing at some ISPs but not others, and how can I fix it?
Why are ESPs enforcing DMARC policies and what are the implications?
