How to fix SPF record exceeding DNS lookup limit?

Key findings

• Nested includes: Even with a small number of direct include mechanisms, the limit can be exceeded if those included domains themselves have complex SPF records with multiple nested lookups.
• Impact on deliverability: Exceeding the 10-lookup limit results in an SPF PermError, causing emails to be rejected or sent to spam, significantly affecting email deliverability. For more details, see our guide on how broken SPF records affect deliverability.
• Standard compliance: The 10-lookup limit is a fundamental requirement outlined in RFC 7208, Section 4.6.4, ensuring efficient processing of SPF records.
• Redundancy: Using both include mechanisms and direct IP addresses for the same sending service can be redundant and contribute to unnecessary lookups.

Key considerations

• Record audit: Regularly review your SPF record to identify and remove unnecessary include mechanisms, especially for services no longer in use for email sending from that domain.
• Consolidation: Where possible, consolidate multiple include statements or use IP addresses directly if the number of IPs is small and stable. Our article on options for overstuffed SPF records provides further strategies.
• Subdomain strategy: Consider creating subdomains for different email sending services, each with its own specific SPF record, to distribute the lookup burden.
• SPF flattening: Utilize SPF flattening services or manual methods to replace domain-based include mechanisms with direct IP addresses, effectively reducing DNS lookups to one for the SPF record itself.

Key opinions

• Hidden lookups: Marketers frequently discover that even a small number of include statements can lead to exceeding the 10-lookup limit due to nested lookups within those included domains. Some ESPs contribute significantly to this. Setting up SPF with multiple ESPs can be complex.
• Service consolidation: It's often found that certain services like Mailchimp or HubSpot do not require direct SPF includes if emails are sent via their own domains, reducing the pressure on the primary domain's SPF record.
• CNAME for delegation: Using a CNAME record for email sending services, such as SendGrid, can help mitigate the lookup issue by shifting the SPF validation to a subdomain or the service provider's domain. More on this process can be found in SendGrid's domain authentication documentation.
• IP vs. include: There's a debate among marketers whether to list direct IPs or rely on includes. Many agree that direct IPs can be better for static infrastructure but are impractical for dynamic cloud-based ESPs.

Key considerations

• Identify primary senders: Focus on including only the domains of services that directly send emails on behalf of your primary domain to minimize unnecessary lookups.
• Use subdomains: Implement separate subdomains for different email sending purposes or ESPs. This approach allows each subdomain to have its own SPF record, preventing the main domain from hitting the lookup limit. Our guide on troubleshooting SPF authentication with subdomains may be useful.
• Monitor changes: Be vigilant about changes from ESPs that might affect their SPF records and, consequently, your lookup count. Unexpected additions can lead to immediate issues.
• Leverage tools: Utilize online SPF lookup tools to quickly check your SPF record's lookup count and identify problematic includes. This proactive approach helps in maintaining compliance.

Key opinions

• Nested complexity: Experts agree that nested DNS lookups are the primary cause of exceeding the SPF limit, even with a seemingly short SPF record. Each include can hide multiple additional lookups.
• Redundancy avoidance: It is redundant to list both include mechanisms and direct IP addresses for the same services; this practice unnecessarily consumes lookups.
• Tool utilization: Using dedicated SPF lookup tools, like the one offered by dmarcian.com, is crucial for accurately counting all lookups (including nested ones) and identifying issues.
• Best practices for DNS lookups: Experts stress the importance of carefully managing DNS records, including SPF, to ensure optimal email deliverability and authentication. Read more about best practices for DNS lookups and SPF records.

Key considerations

• Aggressive ESP records: Some ESPs publish SPF records that contain many includes, making it challenging for customers to stay within the limit when also including other services.
• SPF flattening caution: While SPF flattening can resolve lookup issues, experts advise caution as it can lead to SPF records exceeding the DNS TXT record length limit, which is a separate but related issue. For comprehensive solutions, see our guide on options for overstuffed SPF records.
• Subdomain benefits: Utilizing subdomains for different sending purposes (e.g., marketing.example.com, transactional.example.com) each with its own SPF record, is a robust strategy to manage lookup counts.
• Ongoing vigilance: Due to the dynamic nature of ESP infrastructure, SPF records require periodic review to ensure they remain compliant and do not suddenly exceed lookup limits.

Key findings

• Hard limit: The SPF specification (RFC 7208) imposes a strict limit of 10 DNS lookups. This is a technical boundary, not a recommendation, and exceeding it causes authentication failure.
• Mechanism count: Mechanisms like include, a, mx, ptr, and exists each count towards this limit. Other mechanisms like ip4, ip6, and all do not.
• PermError: If the DNS lookup limit is exceeded, the SPF evaluation results in a PermError, which instructs receiving mail servers to treat the SPF record as invalid.
• CNAME impact: Using a CNAME record at the root domain level for SPF can cause issues, as SPF expects an A or MX record. While CNAMEs are often used for subdomains, their interaction with SPF records for the root domain can be problematic. Our article Why does SPF resolution fail with CNAME records explains this further.

Key considerations

• Recursive nature: Understand that the 10-lookup limit applies to the total number of DNS lookups performed during the SPF evaluation, including any nested lookups triggered by include or mx mechanisms.
• TXT record length: While distinct from the lookup limit, the overall length of the SPF TXT record can also cause issues if it exceeds DNS provider limits or the maximum TXT record size. This is particularly relevant when attempting SPF flattening. See our discussion on managing DNS TXT record length limits.
• Performance impact: The limit is in place to prevent performance degradation and potential abuse on DNS servers during email validation, as explained by Cloudflare's documentation on DNS lookup limits.
• Regular validation: It's imperative to regularly validate your SPF record to ensure it complies with the 10-lookup limit and avoids PermErrors as service configurations may change.