Skip to content

Commit

Permalink
fix: updated MX codebase to do rudimentary bounce detection/greylisting
Browse files Browse the repository at this point in the history
  • Loading branch information
@titanism
titanism committed Dec 5, 2024
1 parent eab90cd commit cfb020b
Show file tree
Hide file tree
Showing 11 changed files with 615 additions and 113 deletions.
20 changes: 17 additions & 3 deletions app/views/faq/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3681,13 +3681,27 @@ We routinely monitor our [IP addresses](#what-are-your-servers-ip-addresses) aga
You can try to use one or more of these tools to check your domain's reputation and categorization:

* [Cloudflare Domain Categorization Feedback](https://radar.cloudflare.com/domains/feedback)
* [Spamhaus IP and Domain Reputation Checker](https://check.spamhaus.org/)
* [Spamhaus IP and Domain Reputation Checker](https://check.spamhaus.org/) (DNSBL)
* [Cisco Talos IP and Domain Reputation Center](https://talosintelligence.com/reputation_center)
* [Barracuda IP and Domain Reputation Lookup](https://www.barracudacentral.org/lookups/lookup-reputation)
* [Barracuda IP and Domain Reputation Lookup](https://www.barracudacentral.org/lookups/lookup-reputation) (DNSBL)
* [MX Toolbox Blacklist Check](https://mxtoolbox.com/blacklists.aspx)
* [Google Postmaster Tools](https://www.gmail.com/postmaster/)
* [Yahoo Sender Hub](https://senders.yahooinc.com/) (includes Verizon/AOL)
* [MultiRBL.valli.org Blacklist Check](https://multirbl.valli.org/lookup/)
* [MultiRBL.valli.org Blacklist Check](https://multirbl.valli.org/lookup/) (DNSBL)
* [Sender Score](https://senderscore.org/act/blocklist-remover/)
* [Invaluement](https://www.invaluement.com/lookup/) (DNSBL)
* [SURBL](https://www.surbl.org/) (DNSBL)
* [Apple/Proofpoint IP removal](https://ipcheck.proofpoint.com/)
* [Cloudmark IP removal](https://csi.cloudmark.com/en/reset/)
* [SpamCop](https://www.spamcop.net/bl.shtml) (DNSBL)
* [Microsoft Outlook and Office 365 IP removal](https://sendersupport.olc.protection.outlook.com/pm/Postmaster) – also see their sender portal at <https://sendersupport.olc.protection.outlook.com/pm/Postmaster>
* [UCEPROTECT's Levels 1, 2, and 3](https://www.uceprotect.net/en/rblcheck.php) (DNSBL)
* [UCEPROTECT's backscatterer.org](https://www.backscatterer.org/) (please read usage; it's not a spammer/DNSBL, it is used by some mail servers for protection against open relays and misdirected bounces – also known as "backscatter")
* [UCEPROTECT's whitelisted.org](https://www.whitelisted.org/) (requires a fee)
* AT\&T includes `[email protected]` in SMTP error messages, which is the best address to email to request removal
* AOL/Verizon includes `[IPTS04]` in SMTP error messages, which indicates you need to submit removal request form at <https://senders.yahooinc.com/>
* Cox Communications includes `[email protected]` in SMTP error messages, but they are merging with Yahoo mail and this will be deprecated
* t-online.de (German/T-Mobile) includes `[email protected]` in SMTP error messages, but they have not been reliable per our removal requests in the past

If you need additional help or find that we are false-positive listed as spam by a certain email service provider, then please <a href="/help">contact us</a>.

Expand Down
220 changes: 152 additions & 68 deletions helpers/get-attributes.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,90 +3,174 @@
* SPDX-License-Identifier: BUSL-1.1
*/

const punycode = require('node:punycode');
const os = require('node:os');

const _ = require('lodash');
const isSANB = require('is-string-and-not-blank');
const { spf } = require('mailauth/lib/spf');

const checkSRS = require('#helpers/check-srs');
const getHeaders = require('#helpers/get-headers');
const logger = require('#helpers/logger');
const parseAddresses = require('#helpers/parse-addresses');
const parseHostFromDomainOrAddress = require('#helpers/parse-host-from-domain-or-address');
const parseRootDomain = require('#helpers/parse-root-domain');
const parseAddresses = require('#helpers/parse-addresses');

function getAttributes(headers, session) {
const HOSTNAME = os.hostname();

//
// NOTE: the `isAligned` option can be set to `true` if we want to
// only return metadata that is verified and aligned
// (e.g. a MAIL FROM that has aligned SPF or DKIM)
// (e.g. a From header that has aligned SPF or DKIM)
// (e.g. a Reply-To header that has aligned SPF or DKIM)
//
async function getAttributes(headers, session, isAligned = false) {
const replyToAddresses = parseAddresses(getHeaders(headers, 'reply-to'));

//
// check if the From, Reply-To, MAIL FROM, sender IP/host, or RCPT TO were silent banned
// (and filter out RCPT TO while parsing so we don't send twice)
// (and also check root domains as well for each of these)
//
return _.uniq(
_.compact(
[
// NOTE: we don't check HELO because it's arbitrary
// check the From header
session.originalFromAddress,
// check the From header domains
parseHostFromDomainOrAddress(session.originalFromAddress),
// check the From header root domains
parseRootDomain(
parseHostFromDomainOrAddress(session.originalFromAddress)
),
// check the Reply-To header
...replyToAddresses.map((addr) => checkSRS(addr.address).toLowerCase()),
// check the Reply-To header domains
...replyToAddresses.map((addr) =>
parseHostFromDomainOrAddress(checkSRS(addr.address))
),
// check the Reply-To header root domains
...replyToAddresses.map((addr) =>
parseRootDomain(parseHostFromDomainOrAddress(checkSRS(addr.address)))
),
// check the sender client host (if provided)
// (only applicable if not allowlisted)
!session.isAllowlisted && session.resolvedClientHostname
? session.resolvedClientHostname
: null,
// check the sender client host root (if provided)
// (only applicable if not allowlisted)
// (but only if the root domain was not equal to the parsed host)
!session.isAllowlisted &&
session.resolvedClientHostname &&
session.resolvedClientHostname !== session.resolvedRootClientHostname
? session.resolvedRootClientHostname
: null,
// check the sender client IP address
// (only applicable if not allowlisted)
session.isAllowlisted ? null : session.remoteAddress,
// check the MAIL FROM (if provided; lowercased)
isSANB(session.envelope.mailFrom.address)
? checkSRS(session.envelope.mailFrom.address).toLowerCase()
: null,
// check the MAIL FROM host (if provided; lowercased)
isSANB(session.envelope.mailFrom.address)
? parseHostFromDomainOrAddress(
checkSRS(session.envelope.mailFrom.address).toLowerCase()
)
: null,
// check the MAIL FROM host root (if provided; lowercased)
// (but only if the root domain was not equal to the parsed host)
isSANB(session.envelope.mailFrom.address) &&
parseRootDomain(
parseHostFromDomainOrAddress(
checkSRS(session.envelope.mailFrom.address).toLowerCase()
)
) !==
// NOTE: we don't check HELO command input because it's arbitrary and can be spoofed

const arr = [
session.resolvedClientHostname,
session.resolvedRootClientHostname,
session.remoteAddress
];

const from = [
// check the From header
session.originalFromAddress,
// check the From header domains
session.originalFromAddressDomain,
// check the From header root domains
session.originalFromAddressRootDomain
];

const replyTo = [
// check the Reply-To header
...replyToAddresses.map((addr) => checkSRS(addr.address).toLowerCase()),
// check the Reply-To header domains
...replyToAddresses.map((addr) =>
parseHostFromDomainOrAddress(checkSRS(addr.address))
),
// check the Reply-To header root domains
...replyToAddresses.map((addr) =>
parseRootDomain(parseHostFromDomainOrAddress(checkSRS(addr.address)))
)
];

const mailFrom = [];
if (isSANB(session.envelope.mailFrom.address)) {
mailFrom.push(
// check the MAIL FROM (if provided; lowercased)
checkSRS(session.envelope.mailFrom.address).toLowerCase(),
// check the MAIL FROM host (if provided; lowercased)
parseHostFromDomainOrAddress(checkSRS(session.envelope.mailFrom.address)),
// check the MAIL FROM host root (if provided; lowercased)
// (but only if the root domain was not equal to the parsed host)
parseRootDomain(
parseHostFromDomainOrAddress(
checkSRS(session.envelope.mailFrom.address)
)
)
);
}

if (isAligned) {
//
// if From header has SPF pass (or) DKIM alignment then push it
// <https://github.com/postalsys/mailauth/blob/41b8e03207fa175d3bc8998ed13e2ca40ac793f2/lib/spf/index.js#L214-L252>
//
if (
session?.spfFromHeader?.status?.result === 'pass' ||
(session?.signingDomains?.size > 0 &&
(session.signingDomains.has(session.originalFromAddressDomain) ||
session.signingDomains.has(session.originalFromAddressRootDomain)))
)
arr.push(...from);

//
// NOTE: it's typically bad practice to include multiple reply-to values, but we generalize this for simplicity
// if any Reply-To has SPF pass (or) DKIM alignment then push _all_ of the addresses
//
let hasAlignedReplyTo = false;
for (const sender of replyToAddresses) {
if (
session?.signingDomains?.size > 0 &&
(session.signingDomains.has(
parseHostFromDomainOrAddress(checkSRS(sender))
) ||
session.signingDomains.has(
parseRootDomain(parseHostFromDomainOrAddress(checkSRS(sender)))
))
) {
hasAlignedReplyTo = true;
break;
}

try {
// eslint-disable-next-line no-await-in-loop
const result = await spf({
ip: session.remoteAddress,
helo: session.hostNameAppearsAs,
mta: HOSTNAME,
resolver: this.resolver.resolve,
sender: checkSRS(sender).toLowerCase()
});
if (result?.status?.result === 'pass') {
hasAlignedReplyTo = true;
break;
}
} catch (err) {
logger.warn(err);
}
}

if (hasAlignedReplyTo) arr.push(...replyTo);

// if MAIL FROM has SPF pass|neutral|none (or) DKIM alignment then push it
if (isSANB(session.envelope.mailFrom.address)) {
if (
session?.signingDomains?.size > 0 &&
(session.signingDomains.has(
parseHostFromDomainOrAddress(
checkSRS(session.envelope.mailFrom.address).toLowerCase()
checkSRS(session.envelope.mailFrom.address)
)
? parseRootDomain(
) ||
session.signingDomains.has(
parseRootDomain(
parseHostFromDomainOrAddress(
checkSRS(session.envelope.mailFrom.address).toLowerCase()
checkSRS(session.envelope.mailFrom.address)
)
)
))
) {
arr.push(...mailFrom);
} else {
try {
const result = await spf({
ip: session.remoteAddress,
helo: session.hostNameAppearsAs,
mta: HOSTNAME,
resolver: this.resolver.resolve,
sender: checkSRS(session.envelope.mailFrom.address).toLowerCase()
});
if (result?.status?.result === 'pass') arr.push(...mailFrom);
} catch (err) {
logger.warn(err);
}
}
}
} else {
arr.push(...from, ...replyTo, ...mailFrom);
}

return _.uniq(
_.compact(
arr.map((str) =>
typeof str === 'string'
? punycode.toASCII(str).toLowerCase().trim()
: null
].map((str) =>
typeof str === 'string' ? str.toLowerCase().trim() : null
)
)
);
Expand Down
81 changes: 81 additions & 0 deletions helpers/get-bounce-info.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@ const REGEX_DENYLIST = new RE2(/denylist|deny\s+list/im);
const REGEX_BLACKLIST = new RE2(/blacklist|black\s+list/im);
const REGEX_BLOCKLIST = new RE2(/blocklist|block\s+list/im);

//
// NOTE: we have access to `err.truthSource` if needed here
// (e.g. google.com, qq.com, is the value)
// but note we don't need to rely on it because we only
// take action if a given `truthSource` value actually exists
//

// eslint-disable-next-line complexity
function getBounceInfo(err) {
//
Expand Down Expand Up @@ -82,6 +89,80 @@ function getBounceInfo(err) {
' ; Resolve this issue by visiting https://learn.microsoft.com/en-us/exchange/troubleshoot/email-delivery/configure-proofpoint-with-exchange#specify-a-limit-for-the-number-of-messages-per-connection ;';
bounceInfo.action = 'defer';
bounceInfo.category = 'network';
} else if (err.truthSource === '163.com' && response.includes('DT:SPM')) {
bounceInfo.category = 'spam';
} else if (err.truthSource === 'orange.fr' && response.includes('[506]')) {
// <https://github.com/sisimai/p5-Sisimai/issues/243>
bounceInfo.category = 'spam';
} else if (
err.truthSource === 'qq.com' &&
(response.includes('mailbox unavailable') ||
response.includes('Access denied') ||
(response.includes('550 recipient') && response.includes('denied')))
) {
bounceInfo.category = 'recipient';
} else if (err.truthSource && response.includes('Too many emails')) {
bounceInfo.category = 'greylist';
} else if (
err.truthSource === 'qq.com' &&
response.includes('Suspected bounce attacks')
) {
bounceInfo.category = 'spam';
} else if (
err.truthSource === 'qq.com' &&
response.includes('Mail is rejected by recipients')
) {
bounceInfo.category = 'blocklist';
} else if (
err.truthSource === 'yahoodns.net' &&
response.includes('mailbox is disabled')
) {
bounceInfo.category = 'recipient';
} else if (
err.truthSource === 'secureserver.net' &&
response.includes('judged to be spam')
) {
bounceInfo.category = 'spam';
} else if (
err.truthSource === 'secureserver.net' &&
response.includes('Recipient not found')
) {
bounceInfo.category = 'recipient';
} else if (
err.truthSource === 'synchronoss.net' &&
response.includes('Resources restricted')
) {
bounceInfo.category = 'blocklist';
} else if (
err.truthSource === 'yandex.net' &&
response.includes('No such user')
) {
bounceInfo.category = 'recipient';
} else if (
response.includes('RFC') &&
(response.includes('compliance') || response.includes('complaint'))
) {
bounceInfo.category = 'spam';
} else if (
err.truthSource === 'google.com' &&
response.includes('this message exceeded its quota') &&
response.includes('messages with the same Message-ID')
) {
// Gmail has detected this message exceeded its quota for sending
// ...
// messages with the same Message-ID
bounceInfo.category = 'spam';
} else if (
err.truthSource === 'google.com' &&
response.includes('originating from your DKIM') &&
response.includes('temporarily rate limited')
) {
bounceInfo.category = 'spam';
} else if (
err.truthSource === 'google.com' &&
response.includes('very low reputation')
) {
bounceInfo.category = 'spam';
} else if (
response.includes('Your IP subnet has been temporarily deferred')
) {
Expand Down
5 changes: 5 additions & 0 deletions helpers/get-error-code.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ function getErrorCode(err) {
return 550;
}

//
// TODO: we should probably not do this property assignment here
// and instead manually do it elsewhere and simply just
// have `const bounceInfo = getBounceInfo(err)` below
//
// get bounce info
err.bounceInfo = getBounceInfo(err);

Expand Down
4 changes: 2 additions & 2 deletions helpers/get-greylist-key.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

const revHash = require('rev-hash');

function getGreylistKey(clientRootDomainOrIP) {
return `greylist:${revHash(clientRootDomainOrIP)}`;
function getGreylistKey(value) {
return `greylist:${revHash(value)}`;
}

module.exports = getGreylistKey;
Loading

0 comments on commit cfb020b

Please sign in to comment.