Mail Routing Fundamentals
E-mail can be accessed multiple ways from the e-mail program installed on your desktop to the web application you use when you are traveling. Even though these methods of accessing e-mail have a different interface their method for sending e-mail is very similar. The process of directing a message to the recipient’s host is called routing. Apart from finding a path from the sending site to the destination, it involves error checking as well as potentially virus and spam filtering. There are many processes, systems and protocols involved in the successful delivery of one e-mail message. Sometimes these messages arrive at their destination almost immediately, but sometimes they may be delayed or even rejected somewhere in the process. We are going to discuss the basic process for e-mail delivery and look at a few reasons that messages can be delayed or rejected along the way.
When you click the “Send” button your e-mail client initiates a two-way conversation with your e-mail provider’s SMTP (Simple E-mail Transfer Protocol) servers over standard SMTP ports – TCP ports 25 and 587. SMTP servers only understand very basic commands, so your mail client must initiate this connection using only commands that the SMTP server can interpret. The most common SMTP commands are:
[table]
SMTP Command, Description
HELO / EHLO, This command is used to identify the sender (client) to the SMTP server. EHELO is a newer command than HELO and includes more functionality\, however most servers still accept either command as an introduction.
MAIL FROM, Specifies the e-mail address of the sender.
RCPT TO, Specifies the e-mail address of the recipient.
DATA, Starts the transfer of the actual data (body text\, attachments etc).
RSET (RESET), Specifies that the current mail transaction will be aborted.
VRFY (VERIFY), Asks the receiver to confirm that the argument identifies a user or mailbox.
HELP, This command causes the server to send helpful information to the client.
QUIT, Quits the session.
AUTH LOGIN, Initiates SMTP Authentication when sending
[/table]
The e-mail client sends these commands to the mail server and the server replies with numerical codes and additional information after the numerical code. These reply codes tell the client if the previously sent command succeeded or failed. Commands and replies are composed of characters from the ASCII character set. EHELO is a newer command than HELO and includes more functionality, however most servers still accept either command as an introduction.
Once the server connection is initiated it will then authenticate your username and password. SMTP authentication is initiated with the AUTH LOGIN command. The server will then responds requesting your username, 334 VXNlcm5hbWU6, which is the phrase Username: in base64 encoding. You then send your base64 encoded username, the server then responds with 334 UGFzc3dvcmQ6 which is Password: in base64 encoding. You then send your base64 encoded password and should receive a 235 Authentication Succeeded message. If the e-mail server accepts your message it will proceed to the MAIL FROM command. This process is done inside your mail client, however with some know-how and command prompt this can be done manually as well.
The output of a telnet session will look something like this:
EHLO example.mailserver.com
250-server1.exampledomain.com Hello [1.1.1.2]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250-STARTTLS
250 HELP
AUTH LOGIN
334 VXNlcm5hbWU6dXNlcm5hbWUuY29t
334 UGFzc3dvcmQ6bXlwYXNzd29yZA==
235 Authentication succeeded
You will know if your message is rejected by your SMTP server and why by looking at the undeliverable message that you receive back at the e-mail account you sent the message from. An undeliverable message will look something like this though the information and formatting will vary based on the provider. It is also possible that the message will be rejected before ever reaching your SMTP server, which will appear in your browser as a pop-up notification showing the error. (see error section below)
If you are using an outside company as a smart host the rejections that you will see should be checked for the rejecting server which could now be your own internal mail server, the smart host’s server or the recipient’s servers.
Every domain that is used for e-mail, datayardworks.com for example, needs to have an MX-record (Mail Exchanger Record) in order to send and receive e-mail. An MX-record is a DNS-entry that tells the sending SMTP server where to deliver the e-mail for a domain. Your MX record will be determined by where your mail is hosted, each company has a different MX record, larger companies will have several. An MX record looks like this:
[table]
HOSTNAME, TTL, TYPE, PRIORITY, Host
datayardworks.com, 3600, MX, 10, mail.donet.com.
[/table]
In our example the hostname tells querying SMTP servers that e-mail addressed to someone @datayardworks.com should be delivered to the server ‘mail.donet.com’. This requires that ‘mail.donet.com’ exists (either as a CNAME or an A-record) in the authoritative zone.
There are two numeric values in the above record. The first one is 3600, which is the TTL (Time to Live). This record tells other DNS-servers that it is OK to cache the above record for up to 3600 seconds (or one hour). The second numerical value is ’10’ and is the MX-record priority. In this example, it doesn’t matter, as we only have one record, but if we were to have multiple MX records, it would determine the priority order of the servers. If the first one is unresponsive, the second one will be used in order of priority with the lowest priority being used first (example: a record with the priority of 10 would be used before the record with a priority of 30). In a server with multiple MX records you would see this:
[table]
HOSTNAME, TTL, TYPE, PRIORITY, Host
google.com., 14400, MX, 20, ALT1.ASPMX.L.GOOGLE.COM.
google.com., 14400, MX, 30, ASPMX4.GOOGLE-MAIL.COM.
google.com., 14400, MX, 30, ASPMX5.GOOGLE-MAIL.COM.
google.com., 14400, MX, 30, ASPMX2.GOOGLE-MAIL.COM.
google.com., 14400, MX, 30, ASPMX3.GOOGLE-MAIL.COM.
google.com., 14400, MX, 20, ALT2.ASPMX.L.GOOGLE.COM.
google.com., 14400, MX, 10, ASPMX.L.GOOGLE.COM.
[/table]
As you can see, there are multiple servers listed for the same domain (google.com). They all have the same TTL, 14400 seconds (four hours). The difference is the priority and the server. The sending server would try to deliver e-mails to the servers in the order from lowest priority value to highest until it finds one server who accepts the connection. This means if the server ‘aspmx.l.google.com’ goes down or is too busy to accept the connection the message will then try to deliver to ‘alt2.aspmx.l.google.com’ and ‘alt1.aspmx.l.google.com’ since these two servers have the same priority they will be chosen at random. In the case of large providers with multiple domains relaying on their servers for SMTP they are likely using a clustered mail environment which means many servers with different IP addresses can send mail from the same host, such as ‘mail.donet.com’.
If you want to know how your mail server might determine the DNS record read my article on DNS resolution here: https://www.datayard.us/blog/article/domain-name-resolution-introduction
Once your SMTP server has determined the server for the mail to be sent to it will then attempt to make a connection with that server using the same basic commands as your mail client used to connect to your SMTP server. The message will first pass through the mail provider’s content filtering servers before even making it to their SMTP server where it may be rejected. There are a few reasons that mail can be rejected or even delayed by a provider’s content filters. First, if a content filtering server deems your mail content to be junk mail it will likely tag the message as junk and deliver it according to the domains preferred settings, most domains choose to have spam messages delivered to a specific “Junk Mail” folder.
If you are on a blockhole list you may receive undeliverable email notifications stating that the mail server you use is on a blackhole list. The first course of action to get removed from the blackhole list is to resolve any outstanding issues with your mail server. If this mail server is provided by an outside company or managed by your IT department you want to send them the bounceback you received so they can address the issue. If you manage this mail server you can check some of the major blacklists for more information and to see if your server is listed in their database.
Once the server has determined that your mail is authorized and not on a blackhole list it will attempt to deliver this message to the recipients’ mailbox. The mail server will check to make sure that the recipient address specified does exist as a mailbox on the server. As long it does exist, the message will be delivered to the recipient’s mailbox for their mail client to retrieve. However, the message can again be rejected at this point. For example if the recipient email address is spelled incorrectly, has been deleted or even if the users’ mailbox is full you may receive a bounceback stating why your message was rejected.
Alternate delivery methods and setting up your own mail server
A smart host is an outside mail server that accepts inbound SMTP and routes it out again however the point of sending your mail to one of these is that you may offload the task of routing and retrying mail to another location. Directing your outbound traffic to an SMTP host at your ISP will allow your outgoing internet connection to free up much quicker than sending through your own network. A relay SMTP server is usually the target of a DNS MX record that designates it, rather than the final delivery system. The relay server may accept or reject the task of relaying the mail in the same way it accepts or rejects mail for a local user. If it accepts the task, it then becomes an SMTP client, establishes a transmission channel to the next SMTP server specified in the DNS and sends it the mail. If it declines to relay the message a 550 error should be returned to the sender.
Common Rejections
Bounceback
From: Mail Delivery System <[email protected]>
To: <[email protected]>
Subject: Undelivered Mail Returned to Sender
Attachment:
file-2 , 416 Bytes
file-3 , 5.9 KBytes
file-4 , 16.2 KBytes
This is the mail system at host smtp.donet.com.
I’m sorry to have to inform you that your message could not
be delivered to one or more recipients. It’s attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The mail system
<[email protected]>: host mail.donet.com[64.56.96.133] said: 554
5.7.1 <[email protected]>: Relay access denied (in reply to RCPT TO
command)
From this delivery failure notice you can find the following information
1. The name of the server that is rejecting your message (this message can either be rejected by your server or by the recipients’ mail server): This is the mail system at host smtp.donet.com, in this case the message was rejected by smtp.donet.com.
2. The cause of the rejection: said: 554 5.7.1 <[email protected]>: Relay access denied (in reply to RCPT TO command, this particular rejection is rejecting relay access which means you do not have permission to send your mail through this mail server.
SMTP errors from a Mail client
No Relaying Allowed: This rejection message tells you that the credentials (username and password) that you are sending to the e-mail server are not authenticating. You should check the settings in your e-mail client to ensure your username and password fields are not blank, the correct credentials are entered and that you have outgoing server authentication enabled.
<[email protected]>: host mail.donet.com[64.56.119.17] said: 554
5.7.1 <[email protected]>: Relay access denied (in reply to RCPT TO
command)
Could not connect to SMTP server: This rejection tells you that the connection from your e-mail client to your e-mail providers’ server failed. You should check the SMTP server name and ensure that your SSL types match the port numbers specified and that you have no firewall or antivirus program that is blocking outbound connections.
Greylisting
Greylisting is a way mail providers attempt to limit the amount of spam that gets through. Grey listing checks the legitimacy of an e-mail message by temporarily rejecting the incoming message with a 451 error. The sending SMTP server will recognize this as a temporary rejection and attempt to re-send the message. If there is a prior relationship between the sending server and the recipient server, the message will get delivered without any issue. At the same time the 451 error is sent a triplet is recorded in the Greylisting database as an unconfirmed triplet.
A triplet consists of:
- IP address of the sending server
- Sender e-mail address
- Recipient e-mail address
This is where delays come in, this response can take up to 24 hours depending on the settings of the server and how it is set to respond. If you have a small one server mail setup then the greylisting process generally doesn’t cause a significant delay. However, in a clustered mail environment where many different servers, with different IP addresses, can be used to send and receive mail the delays can be significant. If the rejected message is re-sent and all information matches the prior unconfirmed triplet then the message will pass through greylisting and be delivered. If any of the three items do not match then it will be seen as a new message and will begin the greylisting process again.
RBLs (Realtime Black Hole Lists)
There are specific companies who run blackhole lists and those lists are used by mail providers to determine when a server’s traffic has become unfavorable as a way to protect them from accepting unfavorable traffic. It’s common for a server’s mail IP address to temporarily end up on a public blackhole list. It could be because of the overall volume of mail that is coming from that server, messages that seem to have characteristics of junk in them, or that list has received messages to spam traps it operates.
How can you identify if your mail server is blackholed?
Spamhaus (http://www.spamhaus.org/)
MXtoolBox (http://www.mxtoolbox.com/blacklists.aspx)
Sorbs (http://www.sorbs.net/
http://multirbl.valli.org/lookup/
