Prolateral Consulting Ltd
Prolateral Consulting Ltd
Knowledgebase : proFilter
   

A quick overview of SMTP

SMTP is the Internet protocol used to transfer electronic mail between computers, much like HTTP is the Internet protocol used to transfer web pages between computers. Like HTTP, there has been more than one generation of SMTP; the second generation is called ESMTP (for Extended SMTP), but the differences are not important for this introduction.

This attempts to be a quick overview of SMTP and related concepts, explaining enough of how it works so that the reader can follow reasonable technical discussions.

In SMTP (and in the rest of this discussion), the client is the computer that is sending email, and the server is the computer that is receiving it. Thus we say SMTP clients (or SMTP senders) send email to SMTP servers, although the machines involved may both be servers in the general sense; for example an ISP's mail server sending email to the SMTP servers for utoronto.ca.

The envelope versus the letter

Just like physical letters, SMTP email has two different sets of address information: the envelope headers, like the addresses on the outside of an envelope, which are used by mail transport software to route and deliver the email, and the normal headers, which are part of the mail message and which are only read and interpreted by the user and his software, just like the address attached to a salutation at the start of a physical letter. Unlike the post office, SMTP usually throws away most of the envelope before it hands the message to the user, so many users are not aware of the envelope headers.

In fact, SMTP never looks at the message headers at all; as far as SMTP is concerned, the email message (headers and all) is just one big blob that it shuttles around. Many SMTP clients are perfectly happy to deliver email with badly broken or entirely nonexistent message headers.

The SMTP protocol

Like many Internet protocols, SMTP operates by sending lines of text back and forth between the client and the server. The client sends commands and eventually the email message, and the server sends back responses to tell the client if the server accepted the command or if something went wrong.

Server responses always come in a special format: three digits, a space (or a dash), and then some free-format text (in error messages, this is usually intended for users to read; otherwise it is generally just noise). If there is a dash after the third digit instead of a space, further lines of response follow; otherwise, this is the last line. The only really important thing about the response is the first digit, like so:

Code Meaning
2xx everything is fine, go on
4xx temporary problem, try again later
5xx permanent error, give up

Errors can happen at any time, so at any response a server can send a temporary or a permanent error instead of the go ahead indication the client was expecting. A proper client must be able to cope with this, retrying temporary failures (but not too soon or too often) and giving up gracefully on permanent failures. (Tragically, there are improper clients out there in the world.)

A SMTP conversation between the client and the server goes in stages, each one initiated by the client doing something. A typical conversation will look like:

Client does: Server normally responds with:
Connects to the server 220 Helo there
HELO client-hostname
250 Pleased to meet you
MAIL FROM:<Sender address> 250 OK
RCPT TO:<Recipient address>
(May be repeated)
250 OK
DATA 354 Start mail input; end with <CRLF>.<CRLF>
Sends the actual email message (Nothing, it's waiting for the . that ends the message)
. 250 OK, accepted for delivery

At this point the email message has been sent. The client can now disconnect with a QUIT command, or it can send another email message by starting with the MAIL FROM step again (optionally sending a RSET command first).

The sender address is the email address that will receive email about delivery problems (mailing lists change this but not the From: email header so that they, and not the people sending to them, get messages about delivery problems). A special null sender address (MAIL FROM:<>) is used to signal that no one cares and no bounce notifications should be sent. Null senders are used when sending bounce messages themselves, and sometimes at other times.

There can be multiple recipients of the same message on the same computer. So that the actual email message only has to be transfered once (saving bandwidth), there can be several RCPT TOs for a message. (There has to be at least one, just like there has to be a MAIL FROM.) The client has to keep track of which recipient addresses have problems, if any, and retry them later if necessary.

The envelope headers are the MAIL FROM and RCPT TO parts of the SMTP conversation. The envelope sender is the MAIL FROM address, and the envelope recipients are the RCPT TO addresses.

The client-hostname, the sender address, and the recipient addresses should all be fully qualified. A fully qualified host or domain name is one that anyone on the Internet could use to look up information, not a shortened name useful only on machines inside an organization; for example, server.example.com instead of just server. A fully qualified email address is an email address with a fully qualified host or domain name, not just an email login; for example, MAIL FROM:<postmaster@server.example.com> instead of MAIL FROM:<postmaster> or MAIL FROM:<postmaster@server>. If the host or domain name is left off an email address, the SMTP server usually has no choice but to interpret it as an address on itself.

Email routing, or welcome to DNS

All of this is very well and good, but it doesn't tell us how a client machine with email to send to user@utoronto.ca decides which SMTP server to deliver it to. That is decided by looking various pieces of information up in the Domain Name System, DNS, which is another Internet protocol and system.

DNS exists to give out various sorts of information about names; you give it a name and what type of information you want, and it tries to give you back an answer. For our purposes (and simplifying a bit), there are three interesting types of information, conventionally called record types:

  • NS records for a domain, which tell you what hostnames can give you further information about that domain and names inside that domain, such as MX records or A records.
  • MX records for a name, which tell you what hostnames should accept SMTP email for user@name, and which order you should try them in.
  • A records for a hostname, which give you the IP addresses associated with the host.

Reduced to its simplest form, a SMTP client with email to send to user@example.com looks up NS records until it finds the nameservers for example.com, then asks them for the MX record for example.com, and finally asks for A records to determine the IP addresses of the names in the MX record. If a name has no MX record but does have an A record, email is delivered straight to the IP addresses listed.

The phrase host or domain name means a name that has one or both of an A record (an IP address) or an MX record (a place of record to deliver mail to). Such names are valid as the name to the right of the @ in email addresses. Names with just NS records are not; postmaster@ca is a nonexistent email address, although .ca certainly has NS records.

Problem:

You want to add extra user accounts to the profilter portal so they can manage their own email, or you can separate the email for different users.

Solution:

You can add any addresses within the domain for which you are an administrator. If you can see the '@example.com' in the Switch Account box - then you are an administrator for the example.com domain.

To add users to the account so that they can login themselves, or to allow you to view each user's mail separately, do the following:-

  1. Login to profilter as the administrator of your domain.
  2. Click on the Settings tab.
  3. Click on the Users subtab.
  4. Type the user's full email address (e.g. fred@example.com) into the New Email address/alias: text box.
  5. Click on the Add New Email Address/Alias button.
  6. Repeat this for each address you wish to add.

If you want the users to be able to login themselves you will need to assign an initial password. This can be done by:-

  1. Click on the Welcome tab. The new users will be listed under the domain in the Switch Account box on the left.
  2. Click on the required user to 'become' that user.
  3. Click on the Settings tab.
  4. Enter a password in the New Password and Confirm New Password text boxes.
  5. Click on the Update Login Credentials button.

The new users will now be able to login with their own, full email address (e.g. fred@example.com) and the password you assigned.

If you regularly get sent email from someone which profilter identifies as Spam, you can use the Whitelist to force profilter to pass it straight through and not quarantine it.

You can access your Whitelist under the Settings tab, then the White&Black Lists tab.

Senders can be added to the Whitelist in two ways:-

  1. Releasing an item from the Spam Quarantine will add the sender to your Whitelist; so it won't get stopped again (assuming it is sent from the same address).
  2. Manually add the address under the Settings->White&Black Lists section. Type the sender's email address in, mark the Whitelist radio-button, and click the Add to List button.

Note: The email address displayed as 'From:' on the email, is not necessarily the email address reported to proFilter as the sender. Every email has two sender-addresses; just like the sender's address on an envelope and on the letter inside. You should use the sender's address as listed inside proFilter.

Whitelisting Domains

This option to whitelist a domain can be useful with automated systems which send out email from different addresses every time in order to track bouncebacks for their own purpose.

Email addresses added can be full addresses, or you can whitelist a whole domain using an address of the format '@somedomain.com' which will allow all emails from the somedomain.com to be delivered without scanning. This option should be used with caution; and you should not add your own domain as a whitelisted domain.

Exchange 2003 introduced a new facility to filter email messages if the email address doesn't exist in the Active Directory. This will stop spammers from sending messages to non-valid addresses. It can also be used by profilter to immediately detect email to non-existent users, thus keeping your quarantines smaller and minimising processing requirements.

The rejection is done at the SMTP level - so the email message isn't even delivered.

This feature would expose your server to "Directory Harvest" attacks, allowing spammers to find valid email addresses on your server - however because it is behind profilter, it is protected. No information on existence of users is made available to the originating system

Exchange 2000 and users on Windows 2000 with Exchange 2003

If you are using the older version of Windows or Exchange, then you should look at third party tools to do the same thing.

Vamsoft's ORF is one such product that can filter on the active directory.

Enabling the Option on Exchange 2003

To enable this option:

  1. Expand ESM, Message Delivery.
  2. Right click on "Message Delivery" and choose Properties.
  3. Click on the tab "Recipient Filtering".
  4. Enable the option "Filter Recipients who are not in the directory."

You then need to enable the Recipient Filter on the SMTP Server.

  1. Still in ESM, Expand Admin Groups, <your admin groups>, Server, <your server>, Protocols, SMTP.
  2. Right click on SMTP Virtual Server and choose Properties.
  3. Click on "Advanced" next to the IP address on the first tab.
  4. With the IP address selected, choose "Edit"
  5. Enable "Apply Recipient Filter".
  6. Click Apply/OK until clear.
  1. Log in to the Office 365 admin center, and go to Admin centers > Exchange.

  2. In the left pane, click mail flow, and click rules.
  3. Click the + symbol, and click Bypass spam filtering:
    Setup Profilter with Office365

  4. In the new rule page, enter a Name to represent the rule.

  5. From the Apply this rule drop-down menu, select The sender > IP address is in any of these ranges or exactly matches:
    Setup profilter with Office365

  6. In the specify IP address ranges page, enter the IP address for the Profilter Clusters, and click the symbol

    Please see the list of the IP addresses for the Profilter Clusters

    Setup profilter with Office365

  7. Click OK, and click Save to create the transport rule.

Problem:

How do I setup Microsoft Exchange to only accept email from the proFilter clusters?

Solution:

  1. Run Microsoft Exchange System Administrator:-

    Go to Start->Programs->Microsoft Exchange->System Manager

  2. Expand the following trees:-

    Servers -> (Your Server) -> Protocols -> SMTP

  3. Right-click on Default Virtual SMTP Server

  4. Click on Properties

  5. Click on the Access tab

  6. Click on the Connections button

  7. Ensure that the Only the list below radio-button is selected

  8. Use the Add button repeatedly to add all IP addresses from the list below

  9. Click OK on all dialog boxes.

  10. Right-click on Default Virtual SMTP Server

  11. Click on Stop

  12. Wait for the service to stop

  13. Right-click on Default Virtual SMTP Server

  14. Click on Start

  15. Wait for the service to restart.

The IP addresses of the proFilter clusters can be found in the article: Profilter IP Addresses

The full current list of Profilter IP addresses is given below:-

  • 212.227.86.52
  • 213.165.83.174
  • 217.160.92.157
  • 209.143.135.31
  • 209.143.135.38
  • 209.143.135.83
  • 209.143.135.91
  • 209.143.135.162 - 209.143.135.163
  • 209.143.135.174
  • 209.143.135.180 - 209.143.135.189
  • 46.105.30.131

IPv6 Addresses

  • 2001:41d0:401:3100:0:0:0:1a26
  • 2001:8d8:87e:f00::94:c94a

 

Ensure that profilter is configured to send email to the mail server host name assigned to your domain by Microsoft.  It will look like this:

example-com.mail.protection.outlook.com 

Configuring Office 365 to only allow email from Profilter (optional).

1. Login to your Office 365 account by going here:

https://outlook.office365.com/ecp

2. Go to Mail flow > Rules.

3. Click on the + sign to add a new rule, and then click more options.

4. Name the rule:

Office 365 setup with profilter - Step 4

5. Choose the "The sender is located..." and "is external/internal" options:

Office 365 setup with profilter

6. Make the rule apply to senders that are "Outside the organization":

Office 265 setup with Profilter - Step 6

7. Create an exception to this rule for all Profilter IPs, so that filtered email still gets through. 

Where it says "Except if..." select the "Sender's IP is in the range..." option and add each Profilter IP:

 

Office 365 setup with Profilter - Step 7

All Profilter cluster IPs need to be added. To enter multiple IPs, click on the + button. 

See: The full list of Profilter IP addresses

8. Where it says "Do the following...", select the "Redirect the message to..." and "hosted quarantine" options:
  Warning: Newer versions of Office365 may not have this exact option.  You may need to choose "deliver the messages to the hosted quarantine" if available.

Office365 setup with Profilter Step 8

Outlook keeps all the Internet Headers of an email that tell the history of how the email arrived into your mailbox. This history is invaluable to technical support in finding out where any potential problems are.

In order to view the Internet Headers in Outlook, use the table below for the relevant version of Outlook you are running.

Select all the text in the 'Internet Headers' box and press CTRL+C to copy it; you can now paste this text into an email, or anywhere else to keep a record of the email history.

Outlook 2010, 2013, 2016, 2019, Office365

  • Double-click the message to open it in a new window.
  • Select the File tab and then click Properties
  • The Internet Headers can be copied from the box

Outlook 2007

Method 1: Right-click the message in the folder view, then choose Options.

Method 2: In an open message, click the arrow in the bottom-right corner of the Options pane in the ribbon bar.

Outlook 2003, 2002, 2000

Method 1: Right-click the message in the folder view, then choose Options.

Method 2: In an open message, choose View | Options.

With either method, you'll see the Internet headers portion of the Message Options dialog.

Outlook Express/ Internet Mail  

Method 1: Open the message and Select Properties beneath the File menu

Method 2: Right-click the message header and select Properties.

Problem:

What is Greylisting?

Solution:

Greylisting is a method of blocking significant amounts of spam at the mailserver level, but without resorting to heavyweight statistical analysis or other heuristical approaches.

Greylisting relies on the fact that most spam sources do not behave in the same way as legitimate mail systems. Although it is currently very effective by itself, it will perform best when it is used in conjunction with other forms of spam prevention such as profilter.

Problem:

What is SMTP and understanding the error codes

Solution:

Ever wondered why your emails don't send or your email marketing campaign fails and all you're left with is something called an SMTP error code and a short meaningless message?

Before you begin to panic this article breaks down the confusion of the SMTP error messages.

What is SMTP?

First let's discuss what SMTP actually is and how it fits into the world of sending and receiving emails.  SMTP stands for Simple Mail Transfer Protocol. SMTP is the mechanism that is used between mail servers to exchange emails.

All emails sent by a client, mobile device or server use SMTP to deliver the messages from the source to the destination mail server.  Think of SMTP as your local postal service and sorting office ensuring your mail gets to the right location and person.

Once the message has been received by a mail server, the user collects that message using their client application (Outlook, Windows Mail, Mac Mail, etc) or mobile device (iPhone, iPad, Windows Mobile, Android, etc) using a protocol called POP3 (Post Office Protocol) or IMAP (Internet Message Access Protocol).

Every time an email is sent using SMTP the receiving server will respond with an SMTP error code.  Not all SMTP error codes actually indicate failures, for example an SMTP error code 250 (2.5.0) means the message was accepted and the delivery action was complete.

So not all SMTP error messages are bad. SMTP error codes usually consist of three digits that carry different meanings.

First digit

The first digit has the following meaning

1xx - Command accepted by mail server but confirmation message is required.

2xx - Task was completed without errors.

3xx - Mail server accepted the request but requires more information.

4xx - Temporary failure encountered by mail server and you have to try again.  If you see one of these errors, the mail server or client should tell you that it will automatically attempt to resend the email.

5xx - Fatal error encountered, Mail server cannot process request.

Second digit

The second digit gives more information and there are six possible values.

x0x - A syntax error has occurred.

x1x - Indicates an informational reply, for example to a HELP request.

x2x - Refers to the connection status.

x3x - unspecified.

x4x - unspecified.

x5x - Refers to the status of the mail system as a whole and the mail server in particular.

Third digit

The last digit is even more specific and shows more graduations of the mail transfer status

SMTP error codes

Below is a list of common STMP error codes that you will most likely see when sending emails and using email marketing software such as Mach5 Mailer or SendBlaster to name a few.

2xx codes

211 - A system status message.

214 - A help message for a human reader follows.

220 - SMTP Service ready.

221 - Service closing.

250 - Requested action taken and completed. This is the best message of them all because it means the message was delivered with no errors.

251 - The recipient is not local to the server, but the server will accept and forward the message.

252 - The recipient cannot be verified (VRFY), but the server accepts the message and attempts delivery.

3xx codes

354 - Indicates mail server is ready to take message or start mail input and end with a dot.

4xx codes – Temporary Failure: retry will occur

421 - The service is not available and the connection will be closed.

450 - The requested command failed because the user's mailbox was unavailable (for example because it was locked). Try again later.

451 - The command has been aborted due to a server error.

452 - The command has been aborted because the server has insufficient system storage.

5xx codes – Fatal Errors: No retry will occur

500 - The server could not recognize the command due to a syntax error.

501 - A syntax error was encountered in command arguments.

502 - This command is not implemented.

503 - The server has encountered a bad sequence of commands.

504 - A command parameter is not implemented.

550 - The requested command failed because the user's mailbox was unavailable. This error can be caused by quite a few situations, some of which are Relay Denied, No such user, Account not available and Host rejected.

551 - The recipient is not local to the server.

552 - The action was aborted due to exceeded storage allocation.

553 - The command was aborted because the mailbox name is invalid.

554 - The transaction failed.

More information

For more information on SMTP error codes and the Simple Mail Transfer Protocol as laid down in RFC 821 please visit http://www.faqs.org/rfcs/rfc2821.html

Additional information on POP3 (RFC 1939 & RFC 2449) and IMAP (RFC 3501) can be found below:

http://www.faqs.org/rfcs/rfc1939.html

http://www.faqs.org/rfcs/rfc2449.html

http://www.faqs.org/rfcs/rfc3501.html