Outgoing TLS settings in the IIS SMTP server

So far, we covered the general installation flow and the incoming security settings related to the IIS SMTP service. Today’s article will cover the outgoing security settings, mostly the TLS support and mail domains.

On top of that, we will see how to troubleshoot any failed email connection, while monitoring the server behaviour and reading the relevant logs.

Let’s begin!

When I guided you through the configuration process of IIS SMTP service, I focused only on the scenario where we need to send an email inside company. However, if we want to send the email over the Internet, especially for password resets or with any similar highly sensitive information inside it, we would like to protect such communication.

The easiest, yet very efficient way to achieve that goal is to use the TLS encryption. TLS is today’s de facto standard for the secured communications. The process is similar to the one described in the article about incoming security settings, yet we will have a few distinctions.

The first distinction is that our IIS SMTP service will be here actually the email client. It will send the email to other email servers. Like any other email client, it can establish the TLS connection with the remote server using the certificate of that far server. That means that you do not need to have the local machine certificate for the outgoing TLS connections.

Enabling the outgoing TLS encryption

We will again open the IIS 6.0 Manager and navigate to our SMTP Server service. We will open the configuration dialog for the SMTP Virtual Server #1 and navigate to the tab named Delivery.

Today, we will choose the button named Outbound Security… We ignored it in the previous case.

The new window will pop-up. The look and feel of this dialog is the similar to that one we used to configure the incoming TLS settings.

The second distinction is that (generally speaking) we will not need to configure any other authentication method except the default Anonymous access. The good news is that there is the way how to send email to any specific server which requires any authentication other than Anonymous.

To enable the outgoing TLS encryption, we simply need to click on this checkbox named TLS encryption and then to click on the button [ OK ]. That’s it!

After that moment, every outgoing connection will be encrypted using the TLS protocol.

Testing outgoing TLS settings

The next step was to test the operations. The process is the same as one described in my other post. I’m using now both SMTP servers in my lab and I will directly send all test emails to each server.

I used the same ASCII based email file and placed it in the Pickup folder. It vanished immediately.

The next step was to check the Outgoing queue folder.

Well, one email is stacked here. That’s not good. Let’s check the log file.

As you can see, the communication with the server named mail.contoso.com (198.51.100.3) passed OK. We can see that the TLS channel was established and email is sent to the destination server.

In the other hand, the communication with the server named mail.fabrikam.com (203.0.113.11) failed. In this moment, I can’t tell why, as this log lacks many useful information.

Luckily, I can see the both sides of this communication and I will show you what happened here.

Troubleshooting outgoing communications

I jumped on the other mail server. This machine running hMail Server version 5.6.7 – Build 2425. All settings are out-of-the box, except that I added the server certificate and enabled the optional TLS communication on the port 25.

I opened the log file and found a few lines related to the communication with the machine named NPS19.contoso.com.

Well, all lines are here, yet this one caught my eye. What the hell this message mean?

I tried the hMail forums yet all I found is that those errors are OpenSSL related. Damn! Any further investigation through the search engines returned mostly correct and absolutely useless answers. They are usually related to the usage of the OpenSSL libraries in the software. Crap! I’m again on my own…

Well lads and lasses, as you can already assume, I never give up too easy. This message started to bugs me.

After a while, I found that the message TCPConnection – TLS/SSL handshake failed… Error code: 335544539, Message: short read found in the hMail Server log actually means – the remote client/side focebly closed connection. Now you know.

After I shorten the resending period on NPS19, I found this in the IIS SMTP server’s log:

As we can see, the NPS19 server (IIS SMTP service) established the connection and then just crushed after the STARTTLS command. This narrowed down the problem to the TLS connection sequence.

As I told you, hMail Server was installed with the default settings. That means that hMail Server supports the TLS versions 1.0, 1.1 and 1.2. The other server is based on the older Kerio MailServer 5.7.10 which only supports TLS 1.0.

When I disabled the TLS 1.2 support on hMail Server, everything worked fine. After all those tests, I made conclusions that

• IIS SMTP Server is extremely rigid with TLS
• if the TLS encryption is on – there is no fallback to no TLS (a.k.a. relaxed TLS connection)
• it supports TLS up to the version 1.1

In my lab I can do whatever I want, yet if we want to have the solution for the any server out there, we need the viable solution. In case that the destination server cannot accept your TLS connection, it’s a small chance that you can persuade other party to lower their security settings.

The good news, my friends! We can make a workaround.

Adding the mail domain

Here is another feature which we barely mentioned so far – the mail domains. Actually, those domains are used for the efficient email routing. Bear in mind that IIS SMTP server can be used also as the email proxy/router for any machine in the local network.

Why we need those additional domains? Those domains allow us to accept and forward emails for the specific mail/Internet domains (like contoso.com, fabrikam.com, etc.) to any email server other then default. On top of that, we can specify completely different outgoing settings, including the TLS encryption, authentication method, etc.

We need to go back in the IIS 6.0 Manager console and to expand the whole branch. Under the SMTP Virtual Server #1 we will see two more options. The first one is named Domains. Right-click on it and from the context menu choose the option named New > Domain.

The new domain can be either remote domain or alias on any existing domain. In this case, we need the Remote domain.

The next step is to specify the name of the remote domain, in this case – fabrikam.com.

After we click on [ Finish ], the new domain is added in the list.

This new domain is created with all default settings. We need to make some changes inside it.

We will again right-click yet this time on the name of the domain itself. From the context menu we will choose the option named Properties. The new dialog will open.

As you can see, we can setup here all those options we already configured on the server level. We can send emails through the smart host (yes, this is another way to override such TLS problem) or we can configure the separate outgoing settings for that domain. I will now show you the second option, as it covers also other scenarios.

When we click on [ Outbound Security… ], the new yet familiar dialog will pop-up.

As you can see, we can specify completely distinct outgoing security settings for this one domain. That can include the scenario that you need to authenticate yourself against the destination server (where you will use Basic authentication settings, for instance) or to enable or disable TLS settings per connection.

By default, the TLS encryption is not enabled and in this specific case we will leave it unchecked. What does it means?

The entire server settings are configured to use mandatory TLS encryption, yet the connection to this specific domain (fabrikam.com) will not use TLS encryption at all.

Testing the settings

Will it work? Yes, and I will show you now. Remember, I restored the default TLS 1.2 support on the side of hMail server, i.e. mail.fabrikam.com.

When I opened the log file, I can see the entire message exchange for that SMTP session. As you can see, there was no STARTTLS command. The server NPS19.contoso.com was instructed to connect to mail.fabrikam.com without TLS, using the plain email exchange.

When I opened the email client on the workstation named Fabrikam_WS01, it properly received that test email:

What have we covered in this article?

The main topic in today’s article are the outgoing security settings and the TLS support. It’s very important topic in today’s world and many administrator would like to encrypt outgoing emails, especially if they may contain any sensitive information.

On top of that, I show you the complete troubleshooting process in case when such settings are in collision between the local IIS SMTP server and any remote server.

I also included the mail domain configuration and the possible workaround in those situations when our server cannot send email using the general settings.

Stay tuned.