This is a repro of a customer reported issue where they were having issues sending email from Office 365 to on-premises Exchange servers. Apparently this had been working previously with no issues, then mailflow started to have problems. Some email was flowing from Office 365, but some was delayed or not delivered at all.
OS Error: 10060 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. The 10060 looks to be a windows socket error, but no idea why that's occuring when the mediaagent is trying to bind to a local port to create the index. Also no idea why it's creating a port on 58240 which is not in our CV firewall range - maybe that's only for external ports not local?
There were no issues sending email from the on-premises Exchange servers to Office 365. The issue was the mail flow from cloud to on-premises.
In order to look at the mail queues on-premises, we can use the Get-Queue cmdlet or the Queue Viewer. Queue Viewer can be found under the Exchange toolbox, which is built into the Exchange 2010 MMC and as a separate Start Menu item in Exchange 2013/2016.
Reviewing Office 365 Message Queues
Reviewing the Queues in Office 365 is also straightforward. In the below example note that the focus is on the Mail Flow, and then message trace. There are pre-canned queries to search emails from the last 24 or 48 hours. This can also be customised to suit the specific requirement. This was sufficient to troubleshoot this issue. For more complex situations please review Andrew's excellent EOP blog, specifically the Parsing an extended message trace post.
After entering the relevant time slot, and expected recipient the trace was executed. Multiple messages were then seen in the queue.
Taking one message as example, we can see that the message delivery to on-premises failed and that a specific error code of 4.4.316 was reported. We can see this by clicking to expand the entry using the arrow on the left hand side of the Message Events table.
If we drill into the Message Events table, and expand the Defer event entry we can see the below details:
The reported error was:
Reason: [{LED=450 4.4.316 Connection refused};{MSG=Socket error code 10061};{FQDN=smtp.tailspintoys.ca};{IP=13.92.177.139};{LRT=3/16/2017 4:59:18 PM}]. OutboundProxyTargetIP: 13.92.177.139. OutboundProxyTargetHostName: smtp.tailspintoys.ca
IP 13.92.177.139 corresponds to the on-premises Exchange infrastructure.
IP 104.47.34.97 is the Office 365 IP address which is attempting to send the email to on-premises.
Reviewing the reported error LED=450 4.4.316 Connection refused};{MSG=Socket error code 10061, indicates that Office 365 was unable to connect to on-premises Exchange so let's verify that configuration.
Reviewing On-Premises Infrastructure
As always, start with the simple things. The IP address of 13.92.177.139 is correct and does point to the on-premises Exchange servers. This was validated using nslookup. We can see the name which was used by Office 365 in the Message Event details – smtp.tailspintoys.ca. This resolves to the external IP of the on-premises environment. Since email is encrypted between Office 365 and on-premises Exchange we also need to verify the certificate used by the encrypted SMTP connection to ensure that it is valid.
Reviewing the certificate bound to the SMTP service we can see that the name on the certificate is also correct – mail.tailspintoys.ca. Also we ensure that the certificate is within the validity period, has a private key and chains correctly to the issuing CA. We can check the chaining on the Certification Path tab. Some CA vendors have their own tools to assist with this validation process. Check with the CA vendor which issued your certificate.
Also the certificate was not changed recently. Changing or updating the certificate which is used requires that the Exchange Hybrid Wizard is executed to update the new certificate thumbprint in Office 365.
What is logged in the Exchange Receive Connector logs? We need to review this to ensure that the Office 365 traffic is being processed by the correct receive connector. It is recommended that you enable the SMTP send and receive logging on all Exchange servers so that log data is available to troubleshoot an issue. Else you then need to enable the logging and wait for the issue to re-occur. The logs will be located under the Exchange installation folder, which is slightly different between Exchange 2010 and 2013/2016.
Exchange 2010
C:Program FilesMicrosoftExchange ServerV14TransportRolesLogsProtocolLogSmtpReceive
Exchange 2013/2016
C:Program FilesMicrosoftExchange ServerV15TransportRolesLogsFrontEndProtocolLogSmtpReceive
C:Program FilesMicrosoftExchange ServerV15TransportRolesLogsMailboxProtocolLogSmtpReceive
We can take the sending IP address from the Message Events and then search for it in the SmtpReceive log. In the example the IP we wish to search for is 104.47.34.97. The IP is shown below so you can see where it was obtained.
There were no connections from this IP address on any of the Exchange logs. This means that the traffic was not getting to Exchange.
Now that we have done our due diligence and validation, time to speak to the firewall administrators.
Office 365 IP and URL Restrictions
Microsoft documents the IP addresses and URLs which are required to access the various components of Office 365. The addresses and IPs are often modified, and you can subscribe to the RSS feed to be notified of changes.
Note: Microsoft is developing a REST-based web service for the IP address and FQDN entries on this page. This new service will help you configure and update network perimeter devices such as firewalls and proxy servers. You can download the list of endpoints, the current version of the list, or specific changes. This service will eventually replace the XML document, RSS feed, and the IP address and FQDN entries on this page. To try out this new service, go to Web service.
Socket Error 10060 Citrix
The firewall admins were asked to review the drop log on their devices to review the connections from the Office 365 IPs identified above. And lo! The firewall was indeed blocking these connections.
In this case the customer did not update the firewall correctly when they made a recent change to their external firewall ACLs. For some reason they removed some of the EOP IP objects from the ACL. As a result only some of the EOP servers were allowed to communicate with the on-premises SMTP endpoint.
Email Socket Error 10060
Once the firewall objects had been corrected, all email was then delivered without further issue.
Cheers,
Rhoderick
Sometimes your SMTP server may return a particular error message. The problem is that it will generally be very cryptic, like “550 Requested action not taken: mailbox unavailable” or “421 Try again later”. What does these numbers mean?
First of all: not any reply code is an error. Sometimes it’s just a response containing a detail about the server or an answer to a command. Secondly: any code consist of three digits, and each conveys a particular information. The first one defines whether the server has accepted the command, fulfilled an action, run into a temporary issue, encountered an error etc; the second and the third one refine the description further, stating if there’s been a syntactic problem, or a connection trouble etc.
Unfortunately, different servers sometimes use these codes in a different way, making the whole thing even more complicated… Anyhow, the most critical series of error messages is the 5xx one, and especially the ones from 550 to 559. In particular, you will probably get a lot of 550 SMTP error codes – that is, a problem that concerns the recipient’s email address.
Finally, remember that it’s much easier to deal with these error codes if you choose to rely on a professional SMTP server that will help you solve any issue. turboSMTP, for instance, comes with a 24/7 customer support: you can try it free and forget once for all these issues.
And here’s a list of the main SMTP error or reply messages, with an explanation and a tip about what to do.
CODE | MEANING | HOW TO SOLVE IT / WHAT TO DO |
101 | The server is unable to connect. | Try to change the server’s name (maybe it was spelt incorrectly) or the connection port. |
111 | Connection refused or inability to open an SMTP stream. | This error normally refers to a connection issue with the remote SMTP server, depending on firewalls or misspelled domains. Double-check all the configurations and in case ask your provider. |
211 | System status message or help reply. | It comes with more information about the server. |
214 | A response to the HELP command. | It contains information about your particular server, normally pointing to a FAQ page. |
220 | The server is ready. | It’s just a welcome message. Just read it and be happy that everything is working (so far)! |
221 | The server is closing its transmission channel. It can come with side messages like “Goodbye” or “Closing connection”. | The mailing session is going to end, which simply means that all messages have been processed. |
250 | Its typical side message is “Requested mail action okay completed”: meaning that the server has transmitted a message. | The oppsite of an error: everything has worked and your email has been delivered. |
251 | “User not local will forward”: the recipient’s account is not on the present server, so it will be relayed to another. | It’s a normal transfer action. For other information check out our article on what is an SMTP server. |
252 | The server cannot verify the user, but it will try to deliver the message anyway. | The recipient’s email account is valid, but not verifiable. Normally the server relays the message to another one that will be able to check it. |
354 | The side message can be very cryptic (“Start mail input end <CRLF>.<CRLF>”). It’s the typical response to the DATA command. | The server has received the “From” and “To” details of the email, and is ready to get the body message. |
420 | “Timeout connection problem”: there have been issues during the message transfer. | This error message is produced only by GroupWise servers. Either your email has been blocked by the recipient’s firewall, or there’s a hardware problem. Check with your provider. |
421 | The service is unavailable due to a connection problem: it may refer to an exceeded limit of simultaneous connections, or a more general temporary problem. | The server (yours or the recipient’s) is not available at the moment, so the dispatch will be tried again later. |
422 | The recipient’s mailbox has exceeded its storage limit. | Best is to contact contact the user via another channel to alert him and ask to create some free room in his mailbox. |
431 | Not enough space on the disk, or an “out of memory” condition due to a file overload. | This error may depend on too many messages sent to a particular domain. You should try again sending smaller sets of emails instead of one big mail-out. |
432 | Typical side-message: “The recipient’s Exchange Server incoming mail queue has been stopped”. | It’s a Microsoft Exchange Server’s SMTP error code. You should contact it to get more information: generally it’s due to a connection problem. |
441 | The recipient’s server is not responding. | There’s an issue with the user’s incoming server: yours will try again to contact it. |
442 | The connection was dropped during the transmission. | A typical network connection problem, probably due to your router: check it immediately. |
446 | The maximum hop count was exceeded for the message: an internal loop has occurred. | Ask your SMTP provider to verify what has happened. |
447 | Your outgoing message timed out because of issues concerning the incoming server. | This happens generally when you exceeded your server’s limit of number of recipients for a message. Try to send it again segmenting the list in different parts. |
449 | A routing error. | Like error 432, it’s related only to Microsoft Exchange. Use WinRoute. |
450 | “Requested action not taken – The user’s mailbox is unavailable”. The mailbox has been corrupted or placed on an offline server, or your email hasn’t been accepted for IP problems or blacklisting. | The server will retry to mail the message again, after some time. Anyway, verify that is working on a reliable IP address. |
451 | “Requested action aborted – Local error in processing”. Your ISP’s server or the server that got a first relay from yours has encountered a connection problem. | It’s normally a transient error due to a message overload, but it can refer also to a rejection due to a remote antispam filter. If it keeps repeating, ask your SMTP provider to check the situation. (If you’re sending a large bulk email with a free one that can be a common issue). |
452 | Too many emails sent or too many recipients: more in general, a server storage limit exceeded. | Again, the typical cause is a message overload. Usually the next try will succeed: in case of problems on your server it will come with a side-message like “Out of memory”. |
471 | An error of your mail server, often due to an issue of the local anti-spam filter. | Contact your SMTP service provider to fix the situation. |
500 | A syntax error: the server couldn’t recognize the command. | It may be caused by a bad interaction of the server with your firewall or antivirus. Read carefully their instructions to solve it. |
501 | Another syntax error, not in the command but in its parameters or arguments. | In the majority of the times it’s due to an invalid email address, but it can also be associated with connection problems (and again, an issue concerning your antivirus settings). |
502 | The command is not implemented. | The command has not been activated yet on your own server. Contact your provider to know more about it. |
503 | The server has encountered a bad sequence of commands, or it requires an authentication. | In case of “bad sequence”, the server has pulled off its commands in a wrong order, usually because of a broken connection. If an authentication is needed, you should enter your username and password. |
504 | A command parameter is not implemented. | Like error 501, is a syntax problem; you should ask your provider. |
510/511 | Bad email address. | One of the addresses in your TO, CC or BBC line doesn’t exist. Check again your recipients’ accounts and correct any possible misspelling. |
512 | A DNS error: the host server for the recipient’s domain name cannot be found. | Check again all your recipients’ addresses: there will likely be an error in a domain name (like mail@domain.coom instead of mail@domain.com). |
513 | “Address type is incorrect”: another problem concerning address misspelling. In few cases, however, it’s related to an authentication issue. | Doublecheck your recipients’ addresses and correct any mistake. If everything’s ok and the error persists, then it’s caused by a configuration issue (simply, the server needs an authentication). |
523 | The total size of your mailing exceeds the recipient server’s limits. | Re-send your message splitting the list in smaller subsets. |
530 | Normally, an authentication problem. But sometimes it’s about the recipient’s server blacklisting yours, or an invalid email address. | Configure your settings providing a username+password authentication. If the error persists, check all your recipients’ addresses and if you’ve been blacklisted. |
541 | The recipient address rejected your message: normally, it’s an error caused by an anti-spam filter. | Your message has been detected and labeled as spam. You must ask the recipient to whitelist you. |
550 | It usually defines a non-existent email address on the remote side. | Though it can be returned also by the recipient’s firewall (or when the incoming server is down), the great majority of errors 550 simply tell that the recipient email address doesn’t exist. You should contact the recipient otherwise and get the right address. |
551 | “User not local or invalid address – Relay denied”. Meaning, if both your address and the recipient’s are not locally hosted by the server, a relay can be interrupted. | It’s a (not very clever) strategy to prevent spamming. You should contact your ISP and ask them to allow you as a certified sender. Of course, with a professional SMTP provider like turboSMTP you won’t ever deal with this issue. |
552 | “Requested mail actions aborted – Exceeded storage allocation”: simply put, the recipient’s mailbox has exceeded its limits. | Try to send a lighter message: that usually happens when you dispatch emails with big attachments, so check them first. |
553 | “Requested action not taken – Mailbox name invalid”. That is, there’s an incorrect email address into the recipients line. | Check all the addresses in the TO, CC and BCC field. There should be an error or a misspelling somewhere. |
554 | This means that the transaction has failed. It’s a permanent error and the server will not try to send the message again. | The incoming server thinks that your email is spam, or your IP has been blacklisted. Check carefully if you ended up in some spam lists, or rely on a professional SMTP service like turboSMTP that will nullify this problem. |