Exchange

In my last blog post I have highlighted how HAProxy can be used to distribute client connections to two or more servers with Exchange 2013 CAS role. But there is another common use case for load balancers in a Exchange environment: SMTP. Let’s take a look at this drawing:

The inbound SMTP connections are distributed to two Mail Transfer Agents (often a cluster of appliances, like Cisco IronPort or Symantec Messaging Gateway) and the MTAs forward the e-mails to the Exchange servers. Sometimes the e-mails are not directly forwarded to the Exchange servers, but to mail security appliances instead (like Zertificon Z1 SecureMail Gateway). After the e-mails have been processed by the mail security appliances, they are forwarded to the Exchange backend. Such setups are quite common. If a load balancer isn’t used, the MX records often point to the public IP address of a specific MTA. In this case, two or more MX records have to be set to ensure that e-mails can be received, even if a MTA fails.

Since Exchange 2007 client connections are handled by the Client Access Server role. With Exchange 2010, Microsoft has introduced the concept of the Client Access Server Array (CAS Array). A CAS Array is required, when internal and external client connections should be load balanced over multiple client access servers. Many client access protocols in Exchange 2010 require session affinity. This means, that the connection between the client and a particular client access server must persist. This requires application-level load balancing for Exchange 2010 and Microsoft recommends this explicitly. Microsoft dropped the concept of the CAS Array in Exchange 2013 and implemented much more logic into the Exchange 2013 Client Access Server role. There is no more need for session affinity in any client access protocol used in Microsoft Exchange 2013. Connections to a Exchange 2013 client access servers can be directed to an available server. A simple DNS round-robin works, but if a server fails, DNS would not handle this.You can use Windows Network Load Balancing (WNLB), but it has several limitations and downsides. I blogged about one of them in my blog post Flooded network due HP Networking Switches & Windows NLB. The other point is, that you can’t use it when you build a two server CAS/ DAG Exchange 2013 environment: You can’t use WNLB on servers that have the Microsoft Failover Cluster role installed. At this point HAProxy comes into play.

The recovery of an Exchange mailbox using a recovery database is usually no big deal. Simply restore the database, create a recovery database and recover the mailbox or items from the mailbox. Sometimes you have the luck that the customer has licensed the Data Protector Exchange 2010 Granular Recovery for Exchange (GRE). This was unfortunately not true in my case. Okay, so let’s do it the old way. The needed tape was available in the library and luckily it was a full backup. So I quickly added a disk to the VM and started the recovery of the database to a temporary location. At this point, the disaster took its course…

When moving users to Exchange 2013 it can happen, that they can’t access public folders housed on the old Exchange 2010 or 2007 server. The same can happen to shared mailboxes (mailboxes with Full Access permissions). The users are constantly prompted for credentials or they get this message:

Cannot expand the folder. Microsoft Exchange is not available. Either there are network problems or the Exchange server is down for maintenance.

There is an advantage, if you solves problems: You can learn something. I’m currently migrate a small Exchange 2007 environment to Exchange 2013. The first thing I learnt was, that IT staff still uses their own accounts for administration, and sometimes they assign administrator rights to users for testing and troubleshooting purposes. This can be a problem, as I described in my last posting. Today I learnt something different: Sometimes it’s the little things that bring you to despair.

During an Exchange 2013 migration project the  first attempt to migrate a mailbox failed with the following error:

Error: MigrationPermanentException: Active Directory property 'homeMDB' is not writeable on recipient 'testing.local/Users/Dummy'. --> Active Directory property 'homeMDB' is not writeable on recipient 'testing.local/Users/Dummy'.

The error message clearly stated, that this was a permission issue. A quick search pointed me to the right direction. I found a thread in the TechNet forums, in which the same error message were discussed. This error occurs, if the Exchange Trusted Subsystem group is missing in the ACL of the user object. This group contains the exchange server and it’s usually inherited from the domain object to all child containers and objects. I checked the ACL of the user and the Exchange Trusted Subsystem group was missing in the ACL. This was caused by the disabled permissions inheritance. An object ACL with disabled permissions inheritance is sometimes called a protected ACL. Bill Long wrote a nice Power Shell script to search for object which have permissions inheritance disabled.

Problem description

I got a couple of warnings (source MSExchange ADAccess, Event ID 2937) after removing a Exchange 2007 server at the end of a Exchange 2007 > 2013 migration. The details of the warning told me, that there was a faulty value set to a attribute of the mailbox database object. Because the public folder migration was part of the migration, the error message seemed plausible.

Process w3wp.exe (PID=4652). Object [CN=Mailbox Database E2K13,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Testing,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=testing,DC=local. Property [PublicFolderDatabase] is set to value [testing.local/Configuration/Deleted Objects/Public Folder Database DEL:4a45b7c2-10fc-42df-bdaa-82ae8a12e66e], it is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible.

A quick check with ADSI Edit confirmed the message. To be honest: I made a mistake and searched for the attribute PublicFolderDatabase in the database object, but in the end I found the wrong entry as a value of the msExchHomePublicMDB attribute in the database object. It must be set to the distinguished name of the mailbox database that houses the public folder mailboxes. If you don’t have any public folders in your Exchange 2013 org, then you have to clear the value!