Microsoft

Virtualization is an awesome technology. Last weeks I visited a customer and we took a walk through their data centers. While standing in one of their data centers I thought: Imagine that all server, that they are currently run as VMs, would be physical?. I’m still impressed about the influence of virtualization. The idea is so simple You share the resources of the physical hardware between multiple virtual instances. I/O, network bandwidth, CPU cycles and memory. After nearly 10 years of experience with server virtualization I can tell that especially the memory resources is one of the weak points. When a customer experiences performance problems, they were mostly caused by a  lack of storage I/O or memory.

When moving users to Exchange 2013 it can happen, that they can’t access public folders housed on the old Exchange 2010 or 2007 server. The same can happen to shared mailboxes (mailboxes with Full Access permissions). The users are constantly prompted for credentials or they get this message:

Cannot expand the folder. Microsoft Exchange is not available. Either there are network problems or the Exchange server is down for maintenance.

There is an advantage, if you solves problems: You can learn something. I’m currently migrate a small Exchange 2007 environment to Exchange 2013. The first thing I learnt was, that IT staff still uses their own accounts for administration, and sometimes they assign administrator rights to users for testing and troubleshooting purposes. This can be a problem, as I described in my last posting. Today I learnt something different: Sometimes it’s the little things that bring you to despair.

During an Exchange 2013 migration project the  first attempt to migrate a mailbox failed with the following error:

Error: MigrationPermanentException: Active Directory property 'homeMDB' is not writeable on recipient 'testing.local/Users/Dummy'. --> Active Directory property 'homeMDB' is not writeable on recipient 'testing.local/Users/Dummy'.

The error message clearly stated, that this was a permission issue. A quick search pointed me to the right direction. I found a thread in the TechNet forums, in which the same error message were discussed. This error occurs, if the Exchange Trusted Subsystem group is missing in the ACL of the user object. This group contains the exchange server and it’s usually inherited from the domain object to all child containers and objects. I checked the ACL of the user and the Exchange Trusted Subsystem group was missing in the ACL. This was caused by the disabled permissions inheritance. An object ACL with disabled permissions inheritance is sometimes called a protected ACL. Bill Long wrote a nice Power Shell script to search for object which have permissions inheritance disabled.

Problem description

I got a couple of warnings (source MSExchange ADAccess, Event ID 2937) after removing a Exchange 2007 server at the end of a Exchange 2007 > 2013 migration. The details of the warning told me, that there was a faulty value set to a attribute of the mailbox database object. Because the public folder migration was part of the migration, the error message seemed plausible.

Process w3wp.exe (PID=4652). Object [CN=Mailbox Database E2K13,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Testing,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=testing,DC=local. Property [PublicFolderDatabase] is set to value [testing.local/Configuration/Deleted Objects/Public Folder Database DEL:4a45b7c2-10fc-42df-bdaa-82ae8a12e66e], it is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible.

A quick check with ADSI Edit confirmed the message. To be honest: I made a mistake and searched for the attribute PublicFolderDatabase in the database object, but in the end I found the wrong entry as a value of the msExchHomePublicMDB attribute in the database object. It must be set to the distinguished name of the mailbox database that houses the public folder mailboxes. If you don’t have any public folders in your Exchange 2013 org, then you have to clear the value!

I use Microsofts Deployment Toolkit (MDT) in my lab to deploy Windows VMs with Windows Server 2008 and Windows Server 2012. I described the installation and configuration of MDT in a small blog post series. Take a look into the intro post, if you’re a new to MDT. But the OS installation isn’t the time consuming part of a deployment: It’s the installation of patches. Because of this, I decided to automate the patch installation and make it part of the OS installation.

The last two days I had a lot of trouble with Microsoft Remote Desktop Services (RDP), or to use the older wording, terminal services. To be honest: Terminal servers are not really my specialty, and actually I was at the customer to help him with some vSphere related changes. But because I was there, I was asked to throw a closer look at some problems with their Microsoft Windows 2008 R2 based terminal server farm. Some problems with removable media (USB sticks etc.) and audio on IGEL thin clients were hard to troubleshoot, but we were able to fix them. The main problem was none at first glance.

While I was poking around in my Twitter timeline, a tweet from Victor van den Berg (VCDX #121) got my attention.

My first though “What a step backwards!”. I have installed a bunch of Microsoft clusters in Virtual Infrastructure and vSphere enviroments and most times it was PITA. Especially with Raw Device Mappings (RDM) and bus sharing, which prevents vMotion a VM to another host (regardless of this: it’s not supported!). It’s ironic to invest a significant amount of money into a technology, which  increases availability and manageability, and another technology lowers availability due additional maintenance windows for cluster failovers. But that’s exactly what you get, when you use MSCS with SCSI bus sharing (RDM or VMFS). A way to address this issue is to use in-guest iSCSI. This means that you access the shared disks directly from the VM due a iSCSI initiator running in the VM. To do so, you have to present the disks for the cluster to the VMs, not to the ESXi hosts. To be honest: In-guest increases complexity. Especially then, when the customer doesn’t have a iSCSI infrastructure. A second method is in-guest SMB, which is currently only supported with Windows Server 2012. Just to clear up the matter with in-guest iSCSI and W2K12(R2). Mostafa Khalil provided the crucial information:

Today I was onsite at a customer to bring a tiny VMware vSphere cluster to life (HP BladeSystem c7000 with 7 HP ProLiant BL460 Gen8). Normally no big deal, but it started with two unavailable Onboard Administrator (OA) network interfaces. I switched from static ip addresses to DHCP, but I had no luck. I noticed that both interfaces were available if I connect my notebook directly to the interfaces. I even noticed that the Insight Display was unresponsive after connecting one or both OA to the network. The customer told me, that they had yesterday network related problems with virtual AND physical machines. Short outages, lost pings, things like that. This morning, before I arrived on site, the problems were worse. The customer told me that they had this network problems for a while. They had a lot of work and the outages were annoying, but not a big problem. The network of the BladeSystem were already connected (HP 10GbE Pass-Thru modules), but this kind of interconnect couldn’t cause this kind of problems. I checked the Switches and found on EVERY SINGLE ACTIVE port an enormous amount of “Drops TX”. But I found no loops or something like that. The network was flat. One VLAN and a /16 network. Not nice, but functional. I asked the customer to start Wireshark. I wanted to take a look around, get a feeling for what was going on in the network. Wireshark started and… stopped responding. After a couple of seconds it came back and I saw traffic that was… spooky. Usually I expect things like broadcasts, ARP, traffic from my client or for my client. But I saw traffic from a domain controller to a Windows NLB cluster and Citrix traffic to a Windows NLB cluster. I checked if the workstation was connected to a monitoring port, but it wasn’t. And it was only traffic with destination to the Windows NLB cluster. Our network problems had something to do with the Windows NLB. The customer and I decided to stop both NLB nodes. After that: Silence… I saw the expected traffic in Wireshark and my OA were both responding. Everything was fine… until we started the NLB again.

In part I and part II of this series I showed how to install the WDS role, MDT 2013 and ADK for Windows 8.1. I showed the process to import the OS images and the necessary drivers for our deployment. Now it’s time to bring MDT to life. Let’s start with part III of this series.