Vmware

On the HP Discover in June 2013 (I wrote 2014, sorry for that typo). HP has announced the HP 3PAR StoreServ 7450 All-Flash Array. To optimize the StoreServ platform for all-flash workloads, HP made some changes to the hardware of the nodes. The 7450 uses 8-core Intel Xeon CPUs instead 6-core 1.8 Ghz CPUs, the cache was doubled from 64GB to 128GB and they added some changes to the 3PAR OS: HP added additional cache flush queues to separate the flushing of cache for rotating rust and SSD devices. They also made some write I/O optimizations and added the ability to perform fragmented writes. Instead of writing 16 KB blocks, 3PAR OS is now able to write only 4 KB of a 16 KB block. This software-based changes may be used also on the 7200 and 7400. This leads to the new…

Virtualization is an awesome technology. Last weeks I visited a customer and we took a walk through their data centers. While standing in one of their data centers I thought: Imagine that all server, that they are currently run as VMs, would be physical?. I’m still impressed about the influence of virtualization. The idea is so simple You share the resources of the physical hardware between multiple virtual instances. I/O, network bandwidth, CPU cycles and memory. After nearly 10 years of experience with server virtualization I can tell that especially the memory resources is one of the weak points. When a customer experiences performance problems, they were mostly caused by a  lack of storage I/O or memory.

Today I stumbled over a nice workaround. While installing a CentOS 6 VM, I needed to install the VMware Tools. I don’t know why, but I got an error message, regarding a non accessible VMware Tools ISO.

I remembered a blog post I read a few months ago, about a VMware online repository, from which VMware tools can be installed. You can download the repository information here. The RPM for RHEL can also be used for CentOS. Simply download and install the RPM:

In April 2014 was a bug in vSphere 5.5 U1 discovered, which can lead to APD events with NFS datastores.iSCSI, FC or FCoE aren’t affected by this bug, but potentially every NFS installation running vSphere 5.5 U1 was at risk. This bug is described in KB2076392. Luckily none of my customers ran into this bug, but this is more due to the fact, that most of my customers use FC/ FCoE or iSCSI. Until today the only solution was to avoid the upgrade to U1 and to use vSphere 5.5 GA (with some patches to fix the Heartbleed bug).

Today I observed a strange behaviour of several VMs at a customer. Several VMs in a cluster showed an alarm, but neither on the alarm tab of the VM, nor the alarm section at the bottom of the C# client showed an error.

The customer still uses vSphere 5.0. An upgrade to 5.5 is on the roadmap. The symptoms:

Yesterday I received the following tweet:

Later Craig Kilborn joined the conversation and I decided to clarify this 100 or 1 IOPS myth the next morning.

In order to give you some context: I wrote a blog post about adding a custom SATP claimrule for HP 3PAR StoreServ storage on ESXi. In this blog post I pointed out, that the claim rule is usually used to change the default behaviour for switching the path for active IO. For the VMW_PSP_RR this is 1000 IOPS, which means, that after 1000 IOPS for a specific device, the path for the active IO to this device ist changed to the next active and optimized IO path. I recommend to read this blog post from Duncan Epping for more information.

One of the tasks that I finish before I present the first Virtual Volumes (VV) to hosts is to discuss the need of a custom SATP claimrule with the customer. Requirement for a custom claimrule is usually, that the active and optimized path should be switched after each IO and not after 1000 IOs. Duncan Epping wrote a nice blog post some years ago. I recommend to read it.

Some basics

The Storage Array Type Plug-In (SATP) is responsable for array-specific operations, like health monitoring of physical paths, reporting of path state changes and path failover. Each SATP is linked to a Path Selection Policy (PSP), which controls the selection of active paths for IO. VMware ESXi provides a couple of SATPs:

Yesterday I stumbled over a forum post in a german VMware forum. A user noticed after a vSphere 5.5 update, that a newly updated ESXi 5.5 hosts wasn’t able to mount some datastores. The host was updated with a HP customized ESXi 5.5 Image. The other two hosts, ESXi 5.1 installed from a HP customized image, had no problems. A HP P2000 G3 MSA Array with iSCSI was used as shared storage. The datastores with VMFS version 5.54 were mounted. Only datastores with VMFS 5.58 were not mouted. The user evacuated the VMs off one of the datastores, and then deleted and recreated the datastore. The recreated datastore appeared for a short moment and than disappered again.

VMware vCloud Hybrid Service (vCHS) stands in one line with Amazon Web Services, Microsoft Azure, Rackspace Cloud or other cloud offerings. I don’t want to compare the different provider with vCHS. To be honest: This article is more a summary for myself, than really new content. I just want to summarize information about the IaaS offering of VMware. If you want a comparison of vCHS and AWS, I recommend to read this article written by Alex Mattson (AHEAD).

I faced today a really nasty problem. I have four HP ProLiant DL360 G6 in my lab. This server type has two 1 GbE NICs with the Broadcom NetXtreme II BCM5709 chip onboard, which are usually claimed by the bnx2 driver. While applying a host profile to three of the hosts, one hosts reported an error. Supposedly the host hasn’t a vmnic0 and because of this the host profile couldn’t be applied. Okay, quick check in the vSphere Web Client: Only three NICs. C# client showed the same result. Now it was interesting: