Vmware

In December 2013, VMware made an christmas gift to the community by releasing pyVmomi. pyVmomi is a SDK that allows you to manage VMware ESXi and vCenter using Python and the VMware vSphere API. Nearly 18 months are past since then and pyVmomi has developed over time.

I’ve started to play around with Python, and I’ve written about the reasons in one of my last blog posts (Hey infrastructure guy, you should learn Python!).

In March 2016, HPE CEO Meg Whitman announced a ProLiant-based HCI solution, that should be easier to use and cheaper than Nutanix.

This isn’t HPEs first dance on this floor. In August 2015, HP launched the Hyper Converged 250 System (HC250), which is based on the Apollo server platform. The HW design of the HC250 comes close to a Nutanix Block, because the Apollo platform supports up to four nodes in 2U. Let me say this clear: The Hyper Converged 380 (HC380) is not a replacement for the HC250! And before the HC250, HPE offered the Converged System 200-HC StoreVirtual and 200-HC EVO:RAIL (different models).

Last year in December, I updated the first customer from HPE Data Protector 9.04 to 9.05. Immediately after the first tests I noticed, that backups were made using the NBDSSL transport. I expected that the SAN transport would be used, because the prerequisites were met and it has worked until the update. I opened a case at the HPE support und I was advised to install the hotfix QCIM2A65619. With this hotfix, several files were replaced:

The Virtual Environment Integration (VE Integration) provides protection of VMs in virtual server environments. It is used o integrate HPE Data Protector with various virtualization environments, currently VMware vSphere and Microsoft Hyper-V. For Citrix XenServer is a script solution available. I will focus on VMware vSphere.

What is possible?

I took this table from the “HPE Data Protector 9.00 Integration Guide for Virtualization”.

Feature	VE Integration
Online backup	✓
Crash-consistent backup	✓
Application-consistent backup	✓
Granularity	vmdk, vmx
Full/ Incremental/ Differential	✓/ ✓/ ✓
Support for changed block tracking (CBT)	✓
Where does the Data Protector component need to be installed?	backup host
Extra licenses needed	1x On-Line Extension per ESXi host

As you can see, Data Protector offers all you need to create a crash-consistent backup of your VMs. HPE Data Protector relies on the VMware vSphere Storage APIs – Data Protection (formerly known as VMware vStorage APIs for Data Protection or VADP). Data Protector has to use the same API as Veeam, CommVault Simpana or any other product that can be used to backup VMs in a VMware vSphere environment. Therefore, most software products offer the same features.

After updating my lab to VMware vSphere 6.0 U2, one of my hosts continuously thrown an error during an update scan.

Host returns ESX error code 99, unhandled exception has occurred

The first thing I’ve checked was the esxupdate.log on the affected ESXi host. This is the output, that was logged during a scan operation.

2016-04-04T13:42:13Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2016-04-04T13:42:14Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2016-04-04T13:42:14Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2016-04-04T13:42:14Z esxupdate: esxupdate: INFO: ---
Command: scan
Args: ['scan']
Options: {'nosigcheck': None, 'retry': 5, 'loglevel': None, 'cleancache': None, 'viburls': None, 'meta': ['http://vum.lab.local:9084/vum/repository/hostupdate/10960002/metadata_1456989617.zip', 'http://vum.lab.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip'], 'proxyurl': None, 'timeout': 30.0, 'cachesize': None, 'hamode': True, 'maintenancemode': None}
2016-04-04T13:42:14Z esxupdate: BootBankInstaller.pyc: DEBUG: Creating an empty ImageProfile for bootbank /bootbank
2016-04-04T13:42:14Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2016-04-04T13:42:14Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/bootOption', '-ro']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2016-04-04T13:42:14Z esxupdate: downloader: DEBUG: Downloading http://vum.lab.local:9084/vum/repository/hostupdate/10960002/metadata_1456989617.zip to /tmp/tmpWW6WJC...
2016-04-04T13:42:17Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2016-04-04T13:42:17Z esxupdate: downloader: DEBUG: Downloading http://vum.lab.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip to /tmp/tmpKfdI64...
2016-04-04T13:42:20Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2016-04-04T13:42:21Z esxupdate: BootBankInstaller.pyc: DEBUG: Creating an empty ImageProfile for bootbank /bootbank
2016-04-04T13:42:22Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/usr/sbin/vsish', '-e', '-p', 'cat', '/hardware/bios/dmiInfo']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2016-04-04T13:42:22Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/smbiosDump']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2016-04-04T13:42:22Z esxupdate: BootBankInstaller.pyc: DEBUG: Creating an empty ImageProfile for bootbank /bootbank
2016-04-04T13:42:22Z esxupdate: BootBankInstaller.pyc: DEBUG: Creating an empty ImageProfile for bootbank /bootbank
2016-04-04T13:42:23Z esxupdate: BootBankInstaller.pyc: DEBUG: Creating an empty ImageProfile for bootbank /bootbank
2016-04-04T13:42:23Z esxupdate: Transaction: DEBUG: Populating VIB list from all VIBs in metadata http://vum.lab.local:9084/vum/repository/hostupdate/10960002/metadata_1456989617.zip; depots:
2016-04-04T13:42:23Z esxupdate: downloader: DEBUG: Downloading http://vum.lab.local:9084/vum/repository/hostupdate/10960002/metadata_1456989617.zip to /tmp/tmpFQrdX3...
2016-04-04T13:42:23Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2016-04-04T13:42:23Z esxupdate: Transaction: DEBUG: Populating VIB list from all VIBs in metadata http://vum.lab.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip; depots:
2016-04-04T13:42:23Z esxupdate: downloader: DEBUG: Downloading http://vum.lab.local:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip to /tmp/tmplZxgm6...
2016-04-04T13:42:23Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR: An unexpected exception was caught:
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 238, in main
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR:     cmd.Run()
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/Cmdline.py", line 113, in Run
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/MetadataScanner.py", line 244, in Scan
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/MetadataScanner.py", line 106, in _generateOperationData
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/MetadataScanner.py", line 88, in _getInstallProfile
2016-04-04T13:42:24Z esxupdate: esxupdate: ERROR: AttributeError: 'NoneType' object has no attribute 'Copy'
2016-04-04T13:42:24Z esxupdate: esxupdate: DEBUG: <<<

You might notice the “Unrecognized file vendor-index.xml in Metadata file” error. I also found this error message on the other hosts, so I excluded it from further research. It was unlikely, that this error was related to the observed problem. I started searching differences between the hosts and found out, that the output of “esxcli software vib list” was different on the faulty host.

Nearly two weeks ago, I wrote a blog post (VMware Horizon View space reclamation fails) about failing space reclamation on automated desktop pools with linked clones. Today I write about the same error, but caused by another problem. In both cases, the error is logged in the View Administrator and the vSphere (Web) Client. On the View Administrator, the following error is shown:

"Failed to perform space reclamation on machine COMPUTER NAME in Pool POOL NAME"

A customer notified me, that he observed an issue with the space reclamation on two automated desktop pools with linked clones. His environment is based on Horizon View 6.2.1 and vSphere 5.5U3. The error was logged in the View Administrator and the vSphere (Web) Client. In the View Administrator, the following error was visible:

"Failed to perform space reclamation on machine COMPUTER NAME in Pool POOL NAME"

VMware vSphere 6 is now an year old and it was time to update my lab to vSphere 6. The update went smooth, and everything has worked as expected. Some days later, I updated the master VM of a small automated desktop pool. I’m using VMware Horizon 6.2.1 in my lab to deploy a small number of Windows 8.1 VMs for tests, administration etc. The recompose of the pool failed during the guest customization.

Yesterday I did an upgrade of my vCenter Server Appliance 5.5 U3 to 6.0 U1. This was the first step to update my lab infrastructure to vSphere 6.0. A bit late, but better late than never. The update of the VCSA itself went smooth. No problems with certificates, hosts, VMs or PernixData FVP. But then, I discovered two errors on the old vSphere C# client (I know that I should use the Web Client…)

Today, I was at a customer to prepare a two node vSphere cluster for some MS SQL server tests. Nothing fancy, just two HP ProLiant BL460c Gen9 blades and two virtual volumes from a HP 3PAR. Each blade had two 400 GB SSDs, two 64 GB M.2 SSDs and a 1 GB MicroSD card. Usually, I install ESXi to a SD card. In this case, a MicroSD card. The SSDs were dedicated for PernixData FVP. Although I saw the MicroSD card in the boot menu, ESXi doesn’t showed it as a installation target.