What to consider when implementing HP 3PAR with iSCSI in VMware environments

Some days ago a colleague and I implemented a small 3-node VMware vSphere Essentials Plus cluster with a HP 3PAR StoreServ 7200c. Costs are always a sore point in SMB environments, so it should not surprise that we used iSCSI in this design. I had some doubt about using iSCSI with a HP 3PAR StoreServ, mostly because of the performance and complexity. IMHO iSCSI is more complex to implement then Fibre Channel (FC). But in this case I had to deal with it.

iSCSI options for HP 3PAR StoreServ 7000

If you decide to use iSCSI with a HP 3PAR StoreServ, you have only one option: Adding a 2-port 10GbE iSCSI/ FCoE adapter to each node. There is no other iSCSI option. The available 2-port 10GbE ethernet adapter and 4-port 1GbE ethernet adapter can’t be used for iSCSI connectivity. These adapters can only be used with the HP 3PAR File Persona Software Suite.

The 2-port 10GbE iSCSI/ FCoE adapter is a converged network adapter (CNA) and supports iSCSI or Fibre Channel over Ethernet (FCoE). The adapter can only be used for host connectivity and you have to select iSCSI or FCoE. You can’t use the CNA for remote copy. You have to add a CNA to each nodes in a node pair. You can have up to four 10 GbE ports in a 3PAR 7200 series, or up to eight 10 GbE ports in a 3PAR 7400 series.

Network connectivity

10 GbE means 10 GbE, there is no way to connect the CNA to 1 GbE transceivers. The 2-port 10GbE iSCSI/ FCoE includes two 10 GbE SR SFP+ transceivers. With 3PAR OS 3.1.3 and later, you can use Direct Attach Copper (DAC) cables for network connectivity, but not for FCoE. Make sure that you use the correct cables for your switch! HP currently offers the following cables in different length:

• X242 for HP ProVision switches
• X240 for HP Comware switches
• HP B-series SFP+ to SFP+ Active Copper for Brocade switches, or
• HP C-series SFP+ to SFP+ Active Copper for Cisco switches

If you use any other switch vendor, I strongly recommend to use the included 10 GbE SR SFP+ transceivers and 10 GbE SR SFP+ transceivers on the switch-side. In this case you have to use fiber cable to connect the 3PAR to the network. In any other case I recommend to use DAC for network connectivity.

It’s a common practice to run iSCSI traffic in its own VLAN. Theoretically a single iSCSI VLAN is sufficient. I recommend to use two iSCSI VLANs in case of a 3PAR, one for each iSCSI subnet. Why two subnets? The answer is easy: Persistent Ports. Persistent Ports allows an host port to assume the identity (port WWN for Fibre Channel or IP address for iSCSI ports) of a failed port while retaining its own identity. This minimizes I/O disruption during failures or upgrade. Persistent Ports uses the NPIV feature for Fibre Channel/ FCoE and IP address failover for iSCSI. With the release of 3PAR OS 3.1.3, Persistent Ports was also available for iSCSI. A hard requirement of Persistent Ports is, that the same host ports of nodes of a node pair must be connected to the same IP network on the fabric. An example clarifies this:

Host port (N:S:P)	VLAN ID	IP subnet
0:2:1	11	192.168.173.0/27
0:2:2	12	192.168.173.32/27
1:2:1	11	192.168.173.0/27
1:2:2	12	192.168.173.32/27

The use of jumbo frames with iSCSI is a often discussed topic. It’s often argued that complexity and performance gain would be disproportionate. I’m a bit biased. I think that the use of jumbo frames is a must when using iSCSI. I always configure jumbo frames for vMotion, so the costs for configuring Jumbo frames is low for me in an iSCSI environment. Don’t forget to configure jumbo frames on all devices in the path: VMkernel ports, vSwitches, physical switches and 3PAR CNAs.

Always use at least two physical switches for iSCSI connectivity. This concept is compareable to a Fibre Channel dual-fabric SAN. I like the concept of switch aggegration (the wording may vary between vendors). I often work with HP Networking and I like the HP 2920 or 5820 Switch Series. These switches can form stacks in which multiple physical switches act a as a single virtual device. These stacks provide redundancy and operational simplicity. In combination with two VLANs you can build a powerful, redundant and resilient iSCSI SAN.

Host port configuration

The CNA ports can only be used for host connectivity, therefore there is no way to use them for disk or remote copy connectivity. Before you can use the port for host connectivity, you have to select iSCSI or FCoE as storage protocol.

Host and Virtual Volume sets

You can organize hosts and volumes in host sets and volume sets. I recommend to create a host set for all ESXi hosts in a vSphere cluster. I also recommend to create a volume set to group all volumes that should be presented to a host or host set. When exporting Virtual Volumes (VV), you can export a volume set to a host set. If you add a host to the host set, the host will see all volumes in the volume set. If you add a volume to a volume set, the hosts in the host set will all see the newly added volume. This simplifies host and volume management and it reduced the possibilty of human errors.

Custom SATP rule for ESXi 5.x and ESXi 6.0

3PAR OS 3.1.2 introduced the new Host Persona 11 for VMware which enables asymmetric logical unit access (ALUA). Beside Host Persona 11, Host Persona 6 for VMware is also available, but it doesn’t support ALUA. 3PAR OS 3.1.3 is the last release that included support Host Persona 6 AND 11 for VMware. All later releases only include Host Persona 11. I strongly recommend to use Host Persona 11 for VMware. You should also add a custom SATP rule. This rule can be added by using ESXCLI.

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -P "VMW_PSP_RR" -O iops=1 -c "tpgs_on" -V "3PARdata" -M "VV" -e "HP 3PAR Custom iSCSI/FC/FCoE ALUA Rule"

This custom rule sets VMW_PSP_RR as the default PSP and it evenly distribute the IOs over all active paths by switching to the next active path after each IO.

iSCSI discovery

Before you can use an exported volume, the host needs to discover the volume from the target. You have to configure the iSCSI discovery in the settings of the software iSCSI initiator. Typically you will use the dynamic discovery process. In this case, the initiator uses SendTargets request to get a list of available targets. After adding the IP addresses of the 3PAR CNAs to the dynamic discovery list, the static discovery list is filled automatically. In case of multiple subnets, the dynamic discovery process can carry some caveats. Chris Wahl has highlighted this problem in his blog post “Exercise Caution Using Dynamic Discovery for Multi-Homed iSCSI Targets”. My colleague Claudia and I stumbled over this behaviour in our last 3PAR project. Removing the IP addresses from the dynamic discovery will result in the loss of the static discovery entries. After a reboot, the entries in the static discovery list will be gone and therefore no volumes will be discovered. I added a comment to Chris blog post and he was able to confirm this behaviour. The solution is to use the dynamic discovery to get a list of targets, and then add the targets manually to the static discovery list.

Final words

HP 3PAR and iSCSI is a equivalent solution to HP 3PAR and Fibre Channel/ FCoE. Especially in SMB environments, iSCSI is a good choice to bring to the customer 3PAR goodness to a reasonable price.