It’s been a busy couple of weeks! I was in Vienna, Austria, all last week, and I’m on the US West Coast this week. Even though I’ve been on the go, I’ve still been collecting various virtualization-related posts and tidbits. Here they are for you in Virtualization Short Take #36! I hope you find something useful.

• You might recall that in early 2008 I wrote about how thin provisioned VMDKs on NFS storage tend to inflate. In a recent post, Chad Sakac pointed out that VMware has addressed this problem, which is caused by the use of the eagerzeroedthick VMDK format instead of the zeroedthick format. The fix requires ESX 3.5 Update 5 and VirtualCenter 2.5 Update 6, plus a configuration change that is outlined in this VMware KB article. Kudos to VMware for fixing the underlying issue instead of just forcing customers to upgrade to vSphere.
• VMware’s Scott Drummonds provides a bit more information on the memory compression technology previewed by Steve Herrod at Partner Exchange 2010 a few weeks ago. In my opinion, anyone who says that the hypervisor is a commodity isn’t paying attention to the fact that VMware is still innovating in this space.
• If you perform virtualization assessments using VMware’s Capacity Planner tool, you’ll find Gabe’s Capacity Planner troubleshooting tips helpful.
• Gabe also published a “wish list” for VMware datastores. If you take a deeper look at what Gabe is really trying to address, though, a great deal of the functionality he’s looking for could be achieved through a combination of policy-based storage tiering and greater integration between VMware and the storage array. Would you really need category labels on VMware datastores if the underlying storage was tiering data effectively based on utilization? Probably not. It might still make sense in some cases, but I think the vast majority of cases would be addressed. I think that you are going to see some very cool innovation in this space over the course of this year.
• Simon Gallagher recently asked this question: with the move to ESXi, is NFS more useful than VMFS? It appears that a large part of Simon’s argument centers around the speed at which files can be transferred into VMFS using ESXi, and it seems to me that VMware needs to do some optimization there. I’m not knocking NFS—I’ve used it extensively in the past and I have and continue to recommend it to customers where it is appropriate—but I’m not sure that you can build an argument for NFS based on ESXi’s file transfer performance. My friend and former colleague Aaron Delp (whose blog was recently added to Planet V12n; congrats!) points out that fixing VMDK alignment using ESXi could be an issue; now that’s a great point. Even third-party utilities like vOptimizer don’t work with ESXi. In my opinion, these points underscore the need for VMware to concentrate very heavily on ESXi if that is indeed going to be the “platform moving forward”.
• I came across an interesting VMware KB article while browsing the weekly VMware KB digest for the week ending 2/28/10. The article, which discusses a situation in which VMware HA would fail to configure at 90% completion, describes how some network switches—HP ProCurve 1810G switches with automatic denial-of-service protection enabled and Cisco Catalyst 4948 switches with ICMP rate limiting enabled—can drop packets that are necessary for VMware HA to configure and start correctly.
• Unfortunately, the latest VMware KB weekly digest (for the week ending 3/6/2010) didn’t include links to the actual articles that were published. Bummer! Still, it’s easy enough to simply look up the articles directly.
• EMC today released a couple of plug-ins for vCenter Server. The Celerra plug-in for VMware Environments brings Celerra NFS provisioning into the vSphere Client. The Celerra Failback Plug-in for SRM automates failback in VMware SRM environments. The official press release is here, which contains links to more information on the individual plug-ins. (Disclaimer: I work for EMC.)
• Newly-minted VCDX #029 Frank Denneman posted a good article on using reservations on resource pools to bypass slot sizing. As Frank points out, it’s not a recommended practice necessarily, but it might be warranted depending on customer requirements.
• Duncan’s recent article on the behavior of CPU and memory reservations is also helpful, especially for those who might not be familiar with the differences between the two types of reservations.
• Similarly, this guest post on Duncan’s site by VCDX Craig Risinger also helps explain how shares on a resource pool work. This is good information to have if you are unfamiliar with the topic.
• I’m not a security geek, but I did think that the RSA-Intel-VMware announcement at RSAC 2010 (third-party coverage here) was pretty cool. Security experts, I’d love to hear your thoughts on the matter. What was good about the announcement? What was missing?
• If you will be working with distributed vSwitches, this post by EMC’s Gregg Robertson might help; it underscores the need to ensure that your environment is being consistently and thoroughly patched and maintained. vCenter Update Manager, anyone?

I do have a few other articles in my “things to read list” that I haven’t yet gotten around to reading:

The Official Quest Software Desktop Virtualization Group Blog » Blog Archive » How to Integrate ThinApp with Quest vWorkspace 7.0
DRS Resource Distribution Chart
HP Flex-10 versus Nexus 5000 & Nexus 1000V with 10GE passthrough

That’s it for now. I hope that you’ve found something useful here, and—as always—I’d love to hear your thoughts in the comments below.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Virtualization Short Take #36

• Virtualization Short Take #30
• Virtualization Short Take #35
• Virtualization Short Take #31
• Virtualization Short Take #32
• Virtualization Short Take #33

• VMware Labs
  The VMware Labs site contains some unofficial and unsupported (but quite cool) tools from VMware engineers.

I was browsing through an EMC technical document titled “EMC CLARiiON Integration with VMware ESX Server” (download it here) a little while ago and I came across a phrase in the document that caught my attention:

“VMware ESX/ESXi support both Fibre Channel and iSCSI storage. However, VMware and EMC do not support connecting VMware ESX/ESXi servers to CLARiiON Fibre Channel and iSCSI devices on the same array simultaneously.”

What? No Fibre Channel and iSCSI from the same array to a VMware ESX/ESXi host simultaneously? That piqued my curiosity, so I contacted a few people within EMC to question the veracity of that statement. It turns out that the answer is more complicated than it might seem at first glance.

For those of you who aren’t interested in the deep technical details, here’s the short explanation behind this behavior:

• VMware fully supports the use of both Fibre Channel and iSCSI from the same array to the same VMware ESX/ESXi host simultaneously.
• VMware does not support presenting the same LUN via both protocols concurrently to the same host. (I qualified this directly with VMware.)
• For a Celerra, you can use both Fibre Channel (via the CLARiiON side of the array) and iSCSI (via the Celerra side of the array) simultaneously. This is a fully supported configuration.
• A CLARiiON array can easily present the same LUN via both Fibre Channel and iSCSI, but then VMware wouldn’t support it (see earlier bullet).
• With a CLARiiON array, it is possible to present some LUNs via Fibre Channel and some LUNs via iSCSI to the same VMware ESX/ESXi host (i.e., LUN A via Fibre Channel and LUN B via iSCSI), but EMC will only support it if you file an RPQ. Without an RPQ, it’s an unsupported configuration. An RPQ, by the way, is a request to qualify a certain configuration for support.

I’m confident that some other array vendors out there will be very quick to jump on this post and harp on this limitation until the cows come home. I would just ask this question: is it really as big of a limitation as it seems? I’ll come back to that question in a moment.

With the short explanation in mind, here are the more in-depth details. If you like the longer, more technical explanation, then read on!

From EMC’s side, the root of the restriction about using both Fibre Channel and iSCSI devices on the same array simultaneously stems from the interaction of host registration and storage groups.

Host registration is a requirement in the CLARiiON world. In order to present storage to a host from a CLARiiON array, you must first register the host’s initiators with the array in Navisphere. Once the host has been registered, then you can proceed with presenting storage to that host. In theory the CLARiiON could operate without registering hosts and initiators, but EMC chose to require registration. EMC made this choice in order to help simplify host management.

Requiring host registration is a bit different than some of other storage arrays on the market. It’s not better or worse—just different. (Remember, pros and cons come from every technology decision.)

If you’re like me, you’re probably wondering at this point how requiring host registration simplifies anything. Instead of having to manage multiple paths, multiple initiators, and individual hosts every time you want to present storage to a host, you only need to register the host—and all of its initiators—and then you can refer to that same object (the host) over and over again as needed. Yes, host registration does mean a bit more work up front, but the idea is that it will save some work down the road. I guess you can think of host registration kind of like defining aliases in your Fibre Channel zoning configuration: it’s a bit more work up front, but it simplifies things later down the road. If you didn’t create device aliases in your Fibre Channel switch, you’d end up having to re-enter Fibre Channel WWPNs multiple times. You create the aliases so that it’s easier later. The same applies to host registration. Again, it’s a matter of choices.

One might also say that registration is security measure, albeit a weak measure. Rather than allow just any Fibre Channel-attached or iSCSI-attached host to see storage, the array requires that it know about the host (via host registration) in order to present storage to the host. This provides an additional layer of security to ensure that only authorized hosts are presented storage from the array.

Now you have a fairly decent idea of why host registration is necessary. So how does host registration occur? Host registration can occur either manually or automatically. Starting with version 4.0, both VMware ESX and VMware ESXi will automatically register with a CLARiiON array running any recent version of FLARE (ESX 3i version 3.5 also supports this form of push registration). FLARE release 28 and earlier will show these hosts as “Manually registered, unmanaged”; starting with FLARE 29, these hosts are listed as “Manually registered, managed”. In either case, the registration occurs automatically. If the host is Fibre Channel-attached, then the Fibre Channel initiators will be included in the automatic registration. The same goes for iSCSI initiators. Normally, this is a good thing because it saves the administrator the extra steps of registering the host with the storage array. (Also, because VMware ESX/ESXi hosts register automatically, there is no need to install the Navisphere Agent.)

In this case, though, the automatic registration causes a problem. Why? This goes back to the second item I said I needed to discuss: storage groups. Specifically, storage groups have two characteristics that come into play here:

1. First, any given host—not just VMware ESX/ESXi hosts, but all types of hosts—can only be connected to a single storage group at any given time.
2. Second, while the CLARiiON can present Fibre Channel LUNs and iSCSI LUNs simultaneously (including presenting the same LUN via both protocols simultaneously), there is no way within a single storage group to specify which LUNs should be accessed via Fibre Channel and which LUNs should be accessed via iSCSI. This is necessary because VMware won’t support accessing the same LUN via both protocols at the same time (see earlier VMware support statement).

Do you see how all the pieces come together? The only way to control which LUNs should be presented via which protocol is to use multiple storage groups—but a host can only be in a single storage group at a time. With only a single host object for any given VMware ESX/ESXi host, that host can only see either Fibre Channel LUNs (by being in a storage group containing Fibre Channel LUNs) or iSCSI LUNs (by being in a storage group containing iSCSI LUNs), but not both. Hence, the statement in the CLARiiON document I referenced in the very beginning of this blog post that outlines using either Fibre Channel or iSCSI but not both. This behavior is required to enforce the single-protocol LUN access required by VMware.

As with all things, there is a workaround. Because it is a workaround, that’s why the RPQ is necessary to get full support.

To work around this problem, you’ll need to ignore the automatic host registration (or disable the automatic host registration) and instead create two manually registered “pseudo-hosts”: one with the Fibre Channel initiators and one with the iSCSI initiators. These “pseudo-hosts” will need fake IP addresses (if they both use the same IP address, Navisphere will treat them as the same host, thus defeating the purpose of the workaround). Put the Fibre Channel initiators into the Fibre Channel storage group(s), and put the iSCSI initiators into the iSCSI storage group(s). Each “pseudo-host” will be able to see LUNs presented to that storage group and therefore would see both Fibre Channel and iSCSI LUNs at the same time. And, as required by VMware, any given LUN would be accessed only via Fibre Channel or iSCSI but not both. Remember that you need to file an RPQ in order to get support on this configuration.

For VMware ESX/ESXi 4.0 hosts (and ESX 3i version 3.5 hosts), you can disable automatic registration using the Disk.EnableNaviReg advanced configuration option. Setting this value to 0 disables the automatic registration with Navisphere. (Here are screenshots for VMware ESX 3i and VMware ESX/ESXi 4.) If you disable the automatic registration, then you only need to manually register the Fibre Channel and iSCSI initiators as separate “pseudo-hosts” and you’re ready to go.

Let me reiterate again that if you are presenting iSCSI LUNs via the Celerra and not the CLARiiON, none of this applies. Presenting Fibre Channel LUNs via the CLARiiON and iSCSI LUNs via the Celerra to the same VMware ESX/ESXi host is fine. This workaround that I’ve described only applies when you want to present some LUNs via Fibre Channel and some LUNs via iSCSI from a CLARiiON to a single VMware ESX/ESXi host.

Earlier you’ll recall that I asked this question: is this really a limitation? There are a couple of viewpoints:

• One viewpoint states there is no need for both Fibre Channel and iSCSI connectivity to the same array. Since you already have Fibre Channel connectivity to the array, what’s the point in using iSCSI? Conversely, if you already have iSCSI connectivity to an array, why invest in establishing Fibre Channel connectivity? Since you can’t use it for failover (that would violate the VMware support position), running another block protocol against the same array and same sets of disks doesn’t add a great deal of value.
• A second viewpoint argues that the ability to provide a differentiation of service based on the different performance characteristics of Fibre Channel and iSCSI (and NFS, but we’re focusing on block protocols for this discussion) is valuable, and thus the need to be able to easily present LUNs via either protocol from the same array to the same host is a worthwhile function. There are a number of potential use cases here—test/development environments, Tier 2 applications, varying SLAs, etc. This is especially true if you are using different disk pools (fast Fibre Channel drives or EFDs vs. slower SATA drives) on the same array.

I can see both sides of the coin. Personally, I tend to side more with the second viewpoint and would prefer to see the CLARiiON have the ability to easily present Fibre Channel and iSCSI to the same host, especially when multiple disk pools are involved. I think that CLARiiON engineering is now evaluating this possibility; as more information emerges, I’ll be sure to keep you posted.

Courteous and professional comments, clarifications, or corrections are always welcome!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

VMware ESX, EMC CLARiiON Arrays, and Multiple Protocols

• NetApp iGroup Strategies for VMware ESX
• Sanrad Configuration Basics
• 4198: Hyper-V on NetApp Deep Dive
• iSCSI from NetApp to ESX Server
• Understanding NPIV and NPV

• Gift Idea - Speck SeeThru Satin Soft-Touch Hard Shell for MacBook Pro
  This is an attractive hard shell for my MacBook Pro.
• Information Storage and Management
  This book provides in-depth information on information storage topics like protocols, storage virtualization, replication, security, and administration. It's written by a team of authors from EMC.

I recently had the opportunity to work on a proof of concept (PoC) in which we wanted to help a customer streamline the processes needed to deploy new hosts and reduce the amount of time it took overall. One of the tools we used in the PoC for this purpose was PXE booting VMware ESX for an automated installation. Here are the details on how we made this work.

Before I get into the details, I’ll provide this disclaimer: there are probably easier ways of making this work. I specifically didn’t use UDA or similar because I wanted to gain the experience of how to do this the “old fashioned” way. I also wanted to be able to walk the customer through the “old fashioned” way and explain all the various components.

With that in mind, here are the components you’ll need to make this work:

1. You’ll need a DHCP server to pass down the PXE boot information. In this particular instance, I used an existing Windows-based DHCP server. Any DHCP server should work; feel free to use the Linux ISC DHCP server if you prefer.
2. You’ll need an FTP server to host the kickstart script and VMware ESX 4.0 Update 1 installation files. In this case, I used a third-party FTP server running on the same Windows-based server as DHCP. Again, feel free to use a Linux-based FTP server if you prefer.
3. You will need a TFTP server to provide the boot files. The third-party FTP server used in the previous step also provided TFTP functionality. Use whatever TFTP server you prefer.

Make sure that each of these components is working as expected before proceeding. Otherwise, you’ll spend time troubleshooting problems that aren’t immediately apparent.

Preparing for the Automated ESX Installation

First, copy the contents for the VMware ESX 4.0 Update 1 DVD—not the actual ISO, but the contents of the ISO—to a directory on the FTP server. Test it to make sure that the files can be accessed via an anonymous FTP user.

Also go ahead and create a simple kickstart script that automates the installation of VMware ESX. I won’t bother to go into detail on this step here; it’s been quite adequately documented elsewhere. You’ll need to put this kickstart script on the FTP server as well.

At this point, you’re ready to proceed with gathering the PXE boot files.

Gathering the PXE Boot Files

The first task you’ll need to complete is gathering the necessary files for a PXE boot environment.

First, copy the vmlinuz and initrd.img files from the VMware ESX 4.0 Update 1 ISO image. Since I use a Mac, for me this was a simple case of mounting the ISO image and copying out the files I needed. Linux or Windows users, it might be a bit more complicated for you. These files, by the way, are in the ISOLINUX folder on the DVD image.

Next, you’ll need the PXE boot files. Specifically, you’ll need the menu.c32 and pxelinux.0 files. These files are not on the DVD ISO image; you’ll have to download Syslinux from this web site. Once you download Syslinux, extract the files into a temporary directory. You’ll find menu.c32 in the com32/menu folder; you’ll find pxelinux.0 in the core folder. Copy both of these files, along with vmlinuz and initrd.img, into the root directory of the TFTP server. (If you don’t know the root directory of the TFTP server, double-check its configuration.)

You’re now ready to configure the PXE boot process.

Configuring the PXE Boot Environment

Once the necessary files have been placed into the root directory of the TFTP server, you’re ready to configure the PXE boot environment. To do this, you’ll need to create a PXE configuration file on the TFTP server.

The file should be placed into a folder named pxelinux.cfg under the root of the TFTP server. The filename of the PXE configuration file should be named something like this:

01-<MAC address of network interface on host>

If the MAC address of the host was 01:02:03:04:05:06, the name of the text file in the pxelinux.cfg folder on the TFTP server would be:

01-01-02-03-04-05-06

The PoC in which I was engaged involved Cisco UCS, so we knew in advance what the MAC addresses were going to be (the MAC address is assigned in the UCS service profile).

The contents of this file should look something like this (lines have been wrapped here for readability and are marked by backslashes; don’t insert any line breaks in the actual file):

default menu.c32
menu title Custom PXE Boot Menu Title
timeout 30
 
label scripted
menu label Scripted installation
kernel vmlinuz
append initrd=initrd.img mem=512M ksdevice=vmnic0 \
  ks=ftp://A.B.C.D/ks.cfg
IPAPPEND 1

You’ll want to replace ftp://A.B.C.D/ks.cfg with the correct IP address and path for the kickstart script on the FTP server.

Only one step remains: configuring the DHCP server.

Configuring the DHCP Server for PXE Boot

As I mentioned earlier, I used the Windows DHCP server as a matter of ease and convenience; feel free to use whatever DHCP server best suits your needs. There are only two options that are necessary for PXE boot:

066 Boot Server Host Name (specify the IP address of the TFTP server)
067 Bootfile Name (specify pxelinux.0)

In this particular example, I created reservations for each MAC address. Because the values were the same for all reservations, I used server-wide DHCP options, but you could use reservation-specific DHCP options if you wanted different boot options on a per-MAC address (i.e., per-reservation) basis.

The End Result

Recall that this PoC was using Cisco UCS blades. Thus, in this environment, to prepare for a new host coming online we only had to make sure that we had a PXE configuration file and create a matching DHCP reservation. The MAC address would get assigned via the service profile, and when the blade booted then it would automatically proceed with an unattended installation. Combined with Host Profiles in VMware vCenter, this took the process of bringing new ESX/ESXi hosts online down to mere minutes. A definite win for any customer!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

PXE Booting VMware ESX 4.0

• Client-Specific DHCP Options with Linux DHCP Server
• Modifying DHCP Client Operation on OpenBSD
• Badmail and Exchange 2003
• Full VM Recovery with NetApp Snapshots
• Apparent Date/Time Issue With Update 2