Video: What’s New in vCenter Ops Manager 5.7!

This video is an overview of VMware vCenter Operations Manager v5.7, a component of vCenter Operations Management Suite that provides you with operations dashboards allowing insight into health, risk and efficiency of your infrastructure.
It also includes performance management and capacity optimization capabilities.

This latest release comes with the following enhancements:

  • Allocation-based Capacity Planning
  • Greater Scalability
  • New and Improved Widgets

VMware vFabric Blog: 15% Discount for Spring Java Training in May

Training is a great way to speed up development, learn how to improve performance and usability for your applications and generally build confidence in your skills. This month, SpringSource is offering java developers a 15% discount code on all VMware trainings including Core Spring, Spring Web, Enterprise Integration, and Hibernate classes.

To secure your 15% discount, be sure to use the promo code springcustomerpromo during your registration process (promo is not available for partners). All of the following qualifying classes for May, 2013 can be found below:

Step 1: Core Spring


Asia Pacific

Europe, Middle East & Africa

Step 2: Spring Web / Enterprise Integration with Spring / Hibernate with Spring


Europe, Middle East & Africa

Live Online

Note: If you cannot find a professional training near you, you can always request an onsite SpringSource training

Part II: Storage Boon or Bane – VMware View Storage Design Strategy & Methodology

By TJ Vatsa, VMWare EUC Consultant

INTRODUCTION Welcome to Part II of the VMware View Storage Design Strategy and Methodology blog. This blog is in continuation to Part I that can be found here. In the last blog, I had listed some of the most prevalent challenges that impede a predictable VMware View Storage design strategy.. In this blog, I will articulate some of the successful storage design approaches that are employed by VMware End User Computing (EUC) Consulting practice to overcome those challenges.

I’d like to reemphasize the fact that storage is very crucial to a successful VDI deployment. Should the VDI project be made prone to the challenges listed in Part I, Storage, for sure, will seem to be a “bane”. But, if the recommended design strategy listed below is followed, you would be surprised to find VDI Storage being a “boon” for a scalable and predictable VDI deployment.

With that in mind, let’s dive in. Some successful storage design approaches I’ve encountered are the following:

  • 1.     PERFORMANCE Versus CAPACITY Recommendation: “First performance and then capacity”
    Often times, capacity seems more attractive when compared to performance. But, is it really so? Let’s walk through an example.
  • a)     Let’s say vendor “A” is selling you a storage appliance, “Appliance A” that has a total capacity of 10TB, being delivered by 10 SATA drives of 1TB capacity each.
  • b)     On “Appliance A”, let’s say that each SATA drive delivers approximately 80 IOPS. So, for 10 drives, the total IOPS being delivered by the appliance is 800 IOPS (10 drives * 80 IOPS).
  • c)     Now let’s say that vendor “B” is selling you a storage appliance, “Appliance B” that also has a total capacity of 10TB, but it is being delivered by 20 SATA drives of 0.5TB capacity each. [Note: “Appliance B” may be expensive as there is more drives compared to “Appliance A”.]
  • d)     Now for “Appliance B”, assuming that the SATA drive specifications are the same as those of “Appliance A”, you should be expecting 1600 IOPS (20 drives * 80 IOPS)
  • It’s mathematically clear; “Appliance B” will be delivering twice as much IOPS than “Appliance A”. More storage IOPS invariably turns out to be a boon for a VDI deployment. Another important point to consider, is the fact that employing higher tier storage also ensures high IOPS availability. Case in point, replacing the SATA drives in the example above with SAS drives will certainly provide higher IOPS. SSD drives, while expensive, will provide even higher IOPS.
  • 2.     USER SEGMENTATIONRecommendation: Intelligent user segmentation that does not assume “one size fits all approach”.
  • As was explained in Part I, taking a generic user IOPS, say “X” and then multiplying that with the total number of VDI users in an organization say “Y”, may result in an Oversized or an Undersized Storage Array design. This approach may prove costly, either upfront or at a later date.

    The recommended design approach is to intelligently categorize the user’s IOPS as “Small, Medium or High” based on the load a given category of users generate across the organization. As part of the common industry nomenclature for VDI users:

    a)     Task Workers: associated with small IOPS.
    b)     Knowledge Workers: associated with medium IOPS.
    c)     Power Users: associated with high IOPS.

    With these guidelines in mind, let me walk you through an example. Let’s say that Customer A’s Silicon Valley campus location has 1000 VDI users. Assuming that the user % split is:

    a)     15% Task Workers with an average of 7 IOPS each
    b)     70% Knowledge Workers with an average of 15 IOPS each
    c)     15% Power Users with an average of 30 IOPS each

    The resulting calculation of total estimated IOPS required will look similar to Table 1 below.

    Key Takeaways:

  1. It is highly recommended to discuss/consult with the customer and to also make use of a desktop assessment tool to determine the user % distribution (split) as well as the average IOPS per user segmentation.
  2. Estimated capacity growth and the buffer percentage, is assumed to be 30%. This may vary for your customer based on the industry domain and other factors.
  3. This approach to IOPS calculation is more predictable based on user segmentation specific to a given customer’s desktop usage.
  4. You can apply this strategy to customers from Healthcare, Financial, Insurance Services, Manufacturing and other industry domains.

3.     OPERATIONSRecommendation: “Include Operational IOPS related to Storage Storms”.

It is highly recommended to proactively account for IOPS related to the storage storms. Any lapse can result in a severely, painful VDI user experience during the storage storms – Patch Storm, Boot Storm and Anti-Virus (AV) storm.

Assuming that a desktop assessment tool is employed to do the analysis, it is recommended to analyze the user % split targeted during each of the storm operations listed above.

For instance, if the desktop operations team pushes OS/Application/AV patches in batches of 20% of the total user community, and the estimated IOPS is let’s say three times the steady state IOPS (explained in Part I), it will be prudent to include another attribute for operational IOPS to Table 1 listed above.

A similar, strategy should also be employed to account for the boot and the log-off storms.

I hope you will find this information handy and useful during your VDI architecture design and deployment strategy.

Until next time. Go VMware!

TJ Vatsa has worked at VMware for the past 3 years with over 19 years of expertise in the IT industry, mainly focusing on the enterprise architecture. He has extensive experience in professional services consulting, Cloud Computing, VDI infrastructure, SOA architecture planning, implementation, functional/solution architecture, and technical project management related to enterprise application development, content management and data warehousing technologies.


VMware vSphere Blog: Understanding ESXi Patches – Size & Patch Bundles

Patch management for ESXi is very different compared to traditional operating system patches, where incremental updates are made to the base operating system and thus increasing the disk footprint for each patch update. For the ESXi hypervisor, when a patch is applied, the entire ESXi image also known as an Image Profile is replaced. This means that each time you patch or upgrade your ESXi host, you are not adding on top of the original installation size.

As part of the ESXi architecture, there are two independent boot bank partitions that are used to store the ESXi Image Profile. This is primarily used as a fail-safe mechanism for rollback.

Here is a diagram showing what the ESXi boot banks would look like before and after applying a patch (pertains to both updates and upgrades)

Another common question that I see frequently asked is whether ESXi patches are cumulative? The answer is yes, they are cumulative. However, at first glance at the patch downloads on the VMware’s patch website, this may not be obvious.

To help clarify this, it is important to first understand the contents of an ESXi patch download (also known as patch bundle or offline bundle). A patch bundle can contain multiple bulletins and each bulletin will contain either the ESXi Hypervisor OS (esx-base) and/or VMware Tools ISO images (tools-light). On occasion, a patch bundle may also contain driver updates. A bulletin will be categorized as either SG (security) or BG (bug fixes).

An SG bulletin means that ONLY security fixes are included and it excludes any functional bug fixes. The reason for having a security only bulletin is for customers that have a very stringent requirement for fixing known security vulnerabilities in a short time frame that does not allow for vetting of the entire patch release. A BG bulletin contains both the functional bug fixes and security fixes. An example of patch bundle containing all four bulletin categories would be ESXi510-201212001

Lastly, each bulletin is just comprised of VIBs which are cumulative from all previous VIBs. If we take an example of a BG bulletin that was released in July, then it will contain all the cumulative bug fixes and security fixes from Jan to June.

As you can see, with the way the ESXi hypervisor is architected, updates and/or upgrades do not increase the disk footprint like traditional Operating Systems. In addition, VMware also provides a very flexible way of either applying all bug fixes or allowing users to select  security only updates based on customer’s security policies and procedures.

Here are some additional articles that may be useful in regards to ESXi patching:

Get notification of new blog postings and more by following VMware Automation on Twitter:  @VMWAutomation

How to move VMware Single Sign On (SSO) database

Customer had all his VMware databases for vCenter Server, Update Manager, Single Sign On (SSO), vCloud Director and vCenter Chargeback running on one big SQL Server where they shared resources with other databases. They asked me to move the databases to a new MS SQL Server because the load of the VMware databases was more than expected. I prepared the move by reading a few VMware KB’s that described how to move the databases, but still experienced some issues, that is why I wrote this blogpost to have a good manual for the next time I have to move these databases.

This first post shows how to move the SSO database.

Move the SSO database

Before the actual move, make sure that you’ve read the following KB: Updating the vCenter Single Sign On server database configuration (2045528).  Since we also have to move the database, which is not described in the mentioned KB article, there are some extra steps to perform. This post only works for MS SQL Server.

  • Make a connection to the old database server
  • On the SSO server, stop the Single Sign On service.
  • On the old database server make a backup of the SSO database (default name is RSA).
  • After a successful backup, set the SSO database to Offline
  • Copy the backup to the new database server
  • On the new database server, import the SSO database
  • On the security section of the SSO database, check if the user RSA_USER is present.

After this, the SSO database is available, but there is no login user connected. Usually when you create a SQL user, you also make a mapping to a database which then automatically creates the database user. In our case the database user is already present but is not yet mapped to a SQL user. That’s what we’ll do now. First let’s make sure the RSA_USER is not yet mapped:

  • Run the following sql query against the SSO database to show all unmapped users of the database: sp_change_users_login report
  • Now create a new SQL User (SQL Authentication) at the SQL Server level not at database level. Name this user RSA_USER and use the same password the database RSA_USER has. Set the default database to RSA (the SSO database).
  • Run the following sql query against the SSO database to map the user RSA_USER (server level) to the RSA_USER (database level): sp_change_users_login ‘update_one’, ‘RSA_USER’, ‘RSA_USER’
  • To check if things worked out, rerun the query against the SSO database. The RSA_USER should not show up: sp_change_users_login report

When running the queries, make sure you run them against the correct database. See image.


Next step is to create the RSA_DBA user which is only a SQL Server user and not a database user. But this user should become the owner of the SSO database.

  • At SQL Server level create the SQL user RSA_DBA and be sure to use the same password you previously used. (Well, you can always reset it later on).
  • After the RSA_DBA user has been created, open the properties of the user and now set this user as the owner of the SSO database

Your database is now ready for use. We just have to tell the SSO Server that it should look somewhere else from now on. To do this run the following on the SSO Server:

  • Go to the ssoserverutils folder (usually: C:Program FilesVMwareInfrastructureSSOServer) . Run the command: ssocli configure-riat -a configure-db –database-host new_host_name
  • You will be prompted for a password, this is the master password you also used to login to SSO with the user: admin@system-domain or root@system-domain.
  • Check the file for the correct database reference. You can find it in C:Program FilesVMwareInfrastructureSSOServerwebappsimsweb-infclasses
  • Edit the file C:Program and modify all values that need to be updated.
  • To update the SSO DB RSA_USER password, run the command if, for example, the RSA_USER password has expired or the Database has been moved to another SQL instance: ssocli.cmd configure-riat -a configure-db –rsa-user-password new_db_password –rsa-user New_RSA_USER

Now cross your fingers and start the SSO Service. Check if you can logon to the SSO Server: https://<your SSO server>:9443/vsphere-client/ Login with:  admin@system-domain


See full post at: How to move VMware Single Sign On (SSO) database

VMware configuration maximums from 1.0 to 5.1

VMware has really grown in scalability from the early days of ESX 1.0 with each new release of vSphere. I put together this table on how the configuration maximums have increased over the years so you can see just how much it scales over the years. VMware has published there Configuration Maximums documentation with each release starting with VI3 which you should be familiar with especially if you are trying to get a certification. I pieced together the earlier versions from the installation documentation for each release, there isn’t much info available on ESX 1.0 so if you know anything please fill in the blanks for me. Notice how the VM virtual disk size of 2TB has never changed, this is to due file system limitations that VMware has not yet been able to overcome. With their new Virtual Volumes architecture that limit may finally be gone. Also note on the earlier versions the documentation did not state a 2TB virtual disk limit although I’m almost positive it existed, the documentation stated “9TB per virtual disk”, not sure why though.

Configuration Maximums for VMware ESX/vSphere

VMware release: 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.1 5.0 5.1
vCPUs per VM 1 1 2 2 4 4 8 8 32 64
RAM per VM 2GB 3.6GB 3.6GB 3.6GB 16GB 64GB 255GB 255GB 1TB 1TB
NICs per VM ? 4 4 4 4 4 10 10 10 10
VM Virtual Disk ? ? ? ? 2TB 2TB 2TB 2TB 2TB 2TB
VMFS Volume ? 64TB 64TB 64TB 64TB 64TB 64TB 64TB 64TB 64TB
pCPU per host ? 8 16 16 32 32 64 160 160 160
vCPU per host ? 64 80 80 128 128 512 512 2048 2048
RAM per host ? 64GB 64GB 64GB 64GB 256GB 1TB 1TB 2TB 2TB
pNICs per host ? 16 16 16 32 32 32 32 32 32

In addition hears a diagram from VMware that depicts the configuration maximums in a slightly different manner:


VMware Configuration Maximum Published Documents:

Pluggable Storage Architecture (PSA) Deep-Dive – Part 1

In this next series of blog articles, I am going to take a look at VMware’s Pluggable Storage Architecture, more commonly referred to as the PSA. The PSA was first introduced with ESX 4.0 and can be thought of as a set of APIs that allows third-party code to be inserted directly into the storage I/O path.

Why would VMware want to allow this? The reason is straight forward. This allows 3rd party software developers (typically storage hardware vendors) to design their own load balancing techniques and fail-over mechanisms for their own storage arrays. It also means that 3rd party vendors can now add support for new arrays into ESXi without having to provide internal information or intellectual property about the array to VMware.

There was another driving factor also. In the past, if one of our storage array partners wished to make a change to the way their particular array did fail-over or load balancing, it could trigger a re-certification of all arrays since they all shared the same SCSI mid-layer code in the VMkernel – a major undertaking as I am sure you can appreciate. The PSA allows vendor to make appropriate changes to their load balancing and fail-over mechanisms without impacting any other vendor.

Before we start, I need to give you a warning. I don’t believe there is any other component within VMware that has as many acronyms as the PSA. I’ll create a short list at the end of the post for reference. Hopefully the amount of acronyms won’t put you off the post too much. The following diagram shows the relationship between the PSA, NMP and some of NMP’s sub-plugins which we will discuss in a future post.


Role of the PSA

The PSA is responsible for a number of tasks within the VMkernel. Primarily, it is responsible for loading and unloading the multipath plugins, which includes the Native Multipath Plugin (NMP) from VMware and other Multipath Plugins (MPP) from third parties (if they are installed). It handles physical path discovery and removal (via scanning). As the PSA discovers available storage paths (and based on a set of predefined rules which I will show you shortly), it will determine which MPP should be given ownership of that path. It routes I/O requests for a specific logical device to an appropriate MPP. It handles I/O queuing to the physical storage adapters (HBA) & to the logical devices. It also implements logical device bandwidth sharing between Virtual Machines and provides logical device and physical path I/O statistics which can be viewed in esxtop and the vSphere client UI performance views.

Role of the MPP

Once an MPP is given ownership of a path by the PSA, the MPP is responsible for associating a set of physical paths with a logical storage device, or LUN. Its tasks involve managing physical path claiming and un-claiming as well as creating, registering, and de-registering logical devices on the host. It also process I/O requests to logical devices by selecting an optimal physical path for the request (load balance) and performing actions necessary to handle failures and request retries (fail-over). The MPP is also involved in management tasks such as the abort or reset of logical devices.

By default, each ESXi host ships with a default Multipath Plugin (MPP). This is called the Native Multipath Plugin (NMP). However, as mentioned, third parties can plugin their own software rather than use the default NMP. The most common MPPs from third parties are EMC PowerPath/VE & Symantec/Veritas DMP.

NMP Detail

The Native Multipath Plugin (NMP) supports all storage arrays listed on the VMware storage Hardware Compatibility List (HCL). NMP manages sub-plugins for handling multipathing and load balancing. The specific details of handling path fail-over for a given storage array are delegated to an NMP sub-plugin called a Storage Array Type Plugin (SATP). SATP is associated with paths. The specific details for determining which physical path is used to issue an I/O request (load balancing) to a storage device are handled by a sub-plugin of the NMP is called Path Selection Plugin (PSP). PSP is associated with logical devices. The SATP & PSP will be discussed in more detail in a future post.


Here is the default set of claim-rules from an ESXi 5.0 host:

~ # esxcli storage core claimrule list
Rule Class Rule Class   Type      Plugin     Matches
MP          0  runtime  transport  NMP      transport=usb
MP          1  runtime  transport  NMP      transport=sata
MP          2  runtime  transport  NMP      transport=ide
MP          3  runtime  transport  NMP      transport=block
MP          4  runtime  transport  NMP      transport=unknown
MP     101  runtime  vendor  MASK_PATH  vendor=DELL model=Universal Xport
MP     101  file          vendor  MASK_PATH  vendor=DELL model=Universal Xport
MP    65535  runtime  vendor   NMP       vendor=* model=*

The rules are matched in descending order, so basically if a device is discovered to be usb, sata, ide, block, or in fact has an unknown transport type, the PSA will assign the NMP ownership of the path. The  MASK_PATH rule is to hide controller/gateway devices from certain DELL storage.

The “runtime” rules are the ones currently in use by the VMkernel. The “file” claim-rules are the ones in the /etc/vmware/esx.conf file. Claim-rules must be loaded from the /etc/vmware/esx.conf file into the kernel, so the list of rules in each place may not always be the same.

The last rule will assign any other storage it discovers from any vendor and any model to the NMP. This is like a catch-all rule. Therefore, by default, every storage device is managed by the NMP. Then it is up to the NMP to see what it can do with it.

Let’s now look at the same set of rules from an ESXi host which has EMC PowerPath installed (I’ve truncated some of the output, fyi).

~ # esxcli storage core claimrule list
Rule Class Rule Class   Type      Plugin     Matches
MP          0  runtime  transport  NMP      transport=usb
MP          1  runtime  transport  NMP      transport=sata
MP          2  runtime  transport  NMP      transport=ide
MP          3  runtime  transport  NMP      transport=block
MP          4  runtime  transport  NMP      transport=unknown
MP      101  runtime  vendor  MASK_PATH  vendor=DELL model=Universal Xport
MP      101  file          vendor  MASK_PATH  vendor=DELL model=Universal Xport
MP      250  runtime  vendor    PowerPath   vendor=DGC model=*
MP      250  file          vendor   PowerPath   vendor=DGC model=*
MP      260  runtime  vendor    PowerPath   vendor=EMC model=SYMMETRIX
MP      260  file          vendor   PowerPath    vendor=EMC model=SYMMETRIX
MP      270  runtime  vendor    PowerPath    vendor=EMC model=Invista
MP      270  file          vendor    PowerPath    vendor=EMC model=Invista
MP    65535  runtime  vendor     NMP          vendor=* model=*

As before, if a device is discovered to be usb, sata, ide, block, or in fact has an unknown transport type, the PSA will once again assign the NMP ownership of the path. However, if the vendor turns out to be DGC (short for Data General, which EMC purchased many moons ago), or indeed EMC, the PSA will assign PowerPath as the owner. There are in fact other array vendors listed too, since PowerPath will work on non-EMC arrays. And at the end of the list, if there are no other matches before that, we have our catch-all rule which assigns NMP to devices which haven’t matched any other rule.

In a future post, I’ll look at the NMP, SATP & PSP in more detail.


  • PSA – Pluggable Storage Architecture
  • MPP – Multipath Plugin
  • NMP – Native Multipath Plugin
  • SATP – Storage Array Type Plugin
  • PSP – Path Selection Plugin
  • MEM – Multipath Extension Module

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Pluggable Storage Architecture (PSA) Deep-Dive – Part 2

As I highlighted in the PSA part 1 post, NMP, short for Native Multipath Plugin, is the default Multipath Plugin shipped with ESXi hosts. Once the PSA has associated the NMP with particular paths, it uses a number of sub-plugins to handle load balancing and path fail-over. In this post, I will look at the NMP in more detail. I will pay specific attention to the activity of the Storage Array Type Plugin (SATP) which is responsible for handling path fail-over for a given storage array and also the Path Selection Plugin (PSP), which determines which physical path is used to issue an I/O request (load balancing) to a storage device.

If a logical device and its paths are associated with the NMP by the PSA, VMware provides a default SATP for each supported array on the Hardware Compatibility List (HCL). Each SATP supports a specific type of storage array, and make the decision on when to fail-over to an alternate path. Typically, this is determined by the SCSI sense codes returned by the array. Each SATP also has a default PSP associated, although this can be changed if you believe your array would benefit from a different PSP to the default or if your storage array vendor provides you with an alternative PSP.

NMPTo look at the available SATPs on an ESXi host (and to see the default PSP associated with said SATP), you can use the following command:

  • # esxcli storage nmp satp list

satp listNote that only SATPs which are required are loaded. There is no point in loading all these plugins into memory if no storage device is going to use them. In this case, the default Active/Active plugin for SAN devices and the local plugin for direct attached devices are the only ones which are loaded. You may see different ones loaded depending on which storage array your ESXi host(s) is connected to. To examine the full list of PSPs, you can use this command:

  • # esxcli storage nmp psp list

psp listBy default, we ship with three unique PSPs. The PSP handles local balancing operations and is responsible for choosing a physical path to issue an I/O request to a logical drive. There are a number of CLI commands to help you query the plugins used by particular logical devices and paths.

  • # esxcli storage nmp path list
  • # esxcli storage nmp device list

These commands will show you exactly which SATP and which PSP are currently being used by a particular device and path.

Now we know what the SATPs are, but how does NMP choose which SATP to associate to a path. Well, there is another set of rules used for that. To examine the complete set of rules for an SATP, use the command:

  • # esxcli storage nmp satp rule list

This will list all the rules, and it is rather long. To narrow down the list and see the rules for a single SATP, use the -s option:

  • # esxcli storage nmp satp rule list -s VMW_SATP_LOCAL

satp rule listIf a match is made based on transport as per the list above, then NMP will use the VMW_SATP_LOCAL for these devices. As you examine other SATPs, you will see different rules defined. You may observe an SATP referenced based on the array vendor, array models or other options.

By way of recap, what you’ll have observed is that the PSA will choose an NMP depending on a claim-rule. The NMP will then choose an SATP, again based on another set of predefined rules. An SATP then has a PSP associated with it by default. In my next post, I will examine the SATP & PSP in more detail.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Pluggable Storage Architecture (PSA) Deep-Dive – Part 3

So far in this series, we have looked at the Pluggable Storage Architecture (PSA) and MPPs (Multipath Plugins). We have delved into the Native Multipath Plugin (NMP), and had a look at its sub-plugins, the Storage Array Type Plugin (SATP) and Path Selection Plugin (PSP). We have seen how the PSA selects an MPP, and if that MPP is the NMP, how the NMP selects an SATP and PSP.

Note – if you are having trouble following all the acronyms, you are not the first. There is a glossary at the end of the first blog post. And if we haven’t had enough acronyms, you will more recently see the plugins referred to as an MEMs, Management Extension Modules.

MEMsHowever, these names never really caught on and the original names continue to be the ones which are commonly used. The next step is to examine the SATP and PSP in more detail.

Storage Array Type Plugin (SATP)

The role of the SATP can be thought of as falling into three distinct areas. The first task of the SATP is to monitor the hardware state of the physical paths to the storage array. The second task of the SATP is to detect when a hardware component of a physical path has failed. This is detected in the form of SCSI sense codes returned by the array controller to the host. (KB article 1003433 details the various sense codes that can initiate a path fail-over). The final task is to switch the physical path to the array when the currently active path has failed.

If an I/O operation reports an error, NMP calls an appropriate SATP. The SATP interprets the error codes and, when appropriate, activates inactive paths and fails over to the new active path.

Path Selection Plugin (PSP)

A PSP handles load balancing operations and is responsible for choosing a physical path to issue an I/O request to a logical device. When a Virtual Machine issues an I/O request to a storage device managed by the NMP, it calls the PSP assigned to this storage device. The PSP selects an appropriate physical path on which to send the I/O, load balancing the I/O if necessary. I posted an article (which includes a link to a video) on the vSphere Storage Blog which shows how the SATP & PSP interact if a path failure occurs.

As highlighted previously, there are three default PSPs shipped with ESXi.

VMW_PSP_MRU — MRU stands for Most Recently Used. This PSP selects the first working path discovered at system boot time. If this path becomes unavailable, the ESX host switches to an alternative path and continues to use the new path while it is available. This is the default PSP used with Active/Passive arrays. A/P arrays are arrays which have multiple controllers, but only a single controller has ownership of the LUN at any one time. This means that the LUN is only ever visible on paths to one controller. In certain failure scenarios, the LUN ownership may have to move to another controller (referred to as a trespass by some array vendors). This fail-over between controllers can take some time to complete, depending on how busy the storage array is. In a misconfigured environment, the ownership of the LUN can continuously move between array controllers. This behavior is referred to as path thrashing, and can have serious performance implications for the ESXi host.

VMW_PSP_Fixed — Uses the designated preferred path, if it has been configured. Otherwise, it uses the first working path discovered at system boot time. If the ESXi host cannot use the preferred path (because of a path failure, for instance), this PSP selects a random alternative available path. The ESXi host automatically reverts back to the preferred path as soon as the path becomes available. Typically used with Active/Active arrays. A/A arrays are able to present the same LUN on multiple controllers at the same time.

VMW_PSP_RR – RR stands for Round Robin. It uses an automatic path selection rotating through all available paths and enabling load balancing across the paths. While this PSP can be used on both A/A arrays and A/P arrays, it is most typically found on A/A arrays since all paths to the LUN can be used in load balancing the I/O. On A/P arrays, only paths to the controller which is currently the LUN owner are used.What we haven’t discussed here is how PSP handles Asymmetric Logical Unit Access (ALUA) arrays. This will be covered in a future post.

As we have already seen, SATPs have a default PSP. However, it is supported to use other PSPs other than the default. A common scenario is for customers to move from Fixed to Round Robin. However, there has been a long-standing directive around Round Robin that you should discuss any changes to the PSP with your storage array vendor before implementing the change. For EMC customers, this is not necessary. Since 5.1, EMC have introduced Round Robin as the default path policy for their arrays. I posted about it here.

Now a number of alternate, partner specific PSPs also exist. For instance, DELL have had one for their EqualLogic arrays since vSphere 5.0. More recently, Nimble Storage introduced a PSP for their storage arrays.

That completes part 3 the deep-dive into the PSA. I hope that has given you some idea how the various components of the PSA are used, and why we chose to go with this direction for I/O device and path management.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Pluggable Storage Architecture (PSA) Deep-Dive – Part 4

In this post, I want to look at some fail-over and load balancing specific to ALUA (Asymmetric Logical Unit Access) arrays. In PSA part 3, we took a look at the different Path Selection Plugins (PSP), but for the most part these were discussed in the context of Active/Active arrays (where the LUN is available on all paths to the array) and Active/Passive arrays (where the LUN is owned by one controller on the array, and is only visible on the paths to that controller). ALUA provides a standard way for discovering and managing multiple paths to LUNs. Prior to ALUA, hosts need to use array vendor-specific ways to inquiry target port state. ALUA Provides a standard way to allow devices to report the states of its ports to hosts. The state of ports can be used by hosts to prioritize paths and make fail-over/load balancing decisions.

In ALUA arrays, both controllers can receive I/O commands, but only one controller can issue I/O to the LUN. The controller who can issue commands is called the managing controller. The paths to the LUN via ports on this controller are called optimized paths. I/O sent to a port of the non-owning controller must be transferred to the owning controller internally which increases latency and impacts on the performance of the array. Due to this, paths to the LUN via the non-managing controller are called non-optimized paths.

Target Port Group Support (TPGS) TPGS provides a method for determining the access characteristics of a path to a LUN through a target port. It supports soliciting information about different target port capabilities. It also supports routing I/O to the particular port or ports that can achieve the best performance. Target Port Groups allows path grouping and dynamic load balancing. Each port in the same TPG has the same port state, which can be one of the following state: Active/Optimized, Active/Non-optimized, Standby, Unavailable, and In-Transition.


VMW_SATP_ALUA In the context of the PSA, the NMP (Native Multipath Plugin) claims the path, then associate the appropriate Storage Array Type Plugin (SATP). For ALUA arrays, this is VMW_SATP_ALUA.  The VMW_SATP_ALUA plugin sends an Report Target Port Group (RTPG) command to the array to get the device’s TPG identifiers and states. There are 5 such target port groups per device, to reflect the 5 ALUA states that I mentioned earlier: Act/Opt, Act/Non-opt, Standby, Unavailable, and Transitioning. When a TPG’s state changes, the state of all paths in the same target port group will be changed.

Now, when we first introduced ALUA support back in vSphere 4.1, we introduced special SATPs & PSPs for ALUA. For example, the EMC Clariion array had an SATP called VMW_SATP_ALUA_CX which had a default PSP called VMW_PSP_FIXED_AP associated with it, as shown here:

esxcli nmp device listThe fail-over functionality for ALUA arrays has now been rolled up into the generic VMW_ALUA_SATP, and the special load-balancing capabilities for ALUA have been rolled up into the VMW_PSP_FIXED which we discussed previously in part 3.

Now if you list the paths to a device, you will see them organized by TPG state. In this example, there are only two paths to the device, one listed as active and the other as active unoptimized (which means it is a path to the LUN via the non-owning controller):

esxcli nmp path listFor further reading on the configuration parameters that you can have with ALUA arrays in the PSA, you can have a read of this blog article I wrote on the vSphere blog. This post also discusses the difference between Explicit & Implicit ALUA, the follow-over algorithm that we use, and path thrashing. One other ALUA feature that you might like to research is the ability to prioritize the I/O path when a fail-over occurs. There is another post on the vSphere blog which discusses this.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage