Le Gia Vinh’s Blog

Source: 09 Jan 2008 | Greg Schulz

LUNs transform raw storage into usable space, accessible from any attached operating system. However, creating and implementing LUNs can be a challenge. In this handbook you’ll learn what LUNs are, how they’re used, how to implement LUNs, how to manage LUNs, how to partition LUNs and the concepts involved in LUN migration.

LUN basics

For open systems environments, including Windows, fixed-block architectures or fixed-block addressing is the most common basis for performing I/O operations to and from a disk drive. The most common I/O command protocol for open systems is the SCSI command set, not to be confused with parallel SCSI cabling, as there is a difference. The SCSI command set is implemented on different network and storage I/O interfaces or transports, such as Fibre Channel, iSCSI, SAS, InfiniBand and others. Part of the SCSI command set protocol is an initiator (source) that requests an I/O operation (read, write or status inquiry) and a target (destination) device with a subaddress known as a LUN. The target can be a single disk in a JBOD array with each HDD having a different SCSI target and possibly a unique or selectable LUN of the device supports it. A common use of LUNs is in storage systems or arrays that incorporate some type of controller, usually with RAID where multiple HDDs are configured and aggregated into a RAID or volume group and then assigned a unique LUN and accessed via a SCSI target ID. For example, assume you have a storage system with 32 300 GB HDDs configured into two separate RAID groups, each with 16 HDDs. One RAID group is a RAID 5 14+1 and a hot spare, the other being a RAID 10 (8+8). The RAID 5 group would have about 4.2 TB of usable capacity, while the RAID 10 group would be about 2.4 TB. The storage system could present a single LUN or volume to a server or servers for each of the RAID groups or, the RAID groups could be subdivided into multiple smaller LUNs for more granular access of the storage depending on specific application needs or preferences. To learn more, check out chapters 3 (Networking with your Storage) and 4 (Storage and I/O Networks) in my book “Resilient Storage Networks” (Elsevier) along with other SearchStorage.com and Storage magazine tips and FAQs, including Tracking down those missing bytes.

Implementing LUNs

LUNs can be created and implemented in various configurations, such as capacity, address, or boot device, along with what host servers can see or access the LUN or volume. Creation of a LUN can be done using vendor supplied management tools or third-party tools. LUNs can be created on storage systems and virtualization appliances along with other storage devices, including virtual tape libraries (VTL). Some third-party storage management tools enable you to create LUNs on different vendor’s storage systems, while other tools simply provide tracking and storage resource management (SRM) information for LUNs. There are a few generic steps for creating a LUN. First, it is necessary to create a RAID group or volume group on the storage system or virtualization appliance mapping some number of unused HDDs or devices together. The next step is assigning a RAID level and then subdividing or “carving” out LUNs from the RAID group assigning manually or automatically address numbers. The number of LUNs supported by a storage system will vary by vendor implementation and storage I/O interface, such as Fibre Channel, SAS, Parallel SCSI, iSCSI, etc., being used. Likewise, various operating systems and virtual servers, including VMware, will have different limits on how many LUNs can be accessed in total and per adapter port on the physical server. Some operating systems also have a default or preference that certain storage, including boot devices, is on certain targets and LUNs. For example, older versions of Windows NT would not recognize nonzero LUNs until upgraded to newer versions and registry entry adjustments. Some storage systems support the notion of partitions where a group of LUNs or volume groups can be mapped into a partition that appears to the host servers as a unique storage system with indicial LUNs. For example, 20 servers could be attached to an LSI/Engenio-based storage system that has four Fibre Channel ports. Assuming that the storage system has 40 total LUNs (or more), 20 partitions could be created with each partition containing two LUNs with one of the LUNs being LUN 0 as a boot device and mapped to a particular server adapter address. Each server would think that it has its own unique LUN 0 for a boot device yet in reality, there are 20 LUN 0s, each isolated from the other in a shared storage environment.

LUN management

Some storage systems support partitions to help allocate unique LUNs to different servers. Depending on implementation, some vendors support partitions that incorporate LUNs from different volume or RAID groups for flexibility, or if a LUN can span multiple storage enclosures and I/O busses inside the storage system to avoid performance contention. Another feature that will vary by vendor implementation is how LUN expansion takes place if supported. For example, vendors will often support on-line growth of RAID or volume groups to facilitate creation of additional LUNs, while others support dynamic expansion of existing LUNs. However, do not assume that all implementations are the same and that dynamic expansion means while data is being read or written to a LUN. Check the details and ask vendor-specific questions as to what it does or does not support and if there are any extra fees for enhanced LUN functionality. In addition to the number of LUNs supported, the size of LUNs supported will also vary by storage system and operating systems. Most storage devices and operating systems today should be able to support at least a 2 TB LUN. Some operating systems can support addressing and accessing of LUNs larger than 2 TB either natively or via upgrade patch kits. For storage systems, look at the total number of LUNs supported by a particular solution or if that vendor has guidelines or restrictions on the number of LUNs that can be accessed on a given physical or adapter port. Also, check to see what limitations on addressing or flexibility exists with volume mapping and masking software features in storage systems to enable specific LUNs to be seen or hidden from different servers. For LUNs being accessed by nonclustered or failover server nodes, typically you would hide those LUNs from other servers so that they are not seen and accessed. Similarly on the server, path management software from vendors, such as Microsoft’s MPIO, IBM’s SDD, Symantec Corp.’s DMP and EMC Corp.’s Powerpath, should be configured to access and support path failover to LUNs accordingly.

Advanced LUN topics

Many storage systems support data migration or LUN migration, however, the devil is in the details. Some vendors support movement of data while a LUN is being read and written, while others can automatically move data based on policies when there is no write activity. Ask your vendor if it supports movement of the data in an entire LUN while applications are actively reading and writing to the LUN and what its caveats are. Investigate how a vendor supports LUN affinity and cache coherency. Even though a LUN may be accessible via two or more controllers, nodes or processors, find out if one controller owns or acts as a primary to maintain data integrity and if so, how are the I/Os moved or “shipped” between the controllers to maintain cache coherency without negatively impacting performance. Besides using host-based third-party or operating system supplied volume managers, LUNs can be created from standalone storage devices that support selectable subaddressing, in addition to specifying SCSI target addresses. LUNs can also be created and run on standard servers as appliances or on special purpose services blades in SAN switches. Below are some important things to remember:

Avoid performance contention between LUNs by avoiding allocating LUNs from the same RAID group to active and competing workloads if possible

Exercise caution creating RAID groups factoring in where HDDs exist in enclosures and on what internal busses to avoid contention

Exercise caution intermixing various types of HDDs of different capacity, speeds and interfaces adhering to vendor recommendations and guidelines

Understand what performance impacts to LUNs when re-sizing, migrating or performing other operations including snapshot copies or replication

Follow vendor guidelines pertaining to creating large RAID groups of many HDDs vs. multiple smaller ones factoring that vendor’s specific architecture capabilities or limitations.

To learn more, check out chapters 3 (Networking with your Storage) and 4 (Storage and I/O Networks) in my book Resilient Storage Networks (Elsevier) along with other SearchStorage and Storage Magazine tips and FAQs including Tracking down those missing bytes., or send me a note with your questions.

In this entry, I illustrate the patching VMware 3.0.0, we will patch ESX-1006511 package into our VMware ESX 3.0.0 server using CLI. Make sure your ESX host is running in maintenance mode. Proceed as following to extract the file:

• Log in to the ESX Server service console, then “su -” to pass to root.
• Create a local depot directory.

# mkdir /var/updates

Note: VMware recommends that you use this updates directory location.

• Change your working directory to /var/updates.
  • # cd /var/updates
• Download the VMware ESX patches under format tar from VMware website into the /var/updates directory.

• Verify the integrity of the downloaded tar file.

# md5sum ESX-1006511.tgz The md5 checksum output should match the following:efa86b4e30e7700e186c8040fde93381 ESX-1006511.tgz

• Extract the compressed tar archive.

# tar -xvzf ESX-1006511.tgz

• Change to the newly created directory /var/updates/ESX-1006511.

# cd ESX-1006511

Installation Instructions

Once you have downloaded and extracted the archive, and if you are in the directory you created above, install the update using the following command: # esxupdate update If you want patching VMware ESX host from patches coming from another location, for example patch depots/repositories, you need to create the patch depot before patching.

Create VMware patch depots/repositories

Patch Depot can be set up on web, ftp Linux/Windows server. Create the depot directory you want to keep patches. Download patches from VMware website and uncompress patches into different patch names. Then run this command: # esxupdate -r file://<host_depot_name>/ESX-2221 update for example our VMware patching depot ftp server name slx002 # esxupdate –r file:///slx002/var/updates/ESX-221 update During the update process, logs appear on the terminal. You can specify the verbosity of esxupdate logs by using the -v option as shown below. # esxupdate -v 10 file://slx002/ESX-221 update Captures

Below is the capture of patching the ESX-2221.tgz on our VMware ESX 3.0.0 server:

[root@vsesx008 updates]# md5sum ESX-2221.tgz

c2e37a178776f9fc1c066e70d19363e4  ESX-2221.tgz

[root@vsesx008 updates]# tar -xvzf ESX-2221.tgz

ESX-2221/

ESX-2221/descriptor.xml

ESX-2221/VMware-esx-drivers-net-e1000-7.0.33-34124.i386.rpm

ESX-2221/vmware-hwdata-1.00-6vmw.noarch.rpm

ESX-2221/vmware-hwdata-legacy-1.00-6vmw.noarch.rpm

ESX-2221/headers/

ESX-2221/headers/header.info

ESX-2221/headers/VMware-esx-drivers-net-e1000-0-7.0.33-34124.i386.hdr

ESX-2221/headers/vmware-hwdata-0-1.00-6vmw.noarch.hdr

ESX-2221/headers/vmware-hwdata-legacy-0-1.00-6vmw.noarch.hdr

ESX-2221/headers/VMware-esx-scripts-0-3.0.0-34330.i386.hdr

ESX-2221/VMware-esx-scripts-3.0.0-34330.i386.rpm

[root@vsesx008 updates]# cd ESX-2221

[root@vsesx008 ESX-2221]# ls

descriptor.xml

headers

VMware-esx-drivers-net-e1000-7.0.33-34124.i386.rpm

VMware-esx-scripts-3.0.0-34330.i386.rpm

vmware-hwdata-1.00-6vmw.noarch.rpm

vmware-hwdata-legacy-1.00-6vmw.noarch.rpm

[root@vsesx008 ESX-2221]# ls

descriptor.xml

headers

VMware-esx-drivers-net-e1000-7.0.33-34124.i386.rpm

VMware-esx-scripts-3.0.0-34330.i386.rpm

vmware-hwdata-1.00-6vmw.noarch.rpm

vmware-hwdata-legacy-1.00-6vmw.noarch.rpm

[root@vsesx008 ESX-2221]# esxupdate update

INFO: No repository URL specified, going with file:/var/updates/ESX-2221

INFO: Configuring...

INFO: Pre-installing ['VMware-esx-scripts', 'vmware-hwdata', 'vmware-hwdata-legacy']...

INFO: | Gathering header information file(s) from server(s)

INFO: | Server: Esxupdate-compatible repository for VMware ESX Server ESX-2221

INFO: | Finding updated packages

INFO: | Downloading needed headers

INFO: | vmware-hwdata-0-1.00-6vmw 100% |=========================| 6.4 kB    00:00

INFO: | VMware-esx-drivers-net-e1 100% |=========================|  940 B    00:00

INFO: | VMware-esx-scripts-0-3.0. 100% |=========================| 4.7 kB    00:00

INFO: | vmware-hwdata-legacy-0-1. 100% |=========================| 6.6 kB    00:00

INFO: | Resolving dependencies

INFO: | Dependencies resolved

INFO: | I will do the following:

INFO: | [install: vmware-hwdata-legacy 1.00-6vmw.noarch]

INFO: | [update: vmware-hwdata 1.00-6vmw.noarch]

INFO: | [update: VMware-esx-scripts 3.0.0-34330.i386]

INFO: | Downloading Packages

INFO: | Getting vmware-hwdata-1.00-6vmw.noarch.rpm

INFO: | Getting vmware-hwdata-legacy-1.00-6vmw.noarch.rpm

INFO: | Getting VMware-esx-scripts-3.0.0-34330.i386.rpm

INFO: | Running test transaction:

INFO: | Test transaction complete, Success!

INFO: | vmware-hwdata 100 % done 1/5

INFO: | loading /etc/vmware/pci.xml...

INFO: | writing /etc/vmware/pci.xml.merged

INFO: | Creating new /etc/vmware/pci.ids file...

INFO: | Creating new /etc/vmware/pcitable file...

INFO: | Creating new /etc/vmware/pcitable.Linux file...

INFO: | Replacing /etc/vmware/vmware-devices.map file...

INFO: | vmware-hwdata-legacy 100 % done 2/5

INFO: | loading /etc/vmware/pci.xml...

INFO: | loading /etc/vmware/pciid/legacy/3c90x.xml...

INFO: | loading /etc/vmware/pciid/legacy/3c990.xml...

INFO: | loading /etc/vmware/pciid/legacy/BusLogic.xml...

INFO: | loading /etc/vmware/pciid/legacy/DAC960.xml...

INFO: | loading /etc/vmware/pciid/legacy/aacraid.xml...

INFO: | loading /etc/vmware/pciid/legacy/aacraid_esx30.xml...

INFO: | loading /etc/vmware/pciid/legacy/acenic.xml...

INFO: | loading /etc/vmware/pciid/legacy/adp94xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/aic79xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/aic7xxx.xml...

INFO: | loading /etc/vmware/pciid/legacy/asa72xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/bnx2.xml...

INFO: | loading /etc/vmware/pciid/legacy/cciss.xml...

INFO: | loading /etc/vmware/pciid/legacy/e100.xml...

INFO: | loading /etc/vmware/pciid/legacy/e1000.xml...

INFO: | loading /etc/vmware/pciid/legacy/gdth.xml...

INFO: | loading /etc/vmware/pciid/legacy/ide.xml...

INFO: | loading /etc/vmware/pciid/legacy/ips.xml...

INFO: | loading /etc/vmware/pciid/legacy/lpfcdd_732.xml...

INFO: | loading /etc/vmware/pciid/legacy/lpfcdd_7xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/megaraid2.xml...

INFO: | loading /etc/vmware/pciid/legacy/megaraid_sas.xml...

INFO: | loading /etc/vmware/pciid/legacy/mptscsi.xml...

INFO: | loading /etc/vmware/pciid/legacy/mptscsi_2xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/pcnet32.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla2200_7xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla2300_707.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla2300_7xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla4010.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla4022.xml...

INFO: | loading /etc/vmware/pciid/legacy/sata.xml...

INFO: | loading /etc/vmware/pciid/legacy/tg3.xml...

INFO: | writing /etc/vmware/pci.xml.merged

INFO: | Replacing /etc/vmware/pci.ids file...

INFO: | Replacing /etc/vmware/pcitable file...

INFO: | Replacing /etc/vmware/pcitable.Linux file...

INFO: | Replacing /etc/vmware/vmware-devices.map file...

INFO: | VMware-esx-scripts 100 % done 3/5

INFO: | Completing update for vmware-hwdata  - 4/5

INFO: | Completing update for VMware-esx-scripts  - 5/5

INFO: | Installed:  vmware-hwdata-legacy 1.00-6vmw.noarch

INFO: | Updated:  vmware-hwdata 1.00-6vmw.noarch VMware-esx-scripts 3.0.0-34330.i386

INFO: | Transaction(s) Complete        INFO: Restarting /usr/sbin/esxupdate...

INFO: Preparing to install VMware ESX Server ESX-2221...

INFO: Running yum install <1 packages>...

INFO: | Gathering header information file(s) from server(s)

INFO: | Server: Esxupdate-compatible repository for VMware ESX Server ESX-2221

INFO: | Finding updated packages

INFO: | Downloading needed headers

INFO: | Resolving dependencies

INFO: | Dependencies resolved

INFO: | I will do the following:

INFO: | [update: VMware-esx-drivers-net-e1000 7.0.33-34124.i386]

INFO: | Downloading Packages

INFO: | Getting VMware-esx-drivers-net-e1000-7.0.33-34124.i386.rpm

INFO: | Running test transaction:

INFO: | Test transaction complete, Success!

INFO: | VMware-esx-drivers-net-e1000 100 % done 1/2

INFO: | loading /etc/vmware/pci.xml...

INFO: | loading /etc/vmware/pciid/e1000.xml...

INFO: | loading /etc/vmware/pciid/legacy/3c90x.xml...

INFO: | loading /etc/vmware/pciid/legacy/3c990.xml...

INFO: | loading /etc/vmware/pciid/legacy/BusLogic.xml...

INFO: | loading /etc/vmware/pciid/legacy/DAC960.xml...

INFO: | loading /etc/vmware/pciid/legacy/aacraid.xml...

INFO: | loading /etc/vmware/pciid/legacy/aacraid_esx30.xml...

INFO: | loading /etc/vmware/pciid/legacy/acenic.xml...

INFO: | loading /etc/vmware/pciid/legacy/adp94xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/aic79xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/aic7xxx.xml...

INFO: | loading /etc/vmware/pciid/legacy/asa72xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/bnx2.xml...

INFO: | loading /etc/vmware/pciid/legacy/cciss.xml...

INFO: | loading /etc/vmware/pciid/legacy/e100.xml...

INFO: | loading /etc/vmware/pciid/legacy/gdth.xml...

INFO: | loading /etc/vmware/pciid/legacy/ide.xml...

INFO: | loading /etc/vmware/pciid/legacy/ips.xml...

INFO: | loading /etc/vmware/pciid/legacy/lpfcdd_732.xml...

INFO: | loading /etc/vmware/pciid/legacy/lpfcdd_7xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/megaraid2.xml...

INFO: | loading /etc/vmware/pciid/legacy/megaraid_sas.xml...

INFO: | loading /etc/vmware/pciid/legacy/mptscsi.xml...

INFO: | loading /etc/vmware/pciid/legacy/mptscsi_2xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/pcnet32.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla2200_7xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla2300_707.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla2300_7xx.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla4010.xml...

INFO: | loading /etc/vmware/pciid/legacy/qla4022.xml...

INFO: | loading /etc/vmware/pciid/legacy/sata.xml...

INFO: | loading /etc/vmware/pciid/legacy/tg3.xml...

INFO: | writing /etc/vmware/pci.xml.merged

INFO: | Replacing /etc/vmware/pci.ids file...

INFO: | Replacing /etc/vmware/pcitable file...

INFO: | Replacing /etc/vmware/pcitable.Linux file...

INFO: | Creating new /etc/vmware/vmware-devices.map file...

INFO: | Completing update for VMware-esx-drivers-net-e1000  - 2/2

INFO: | Updated:  VMware-esx-drivers-net-e1000 7.0.33-34124.i386

INFO: | Transaction(s) Complete

INFO: Running esxcfg-boot to regenerate initrds...

INFO: --- TOTALS: 4 packages installed, 0 pending or failed, 0 excluded ---

INFO: Install succeeded - please standby for reboot.

INFO: Rebooting in 5 seconds...

Broadcast message from root (pts/0) (Fri Feb  1 00:09:18 2008):

The system is going down for reboot NOW!

[root@vsesx008 ESX-2221]#

Notes:

1. After patching ESX-3083826, ESX-3198250, ESX-5754280, we lose the licenses (/usr/lib/vmware/hostd/vmware-hostd). Do what necessary to recover licenses.