Esxi Awaiting Fast Path State Update Before Retrying Failed Command Again

This posting is ~viii years years one-time. Y'all should proceed this in mind. IT is a short living business. This information might be outdated.

Yesterday I stumbled over a forum post in a german VMware forum. A user noticed after a vSphere 5.5 update, that a newly updated ESXi 5.5 hosts wasn't able to mountain some datastores. The host was updated with a HP customized ESXi five.five Paradigm. The other two hosts, ESXi five.i installed from a HP customized image, had no bug. A HP P2000 G3 MSA Assortment with iSCSI was used as shared storage. The datastores with VMFS version v.54 were mounted. Simply datastores with VMFS 5.58 were not mouted. The user evacuated the VMs off one of the datastores, and then deleted and recreated the datastore. The recreated datastore appeared for a short moment and than disappered again.

 I knew the problem. It's a mixture of a trouble caused by a change in the HP customized images and a known behaviour of ESXi with VMFS v datastores.

Changes in the HP customized ESXi images

HP removed the P2000G3 VAAI-Plug-in for ESXi five.x and four.one in September 2013 due to an incompatibility to HP Smart Array P711M-, P712M and P721M SAS RAID controllers and HP P200 G3 arrays running firmware TS230 or TS240. The incompatibility can cause messages like this:

012-10-25T15:22:34.347Z cpu28:8220)Warning: NMP: nmp_DeviceRetryCommand:133:Device 2012-x-25T15:22:35.249Z cpu24:8839)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry earth failover device "naa.600c0ff0001344cecd99725001000000" - issuing control 0x4124419f0100 2012-x-25T15:22:35.249Z cpu4:8196)<4>hpsa 0000:0c:00.0: Device:C1:B0:T2:L1 Control:0xc2 Command Invalid.  2012-10-25T15:22:35.249Z cpu28:8220)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0xc2 (0x4124419f0100) to dev "naa.600c0ff0001344cecd99725001000000" failed on path "vmhba8:C0:T2:L1" H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2012-ten-25T15:22:35.249Z cpu28:8220)Alarm: NMP: nmpCompleteRetryForPath:378:Logical device "naa.600c0ff0001344cecd99725001000000": awaiting fast path state update earlier retrying failed command again...

These letters are associated with problems regarding the cosmos of datastores or mounting datastores. HP published a customer advisory in February 2013 to address this issue.

VMFS Locking Mechanisms

VMFS supports two locking mechanisms:

• SCSI reservations
• Atomic Examination and Ready (ATS)

Locking is necessary in an environment, where multiple hosts writes to the same filesystem. It should prevent a situation, where multiple hosts meantime writing to the same blocks. SCSI reservations are the practiced old mode. SCSI reservations are used, if a storage system doesn't support VAAI. A SCSI reservation locks a whole LUN. No other host can write to it until the reservation is removed. This can pb to performance bug. Many of you know the trouble with to many SCSI reservations. Atomic Exam and Set (ATS) is more intelligent. ATS is capable to lock only a specific sector of a LUN. Merely the storage system has back up this characteristic. For more than information almost VAAI, I recommend a blog article written by Chris Wahl.

Y'all possibly know that at that place are different VMFS versions. And because of this, there are several situations where SCSI reservations are used instead of ATS, simply also where ATS-only locking is used. I took this table from the vSphere documentation.

Storage Devices	New VMFS5	Upgraded VMFS5	VMFS3
Single extent	ATS but	ATS, but tin revert to SCSI reservations	ATS, just can revert to SCSI reservations
Multiple extents	Spans only over ATS-capable devices	ATS except when locks on non-head	ATS except when locks on non-caput

An extend is a LUN which is used to expand a VMFS datastore by concatanating multiple LUNs together. Multiple extends tin can served from different storage systems (with limitations…). Some words to the VMFS 5 versions:

ESXi Release	ESXi 5.0	ESXi 5.1	ESXi 5.5
VMFS 5 Version	5.54	v.58	5.threescore

Putting the pieces together

Nosotros have to assume, that the datastores, that couldn't exist mounted, were created on a ESXi 5.1 host with VAAI plug-in for the P2000 G3 installed. So it was single-extend and ATS-but. One host was updated with a HP customized prototype to ESXi 5.v. This image doesn't include the VAAI plug-in. So the new 5.five host doesn't support VAAI. No VAAI, no ATS. Because of this, the the host was able to mount the older, VMFS five.54 datastores, but not the newer 5.58. I presume that the 5.54 datastores were updated from an older VMFS, so that they were ATS capable, but can also revert to SCSI reservation. Equally the user deleted the datastore and recreated it on a v.1 host, the datastore was once again version 5.58 and ATS-only.

The workaround and the solution

The workaround is to install the HP P2000 Software Plug-in for VMware VAAI. It isn't a workaround to disable the Diminutive Test and Fix (ATS) primitive. If you lot take VMFS 5 ATS-only datastores, y'all wouldn't be able to mount them. In this example you lot besides have to disable the ATS-but mode. A solution is to do a firmware update on the P2000 G3. A few weeks ago HP released a new firmware for the P2000 G3. With the firmware release TS251R004 the P2000 G3 VAAI plug-in is no longer supported, because T10-compliance for VAAI was added.

I similar to outline, that this problem could occur with any other storage that needs a VAAI plugin. If yous update you hosts and you lot forget to install this plugin, you lot would have the aforementioned issues.

• Author
• Contempo Posts

vcloudnine.de is the personal blog of Patrick Terlisten. Patrick has a stiff focus on virtualization & deject solutions, but as well storage, networking, and IT infrastructure in general. He is a fan of Lean Management and agile methods, and practices continuous improvement whereever it is possible.

Feel free to follow him on Twitter and/ or get out a comment.

moorearnimedly.blogspot.com

Source: https://www.vcloudnine.de/vmware-esxi-5-5-host-doesnt-mount-vmfs-5-datastore/