vSphere 5.0 Storage Features Part 1 - VMFS-5

One of the primary objectives of Storage enhancements in 5.0 is to make the management of storage much simpler. One way to do this is to reduce the number of storage objects that a customer has to manager, i.e. enable our customers to use far fewer and much larger datastores. To that end, we are increasing the scalability of the VMFS-5 filesystem. These scalability features are discussed here. In future postings, I will discuss further features which aim to fulfil this vision of simplifying storage management.

VMFS-5 Enhancements

• Unified 1MB File Block Size. Previous versions of VMFS used 1,2,4 or 8MB file blocks. These larger blocks were needed to create large files (>256GB). These large blocks are no longer needed for large files on VMFS-5. Very large files can now be created on VMFS-5 using 1MB file blocks.
  
• Large Single Extent Volumes. In previous versions of VMFS, the largest single extent was 2TB. With VMFS-5, this limit has been increased to ~ 60TB.
  
• Smaller Sub-Block. VMFS-5 introduces a smaller sub-block. This is now 8KB rather than the 64KB we had in previous versions. Now small files < 8KB (but > 1KB) in size will only consume 8KB rather than 64KB. This will reduce the amount of disk space being stranded by small files.
  
• Small File Support. VMFS-5 introduces support for very small files. For files less than or equal to 1KB, VMFS-5 uses the file descriptor location in the metadata for storage rather than file blocks. When they grow above 1KB, these files will then start to use the new 8KB sub blocks. This will again reduce the amount of disk space being stranded by very small files.
• Increased File Count. VMFS-5 introduces support for greater than 100,000 files, a three-fold increase on the number of files supported on VMFS-3, which was ~ 30,000.
• ATS Enhancement. This Hardware Acceleration primitive, Atomic Test & Set (ATS), is now used throughout VMFS-5 for file locking. ATS is part of the VAAI (vSphere Storage APIs for Array Integration), and will be revisited in a future posting. This enhancement improves the file locking performance over previous versions of VMFS.
  

Here is a vmkfstools output of a newly created VMFS-5 volume showing many of the new scalability characteristics:

~ # vmkfstools -Pv 10 /vmfs/volumes/newly-created-vmfs5/
VMFS-5.54 file system spanning 1 partitions.
File system label (if any): newly-created-vmfs5
Mode: public
Capacity 3298534883328 (3145728 file blocks * 1048576), 3297500987392 (3144742 blocks) avail
Volume Creation Time: Tue Jun 14 14:35:53 2011
Files (max/free): 130000/129992
Ptr Blocks (max/free): 64512/64496
Sub Blocks (max/free): 32000/32000
Secondary Ptr Blocks (max/free): 256/256
File Blocks (overcommit/used/overcommit %): 0/986/0
Ptr Blocks  (overcommit/used/overcommit %): 0/16/0
Sub Blocks  (overcommit/used/overcommit %): 0/0/0
UUID: 4df771c9-f6419df2-81bc-0019b9f1ecf6
Partitions spanned (on "lvm"):
        naa.60a98000572d54724a34642d71325763:1
DISKLIB-LIB   : Getting VAAI support status for /vmfs/volumes/newly-created-vmfs5/
Is Native Snapshot Capable: NO
~ #


VMFS-3 to VMFS-5 Upgrades

• Upgrading from VMFS-3 to VMFS-5 is an online & non-disruptive upgrade operation, i.e. VMs can continue to run on the datastore.
• Upgraded VMFS-5 can use the new 1KB small-files feature.
• Upgraded VMFS-5 can be grown to ~ 60TB, same as a newly created VMFS-5.
• Upgraded VMFS-5 has all the VAAI ATS improvements that a newly created VMFS-5 has.
  

Here is a vmkfstools output on an upgraded VMFS-5 volume:

~ # vmkfstools -Pv 10 /vmfs/volumes/upgrade-testvol
VMFS-5.54 file system spanning 1 partitions.
File system label (if any): upgrade-testvol
Mode: public
Capacity 3298534883328 (3145728 file blocks * 1048576), 3297916223488 (3145138 blocks) avail
Volume Creation Time: Mon Jun 13 13:03:04 2011
Files (max/free): 30720/30713
Ptr Blocks (max/free): 64512/64496
Sub Blocks (max/free): 3968/3968
Secondary Ptr Blocks (max/free): 256/256
File Blocks (overcommit/used/overcommit %): 0/590/0
Ptr Blocks  (overcommit/used/overcommit %): 0/16/0
Sub Blocks  (overcommit/used/overcommit %): 0/0/0
UUID: 4df60a88-8eaa51ea-3108-0019b9f1ecf6
Partitions spanned (on "lvm"):
        naa.60a98000572d54724a34642d71325763:1
DISKLIB-LIB   : Getting VAAI support status for /vmfs/volumes/upgrade-testvol
Is Native Snapshot Capable: NO
~ #


Differences between newly created and upgraded VMFS-5 datastores:

• VMFS-5 upgraded from VMFS-3 continues to use the previous file block size which may be larger than the unified 1MB file block size.
• VMFS-5 upgraded from VMFS-3 continues to use 64KB sub-blocks and not new 8K sub-blocks.
• VMFS-5 upgraded from VMFS-3 continues to have a file limit of 30720 rather than new file limit of > 100000 for newly created VMFS-5.
• VMFS-5 upgraded from VMFS-3 continues to use MBR (Master Boot Record) partition type; when the VMFS-5 volume is grown above 2TB, it automatically & seamlessly switches from MBR to GPT (GUID Partition Table) with no impact to the running VMs.
• VMFS-5 upgraded from VMFS-3 continue to have its partition starting on sector 128; newly created VMFS5 partitions will have their partition starting at sector 2048.

RDM - Raw Device Mappings

• There is now support for passthru RDMs to be ~ 60TB in size.
• Non-passthru RDMs are still limited to 2TB - 512 bytes.
• Both upgraded VMFS-5 & newly created VMFS-5 support the larger passthru RDM.

Misc.

I decided to add this section as I know many of you will have questions about it.

• The maximum size of a VMDK on VMFS-5 is still 2TB -512 bytes.
• The maximum size of a non-passthru (virtual) RDM on VMFS-5 is still 2TB -512 bytes.
• The maximum number of LUNs that are supported on an ESXi 5.0 host is still 256.

These enhancements to the scalability of VMFS should assist in the consolidation of more VMs onto less datastores, reducing the number of storage objects that an administrator has to manage, and in turn making storage management that little bit easier in vSphere.

Recommendation

If you have the luxury of doing so, I would recommend creating a new VMFS-5 filesystem rather than upgrading VMFS-3 to VMFS-5. Storage vMotion operations can then be used to seamlessly move your VMs to the newly created VMFS-5. This way, you will enjoy all the benefits that VMFS-5 brings.