Western Digital’s Advanced Format: The 4K Sector Transition Begins
by Ryan Smith on December 18, 2009 12:00 AM EST- Posted in
- Storage
We had a chance to talk to Western Digital last week about a new hard drive technology they have coming down the pipe called Advanced Format. Ahead of the impending launch of the first Advanced Format drives, Western Digital is engaging in an education campaign about the technology, as it represents the first step in a general transition for hard drives, and is a transition that can’t be done completely behind the scenes. Specifically, Advanced Format is going to bring about some issues where older operating systems (and we’re looking almost exclusively at Windows XP here) will inadvertently harm drive performance by making certain assumptions about the drives that will no longer be true.
We’ll get in to the nitty-gritty about the impact of this in a bit, but first let’s start with Advanced Format. What is it? In a nutshell, it’s replacing the traditional 512 byte hard drive sector with a 4 kilobyte (4,096B) hard drive sector for a number of practical and technological reasons. Advanced Format is the name Western Digital is giving to drives implementing 4KB sectors.
The move to 4K-sectors has been some 10 years in the making, as research in the late 90’s identified that 512B sectors made less and less sense as drives continued to get larger. At a time when hard drives were measured in megabytes, 512B sectors were a good balance between technical factors and a desire to keep wasted space at a minimum (you can’t normally put 2 files in 1 sector), but times have changed and hard drive capacities are several orders of magnitude larger.
The crux of the problem is that there are 3 factors that are in constant need of balancing when it comes to hard drive design: areal density, the signal-to-noise ratio (SNR) in reading from drive platters, and the use of Error Correcting Code (ECC) to find and correct any errors that occur. As areal density is increases, sectors become smaller and their SNR decreases. To compensate for that, improvements are made to ECC (usually through the use of more bits) in order to maintain reliability. So for a drive maker to add more space, they ultimately need to improve their error-correction capabilities, which means the necessary ECC data requires more space. Rinse, wash, repeat.
The SNR for various areal densities
At some point during this process drive manufacturers stop gaining any usable space - that is, they have to add as much ECC data as they get out of the increase areal density in the first place - which limits their ability to develop larger drives. Drive manufacturers dislike this both because it hinders their ability to develop new drives, and because it means their overall format efficiency (the amount of space on a platter actually used to store user data) drops. Drive manufacturers want to build bigger drives, and they want to spend as little space on overhead as possible.
But all is not lost. The principle problem here is that ECC correction takes place in 512B chunks, while ECC can be more efficient when used over larger chunks of data. If ECC data is calculated against a larger sector, even though more ECC data is necessary than for a single 512B sector, less ECC data than the sum of multiple sectors is needed to maintain the same level of operational reliability. One estimate for 4K sector technology puts this at 100 bytes of ECC data needed for a 4K sector, versus 320 (40x8) for 8 512B sectors. Furthermore the larger sectors means that larger erroneous chunks of data can be corrected (burst error correction), something that was becoming harder as greater areal densities made it easier to wipe out larger parts of a 512B sector. As a result, the need for the larger sector is born.
As for why Western Digital and other drive makers are transitioning specifically to a 4K sector, the rationale is based on the earlier problems. 4K sectors are much larger than 512B sectors, which means they benefit more from our earlier ECC optimizations, which in turn brings a greater increase in format efficiency than using smaller sectors (i.e. 1K).
4K also happens to be a magical number elsewhere when it comes to computers – this is primarily rooted in the fact that a normal page of memory on an x86 processor continues to be 4KB (ed: 4MB pages also exist). The x86 page size in turn has also lead to file system clusters (the smallest unit of storage in a file system) becoming 4KB as 4KB clusters neatly fit in to a page of memory, while the need for smaller clusters has subsided amidst a general increase in file size (i.e. fewer files are smaller than 4KB and waste space). NTFS, EXT3, and HFS+ all default to 4KB clusters on modern hard drives. So 4KB physical sectors map perfectly with 4KB file system clusters, which in turn map perfectly with 4KB memory pages. And hence 4KB is the largest practical size for a hard drive sector at this time.
| Old Sector | New Sector | x86 Mem Page | NTFS Cluster | HFS+ Cluster | |
| Size | 512B | 4K | 4K | 4K | 4K |
So what’s the net benefit from all of this for consumers? At the moment, not a lot, which is why this is a low-key launch for Western Digital, and the focus of this is an education effort on what the use of 4K sectors means for older operating systems. The biggest benefit is going to be that this will enable Western Digital to more easily design drives over 2TB in size.
From a numbers perspective, Western Digital estimates that the use of 4K sectors will give them an immediate 7%-11% increase in format efficiency. ECC burst error correction stands to improve by 50%, and the overall error rate capability improves by 2 orders of magnitude. In theory these reliability benefits should immediately apply to all 4K sector drives (making the Advanced Format drives more reliable than regular drives), but Western Digital is not pushing that idea at this time.
IDEMA projected format efficiency and ECC burst sizes
With all of that said, to make this kind of advancement practical, a transition period is necessary. That transition period will be done through the use of 512B emulation technology, which will expose Advanced Format drives to the drive controller and operating system as having 512B sectors, when in reality they will have 4K sectors. Unfortunately, this method has a downside, which is the ultimate reason for Western Digital’s education campaign.
512 byte sector emulation: how the drive is laid out (physical) and what the OS sees (logical)
With the emulation of 512B sectors, there’s the risk that a partition could be misaligned compared to the 4K physical sectors - where it would be unwittingly started in the middle of such a sector. As a result, the clusters of a file system on that partition would end up straddling 4K sectors, which would cause performance problems. Specifically, in testing IDEMA (the drive manufacturers’ trade group) found that random writes in would be particularly impacted, as a Read-Modify-Write(RMW) would need to take place to update each sector, rather than a straight write(ed: This is very similar to writing to a dirty block on a SSD). Although this isn’t mechanically/electronically harmful in any way, the performance hit compared to a straight write makes it undesirable.
A single unaligned 4K cluster straddling 2 physical sectors
A misaligned partition under WinXP. Only LBAs divisible by 8 are aligned
Facebook
Twitter
RSS
86 Comments
View All Comments
dragonf - Monday, November 8, 2010 - link
Correct me if I am wrong, but after reading this article on 4k sectors:http://seagate.com/docs/pdf/whitepaper/tp613_trans...
I assume that it is possible to align Advanced Format drives (those ones, which are not provided with a special aligning software by HD manufacturer) for Windows XP usage with 3rd party 4k-aware partitioning utility:
"The most critical aspect of a smooth and successful transition to 4K sectors used in Advanced Format is to promote the use of 4K-aware hard drive partitioning tools."
"When using third-party software or utilities to create hard drive partitions, check with your vendor to make sure they are updated and confirmed to be 4K aware."
So to align properly for Win XP usage for example Samsung 2TB F4EG HD204UI one has to find a 3rd party software tool for partitioning, which is 4k-aware, and use that software to partition such disk instead of Windows XP Disk Manager.
Are there any free 4k-aware partitioning tools for Windows XP?
Also would re-aligning utilities provided by WD and Seagate work with other manufacturers HDs like Samsung?
From that article I learnt also that even the newer OSes like Windows 7 do not use 4k sectors directly, they still have to use 512 B sectors. Those newer systems are only capable of aligning Advanced Format hard drives properly during their partitioning, without the need to use 3rd party 4K-aware partitioning software, and that's all:
"there are many aspects of modern computing systems that continue to assume that sectors are always 512 bytes. To transition the entire industry over to the new 4K standard and expect all of these legacy assumptions to suddenly change is simply not realistic. Over time, the implementation of native 4K sectors, where both host and hard drive exchange data in 4K blocks, will take place. Until then, hard drive manufacturers will implement the 4K sector transition in conjunction with a technique called 512-byte sector emulation."
"The sector size increase, described by Advanced Format, occurs at the hard drive media level. Host systems will continue to request and receive data from the hard drive in 512-byte sector sizes. However, the translation from 4096-byte sectors in the hard drive to the 512-byte sectors in the host will be managed in the hard drive. This process is called 512-byte emulation. It’s important that every drive partition start with an LBA offset that is aligned to the drive’s physical 4K sector. If partitions are un-aligned, then hard drive performance will be degraded."
"How can partition misalignment conditions be managed?
The first management step is to avoid misaligned conditions in the first place. This can be achieved by creating hard drive partitions with a 4K aware version of your operating system or through a hard drive imaging software product."
"The second method to managing misaligned partitions is to use partition-alignment software to identify and fix misaligned partitions. This technique should be used during the hard drive integration process."
"It’s not practical to make host computer systems talk in 4K native language – at least for a while. There are just too many places the 512 byte assumption is embedded. So, while hard drives will transition to 4K sectors physical sectors on the media, they will still “look and talk” like 512 byte formats to host computers. They will actually emulate 512 byte communications to hosts. This works well as long as the logical 512 byte assumptions from the host computer are aligned with the 4K sectors on the hard drive."
"It turns out that when a hard drive partition is created, the starting position can vary. A 4K drive format is set to work under the assumption that the first 512 byte sector (Logical Block Address = 0) will align perfectly with the first physical 4K sector".
"Alignment 0 ... works well for hard drives & 512 byte emulation because they can neatly map eight 512 byte logical blocks into a single sector. Sometimes hard drive partitions get created so the logical to physical alignment is off...".
"This is called Alignment 1 and it’s not so good for 4K drives when it comes to emulating 512 byte legacy sectors, especially when writing data. Essentially, this alignment can often cause a hard drive to manage a write with extra disc rotations, which slows things down. The results can be sort of dramatic as shown by these test results at HOTHARDWARE.com"
"The Windows XP situation seems fairly well understood in the marketplace. What is much less understood is the situation with cloning and imaging software. System builders, integrators and IT organizations frequently rely on these tools to configure systems for sale or deployment in their organization. Even if you are using Windows Vista or Windows 7, if your hard drive was partitioned with one of these utilities, you’re likely to end up with an un-aligned partition with the potential for poor performance."
http://consumer.media.seagate.com/2010/03/the-digi...
Video on 4K sectors:
http://usingwindowshomeserver.com/2010/07/27/4k-se...
Seagate manages aligment problems in the background without any interaction or knowledge of user.
BTW, a warning against 2TB hard drives:
http://www.clearfoundation.com/component/option,co...
imwithid - Sunday, November 14, 2010 - link
There seems to be much confusion and I feel as though I'm being thrown in. This seems to be, as far as I've been able to find, an x86 issue for XP users (not sure about Vista or 7). For those running Windows XP x64 Edition, this software layer interface to firmware level should not be an issue when it comes to alignment.That said, however, I suspect that Windows XP x64 Edition still formats a new, raw unformatted hard drive into 512 bit sectors during install, thereby requiring either an external live CD or boot CD that can properly format the drive's partition sector mapping into the 4096 bit Advanced Format that exists at the firmware level. Otherwise, the alignment problem will appear as the partition addresses may attempt to locate a sector at a point partway into the larger section.
Can anyone confirm this?
airesearch - Wednesday, January 5, 2011 - link
I have two new un-partitioned 1TB Black WD drives that I'll be using for data storage. Even though they don't indicate Advanced Formatting on them, If I partition them for 4k sectors and quick format them at 4096 allocation unit size will they work ok with XP PRO SP2?bobbozzo - Monday, March 14, 2011 - link
"Farther down the road will be the exposure of 4K-sectors to the operating system itself. Linux and Win 6.x are set (and we believe that Mac OS X is too), the only limit right now is the desire to do a phased transition to make thing easier for legacy users. Since 4K-sector drives won’t work on Win 5.x at all, drive manufacturers can’t put them on the market so long as there’s a significant legacy base. 2014 – the year that extended support ends for WinXP – looks like a good year to finally complete the move to 4K sectors."It's already happening...
I just bought a 3TB Western Digital My Book Studio LX, Model WDBACH0030HAL-NESN. I chose this drive in because I wanted Firewire (it also has USB), and I did not want to go with the Seagate dockable drives due to reports (here) of overheating (the WD LX is in an well-ventilated aluminum case).
Many websites are listing this drive as compatible with Windows XP, so I ordered it from Amazon.
The box says "formatted for Mac OS X"... and "requires reformatting for Windows XP, Vista, or 7".
In much smaller print, it says "Windows XP does not support drives with capacities greater than 2TB".
I took that to mean that I can't create a partition over 2TB, which I expected anyways.
Upon connecting the drive to an XP PC, a 'virtual CD' was mounted with utilities. I ran the quick format utility, but it eventually fails saying something like "the sector size cannot be changed with your firmware version".
After fruitlessly searching for help, I called WD and they eventually told me this drive is not supported in XP, and there is no way to adjust the sector size or enable translation.
The primary purpose of this drive was to take an image of a Win2003 server using VMWare Converter from a laptop running XP. This turned out to be impossible, but I was able to put a firewire card in the server (it only had USB1.0) and make an image directly.
However, after that was done, I was planning to use this drive to backup my PCs, most of which are running XP.
I have one machine running Win7 but it does not have Firewire; I may have to buy a firewire card.
My PC at work is also XP, so there will be no way to use this drive to move data from home to work.
I am also disappointed that no major-brand 2-3tb drives seem to exist with 2 or more of the following:
USB 3
eSATA
Firewire
I know it's possible because some of the external case manufactures make cases with USB 2, eSATA, and Firewire 800 in on, but I've had too many bad experiences with third-party cases (poor ventilation, failed fans, failed power supplies, weak plug connections, ...)
Seagate comes close with their docks, but the base model is USB 3 only, and additional docks are $30-50 EACH, and they are bulky to carry around.
FYI, at least one MAC website lists this drive as being a NAS as well as having Firewire, but that is incorrect. From the pictures, however, it was impossible to tell as the Firewire 800 ports are indistinguishable from CAT5 in small pictures.
Also, the USB port is a mini-USB so one has to always carry an (included) cable around with it if one wants to use USB.
Perhaps one day USB 3 will be the only connector we need for external storage, but I've had bad experience with high CPU usage on USB 2 and I don't see any evidence that USB 3 is better (still no DMA).
Thanks for the informative article, btw.
Bob
James5mith - Friday, May 27, 2011 - link
This article, and the 4K "conversion" has been happening for some time, and yet I have still been unable to find 4K native devices.I'm interested to see a rundown of what is and isn't possible with 4K drives, if there is any kind of throughput benefits and/or IOPS benefits, etc.
Do you know of anything coming within the next 6 months or so?
alpinemobile - Saturday, October 22, 2011 - link
Advanced Format Drive Fix for Windows XP (Solution, workaround).After reading a lot of crap about Advanced Format drives having problems with Windows XP and other old OSes, I decided to post this easy fix. So if you want to use the Advanced Format Drive with other OSes than Vista or 7 do this one easy thing after you purchase the drive.
Create all the partitions on the disk using Acronis Disk Director 11 (or later). Remember to tell Acronis Disk Director that you'll be using the drive for Windows 7, even though you'll use it for XP or other OS. Do not use any obsolete partitioning tools like Acronis Disk Director 10 or Windows XP disk management! The old partitioning tools will create the partitions misaligned!
That's all you have to do!
You can do a test and see if you have done everything right by running this utility: Dell Utility Advanced Format HDD Detection Tool. Just Google it.
You'll see whether your partitions are Aligned or Misaligned.
If you delete a partition using Windows XP and recreate it using Windows XP it will be misaligned again.
So use Acronis Disk Director 11 to recreate it and everything will be fine again.