Filesystem/Disk Limitations


My little blurb: There are 2 partitioning schemes: BIOS+MBR and UEFI+GPT. GPT uses 64-bit LBA, which is now the current standard. The new partitioning scheme is called the GUID Partition Table (GPT). Partitioning software like gparted (and Windows) needs to be compatible with the 16-bit partitioning scheme that we already have AND work with GPT.

for maximum NTFS basic/dynamic disk size articles circa windows 7/8, refer to this and this.

LBA sizes section

MFRs have already started making IDE/SATA drives with sector sizes larger than 512 and currently use 512-byte sector enulation or 512e I think it's called.

4K=4096-byte sectors are the new "Advanced Format".

  • 48-bit LBA:
    144,115,188,075,855,872 byte drives. I think this is the ATAPI 6.0 standard that came out in 2001.
  • 48-bit LBA:
    1,152,921,504,606,846,976 byte drives. I think this is the ATAPI 6.0 standard that came out in 2001
  • 64-bit LBA:
    9,444,732,965,739,290,427,392 byte drives.
    Note that 64-bit LBA goes beyond the standard size of a 64-bit CPU register and can be simply multiplied with normal data structures to get total number of bytes but IF values were to ge this high or max were calculated, could overflow by up to 9 bits (and C/C++ doesn't do overflow detection) but it probably won't be there for quite a while...
    used by [[U]EFI+]GPT
  • 64-bit LBA:
    75,557,863,725,914,323,419,136 byte drives.
    Note that 64-bit LBA goes beyond the standard size of a 64-bit CPU register and can be simply multiplied with normal data structures to get total number of bytes but IF values were to ge this high or max were calculated, could overflow by up to 12 bits (and C/C++ doesn't do overflow detection) but it probably won't be there for quite a while...
    used by [[U]EFI+]GPT

the rest

  • Win9x/ME: 229=536,870,912 bytes=512MiB RAM Maximum.
  • 32-bit NT-family OS's: 232=4,294,967,296 bytes=4GiB RAM
  • 64-bit NT-family OS's: 264=18,446,744,073,709,551,616 bytes=16 EiB RAM
  • Windows server 2008 64-bit: 2TiB RAM for Enterprise, Datacenter and Itanium-based Systems, 32-bit: 64GiB for Enterprise and Datacenter
  • DOS/Win95 DoubleSpace Disk compression: compressed volume size max is 512MiB, due to the fact that software can't handle cluster sizes larger than 8KiB.
  • SCSI: 32-bit block number built into the protocol. With 512 byte blocks, that's a 2TB architectural limit. We can expect to be able to buy a 2 TiB disk off the shelf in another two or three years.
  • FAT12: Maximum 32MiB volume size. 4077 clusters max*8192bytespercluster=33,398,784 bytes per volume.
  • FAT16: given 32KiB clusters and 16-bit FAT, maximum 32768 bytespercluster*65517clusters=<2GiB=2,146,861,056 bytes per volume. 64KiB clusters will result in problems with Windows and DOS filesystems. Don't know about FreeDOS - maybe it's fine.
  • FAT32: given 32KiB clusters and 32-bit FAT, maximum 32768bytespercuster*268,435,445clusters=<8TiB=8,796,092,661,760 bytes per volume. 64KiB+ clusters will result in problems in windows systems, but may be possible in other OS's.
  • FAT32: under Windows, clusters cannot be 64KiB or larger or it would calculate disk space incorrectly.
  • FAT32: under Windows, minimum 65,527 clusters=32768*65527=<2GiB=2,147,188,736 bytes.
  • FAT32: under Windows, You can't decrease the cluster size on a volume so that the FAT itself ends up larger than 16 MB-64 KB in size.
  • FAT32: SCANDISK on win9x is 16-bit program. limited to 4,177,920 clusters(<4Mi)=32768*4177920=<128GiB=136,902,082,560.
  • FAT32: under Windows XP installation/setup, When attempting to format a FAT32 partition larger than 32 GiB, the format fails near the end of the process with the error: "Logical Disk Manager: Volume size too big."
  • FAT32: maximum file size: 4GiB-1byte=4,294,967,295 bytes.
  • FAT32: 255 characters/filename when using LFN/vfat.
  • FAT32: SW limitations: 64GiB limit enforced by 98's FDISK (see replacement repaired FDISK), 127.53GiB limit imposed by SCANDISK (part of WinME, 98) and 32GiB limit within the Windows 2000 Format Utility, even though FAT32 architecture theoretically can support drives up to 8TiB. Windows 2000 can mount FAT32 volumes that are up to 127.53GB in size (max). XP can't format FAT32 more than 32GB during installation , but supposedly can format up to 8TiB later (I had no such success with my external 250GB USB 2.0 - NTFS only). Even if you can't format the disk and you get an error, you can mount a 128GiB FAT32 volume in XP/Vista. You may get continuous disk errors from 98/ME's SCANDISK on FAT32 volumes larger than 32GiB when passing the 32GiB point. see next item on PHEONIX BIOS/BITSHIFT.
  • the bigger FAT32 gets, the slower it gets.
  • FreeDOS32/FAT32+: files up to 256 GiB in size.
  • PHOENIX BIOS+SCANDISK: 32GiB LIMIT on FAT32 volumes because of BITSHIFT translation method. see SCANDISK patch for 9x and article.
  • FAT12/16: 512 entries in the root directory/folder. that means C:\ (I was not sure what exactly this meant. I have always experienced a limit of 105 files in this case. anything more and you get a "disk full" error)
  • DOS/Win95: 105 files in the root directory/folder. at this point you get a "disk full" error.
  • FAT16: 2GiB volume see other limits of FAT on wikipedia
  • FAT32,NTFS: 16384 files per folder/directory if you have Kaspersky installed. see articles here, here, and with more research on Vista, here. solution: not disable, but UNINSTALL Kaspersky if you need to store this many files in a folder/directory. Problems with Vista will disappear.
  • [[U]EFI+]GPT: 64-bit LBA=264=18,446,744,073,709,551,616 sectors.
  • NTFS: 232−1=4,294,967,295 files per volume. see off-hand reference.
  • NTFS: max 2^64−1 clusters=18,446,744,073,709,551,615 clusters*32768bytespercluster=<512ZiB=604,462,909,807,314,587,320,320 bytes 256TiB maximum volume size [not so. see the next limitation below]
  • NTFS: 2.19TB (2TiB=232×512=2,199,023,255,552 bytes) maximum disk size due to 32-bit-LBA addressing limitations of MBR, and this does not apply to GPT. Under GPT the maximum is 256TiB. HOWEVER: this 2TiB limitation increases with the sector size - large-sectored hard disks will probably appear after the 2TB drive.
  • NTFS: maximum number of files: 232-1=4,294,967,295=4GiB-1
  • NTFS: maximum file size: theoretical as of win2k is 16EiB-1KiB = 16×10246B-1024B = 18,446,744,073,709,550,592B theoretical, implemented 16TiB-64KiB = 16×10244B-65536B = 17,592,185,978,880B. according to this data structure, sector sizecan be a maximum of 32 bits=2^32 but you must lop 1 bit off to have a power of 2 so 2^31. so is the current theoretical maximum with win7 and up with large sectors now (16*1024^4-65536)/512*2^31=73,786,976,019,960,299,520 = 63.999999*2^60 = 63.999999EiB? this is over the 16EiB limit of 64 bits, so the filesystem is probably going to be stuck at 64 bits max until we can get some nice 128-bit or 256-bit cpu's. example: a 4k sector "Advanced Format" disk is 4096/512=8x bigger than the old standard. os the implemented maximum may be 128TiB. seems awful small for an EMC Data Center storage array if you want to do RAID... I am guessing the Dynamic Disks also have the same limitations. what this means is, tiny files will now take up 4k bytes each minimum. if you have lots of them, as in the case of software development, this would make a big disk seem small.
  • NTFS: maximum filename length: 255 characters (UTF-16 code units)
  • ATAPI 6: 144PB (petabytes, or 144×1015=512×248=144,115,188,075,855,872 bytes). with ATAPI's current limitation of 133MB/s (13.3MB/s real world), maybe this is not such a hot idea? we'll see how they handle the speed barrier, haven't seen any ATA150 drives or motherboards really, just ATA133. same 40-pin connector. My guess is that means slightly reduced data rate with more address to push on bigger drives — something's gotta give when you have a fixed bandwidth.
  • NTFS on Windows 2000 SP2 & XP pre-SP1: did not support 48-bit LBA hard disks (disks larger than 137GB=512×27×4=137,438,953,472 bytes). Later service packs like SP3 and SP4 of Windows 2000 support 48-bit LBA (drives larger than 137GB).
  • Win2k Disk Defragmenter: can't defrag NTFS with cluster sizes larger than 4KiB. limitation removed for later OS's like XP & server 2003.
  • complete memory dumps are not available on Win2k/xp/2003 computers with more than 2GB RAM.
  • 4GB paging file for windows NT family OS's, such as NT, XP, Win2k. 2003 is the exception. it can take multiple pagefiles.
  • NT-family OS's have a 4 primary partition limit. [could just be 4 partitions, but I think it's primary partitions]
  • NT 4.0 limited to 7.8GB system partition.
  • Win2k - Upgrading a Basic Disk to a Dynamic Disk: You can't upgrade drives that use sector sizes larger than 512 bytes. If the drive has large sector sizes, you'll need to reformat before upgrading.
  • XP - number of handles per user process is limited to 10,000. Webroot SpySweeper opens 18,000 somehow. of all my programs, it is the hog. They have not fixed the problem. The next largest is IE7 at 1,400.
  • ALL microsoft OS's - the following 2 limitations are found in winnt.h within Visual C++ 6.0: #define MAXIMUM_PROCESSORS 32 and #define MAXIMUM_WAIT_OBJECTS 64. I am told the MAXIMUM_WAIT_OBJECTS applies per-process and is there to prevent abuse. There seemed to be an upper limit of about 2000 threads back when I was running NT 4.0. On average, an XP desktop will use about 1054 threads.
  • limitations on FAT long filenames for win95/98/ME: Windows 95/98 had a 255 file name character limit, and a maximum of 260 characters for the full path name (see post). also see microsoft kb article on the allowable character set (also applies to NT family systems).
  • COMMAND.COM/DOS is limited to 8.3 for filenames, and you are pretty much limited to `_0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ. for characters. In FreeDOS, minus(-) is translated to underscore (_) somewhere between mkisofs.exe and FreeDOS xcopy.exe.
  • NTFS: you can't use / \ : * ? " < > | characters in ANY NTFS filename and make sure your total file path does not exceed 255 characters (unless you are specifically addressing a drive using colon). see article.
  • CMD.EXE shell:up to XP SP2: NT family CMD shell command lines cannot be longer than 2048 characters. (you run into problems with some UNIX emulators when compiling programs in gcc)
  • CMD.EXE shell:XP SP3, on 32-bit machine: NT family CMD shell command lines cannot be longer than 8190 characters (which is 2 characters just short of 8KiB. (you run into problems with some UNIX emulators when compiling programs in gcc). This probably also means that environment variables like the PATH can be 8KiB as well.
  • Windows Memory Limitations: see MSDN Article.
  • exFAT aka FAT64 (Vista SP1). Intended for flash devices. 32MiB/cluster. 64-bit FAT. Does not work with Vista's ReadyBoost.
  • ALL PC OS & BIOS: a limitation of the 16-bit IBM partitioning scheme & 32-bit LBA is 2TiB-512bytes=2,199,023,255,040bytes per partition and on total disk size, based on 512 bytes per sector and 4-byte # of sectors of partition and 4-byte LBA starting sector listed here on wikipedia. You will start seeing this limitation start to affect users when the 2.5TB disks start arriving, which won't be long. People will not be able to use large disks on older systems, but worse, the partitioning scheme will have to be layed aside for something better - the GPT+UEFI has not been accepted by the industry yet, and larger drives are coming fast, and the standard isn't ironed out with MS yet it seems. Expect to see large-sector disks and a resulting lot of software patches. Maybe then you can ask the UEFI what the sector size is.
  • the 64-bit FILETIME structure is based on a 12am Jan 1, 1601 epoch and is in 100-ns increments. using seconds as the base, it is good until the year (100e-9*2^64+1601*365.24224*24*60*60)/(60*60*24*365.24224)=60,056.446537 (60k years). I don't think there will be a problem until the elements melt with fervent heat? (God lets go of the atoms and they fly apart?) By him all things consist.

OS limitations:

  • CMD: 2048 (pre XP SP3) or 8190-character (XP SP3 or 64-bit XP) command-line
  • COMMAND.COM: 127-character command-line
  • memory limits on windows

you can see how long the command-line is on your favorite OS by doing this (we are trying to turr a non-square block of characters into as square a block as possible with the characters we have):

  1. start Notepad++ (not Notepad, this is a programmer's editor)
  2. start a cmd.exe by doing [windows-logo-flag-key]-R cmd [Enter]
  3. click the upper left corner menu handle and pick properties and set the window size to 132x121 and buffer size (as big as you can make it) to 132x9999 however long you can make it, I usually choose 9999.
  4. make your changes to the shortcut which started the window, and save your changes
  5. hold down a letter key in the cmd shell prompt. it will take a few minutes.
  6. after it stops, right click and pick Mark, click on the upper left corner of the entire block of text where your characters are, even if it contains a command prompt and some trailing spaces, but no blank lines. hit [Enter] when you are done. this will copy it to the clipboard.
  7. paste it into a new document in Notepad++.
  8. remove the command prompt from the text (drag the mouse over it and hit delete).
  9. if there are characters left over on that line, drag the mouse over the remaining repeated letters on the line and hit Ctrl-x for cut or do Edit, Cut and click on the end of the last partial line and do Ctrl-V or do Edit, Paste. essentially you are stripping out the command-prompt junk, and moving the little chunk of text from the first line onto the last line.
  10. if it overextends past the line ends of the other lines, click in the place where the line should end on the last line and hit [Enter] and the line should split.
  11. go back and click on the first line and see if it is completely blank.
  12. if it is (it should be), delete it by hitting the delete key until the the entire text comes up one line (until that blank line is gone).
  13. NOW we can count blank lines and columns. click in front of (just to the left of the character of) the last character of the longest last line.
  14. now look in the status bar at the bottom. you should see something like Ln:62 Col:132 multiply these numbers with a calculator and write and answer down.
  15. now for adding the last line, if it contains fewer than a full line of characters, you click on the last line just before the last character (or you can hit the down arrow key and then the left arrow key).
  16. look at the Col:6 number and add this number to the previous number you calculated. this is the total command-line length.

If you get something other than 8190, I would be extremely interested in it if you were to email me with the results so I can document it here. If Microsoft has improved the cmd shell on a particular OS, such as 7 or 8 or vista, I am interested.

Overcoming 2TB Limit