living with 2TB of data

 

Overview

Life with 2TB3TB of data can get bad at times on an old Pentium 4 HT 2.8GHz (1C 2-thread), especially when it rolls around time to do a scan or do a backup or virus/malware scan.

If you have an old XP or 2000 box and you are thinking of upgrading the drive, instead think about upgrading the machine AND the drive. you will thank yourself in the end.

snapshot of my system

Fri 08/31/2012 19:07:18.43||E:\prj\df\df-3.19\win|>32\df -a -iec *
       C:--------------------------------------**  42.59GiB/ 834.63GiB(  5.10%)
       D:-----------------***********************   1.07TiB/   1.81TiB( 59.02%)
      *E:-------------------------------------***  93.28GiB/ 935.82GiB(  9.97%)
       F:-----------------------*****************   3.18GiB/   7.21GiB( 44.09%)
       G:------------------------****************   2.96GiB/   6.99GiB( 42.34%)
       H:-----------------------*****************   3.01GiB/   7.04GiB( 42.76%)
       I:-----------------------*****************   3.23GiB/   7.27GiB( 44.49%)
       J:-----------------------*****************   3.13GiB/   7.16GiB( 43.71%)
       K:-----------------------*****************   3.23GiB/   7.27GiB( 44.52%)
       L:-----------------------*****************   3.10GiB/   7.13GiB( 43.46%)
       M:----------------------******************   3.42GiB/   7.46GiB( 45.93%)
     TOT:---------------------------*************   1.23TiB/   3.60TiB( 34.16%)
TOT USED:--------------**************************   2.37TiB/   3.60TiB( 65.84%)

Fri 08/31/2012 19:07:27.34||E:\prj\df\df-3.19\win|>

there is 851GB on c:, 820GB on d:, and 899.91GB on e: which is a totalof 2570GB of data.

antivirus packages

sometimes with windows it's an auto forced reboot right in the middle of work (depending on the program and your windows update settings if it is set to auto). current antivirus software has been written to repeat weekly (unfortunately) and do auto updates (sometimes regardless of update settings). windows sometimes gets a request from the antivirus or windows update to reboot the machine after an auto-update during the scan.

In fact, If you have even 1.5TB-1.7TB of data, antivirus programs will do its weekly scan restart - and break your current scan, thus leaving you with an incomplete job and the message "your computer is at risk".

I hear Norton is 3x faster than Mcafee at one particular point.

scans: 24/7 for over 2 weeks

for 851GB of data, it takes 6 days to scan on my old box. multiply that by 3 partitions (2570GB) and you get 18 days of scanning. no windows machine I know of will run that long and stay stable. so let's find out how many days 4TB will take on my machine: 6days/851GB*4000GB=28.2 days. the results are using microsoft "safety scan" MSERT.

for any scan, it means leaving your pc on 24/7. it depends upon your OS: linux can handle the beating much longer than a windows system can (but not forever without a reboot). windows sometimes gets a request from the antivirus or windows update to reboot the machine after an auto-update (within that 28 day period!). Since I have 2x2TB, it takes a total of anywhere from 2x4=8 days to 2x9=18 days or more for a scan to complete. and I have LOTS of little bitty files and compressed archives within ISOs, which are archives.

XP is disk-priority. if the disk is going,it will shut off any other interrupts practically (like keyboard, mouse, etc). I see windows MSERT running the disk at anywheres from 15%-50% duty cycle. sometimes it goes for long pauses doing nothing with the disk. those aree the cases where a faster processor would make a tremendous difference. my proc is 1/30 the speed of today's fastest enthusiast desktop proc, the i7-3960x, and mine is equivalent to today's lowest-end desktops and laptops. mine was just about tops when I bought it. at some times, it's more like 1% duty cycle.

chkdsk /f

chkdsk /f now takes about 1 hour for both disks. chkdsk /f /r, well, leave it on for 2 days. actually I have 2x2TB drives and I do both because both need it and I have it down to a process. one of the drives is a little on the flaky side so I use it for backup purposes. I don't want to waste my money.

when to shutdown

it's usually a good idea with windows machines to shut them down after a week. you might be able to run them longer if you are doing an antivirus scan of a full 2TB hard drive, MAYBE - but the antivirus update might kick in and cancel/restart the process is usually my experience.

backups

Backups could be done with tape, but that is very expensive. it is fastest and best to mount a drive internally once in a while and back up your machine that way. If the drive is permanently mounted, the backup drive will last exactly as long as your source drive - about 5 years. it's a choice of convenience. hard drives are much cheaper. usb 2.0 is incredibly slower. we will have to see about usb 3.0 (5Gbps) and thunderbolt (10Gbps). eSata is I am told very slow. This is why I said to mount the drive internally. even a removable drive cage would be an idea (but make sure you don't remove the drive while it's on).

4TB hard drives

it takes 24 hours with 1.14TB of data to chkdsk /f /r both drives (that means fix errors [/f], and mark off bad blocks [/r]). I have the OS on c: and the data on d:. unfortunately, windows 7 didn't see fit to partition my 4TB disk as 4TB, it made it a 2TB partition and doesn't let me see the rest of the disk space as usable unpartitioned space - it makes it off limits. might be due to the fact that I needed storage drivers to do this. windows didn't see fit to ask for any. I am using 7200RPM drives with 64MB cache.

the chkdsk /f /r you should do on hard disks ONLY once every 3 months, or even better, use grc.com's spinrite. this refreshes the disk (reads do this). otherwise, you have more chances of "bit rot". a new filesystem out by microsoft called ReFS (right now only in server) is supposed to do its best to alleviate this problem of bit rot. really, it's more appropriate to do this inside of the hard disk, but as Steve Gibson said, it would make hard drives cost about $1000 to solve the problem.

I don't know how long it will take if I end up with more data. the /f operation takes up at least an hour or two by itself if I remember right.

SSD's are not like this. you don't use /r on SSD's (you don't check for bad blocks, this can reduce lifespan). you only need to do /f, and this is done in seconds with only 22GB of data on a Samsung 840 Pro 256GiB.

Backup/drive upgrade Software
NTIBackup Advanced, available from novastor.com $59.95/1computer
Hard Drive/CD/DVD/Blu-ray/Pen Drive/NAS/Network Drive/FTP Server. drive spanning. email notification. backs up whole computer. winbdows explorer for all data selection. create bootable cd/dvd. remote backup. not for windows 8/8.1 yet?
NovaBackup, available from novastor.com $49.95/1computer
they have a variety of backup packages available, including tape, sql, server, data center, pc, etc.
Symantec Backup Exec, available from symantec.com $1162.66-$1597.48
this is for client-server
backup4all, available from backup4all.com $49.95
can backup to any kind of disc media, network drive, ftp,or usb drive or flash drive with zip64 compression. plugins for backing up specific popular programs. 30-day trial.
Casper 7.0, available from fssdev.com $
does backups and drive copies/upgrades, even across disparate sector sizes. minor version updates are free, major version changes require you to buy new version (as with most companies). uses volume shadow copy service
Norton Ghost, available from symantec.com $69.99
does backups and drive copies, even across disparate sector sizes. offers cold backup. cannot duplicate/upgrade a drive that has differing sector sizes. uses volume shadow copy service.
ntbackup.exe
built into the OS (on the cd for XP home). originally meant to backup to tape and floppy. on newer versions of windows, can I think use removable media including maybe JAZ and ZIP (not cost effective) and even optical media.