Would you like to make a Windows PC that has some network-shared folders containing large files look as if there's something horribly wrong with it?
Well, that is, I have just discovered, easy!
Just copy one of the files - let's say they're movies - over the network to another computer.
Then, before the first copy operation is complete, start copying another. And another. And another.
Like, maybe you're copying all of your absolute favourite TV shows and movies, and you're just clicking and dragging whatever catches your eye, and letting the little copy dialogs pile up.
This is exactly the sort of task that a proper file server is specifically designed to handle. Even if the server's storage isn't terribly fast, six people all asking for different things on the same drive at the same time should be put in a queue, rather than asking the waiter to carry every order for the whole restaurant at once, as it were.
But consumer versions of Windows aren't set up to do that. They're meant to serve a single user, and see nothing wrong with just doing exactly what they're told to do if several remote computers - or even just one - ask them to do something like copy 20 large files at once.
(You can probably alleviate at least some of this problem in consumer Windows versions by something like selecting the "background services" and "system cache" options in WinXP's Performance Options -> Advanced tab. Even a cheap Solid State Drive would immensely reduce the problem, too, since the multi-millisecond seek time of mechanical hard drives is a big reason why it happens, and SSDs have near-zero seek time. But SSDs are, of course, still rather too expensive per gigabyte for tasks like bulk video storage.)
Once you've got several large copy operations all happening at once on, let's say as a random example, the Windows XP computer on which I'm writing this, this "target" computer will be flogging to death the drive(s) on which the big files reside. Data to and from those drives probably has a bottleneck or two of its own before it gets to the network, too; the ATA I/O hardware in consumer PCs is not made to deal elegantly with several drives all talking at once.
The upshot of this is that any operation which expects one of those drives to respond snappily to a request - or, quite possibly, which is just trying to talk to some other drive in the computer - will now suddenly find itself waiting a lot longer for that response. And that's one of the standard ways in which programs can misbehave. It's perfectly normal for programs to, say, ignore user input while they're saving a ten-kilobyte file; that should only take a moment. But if it now takes 30 seconds for that tiny file to be saved, as the save operation tries to push through a storm of seeking and reading, the program will appear to have hung. This applies to lots of things besides saving files - if modal windows suddenly take 30 seconds to appear, for instance, the program will seem just as broken.
On the plus side, as soon as you cancel the 20 simultaneous copy operations, the target PC will immediately start working perfectly again.
On the minus side, before you cancel the copies, it'll be doing a very good imitation of a computer with one or more failing hard drives. And practically anything you do to try to figure out what's going on will just add more input/output tasks to the mess, and make things even worse.
Eventually, the user - let's call him Dan - will try to shut the computer down, get sick of waiting for the numerous simple disk tasks this entails to conclude, and just turn the darn thing off. And now, the metaphorical plate-juggler will forget about all of those plates that are still in the air, and leave them to crash to the floor.
In my case, this meant various programs lost some of their configuration data. Firefox, for instance, was back in the default toolbar and about:config state, and all of the extensions thought they'd only just been installed. My text editor forgot about the files I used to have open, and Eudora lost a couple of tables-of-contents and rebuilt them with the usual dismal results.
(If you lose the TOC in a mail client like Eudora that uses the simple mailbox format in which the main mailbox file is just a giant slab of text, and the TOC is the separate file that tells the programs where actual e-mails start and end within that text, rebuilding the TOC probably won't go well. It'll give you a separate "e-mail" for every version of a given message that looks as if it might exist. So if you saved an e-mail you were writing five times before you finished it, you'll get five separate versions of that e-mail, each at a different stage of completion. When you "compact" a mailbox, you're getting rid of all that duplicate data.)
Oh, and the file I'd been writing something in for the last hour was now a solid block of null characters. As, delightfully, was its .bak file.
I lost very little actual data, because I make regular backups. (Remember: Data You Have Not Backed Up Is Data You Wouldn't Mind Losing.) And stuff I'd done since the last backup which I wasn't actually working on at when the Copy Of Doom commenced was all fine. Good old PC Inspector even let me recover a bit more data.
Even quite a lot of data loss would have been preferable to what I originally thought was going on, though. It looked as if the boot drive - at least - was failing, leading me to another of my disaster-prompted upgrades. It's about time for a new PC now anyway, but it's ever so much more civilised to upgrade when the old computer's still alive.
1 February 2009 at 3:58 pm
yea i used FEBE plugin for firefox automated backup now. firefox lost my profile a couple times in the past. so annoying. i think if you dig through the folders the profile info still exists and you have have to manually load it or something, its been a while. still i'm sure if that happened to someone who wasn't techie at all it would freak them out. it certainly pissed me off:P
and yea the inability for windows to chain tasks together is annoying. people like to start several processes/tasks and then walk away or do something else if its a long task. forcing people to babysit tasks is silly.
1 February 2009 at 4:54 pm
This seems like the kind of story that should be told around a campfire, with a flashlight under one's chin.
"And when he restarted his computer...ALL OF HIS CONFIGURATION DATA WAS CORRUPTED!"
*children gasp*
"No!"
"That's why I make regular backups!"
1 February 2009 at 6:40 pm
I keep hitting little unexpected munged-file land-mines. The most recent one was causing Winamp to die on startup, every time; turned out Winamp's main.dat and recent.dat files had, like the thing I'd been writing, turned into blocks of null characters of exactly the same size as the original file.
1 February 2009 at 7:13 pm
haha, Dan you goose.
[Hey - I didn't say anything about me being the person who did the deadly clicking and dragging. I knew there was copying going on, but I didn't know it was a Copy Of Doom - if I'd known how it was being done, the penny would have dropped pretty much immediately and I could have avoided the data loss. The identity of the Doom-Copier will remain a secret, to protect him or her from violent reprisals. -Dan]
Use an alternate file copy handling program
I can recommend "Teracopy" http://www.codesector.com/teracopy.php
Works great for queuing up file transfers so you can just copy over willy nilly as you described (I reccommend turning it off as a default file handler and right click dragging, works better)
Not only that, but it will let you resume files, etc.
I tried a few programs for that, this seems like the lightest one that works the most seamlessly, either way I think you should have a looksie :)
1 February 2009 at 7:35 pm
Teracopy is quite nifty but it's no good for moving files on a network drive as it acts like its REALLY moving files
1 February 2009 at 10:23 pm
In your place I'd nuke from orbit and restore a backup. The deities only know how many other configuration/system files have been munged by this and will cause your system to randomly come crashing down for no apparent reason.
Frasera: I use a Seamonkey/Firefox combo for browsing (SM for light stuff, FF for interactive things like forums and such that occasionally confuse SM into a stupor), and I've found the later versions of both tend to lose their brains a lot less than before. I still do regular exports of my bookmarks, but it's now more than a "just in case" thing than something that's actually needed.
This might have to do with me switching my work computer to Linux exclusively (and the EeePC already had it). Even Linux-for-dummies Ubuntu and Xandros aren't quite the user-friendly experiences that some want you to believe, but they genuinely do not ever crash in the way Windows is prone to do.
Then again, I use the same SM/FF combo on my gaming machine, and those too lose their config data a lot less.
I'll try teracopy on the gaming box, sounds promising.
2 February 2009 at 3:09 am
Teracopy is quite nifty but it’s no good for moving files on a network drive as it acts like its REALLY moving files Comment by hitmouse — February 1, 2009 @ 7:35 pm
I use teracopy locally and to shuffle files PC to PC over gigabit ethernet, and I don't understand what you're getting at. With proper permissions, I've moved and copied files without a single hitch. The one thing I can think of is if you have read-only permissions on a network share and you attempt a move off of that share. I've seen teracopy, during a move on a large file with verify turned on, copy the file, checksum the two, then delete the source (moved) file. When is this undesirable?
Otherwise, yeah, teracopy would have prevented this exact issue Dan. If you start it before doing copies and set a destination folder, you can drag and drop relevant files onto the teracopy window and then start the transfer - it will do the copies sequentially with no drive (or data) destroying seek thrashing.
2 February 2009 at 3:10 am
Re Teracopy: It offers copy and move options on the context menu. What´s "it acts like it´s really moving files" mean?
2 February 2009 at 6:33 am
Oi, for the love of Mike that sounds terrible. You have to love the way it's decided that files, in event of emergency, should have their state written to the disk, even though total abandonment would likely be soooo much better.
2 February 2009 at 3:14 pm
Teracopy will not just move a file pointer if you are on a network share, and move a file from folder 'a' to 'b' (both on the share). It will actually move data over the connection (slow). I recently used TeraCopy to populate a new NAS, worked perfectly for many GBs of photos etc, it will simply que additional operations.
2 February 2009 at 5:21 pm
Somewhere in my system is a little memory overrun or something that will, after many hundreds of hours of uptime, and certain other precipitating factors, cause Maxthon (my browser) to sort of lose its mind and stop displaying my entire favorites menu. It will also not open the right click menu -sometimes only part of it shows up, sometimes it gives me a bizarre dialog "Error load browser menu". If this goes on long enough the right-click menu becomes weirded out sytem-wide. A restart solves this problem. It is often exacerbated by my girlfriend leaving 87,000 Firefox tabs open at the same time, many of them with Flashcrap and the like going in them, eating up hundreds of thousands of K of RAM.
2 February 2009 at 8:25 pm
"The identity of the Doom-Copier will remain a secret, to protect him or her from violent reprisals."
It was one of the cats, wasn't it?
2 February 2009 at 8:54 pm
I don't use terracopy, I use Total Commander. It has a "Queue" instruction when copying or moving. I used it for moving very larges files and never had any problem.
And you can even specify buffer size,Pause it, or limit copy's "speed".
The best about TC, it looks like nc in dos or mc in linux...
Btw Dan, as I learned in woodworking classes, "Always blame the tool."
2 February 2009 at 11:15 pm
One of you bastard readers reminded me, via e-mail, that I really ought to run chkdsk over every drive in the computer now.
So I did that, at eight o'clock in the morning.
At eleven o'clock at night, it finished!
And I wasn't even doing every disk. A couple weren't locked, so I chkdsked them in the background while doing other stuff. Four disks remained that had to be done on the next restart, though, and those four were apparently chkdsked in a process involving telegrams and sea mail. It got to the point where I was checking to see if the damn thing was in a loop.
Still, it's one of those better-out-than-in things.
3 February 2009 at 3:04 am
This is the difference between file servers and home computers, and why UPS's are so important on file servers.
The average home computer has only a few files in memory and not written to disk. And likely not all that important if the power fails and stuff doesn't get written.
File servers, on the other hand, generally have lots of imporant stuff still in memory and not written to disk.
3 February 2009 at 9:35 am
Ah, the kind of problem I ran into back in the good old days of network sharing mp3's in my dorm in '97, if you get a few people streaming different files off your computer at the same time the computer ground to a halt.
4 February 2009 at 10:29 pm
Times like this, Windows needs someone like Mr. Resetti to pop up and berate you for corrupting the file table, then encourage you to restore from the backups you'd better have made, because otherwise he's coming after you with a rake. Not for Dan's benefit of course but it would be a more effective deterrent than an enforced chkdsk.
It's shocking in this day and age that OSes don't have a built in sense for which disk jobs simply should not be done in parallel.
6 February 2009 at 4:50 am
Dan: you of all people should know the CTRL-click-everything-you-want-before-copying rule. Windows makes a nice, sane, orderly queue for copies *if* you do it all in one go.
6 February 2009 at 12:06 pm
Yeah, one more time: It wasn't me doing the copying.
I wouldn't have started umpteen parallel copies, if only because of the obvious horrible inefficiency.
8 February 2009 at 12:03 pm
Obviously, no software yet invented can supply the performance you *really* want in this situation, but yeah, consumer versions of Windows are particularly lame about it.
A better approach (on the part of the OS) is to copy large files in large-chunk-at-once mode but give your-chunk-is-next scheduling priority to small chunks, which will then be smaller files. That would probably have greatly reduced the collateral damage in your situation, though the large file copies would still have taken a goodly while to complete.
Various OSes have various perf characteristics, but I suspect almost any server-grade OS will do better here than Windows XP. Ideally you don't want to use Windows desktops as file servers, but you know, sometimes you've GOT the computer sitting around turned on almost all of the time anyway, and it happens to have a bunch of free space on its drive, and it's just so tempting to put a shared folder on there and use some of that available space. I do that myself.
The protocol does also matter, but if you want to access the files from desktop Windows systems, you pretty much have only one choice there (CIFS). I believe Microsoft supports other protocols (e.g., NFS) in some of its server-OS products, but if Windows XP or Vista can connect to NFS shares, it's news to me.
14 February 2009 at 11:01 am
hm nice teracopy recommendations:) helped me a bit with network transfers of large files.