Mourning the Loss of Your Written Work

[Rilbur]

What you say is correct when you begin writing a story file to a clean, newly formatted flash drive. As the story get longer the story file gets bigger and it's saved to the flash drive both with manual save commands and with autosave. Each of these saves is done with multiple intermittent writes. Then another story is written and rewritten to the flash drive, also with multiple intermittent writes. The way a FAT formatted flash (or hard disk) drive works is that each one of the manual saves writes the complete file to disk, and the old file is marked as deleted and the sectors it used are returned to the file allocation table which is written to the flash drive. Eventually the flash drive become fragmented; large files written the flash drive will be fragmented across many sectors. Each sector is written as a series of writes; as a file is written the links for the sectors being used are updated to the file allocation table on the flash drive. For a large MS Word doc file write (say, 10,000 words and about 112KB) there will be sustained writes as the file is written across multiple sectors and the links are updated to the file allocation table, plus the old file is marked for deletion and it's links are written to the file allocation table; i.e., this generates a series of sustained writes to the flash drive, the same as a hard disk drive formatted as FAT32. A hard disk drive is designed for sustained writes; a flash drive is not.

 

A few details / habits that explain our differences of opinion.

 

1. Thumb drives don't suffer from fragmentation. They can't. Not because the data won't be fragmented over the device, but because it doesn't matter. it only matters on 'traditional' hard drives, because in-order sequential reading (writing) is the quickest way to do it. You have to physically move the head along the platter, and wait for the magnetic platter to reach the right point to read from the disk. On a thumb drive, this is a non-issue.

 

2. I don't keep my stories in one large file. I have every chapter separately labeled as it's own file, along with a file just for writing (hold over from when I wrote in a hand held device 'on the go', something I don't do anymore). As a result, the average read / write is much smaller, especially since I aim my chapters to be about 5000 words long.

 

3. I've never actually faced such a break down. I make a habit of replacing my thumb drives on a regular basis (usually for size / speed increases, about once every two years), and I actually have software specifically to automatically back my data up (twice; once to my primary HD, and then every once in a while my primary HD is backed up to my 'backup' drive). If I'd been hit by this problem, I'd probably be more careful.

 

4. Until recently, I did a lot of bouncing between computers to work. A thumb drive was the only device that made sense for that. In fact, it's still the only data storage device that makes sense, in a lot of ways, because I can't reliably say 'I WILL have access to my desktop'.

[Hoskins]

Rilbur, I'm glad you have found a way to keep your data with you. Please change your flash drives more often, or get a few of them and rotate them. They are only programmed to be written to x number of times, and then they quit. Usually by going totally south on you. I have three, and I rotate them, usually by month.

And please don't consider them as a reliable place to store valuable files, because they are badly made (unless you pay a lot for them) and can die without any notice. I've seen it happen quite a few times.

 

Also, if you're moving from computer to computer a lot, use a service like dropbox.com. Its free for up to 2 gig, installs on your computer so you just "save" files to the cloud within your applications, and can be accessed via a web interface if you're working from someone else's machine. I've found it's much much more convenient than a flash drive. Just go to a website, download your file, do your work, and upload it back to the web.

 

Also, what Colinian says is correct about file saves. When you save a Word document for the first time, it saves the file in total. When you make changes and save it again, it saves only the changes you've made, appending them to the file each time. As you work on a file, it grows.

 

Every once in a while use the "save as" command on a document, which causes all the changes to be committed to the file in total, and you will see that its size will drop, sometimes dramatically.

Unless a flash drive has firmware that overrides the disk format (usually FAT32), disk writes aren't necessarily automatically sequential and a flash drive can fragment. Generally, the cheaper the flash drive, the more likely this is the case.

Edited June 10, 2010 by Hoskins

[Rilbur]

I actually do have a dropbox account, the problem is that I frequently don't have internet access either. It certainly isn't 'reliable'.

 

My thumbdrive is set up to automatically backup every couple of hours to my HD (reads don't consume lifespan, only writes, from what I understand, so I'm exploiting this). Also, I tend to pay the premium for good, high-end hardware. As such, I can loose practical maximum of one day's work on a given file, especially since my hardware tends to last me for several years.

 

Eventually, I will loose work when a device fails. But... rewriting a scene, with rare exceptions, is generally quite a bit easier than writing it the first time. (The rare exception would be the major 'trauma' scenes, where something so traumatic happens to your characters that it bounces back into you. For example, I could not re-write the rape scene in Guardians with ease.)

 

Edit: Oh, and I nearly forgot the point I was going to make with this post. I think we're going to have to agree to disagree, because using the thumb drive like I do now is just too appropriate to my lifestyle to give up. All I can do is make sure I have regular, thorough backups done.

Edited June 25, 2010 by Rilbur

[Hylas]

I wonder why you guys have to resort to those kinds of backup though. Text files are usually very small. I am usually just content with copying the entire folders over to different hard drives every once in a while or posting them somewhere in the internet. For example, for text documents related to work, I usually save them in an EtherPad* hosted by our own servers, for stories I just upload them bit by bit to GA heh. It does makes sense though if you want to save other larger files as well. I have a 500 GB external hard drive for that but I don't set it to auto, I just move stuff there manually when needed. And its mostly used for my large work files.

 

I also utilize a habit I have learned from my work as a 3d artist. Save often and save multiple copies per save, have as many backups as you possibly can and name them correctly (don't be lazy and just type a random string of numbers after each copy, name them sequentially or by the date that they had been saved). This means utilizing the Save As button. In my 3d apps, we have a Save As Copy button (which Word doesn't have AFAIK). It's an essential discipline for us, as overwriting or losing something could cause us to lose valuable work data. I've learned the hard way on that myself around 3 years ago, when I lost files for a massive space station I had been working on for a whole month, all because I had clicked the wrong button and overwrote it LOL.

 

*For those unfamiliar with it, it's a real-time collaborative text editor, with a chatbox on the side. Very handy for writing up something with people over the internet. You can password protect pads being currently worked on, and it has an incremental save system (it saves literally everything done to the pad) which you can rollback at any time. You can resume working on a text document, as long as you know the url (you can bookmark all your pads if you want). It also has basic formatting options. This might be of interest to authors/editors who want to work more closely together or for authors who want to collaborate on a story. It has since been purchased by Google for Google Wave but has allowed (one of the reasons why i love Google) the original Etherpad team to release its source code as GPL (that's 'completely free' for non-geeks). There are now other sites currently hosting it for those who have no access to privately hosted EtherPad servers, the most prominent being PiratePad.

[Rilbur]

In general, you don't 'have' to. It just really sucks when you don't, and then turn around and discover something went wrong.

[Hoskins]

Hylas, you're way far ahead of the curve by understanding the difference between "save" and "save as". Your experience as a 3D artist puts you in good stead amongst those who have lost stuff to simple mistakes.

 

Most people are so ignorant about computers and backups that, if there were a way, I'd revoke their license to use a computer. Put succinctly and being whiny about it, what usually happens is that a customer takes a "it won't happen to me because it never has before" attitude, gets upset when the data is lost, and then the complaining starts because they need to actually DO something (or spend money) about backing up their data, because as far as they are concerned, it should just be magically backed up in the background while they go on shopping on overstock.com.

 

I can't count how many times I've said "Do you have a backup of this?" and got nothing but a blank stare back. It's not like magazines, TV, and Internet sites aren't advertising online backup services. There are even free ones.

 

I can't imagine being a college student and having someone walk away with my backpack with a laptop in it during the middle of an important term.

 

Okay so maybe that wasn't so succinct...