The Problems With Digital Data Storage

The odd floppy discs and even mini discs that are lying around your office might tell you all you need to know about how quickly new storage formats can become obsolete. But this problem doesn't just affect consumers. It also presents a big problem for archivists who are trying to store digital material into the future. According to Dr. Leo Enticknap who's a lecturer in cinema from University of Leeds, this is certainly a problem in the archiving of movies and television programmes as he explained to Dominic Ford.

Dominic - Now, we're used to hearing about classic episodes from the 1960s being burnt because there simply wasn't storage for the film with those programmes on.  Surely, with the advent of modern hard discs and DVDs, it's much easier to store vast quantities of video information.

Leo - In the short term, yes, but I'm afraid there always is a 'but' whenever we're talking about digital media. Firstly, you have a problem with chemical decomposition of the discs themselves. For example, in pre-recorded DVDs, the ones that are pressed in a factory, you can get into problems with the chemicals used to form the dye for the label on top, attacking the substrate of the disc for example and causing problems. And with consumer recorded DVDs, the DVDRs for example, the recording medium is a dye. And that dye is very slightly changed in its light reflectivity characteristics by being burned by a laser. The problem is, that that reaction that's started by the laser being burned can't be stopped completely. It can only be slowed down almost to the point of being stopped. So, that dye layer is continuing to change very, very slowly even after it's being recorded and as a result of that, basically, your DVDRs have relatively small shelf lives.

Dominic - So, what timescales are we talking about here? If I'd burn a video onto a DVD, how long will it stay there?

Leo - There are a number of variables but most archivists and librarians would not want to trust one for longer than say, 3 to 5 years and even that stored in optimal conditions.

Dominic - What about hard discs? Surely, those are more durable.

Leo - They're slightly more durable, but again, I'm afraid there's always a gotcha with all of these media. If you think about a hard disc, it's a complicated electromechanical device. You've got two or more, sometimes several glass platters which have the magnetically sensitive coating on them. You've got the head assembly, the platters themselves are on a spindle, which has a bearing and which has lubricant in it, and on top of that, you've got a PCB which contains the control electronics and of course, your interface to the computers.

Think about all the possible things that can go wrong on that. You can have a failure of any one of these moving parts which are incredibly tiny and also, you can have the issue of hardware or software obsolescence. What happens if in 20, 30 years' time, computers quite simply don't have USB sockets on them anymore. In the same way that computers now, simply don't have 5 1/4 inch or 8 inch disc drives in them as they did 30 years ago.

Dominic - And I guess we've touched on the question of software here.  Obviously, file formats are forever changing. Does that mean that you're having to continually reprocess this material into newer file formats?

Leo - Really, we're into some quite unknown territory here because so far, we haven't had a major instance of an audio or video file format in widespread use quite simply, dropping out of existence in terms of being supported by the latest generation of playback software, but you'll never know it could happen.

I mean, just to give you a non-audio visual example, if you have documents that were produced using only like version 1 or version 2 of Microsoft word, and you try to open them using Microsoft word 2010, well, it can be done, but you have to get involved in some quite complicated geekery in order to do it, involving downloading plug-ins and things like that. It won't do it straight out of the box. So, even with probably the world's most widespread word processing system, if you're trying to open files that were produced using the version that was in use 15 to 20 years ago, you're going to have a struggle. In 20 years' time, a software manufacturer could be saying to itself, "Well, so few people now are still using mp3 that we're not going to push our production costs up by supporting this one." And yes, you could be in a position whereby, it's going to be very, very difficult to read back that software format.

Dominic - So, this sounds like it's problem not just for the media wanting to keep videos on file, but also for academics who want to keep documents that they were discussing 10, 15 years ago.

Leo - Yes, absolutely. I think archivists are going to have to keep their collections under absolute constant review and look at carrying out migration both in a hardware sense and in a software sense, as and when support issues really do start to emerge.

Dominic - So, what are companies doing to try and solve this problem?

Leo - There is no silver bullet. There is no widely accepted store and ignore format. In short, you've got a problem. And the moment you decide that you want to preserve this digital movie or this sound recording or whatever in the long term, you are making a long term commitment to manage actively the integrity of that digital asset.

Dominic - I guess this affects not only big companies who are making videos professionally, but also, people at home. If you're shooting a video of your wedding for example, what can you do to make sure that future generations of your family will be able to see that?

Leo - You basically just got to do a scaled version of what the professionals do. You need to take ownership of that material and decide that you are going to make a long term commitment to keeping it.

What I do personally for example is that I backup all of my data to a separate portable sets of hard drives every week. One of them then goes into work with me on Monday morning and sits on my desk drawer a week. So that if my flat burns down, then I've got a copy of my data in another physical location. And I'm looking at constantly backing up, constantly keeping an eye on the currency of the file format that they are encoded in.

And to be honest, it's partly because I do worry about my ability to do that, that I still take most of my personal photographs on a Leica M3 film camera that I inherited from my grandfather because I know that with a 35mm film negative, yes, obviously, what I'll do is scan those negatives so I can email the photos to people and that sort of thing. But ultimately, if I suffer a complete data loss disaster, I can always scan those films again if the original is a born digital file and all the copies of that, I have go well then that's the end.

Dominic - That's the technology that's being used for over 100 years, so I guess you can be fairly sure that it will probably still be in use in decades to come. I guess this has implications for how historians will look back on the 21^st century. Some people have said that they'll be overwhelmed with the amount of stuff that we record, but you're actually hinting that they will have a problem of not being able to read out records.

Leo - I think there will be that problem as well and furthermore, there's even a phrase for it which is kind of started to do the rounds in the archival community and that is the Digital Dark Age. And it's being speculated that just as we have lost, it is estimated between 2/3 and 3 quarters of the film shot during the first quarter of the 20^th century. So, we might end up in a situation whereby, we've got this kind of black hole from say, the first 1 to 3 decades of the digital era where in fact, very little survives.