Back-Up Project

Cori · Post by **Cori** » July 26th, 2007, 12:15 am

While talking of backups ... it has been mentioned that ibiblio and gutenberg both hold some files.

Perhaps it might be incredibly useful, though potentially tedious, to check what files of ours they do hold? This has the following benefits:

* Gutenberg hold a variety of formats. (Not sure about ibiblio.)
* Both are high-profile storage servers, with mirrors.
* Knowing where the gaps are will make it easier for Josh to submit our complete collection to PG (or train up someone else to help him, possibly..?) I don't know about ibiblio submissions, but it'd be very handy for missing books to be added over there.

I don't mean to cast gloom on this effort, Alex, as it seems very handy to have a snapshot of how things are now ... but I don't think it's scaleable in the long-term, for the reasons that Jim mentions about size, and also because it requires someone to upload every new release manually. That's fine for a few months when everyone's interested, but again, long-term ... not sure.

Finally ... 300GB is a LOT of stuff. Do you have a download limit..? Or is this pure emergency storage, and the only access should be in case of problems at archive..?

Post by **hugh** » July 26th, 2007, 4:48 am

a couple of points:

-i think a complete back-up is a very good idea: while I have faith in archive.org... our files regularly get corrupted/chop in the upload/derivation process. and who knows what could happen. much safer to have all files kept somewhere else.

-i suggest ed use back-up as just that: safety for us in case we lose some files - so there would be lots of upload bandwidth needs, but barely any download bandwidth.

-gutenberg adds stuff ad-hoc, so they are not really a reliable back up (unless we formalize the system with them - which so far has not happened, and does not seem likely, as their process is just a guy adding files).

-ibiblio theoretically should be our back-up, but they have not functioned reliably for a while, with a number of problems ranging from accessibility of the files to disappearing files.

-for a back-up system to work as a back-up system, it has to be sysetmatic and reliable.

-a "systematic and reliable" system suggests a script that automatically goes thru the entire catalog, downloads and stores; and then does the same for all new releases...ad-hoc humans won't do the trick.

-a "systematic and reliable" system could be set up using elbow grease and human effort, but I'm not sure what it would look like.

-for a back up there is no need to have 128 files, 128 zip, 64 files, 64 zip ...all we need is the 128 files (or maybe even the 64 files). the zips and different formats provide choice to the end-user, but in the case of back-up we just want best quality files in case we need to replace.

ab2525 · Post by **ab2525** » July 26th, 2007, 6:21 am

I have 50TB of monthly bandwidth. I'm also working on finding an automated script, but in the mean time, I think ad-hoc humans are better than nothing at all

ab2525 · Post by **ab2525** » July 26th, 2007, 6:24 am

Could someone help me write an rsync command that would only download .mp3 files?

ductapeguy · Post by **ductapeguy** » July 26th, 2007, 6:54 am

You might want to try wget too . I know there is a fairly large community of wget scripters online.

ab2525 · Post by **ab2525** » July 26th, 2007, 7:09 am

I'm going to use wget, that runs weekly in a cron job

Post by **hugh** » July 26th, 2007, 7:30 am

I'm also working on finding an automated script, but in the mean time, I think ad-hoc humans are better than nothing at all

well, i'm not sure: if we figure out a sensible automated system, then whatever effort put in beforehand will be duplicated; if we *don't* figure out an automated system, then having a handful of backups might be good - but probably not much better than the status now (where generally BC's keep their projects for a period of time).

mind you, if we can come up with a comprehensive human-based system that makes sense and is workable ... that would be useful.

so i support the idea and the effort, i just think we need to think about the system more to get it to work properly.

ab2525 · Post by **ab2525** » July 26th, 2007, 7:38 am

Maybe different people can be responsible for backing up each letter of the catalog, so there is an a person, b person, C person, etc. As for now, if my calculations are correct, than if we upload the 128 KB zips, I can hold 4 times the size that the catalog is now before I hit my limit

ab2525 · Post by **ab2525** » July 26th, 2007, 7:39 am

Once everything we have is backed up, I can probably maintain the backups myself, as the weekly turnout is around 12

Kenny · Post by **Kenny** » July 26th, 2007, 8:03 am

I can upload 600GB files on at least several places including one or more server at home. The more, the better, right?

Gothoborgensis · Post by **Gothoborgensis** » August 14th, 2008, 12:38 pm

Could there not be an interest from some official institution to keep a backup of files like the ones at librivox? Library of Congress for example, or any big city library in England or USA... I dont totaly know the US but in Western Europe i think all states and regions have public archives dedicated to preserve written and other documentation of our and older times.

Shipley · Post by **Shipley** » May 21st, 2009, 6:19 pm

Has any thought been given to backing up files on DVD-R's? At 4.7 GB per disc, we would need no more than a couple of hundred for all existing files, and they are easy enough to duplicate so that you could keep multiple copies in widely spaced locations. Alternatively, you can now keep about 7 GB in a gmail account - have a hundred volunteers sign up for an extra Gmail account each, and upload the files via Gspace or some similar Mozilla add-on. I keep several hundred MB of backup files for a church and home business using Gmail in this manner.

Mariane · Post by **Mariane** » September 4th, 2009, 1:37 pm

You can also get one terabyte external hard disk for
about 200 euros now. This is the best backup solution
as it can be put online (while DVDs cannot unless they
are kept in the drive).

You need someone with some apache experience, though,
it is not trivial.

Maybe Cambridge University would be willing to store
the data? They are working on HTK so it would be useful
for them too.

Mariane