torrents

Comments about LibriVox? Suggestions to improve things? News?
hugh
LibriVox Admin Team
Posts: 8006
Joined: September 26th, 2005, 4:14 am
Location: Montreal, QC
Contact:

Post by hugh » February 3rd, 2008, 3:39 pm

we're in the process of working out a bit torrenting thing with the guys at ibiblio.org - where they would have permanent seeds for torrents of our books.

for torrents to work, you need as many seeds as you can get to make it work properly ... and unfortunately our past experience with torrents have proven difficult, mainly because you tend to get a spike of seeders when a book first comes out, then none. torrents are really better for more popular media, beause the more people *want* and *have* a particular torrent at any given time, the better the whole system works.

anyway, I was just checking out some torrent sites, and was ... well ... flabbergasted to see that, at this precise moment anyway:
The Odyssey has 58 seeds
Tale of Two Cities has 40 seeds
Sense and Sensibility has 28 seeds

see:
http://www.mininova.org/search/librivox/seeds

hugh
LibriVox Admin Team
Posts: 8006
Joined: September 26th, 2005, 4:14 am
Location: Montreal, QC
Contact:

Post by hugh » February 3rd, 2008, 4:12 pm

quick & dirty explanation of torrents:
-you have a big media file on your hard drive
-your run it thru a torrent program (eg http://azureus.sourceforge.net/) to "make" a torrent (see further down)
-this "chops" up the big media file into discrete pieces, and the torrent file is like a key that lets someone get bits and pieces of the file and then put it all together in the right order
-then you make the media file (and the torrent file) available to anyone on the net (you "seed")
people all over the world download the torrent file, that lets them look for other people who are seeding the big media file.
-those people (leeches) download bits of the file from all the people who are seeding, taking bits of the file from whichever connection is fastest
-as they start download more of the file, they too become seeders, letting other people download from them, while they continue downloading the whole

so unless you have someone permanently making:
a) the media file available
and
b) the torrent file available

then the media file "disappears" from the system and is no longer available. so for projects like librivox, where there is a limited interest in our books (compared to, say, downloading The Simposons - illegally), seeing usually disappears quickly.

For a while we were adding torrents to:http://legaltorrents.com/ but we stopped when the volunteer who was doing it found that he could not keep up with seeding, and the books then disappeared.

but I don't know much else about the *actual* mechanics of torrents. but basically I guess you have to somehow set up a server to seed permanently.

here is the ibiblio project to do that (i think this is open source software you could use on your server, but i'm not sure).
http://osprey.ibiblio.org/

hugh
LibriVox Admin Team
Posts: 8006
Joined: September 26th, 2005, 4:14 am
Location: Montreal, QC
Contact:

Post by hugh » February 3rd, 2008, 4:44 pm

a good solution.

a nice project for someone would be to set up a network of torrenters who would each take on permanent seeding of a different portion of the catalog, thereby creating a nice distributed system, with all the inherent stability and safety factor that comes with it. ideally 2-3 permanent seeders per catalog portion, so that even if one got taken out, there would be redundancy in the system to ensure the long-term availability of the torrents & the catalog.

Starlite
Posts: 16694
Joined: April 30th, 2006, 2:17 pm
Location: Thunder Bay Ontario, Canada

Post by Starlite » February 3rd, 2008, 5:05 pm

There was more talk of torrents here:

http://librivox.org/forum/viewtopic.php?t=9826
"Reasonable people adapt themselves to the world. Unreasonable
people attempt to adapt the world to themselves. All progress,
therefore, depends on unreasonable people." George Bernard Shaw

Starlite
Posts: 16694
Joined: April 30th, 2006, 2:17 pm
Location: Thunder Bay Ontario, Canada

Post by Starlite » February 3rd, 2008, 7:50 pm

I did some torrenting early on but since my computer is not on 24/7, I gave it up. Also as it is Tax season, I am too busy to really give it a real go right now. I will watch the progress though.

Have you done a torrent search for "librivox"? There are a few still out there being seeded. If you d/l those using a torrenting client, you will become a seeder once your d/l is completed and you begin uploading. (at least that is how I understand it)

Esther
"Reasonable people adapt themselves to the world. Unreasonable
people attempt to adapt the world to themselves. All progress,
therefore, depends on unreasonable people." George Bernard Shaw

delibab
Posts: 227
Joined: August 3rd, 2007, 11:01 am
Location: Hungary

Post by delibab » February 4th, 2008, 4:22 am

If we are going to get more serious about Torrents, do we need to provide some info on hashes for the files?
- file/download integrity
- assurance (well, ahem - kind of) that it really is a Librivox recording/file
- Risk to reputation. Don't want something masquerading as a Librivox recording/file, that is something else, or riddled with viruses or spyware

On a different note - what about the relationship between Librivox and Wikisource.. Any?

Regards, DB
[color=green][b][u][url=http://www.openstreetmap.org]openstreetmap.org[/url][/u] - The Free Wiki World Map..[/b][/color]

NightOwl
Posts: 303
Joined: January 10th, 2008, 9:17 am
Location: Taiwan

Post by NightOwl » February 4th, 2008, 4:30 am

You are doing something really great, Robert!

Thanks!

HC

Starlite
Posts: 16694
Joined: April 30th, 2006, 2:17 pm
Location: Thunder Bay Ontario, Canada

Post by Starlite » February 4th, 2008, 4:41 am

delibab wrote: - Risk to reputation. Don't want something masquerading as a Librivox recording/file, that is something else, or riddled with viruses or spyware
Regards, DB
I doubt this can be avoided. All files should be tagged "Librivox" as that is how I would search for it as well as "audiobook".

I really don't know much about setting it up though - creating files and such. I just downloaded the existing torrents to seed.

Esther
"Reasonable people adapt themselves to the world. Unreasonable
people attempt to adapt the world to themselves. All progress,
therefore, depends on unreasonable people." George Bernard Shaw

Cori
LibriVox Admin Team
Posts: 11950
Joined: November 22nd, 2005, 10:22 am
Location: Great Britain
Contact:

Post by Cori » February 4th, 2008, 11:54 am

I'd suggest having a look at http://en.wikipedia.org/wiki/BitTorrent_%28protocol%29 to get the tech background for this. Torrenting has a bit of jargon to get to grips with, but the background concepts are worth knowing too -- it's all pretty darn cool!

I would then go the opposite route to what you've said above, Robert, and recommend supporting existing (good) seeds. One user at PirateBay, for example, is called librivox, and provides excellent files in catalogued condition, with disclaimers and labelled "librivox" in the torrent-title and information page. (I've downloaded and listened to some of our books that way, as well as seeding on myself.) This boosts the number of people seeding and leeching the same (good) files, rather than splitting that number between two or more seeders (which makes it harder for people to download, and also proportionately reduces the credibility of the torrent. Given a choice between getting an identically-named-something that 12,000 other people have downloaded, and one that 120 people have downloaded -- which would you pick..?)

Only create new torrents if you know a particular book is bad (and of the virusy things I've come across, no MP3 has ever been among them) -- or for new books / torrents that have died out, etc.

Hashes are a security and validity measure, and different ones are generated by all sorts of processes. The Torrent one is different to the archive.org one -- and there's a different hash for each Torrent, even of the same file. Hence Starlite's suggestion of downloading the torrent files first -- this provides you with the same hash other seeders are already using, which supports the overall effort.
There's honestly no such thing as a stupid question -- but I'm afraid I can't rule out giving a stupid answer : : To Posterity and Beyond!

Let's go on an April Adventure!

Post Reply