A Modest Proposal

[harvey]

LibriVox doesn't have hardware decent enough to support on-the-fly conversion.

Yes, I understand what I'm asking for (:-)

(How) is it known that the server at iBiblio is inadequate for dynamic conversion?
(A separate matter from the question of would the proprietors there permit it.)

[marlodianne]

Well, here's the thing, the smaller you go, the more the sound will deteriorate. That's inevitable given mp3 formats. There are those who won't ever use less than 320, so we're never going to please everyone, quality wise.

Given speech and the formats we currently have, I don't notice much of a sacrifice from 128 to 64. There is some sacrifice for 64 to 32, and there's even more if you go on to 16. All of this is worse of course, if you don't use VBR.

But, for many people on narrowband, the choice is low quality or nothing. If they can't get a small version, they can't get anything. I know many of them would choose what was admittedly advertised as a crappy quality file, rather than not have the experience at all.

[HerrSchildkroete]

LibriVox doesn't have hardware decent enough to support on-the-fly conversion.

Yes, I understand what I'm asking for (:-)

(How) is it known that the server at iBiblio is inadequate for dynamic conversion?
(A separate matter from the question of would the proprietors there permit it.)

In fact I wrote: LibriVox doesn't have the $s to pay for hardware decent enough to support on-the-fly conversion. Please quote correctly.
The implication is: If we want a high-profile web-application with on the fly encoding, we very likely will have to pay for it. Of course I can't prove it, but the odds are this won't be free.

Regarding iBiblio: It is possible that their servers are powerful. To assess the possibility of dynamic conversion it would be necessary to know how many hits these servers get, spread across every site that is hosted on them.
Judging by the browsing experience, however, the servers are already close to their limit without lots of dynamic contents - leave alone dynamic audio processing.

[harvey]

Kri wrote (on automated conversion):
>Do you know anyone that can create the software to do this?

I've done this type of programming (NB - disclaimer: this is not to be
contrued as my volunteering for this project >:-)
So it strikes me as a simpler task than the LibriVox catalog database
project, but there are always the inevitable "gotchas" once one gets
started.


>Also, this may put a big load on the ibiblio computers. They may not
>agree to let us put such a system hog in their system.

Yes, it may. No, they may not.

And then again, they may be interested in our doing it if we offer to
share the technology with them, for they may view it as a desirable
improvement to their operation.


>It's a great idea, but I don't think we'll be able to do it.

*more indistinct grousing, again, something about "water-soaked bed coverings"*

Thank you, I like the idea, too. One generally doesn't get big things
one hasn't asked for (:-)

[harvey]

LibriVox doesn't have hardware decent enough to support on-the-fly conversion.

In fact I wrote: LibriVox doesn't have the $s to pay for hardware decent enough to support on-the-fly conversion. Please quote correctly.
The implication is: If we want a high-profile web-application with on the fly encoding, we very likely will have to pay for it. Of course I can't prove it, but the odds are this won't be free.

My sincere apologies: I did not intend to misquote you or distort
what you said. For brevity, I shortened your statement to what I take
to be its essential point, the absence of a computer with sufficient
horse power to handle dynamic conversion.

The Web application for dynamic conversion is a new and separate issue.
Is LibriVox paying for the catalog database or for ReVoxer? I understand
these to be volunteer programming projects.

[HerrSchildkroete]

My sincere apologies: I did not intend to misquote you or distort
what you said. For brevity, I shortened your statement to what I take
to be its essential point, the absence of a computer with sufficient
horse power to handle dynamic conversion.

Thanks. My point acutally was that right now the resources LibriVox uses on anyone's servers (ibiblio, nyip) are neglectible and thus are provided for free. Dynamic conversion however is not neglectible anymore and won't be in the near future. You are right, in the far future it probably will be, but right now it just isn't.

On a side note: To me this also a question of what LibriVox means and what the philosophy is and I am certainly advocating what I personally like about the project: It is technology in use, but the technology itself is nothing more than a tool. In my opinion we are not the group to advance technology by pushing the limits. That's something that should be done elsewhere. But again, this is just my personal opinion.

The Web application for dynamic conversion is a new and separate issue.
Is LibriVox paying for the catalog database or for ReVoxer? I understand
these to be volunteer programming projects.

I wasn't actually thinking about the webapp's development but only about the deployment cost like hardware requirements and about the licensing issues.
As to these other projects: The Database in its core form seems realizable and is probably really needed (guess we need to ask the MCs about that). Revoxer and on-the-fly conversion are very ambitious projects, but (at least to me) in the "nice-to-have" category.

[kri]

Given the near-miraculous increase in computer power and software
functionality, I believe the greater error will be for LibriVox to aim
too low.

The problem isn't the aim. We would need to find a properly skilled programmer to create this software that would do all this converting...possible but may take time. The deciding factor, however, would be whether or not Ibiblio would permitt this on their system. Whether WE see it as a system hog or not isn't the issue. It doesn't matter how advanced the hardware becomes, if Ibiblio doesn't have it it's of no use to us.

Also, this is all assuming that we definitely decide to switch to Ibiblio as well. I think we're still testing it out.

[Yakumo]

regarding the ideal bitrate for encoding, I would say that 48bps is perfect. I have exprimented with this as well, and at that bitrate, I do not here any distortion that comes from compression, provided that a good encoder is used.

Automated file insertion was considered cpu intensive a few years ago, ie. 2001, I do not know how it is now, but I am guessing a computer would have to be prety powerful to insert tags at the begining and end of an enormous amount of files.

These types of aplications are liscensed by seriou companies with cash! google MUIFS and see what I mean.

Figuring out an efficient way to make files bookmarkable for the masses would be nice though, The idea for bookmarking is a good one, I would go for it if it was implemented.

I like the idea, but at the same time, I realize things like this will come about gradually, once the comunity finds an efficient way to work it out.

[harvey]

The difficulty with dynamic audio file conversion

As a professional programmer and Web site developer, I've programmed
CGI scripts from scratch that did sophisticated data access. And I've
done the same with modern Web-specific programming systems. Also,
I've had a number of jobs which were on the cutting edge of technology.

Unless there is something significant I've failed to recognize, I don't
see that the dynamic production of customized audio file copies (ie,
format and bit rate selected by the user) of LibriVox recordings is
either cutting-edge or particularly difficult. It strikes me as a
reasonably straightforward programming task. Consider that LibriVox
volunteers are already doing this conversion by hand. They start with
a Wave file and then use software to convert it to MP3 format. I use
the freeware program RazorLame, which is a GUI front-end for the LAME
encoder. RazorLame calls the command-line version of LAME to do the
actual encoding work, based on parameters I specify in RazorLame.
There are several other alternative programs which do the same thing.
I see no great difficulty in doing this same task in a Web-based application.

[kri]

The difficulty with dynamic audio file conversion

As a professional programmer and Web site developer, I've programmed
CGI scripts from scratch that did sophisticated data access. And I've
done the same with modern Web-specific programming systems. Also,
I've had a number of jobs which were on the cutting edge of technology.

Unless there is something significant I've failed to recognize, I don't
see that the dynamic production of customized audio file copies (ie,
format and bit rate selected by the user) of LibriVox recordings is
either cutting-edge or particularly difficult. It strikes me as a
reasonably straightforward programming task. Consider that LibriVox
volunteers are already doing this conversion by hand. They start with
a Wave file and then use software to convert it to MP3 format. I use
the freeware program RazorLame, which is a GUI front-end for the LAME
encoder. RazorLame calls the command-line version of LAME to do the
actual encoding work, based on parameters I specify in RazorLame.
There are several other alternative programs which do the same thing.
I see no great difficulty in doing this same task in a Web-based application.

Harvey, I believe that those in charge at Ibiblio said they would not let us create and use a file deriver like the one we use at Archive.org (created by us of course) because of its use of CPU cycle. If we switch to Ibiblio, we'll have to be manually creating our files as it is.

If we switch to Ibiblio, it's probably not going to happen.

[harvey]

I believe those in charge at Ibiblio said they would not let us
create and use a file deriver like the one we use at Archive.org. If
we switch to Ibiblio, we'll have to be manually creating our files as
it is. If we switch to Ibiblio, it's probably not going to happen.

Yes, kri, I acknowledge the validity of what you say. I also don't see
it's relevance to the issue of the technical difficulty of implementing
dynamic audio file conversion on a Web site. Even if we aren't allowed
to do it at iBiblio, it's still of interest to the future of LibriVox and so,
there's benefit in discussing it.

It seems that you are objecting to this discussion on the basis that
we would not be allowed to implement a converter at iBiblio. Do I
misunderstand? If not, then I disagree with you.

[JickBahTech]

So this thread has kinda split a little huh...
For both "splits", I really just want to say "128Kbps MP3 is just fine by me".

As for making audio books available, I think its a great idea right now to post the instructions somewhere, and let those who are really interested do it themselves (or with forum help on a one-by-one basis). Maintaining yet another audio format seems a bit taxing, and where do you draw the line on formats (ogg is good, and I really like Speex, and why not also m4a for shorter stories so we can include "album art", heck why not some form of PCM for people that want to listen on older voice recorders, and even though Sony screwed it up ATRAC wasnt horrible, and...). Also I haven't heard anyone jumping at wanting to start this process. I have the kit at home, but i really dont want to volunteer my time for something that tedious. I'm much more interested in bolstering an "Editors Wanted" section. I own an ipod, but so far I haven't really been inconvenienced by playing the chapters individually (kind of how I read books anyhow), and if I really need book marking, well, now I know how to make an audio book. I think its a really cool idea, but right now it sounds like more an idea for individual users than the site as a whole.

As for on the fly conversion, doesn't this mean we'll need to put up a fairly high quality version to start anyway (and server space is still a concern right)? Even on 56K, downloading a collection of files between 5 and 10MB (at 64Kbps) , while not the most convenient, is certainly not a painful experience (we all did it during the first Napster days ). Again I don't really see anyone leaping to build this function (even you, Harvey want to pass this buck on). Maybe if someone wants to build their own off site (a site where you feed it a link to a story and it does the conversion and spits out a file for you), but I dont really see how it enhances the Librivox experience.

Love it or hate it, but MP3 @ 128Kbps has won this round, and for maximum compatibility, its the best space to play in. Maybe when a new audio format rules the roost and Librivox 2.0 is being built, and...

[kayray]

For the record, we're not allowed to do anything CPU-intensive on ibiblio. Also, the MCs' heads will explode if we have to keep track of ONE MORE THING. We're looking forward to ditching the 128s (probably) and dealing only with 64s, oggs, and zips (probably).

However -- our files are in the public domain! Feel free to take them and set them up on some other server that can convert them to whatever you like :)

[hugh]

hi folks just a comment on on-the-fly conversion: ibiblio will give us space but not much porcessing power: so no conversion allowed on those servers ... even once, and definitely not on-the-fly.

[tshirt]

hi folks just a comment on on-the-fly conversion: ibiblio will give us space but not much porcessing power: so no conversion allowed on those servers ... even once, and definitely not on-the-fly.

Why does the conversion have to be at the server side? A straightforward
browser application can arbitrarily downgrade the bitrate of an audio file.
- The servers will keep the highest quality compressed mp3
- Browser (applet/script) will download the high quality mp3 and downgrade to
a smaller file or convert to an alternative format (using user's resources) as
per requested by user.

The users will need to wait longer to download a larger file than they need.
But they can get customized bitrates/file formats with the click of a button.
This will enable less computer savvy users to use alternative file formats.
It will also save others from having to install converters on their computers.

Too bad if such applets don't already exist. They are absolutely doable.

BTW: ReVoxer project still needs volunteer contribution from knowledgeable
programmers. Please contact me if you have experinence in J2ME
programming and/or server side Java programming and/or speech and signal
processing applications.