A Modest Proposal

[Pappy]

Many users of LibriVox will listen to their audiobooks on an mp3 player. Many of those mp3 players will be iPods. Ipods/Itunes have a neat feature known as bookmarking audio files. Basically, these types of audiobook files will remember where you were when you stop your playback, and will automatically start up where you left off when you next begin listening to the audiobook.

This feature is great; particularly for a portable mp3 player. In fact some would say its almost a necessity.

There is a procedure:

http://www.ipodlounge.com/forums/showthread.php?s=&postid=273806#post273806

which walks one through converting a group of mp3 files into a single .M4B file which has this bookmarkable feature (when used on iPods).

This procedure (in simple terms) consists of the following:

1) Merge all mp3 files into one mp3 file (there is freeware s/w to do this)
2) Use iTunes to convert the mp3 file into their AAC format (.M4A file)
3) Use Windows to rename file to .M4B extension (the referenced procedure also supports Apple as well)

Finally to the Modest Proposal --

While everyone interested could go through this procedure everytime they downloaded a series of LibriVox files, wouldn't it be nice if we provided a single converted M4B file that could be downloaded for each book offered. LibriVox could do the conversion once rather than 1000's of people each do the conversion for each book they download.

I realize there may be issues, so I have listed potential problems below:

1) The people generously providing a repository for our files may not like the idea of also storing this new format.
2) Possibly this M4B format is not in the "public domain" and we wouldn't have permission to use it. I think audible.com originally came up with the concept.
3) The effort involved in the conversion -- note: there is a $15 shareware program that will automate the whole conversion. I would also be willing to volunteer to assist in the one-time conversion.
4) This would benefit only a small(?) subset of the LibriVox users and therefore is not appropriate -- I really don't know how many people have iPods versus the other players out there, but I would guess that it is a significant percentage. It is also conceivable that other players do or will support this format -- I just don't know.

Proposal over -- let's discuss.

[hugh]

hi pappy,

yes would be good. we are as we speak in the process of setting up a new server space at iBiblio - which is much more flexible and easy to use than archive.org, and will give us more ability to do things like this if we can find the tech volunteers to help set it up. we should explore this - it is a nice option. some specific comments below:

1) The people generously providing a repository for our files may not like the idea of also storing this new format.

in our new set-up we are considering using just 64kbps mp3s, so that will make "space" for another format. though this does not seem to be a major probelm.

2) Possibly this M4B format is not in the "public domain" and we wouldn't have permission to use it. I think audible.com originally came up with the concept.

not sure, but most formats are effectively usable by anyone, even if there is a long-term legal issue.

3) The effort involved in the conversion -- note: there is a $15 shareware program that will automate the whole conversion. I would also be willing to volunteer to assist in the one-time conversion.

what is the software? does it do ogg & speex conversion as well?

4) This would benefit only a small(?) subset of the LibriVox users and therefore is not appropriate -- I really don't know how many people have iPods versus the other players out there, but I would guess that it is a significant percentage. It is also conceivable that other players do or will support this format -- I just don't know.

small subsets are our specialty at LibriVox. If it's good & we can do it, we should. problem is: time, resources, expertise etc.

[HerrSchildkroete]

Ipods/Itunes have a neat feature known as bookmarking audio files.

I absolutely agree - this is the reason why I opted for the iPod.

There is a procedure:
http://www.ipodlounge.com/forums/showthread.php?s=&postid=273806#post273806

I have done that and have run into trouble. Especially if the input files differ in some of their parameters it crashes and leaves you with incomplete files. For that reason, and since the process still requires many clicks, I have written a shell script to do the work for me. It is available with a brief explanation at http://www.fcengel.de/site/librivox/merge-m4b.htm . However it is a rough hack that works on my machine and probably nowhere else. However, since there seems to be an interest here, I have put it on my website.

While everyone interested could go through this procedure everytime they downloaded a series of LibriVox files, wouldn't it be nice if we provided a single converted M4B file that could be downloaded for each book offered.

That would really be nice - however (as said above) I found the process very error prone. So I didn't dare suggest anything like it. It would introduce a lot of trouble.

2) Possibly this M4B format is not in the "public domain" and we wouldn't have permission to use it. I think audible.com originally came up with the concept.

Neither is mp3. If you use iTunes apple grants you the right to create m4a and m4b files, so that is no problem.

4) This would benefit only a small(?) subset of the LibriVox users and therefore is not appropriate -- I really don't know how many people have iPods versus the other players out there, but I would guess that it is a significant percentage.

I do However since the conversion process is in my opinion non-trivial I suggest not to make these files "officially" available.

[Pappy]

The shareware is called MarkAble. Here is a link to a conversion tutorial using markAble which ends up with a bookmarkable file:

http://markabletutorial.blogspot.com/2006/03/markable-cd-wizard-tutorial-cd-to.html

It will work with a group of mp3 files as a starting point or audio cd's (not of interest to us).

One other potential issue is that if we have a single file for the entire book, then the start and end chapter statments become a llittle redundant. Not a big deal.

[kri]

I suggest not to make these files officially available as well, since the amount of people who want them will be small (I'm guessing anyways). However, we could set something up where people can request that volunteers convert them and send them to them. You can even use my server space to temporarily hold them until the requestor has downloaded them. It takes extra time and seems tedious, but if we get a lot of requests, that will be a sign to perhaps start having an "official" copy.

We could easily alert people to a request procedure of sorts in the Wiki. Explain that they should make their request in the Get Help forum. We have enough active volunteers that will likely see and help. I might even be willing to figure out how to do this file conversion and help when requested.

[harvey]

we are in the process of setting up a new server space at iBiblio - more
flexible and easy to use, and will give us more ability to do things like this.
In our new set-up we are considering using just 64kbps mp3s

Since these big changes are being considered, here are some more ideas.

Recently, I read that some on-line music service (I think it may be
allofmp3.com) stores its files in a lossless format. When a customer
wants a file, he specifies the format and bit rate he wants, and the server
creates it for him on-the-fly from the lossless original. That means only
one version of each file is stored.

I really like this idea because, as has been stated by others elsewhere in
these forums, I want the option of downloading MP3 files in bit rates lower
than 64, which is, based on my long experience with computer audio, higher
than is necessary for good speech reproduction. I have a narrow-band
connection to the Internet, and, consequently, file size is very significant.

Similarly, the server could dynamically bundle up all the files for a
work into a zipped file, when a visitor chooses that option. Again,
saving storage space.

Another possibility is to deal with the listener "annoyance" of having
to sit through -- or fast-forward past -- the credits and title at
both ends of every file in multi-part works. The idea is to have the
server handle this. Presumably, each part (presently in one file)
would exist as several files on the server. When an individual part
(eg, a chapter) is requested, the server combines the required files,
and what it sends to the listener is exactly what they presently get.
However, if the entire work is requested (perhaps in the form of a
zipped file), then only the first chapter file will contain the LibriVox
disclaimer and the title of the work. And it could be an option for
the listener whether the subsequent chapter files contain chapter
numbers and titles. Common items, such as the disclaimer, the title
of the work, and the reader credit for solo works, would exist as
single files for the entire work.

[kri]

we are in the process of setting up a new server space at iBiblio - more
flexible and easy to use, and will give us more ability to do things like this.
In our new set-up we are considering using just 64kbps mp3s

Since these big changes are being considered, here are some more ideas.

Recently, I read that some on-line music service (I think it may be
allofmp3.com) stores its files in a lossless format. When a customer
wants a file, he specifies the format and bit rate he wants, and the server
creates it for him on-the-fly from the lossless original. That means only
one version of each file is stored.

I really like this idea because, as has been stated by others elsewhere in
these forums, I want the option of downloading MP3 files in bit rates lower
than 64, which is, based on my long experience with computer audio, higher
than is necessary for good speech reproduction. I have a narrow-band
connection to the Internet, and, consequently, file size is very significant.

Do you know anyone that can create the software to do this? Also, this may put a big load on the ibiblio computers. They may not agree to let us put such a system hog in their system. It's a great idea, but I don't think we'll be able to do it.

[marlodianne]

On-the-fly conversion wouldn't just be a resource hog, the possibilty of corruption would be huge. The quality control could get unacceptably dodgy.

However, if we could offer 16 bit, or even 32 bit, encodes as a matter of course, that would make a lot of narrowband people very excited.

[kri]

16 might be too low quality, do you think? I remember Chip doing several versions of one file, and I didn't notice a significant difference in quality until 16....or was it 8? I don't remember.

[jimmowatt]

16 might be too low quality, do you think? I remember Chip doing several versions of one file, and I didn't notice a significant difference in quality until 16....or was it 8? I don't remember.

I've done similar tests for my own benefit and similarly noticed no difference until 16.
32 sounded absolutely fine but 16 is pretty horrible and then 8 is unlistenable.

[harvey]

On-the-fly conversion

From the start of the World Wide Web in the early 90s, Web servers
have supported SSI -- the server-side include mechanism, which is
often indicated by the .shtml extension. In those early days, there
was oft-expressed concern about the use of this feature, especially
enabling it for every file on a Web server, on the basis that it was a
resource hog. But because computer horse power was -- and still is --
increasing rapidly, this concern soon faded away.

The point of this short history lesson is two-fold. We should find
out for certain if dynamic conversion is indeed a resource hog and, if
so, to what degree. And, if so, recognize that it likely won't remain
so for very long.

That a commercial music is using dynamic conversion suggests two
things. (1) The trade off between decreased storage space and
increased CPU use is acceptable, if not desirable. (2) The occurence
of corruption due to automated conversion is low. If anyone knows of
specific contrary evidence, please let us know.

Given the near-miraculous increase in computer power and software
functionality, I believe the greater error will be for LibriVox to aim
too low.

[Pappy]

The AAC encoding used for the bookmarkable files is 32k so we gain some space by using this format.

The difficulty in performing the conversion should not be a deal-breaker. The point is: this conversion is only done once and then the new format can be downloaded by anybody. In fact, the difficulty in converting is a good reason for us to do it; since it would only be done once and by "trained professionals".

I'm not sure about how many people would want this ( i.e. how many iPod users). The iPod world is big though.

The only negatives to doing this are:

1) do we have the space for extra format
2) who will do the work for the one-time conversion of each book

Both of those seem solvable.

[HerrSchildkroete]

On-the-fly conversion
That a commercial music is using dynamic conversion suggests two
things. (1) The trade off between decreased storage space and
increased CPU use is acceptable, if not desirable.

That a commercial music store is offering these shows they are looking for something to put them above the other stores.
Please keep in mind that LibriVox doesn't have the $s to pay for hardware decent enough to support on-the-fly conversion. We're not talking about simple time-and-date insertion or efficient database queries but about computationally expensive tasks.

[HerrSchildkroete]

The only negatives to doing this are:

1) do we have the space for extra format
2) who will do the work for the one-time conversion of each book

I'd like to add to your list:
3) Do we want to bother the MCs with yet another file to collect and another bit to care about?

[harvey]

Bit rates

There are two inter-related issues here. One is the dividing line
between good and poor quality audio for speech recordings. The other
is what should be the lowest (and highest) bit rate made available to
listeners.

In agreement with what's been state above, my experience with a lot of
recording of speech from a variety of sources indicates that, in
general, the dividing line is 16 Kbps; that is, by this bit rate, one
starts to hear problems (although I wouldn't call them horrible -- but
that's partly a matter of aesthetic sensibility). And that, in
general, 32 Kbps is a good, all-round standard.

However, it's especially true for music that the "lowest" bit rate one
can get away with, before one starts to hear the ill effects of
compression, depends entirely on the individual musical recording.
Some music can be compressed much more than other before problems show
up. After starting to record for LibriVox, I discovered this to be
true also for speech recordings, but the effect is not so pronounced.

Two examples. (1) I regularly recorded speech in an auditorium for
the purpose of putting it on a Web site. I experimented with various
bit rates and settled on 24, because noticeable sound deterioration
occurred at 16 and I couldn't tell any difference between 24 and 32.
(Later I switched to 16 for the practical reason that the files are
1/3 smaller and this saves upload and download time. And, while the
audio quality is slightly poorer, most people don't notice it.)

(2) In contrast, for the relatively clean recordings I make for
LibriVox in my living room, 48 sounded good and I started hearing
deterioration at 32. For comparison, the LAME MP3 encoder has a
number of highly-tuned presets. Its preset for voice encodes at a
variable bit rate that averages about 56 (and mostly it automatically
down-samples from 44,100 to 32,000 Hz; I think it also converts stereo
recordings to mono).

The other of the two issues is what should be the bit rates LibriVox
makes available. Since I'm all for giving users -- listeners, in this
case -- maximum flexibility and choice, my vote is to give them the
choice of (nearly) anything that's technically possible. Since I'm
thinking in terms of automated conversion, the choices would be every
standard bit rate from 8 to 320 (adjusted at the upper end for
whatever makes sense in terms of the resolution of the lossless format
stored on the server). And there ought to be suitable arm-waving to
alert listeners to the trade-offs for smaller bit rates. Ideally, this would
include short example files at different bit rates.