Project recoding?

Non-reading activities need your help too!
Post Reply
ProDigit
Posts: 10
Joined: May 12th, 2018, 5:57 pm

Post by ProDigit » May 17th, 2018, 10:51 am

Hello,

I don't know if there's any interest in recoding the content of the site for smaller file formats?

I'm making personal backups for my Christian home library, into a Windows file format.
While doing so, I stumbled upon this website, and wanted to add as many files to my personal library as possible, as I just love to listen to them while on the road.

So my second question, aside from interest in recoding the library, is if there is a way I could perhaps receive a blueray disc, or USB stick, with all the files already on it, shipped to my mailbox, rather than having the tedious job of downloading all files?
Not only would it save me time, but it would save you bandwidth as well (of me downloading)

If so, I would return the favor by converting each format, and ship return the item (USB stick, or burn on one or two discs) with all those converted files.

I wanted to write down the results of my research here, just in case someone wants to do the same.
Hoping that I can join in on the whole Librivox project!
My program of choice is Fre:ac. It includes a WMA encoder for free, fully legal.
The free program is able to use 1 CPU core to process audio, however you can start as many programs as you have cores (in my case, I have 4 CPU cores, and often am using 2-3 instances of Fre:ac at a time doing different projects).

My encoder of choice, is the Windows version of MP3, named 'WMA' (or Windows Media Audio).
Unlike MP3, which was invented around 1996, and hasn't seen any major audio quality improvements over the years, other lossy formats like OGG, AAC, MP4, and WMA have succeeded them.
- OGG is mainly focused towards high quality audio, and definitely the format of choice for reconverting any audio files that are of CD-like quality.
- For years M4A/MP4/AAC was the audio format of choice for both low and high bitrate files. Derived from the Sony Discman, AAC and the other similar formats were mostly compatible with Apple devices, and later Windows too. These formats are vastly superior to MP3 in every way. And they not only sound good in the high bitrates, but unlike OGG, they also sound good in the low bitrates.
Plus, they've gained some popularity in the past few years, with media players as well (especially since it's the encoder of choice for Youtube audio and video (MP4)).
- WMA, for years was just in between. Not compatible with anything but Windows, better than MP3, but less good compared to the rest. However, Microsoft was one of the last ones to research in the format, and came out with 2 successors to the original WMA: WMA 9.2, and later WMA 10.
WMA 9.2 introduced variable bitrate encodings, and WMA 10 had expanded on WMA 9 by offering even better compression, quicker encoding, and more options; all while being backward compatible with WMA 9.

In most cases, compressing audio files to WMA makes little sense. I mean, most people have only a few hundred or less audio CDs, and storage is cheap.
Some users prefer the Lossless encodings like FLAC, or even WAV.
However, for large databases, and musicians (people who have vast libraries of music files), even a reduction of 10% in filesize, could result in thousands of megabytes or gigabytes of storage gained; which can help reduce the overhead cost on storage, or in the case of Librivox, network bandwidth.
Websites can actually choose to upload higher quality books while keeping the same file size, or upload the same quality books, at a fraction of the necessary storage space.
Remote areas, where internet is scarce and still running on shared networks (with eg: speeds of less than 10kB per user; you might be surprised, they still exist out there); might also benefit of these space saving formats!

My reason for choosing this WMA 9.2/WMA 10 Pro format, is because it became vastly superior in low bitrate encodings, than any other codec (quality per MB), including MP3, OGG, MP4, AAC, or any other extension, perhaps mainly because of an artifact called 'crystalizer', a way to reconstruct higher frequencies lost in the encoding process.
That and, WMA has become the current number 2 in compatibility nowadays.
Currently more devices support WMA, than M4A/AAC or OGG (which comes dead last) combined together; and while nothing beats MP3 in compatibility, some surveys suggest that the Windows media decoder is installed in about 75% of all audio capable playback devices (vs 99% MP3); most of those devices not including WMA are apple devices, or custom devices that play back their own proprietary formats.

WMA can be decoded (is playable) for free on Android, apple, and windows media players, most modern Car entertainment systems of 2013 or above (and some devices play this back as early as 2003-2004), and on most smart TVs.
WMA ENcoding (which is making the file; and not talking about decoding, which is the playing back of the file, and free everywhere) is mostly free on Windows, and requires a license on most other platforms; most of the time in the form of using a paid program to do the encoding (though there are free programs on every platform, if you look hard enough).

WMA is a lossy codec, which means that it gets rid of audio data, like MP3, but can do this much more efficiently.
WMA files are on average 25-50% smaller than most MP3 files of the same quality, 0-15% smaller than OGG on low bitrates, and 5-10% smaller than M4A/AAC.
WMA files shine at bitrates of below 128kbits, and can go as low as 4 to 5kbits (although I wonder what application there is for such low bitrates).
WMA has expanded and integrated bits of AAC, in it's own encoding, and is currently considered the best codec for low bitrates.

There are 3 main types of audiobooks I encode:
1- Low bitrate MONO recordings (usually <32kbits, <32kHz); these are more like 'web streaming' audio files
2- Low bitrate STEREO recordings (usually 32-128kbits 32-44kHz); these are mostly audio books
3- HIGH bitrate STEREO recordings (usually >128kbits 44kHz or higher); these are mostly with musical pieces included.

In 2 and 3, I generally encode in WMA 10 Professional, VBR encoding 0.
In case 3, the average bitrate ends up being ~64kbits, 32-44kHz, comparable in quality, to 160-192kbits MP3 files, but saving 60-70% of space in the process.
Since it's a variable Bitrate, there is no exact bitrate, but it appears to me that most of the files hover around 64kbits; and can differ as much as 10%.

In case 2, WMA VBR Q0 automatically selects the right settings. At times, it would encode a 32kHz file to 44.1, simply because there's no real file size penalty (perhaps 1kB per MB, or a magnitude of well below 1%).

In case 1, depends on the demand, but I usually try WMA 10 VBR Q0, though occasionally, when the source file quality is too low, I can choose WMA 9.2, which allows me to finetune below 32kHz 32kBits files; usually only done if the source file is a 22.050Hz, MONO file or below.
WMA 10 would actually not be able to detect lower bitrates than 32kbits, so chances are that the new encoded filesize can end up being larger than the source.
In this case, WMA 9.2 comes to play.
The pros of WMA 9.2?
It allows for lower bitrates than WMA 10.
The cons?
No VBR at these low bitrates.


For low bitrate encodings (like in WMA 9.2), I've noticed it being very hard to hear a difference between the original file and the new file using a bitrate equal to the sample frequency.
eg, for MONO low bitrate files, 44.1kHz needs at least 44kbits, 32khz audio needs at least 32kbits, 22.050Hz audio needs at least 22kbits bitrates;
At these numbers, there is a slight audible difference noticed in the high frequencies, when comparing it with the original.
Comparatively, you can hear a difference, but you'd not be able to distinguish you're listening to the low bitrate compressed file when not A/B-ing the files; meaning, the difference is only heard side by side.

Once bitrate goes below the sample frequency (eg: 32kHz, 24kBits, or 44.1kHz 32kBits), sizzling is heard quite clearly to the trained ear.
Also, this applies only for MONO files.
Stereo files need 1 step up on the bitrate;
Even recoding at WMA 10 32kbits/32kHz stereo will result in sizzling.
eg:
22.050Hz stereo files need ~32kBits, 32kHz stereo files need 40-48kBits, 44.1kHz files need 48kBits, and files with 48kHz or higher sampling data, are essentially the same as 44.1kHz, as low bitrate WMA cuts off the frequencies to about 20kHz anyway (40kHz sampling data).
Quite often it doesn't benefit encoding an audio book of
22kHz in a bitrate higher than 32kbits,
32kHz in a bitrate higher than 48kbits,
44kHz in a bitrate higher than 64kbits,
As the Windows encoder does a good enough job, and increasing the bitrate will only create larger files, for a near to inaudible difference in quality.

The variable bitrate part of WMA 10 pro, gains most benefit for spoken (read) audio, as it saves bits on the quiet pieces (in between words, line breaks, chapter breaks..), that it can give to pieces with lots of audio data (spoken word).
Unlike CBR (Constant Bit Rate), where even during silences, the same amount of bytes is allocated per second, regardless of audio data.
When using CBR in an encoding of low bitrate, bits are assigned where they're not needed, and are not assigned to areas where they are needed.
This often results in the higher frequency sizzling or inconsistencies, as high frequencies often require more data than lower frequencies.

Audio books are much more dynamic than music, in that there is more silences than in a piece of music or song.
This means it's best to use VBR where possible.

Audio book encodings using WMA 10 pro VBR Q0, mostly result in ~48kbits bitrates
Audiobooks with background music most of the time, end up averaging around 64kbits in encoding bitrates.

Mono Audio books of 22kHz, is best encoded in WMA 9.2 at 22kbits

I have no experience I can share for bitrates lower than 22kbits.
Last edited by ProDigit on May 18th, 2018, 12:21 am, edited 3 times in total.

TriciaG
LibriVox Admin Team
Posts: 39656
Joined: June 15th, 2008, 10:30 pm
Location: Toronto, ON (but Minnesotan to age 32)

Post by TriciaG » May 17th, 2018, 6:22 pm

I didn't read your quoted portion, but wanted to comment on the suggestion to receive files on hard media rather than downloading.

All the files, from first uploading for proof-listening to the final cataloging, is done by a lot of different people, and we all collaborate online. The in-progress files and the final, cataloged files are on web servers. There is no physical place where we could copy the files from a hard drive to a stick or disc, unless someone took a trip to Archive.org's servers (I'm not sure if they're all in California, or if they have other server locations), figured out which server the files were on, and copied them on site. But that's not realistic. :)

So no, we wouldn't be able to ship a physical item with the audio on it.

Availle
LibriVox Admin Team
Posts: 15562
Joined: August 1st, 2009, 11:30 pm
Contact:

Post by Availle » May 17th, 2018, 8:34 pm

Hi there and welcome!

Umm... I don't think you are realising the magnitude of the task you have set yourself! :shock:

We now have about 12.000 finished projects, with, say, on average 25 sections each. That's 300.000 files.

If I estimate that each of these files has a runtime of 30 minutes, then we're at 150.000 hours of listening time, that's 6250 days or 17 years and 45 days. Nonstop! :D
I doubt that there will ever be a USB stick holding that much data. :wink:

Besides that, what Tricia said above: Our files are stored at archive.org, and we do not have physical access there. So, to send you the files, somebody would have to manually download them via a potentially slow internet connection, just as you would.
I appreciate that you want to give something back, but again, we do not have physical access to the servers, so somebody would have to manually upload all of the files you're sending us, which is not feasible given the sheer numbers. Also, as a worldwide online-only volunteer organisation, we don't have an "office" where you could send your hardware to.

I'm not saying what you're suggesting is not worthwhile, I am sure there are people who would appreciate really small audio files! But from the sheer size of it, I don't think this is something that should be attempted manually on either side.

Archive.org does provide us automatically with a few more file options: We upload 128kb mp3 files and archive automatically generates .ogg files and 64kb mp3 files from them. There is also a torrent link on each project, as well as an m4b, which might make your task a bit easier, should you wish to proceed with it. :thumbs:
Cheers,
Ava.

--
AvailleAudio.com

ProDigit
Posts: 10
Joined: May 12th, 2018, 5:57 pm

Post by ProDigit » May 17th, 2018, 11:52 pm

I understand.
I thought someone had the files on a server, and had physical access to the server.
Does LibriVox rent storage space with archive?

Would it be legal to download, convert, sort or archive them, and share them for free with other people?

My estimation is that one blueray disc of 25GB with WMA encoded files can hold about 800-900 hours of recordings; a 128GB USB stick would hold about 4000-4500 hours.

Not all recordings are worthwhile, and of some audiobooks there are doubles or triples, where I personally would convert only 1 version of.

Is there a torrent link, with a pack of MP3s?
Last edited by ProDigit on May 18th, 2018, 12:14 am, edited 1 time in total.

annise
LibriVox Admin Team
Posts: 30010
Joined: April 3rd, 2008, 3:55 am
Location: Melbourne,Australia

Post by annise » May 18th, 2018, 12:03 am

It's legal as far as LV is concerned , but all our files are held at Internet Archive , and I don't know if they have any restrictions on bulk downloading, you could read their FAQs and if you need further help ask in their forum if you wanted.
Many of people distribute our files in different ways and formats, we are creators not distributers and we neither support or condemn the distribution

Anne

keeeto
Posts: 4
Joined: May 15th, 2018, 9:56 am

Post by keeeto » May 24th, 2018, 9:54 am

Unlike ogg and mp3 (now that the patents have expired), I don't think wma is an open format. Maybe that doesn't matter though...

Post Reply