Problem with the Librivox API

Post your questions & get help from friendly LibriVoxers
Locked
williamjones
Posts: 2248
Joined: April 26th, 2016, 7:47 pm
Location: Florida

Post by williamjones »

A PARABLE

There once was a wealthy citizen who had, over many years, amassed a library of books. Eventually he wanted to make a list of them with their titles and authors, etc.
One morning, he called in his laboratory wretch, Igor, and said, "Igor. Bring me all the books on my office shelf, a few at a time so that I can work with them."
Igor asked, "How many shall I bring at a time, Master?"
"Four," was the reply.
Drooling and limping down the hallway, Igor left and returned with the requisite number of books.
"OK, I'll call when I want more of the books."
And, soon enough, Igor was called upon to bring another group of four books.
And so it went until Igor brought only three books.
"That's all there were left on the shelf, Master."
"Wait until I've finished these last three, then re-shelve all of them, Igor."
"Yes, Master."

Time passes...

"Igor, come here! We've handled 23 books this morning. You've done well. Tell Cook to give you an extra ladle of swill in your food dish. "
"Thank you, Master! Thank you!"

More time passes...

"Dammit, Igor!! I've made a mistake. I need to re-do all the books. This time bring them to me SIX at a time."

And so, Igor once more paraded up and down the hall with armloads of books.
Eventually, Igor reported that there were no more books on the shelf.

"What?!?!" cried The Master. "That can't be! Twenty-three isn't evenly divisible by six. Let's start all over and bring me FIVE books at a time."
The process was repeated and when Igor reported that his last load had only 1 book in it. The master swatted Igor and yelled: "What the holy hell is going on Igor. Are you playing tricks on me?"

Igor was made to parade up and down the streets of the town with a sign hung about his neck:
I AM A FOOL.
My name is
LibriVox API.
.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.

This little story illustrates how non-dependable the Librivox API is. When I try to get *ALL* the 15,000+ books in the LV catalog, retrieving modest gulps at at a time, I get different totals depending on how many I retrieve at a time. I've tried retrieving 5, 50, 100, 200, 500, 1000 and 2000 books at a time. I get a different total number of books depending on the "gulp size". This is not logical. There is something amiss with the API.

Is there anyone out there who knows something I need to know...about the API, I mean :) ?
-- Bill Jones

When you think that you have exhausted all possibilities, remember this: you haven't.
--- Thomas Edison
annise
LibriVox Admin Team
Posts: 38572
Joined: April 3rd, 2008, 3:55 am
Location: Melbourne,Australia

Post by annise »

It would save a lot of time if you would search before you asked however the official and correct answer is
The LibriVox API is available at https://librivox.org/api/info.
When we have sufficient funds it will be further developed but at the moment I have no idea when that will happen

There is a discussion thread in our forum with some unofficial solutions here viewtopic.php?f=24&t=44129 you are welcome to register and join the discussion if you wish

Hope this is some help
Anne
williamjones
Posts: 2248
Joined: April 26th, 2016, 7:47 pm
Location: Florida

Post by williamjones »

annise wrote: March 24th, 2021, 5:01 am It would save a lot of time if you would search before you asked however the official and correct answer is
The LibriVox API is available at https://librivox.org/api/info.
When we have sufficient funds it will be further developed but at the moment I have no idea when that will happen

There is a discussion thread in our forum with some unofficial solutions here viewtopic.php?f=24&t=44129 you are welcome to register and join the discussion if you wish

Hope this is some help
Anne
Thank you for your suggestions.
I *have* thoroughly looked over these threads and, additionally, other commentary on Github.
I am a 79-year-old man who has made his living since 1964 as a programmer, analyst, database designer and professor of computer science. In other words: I know my stuff. What I don't know are the internals of the LV API. What *would* help is adequate documentation. I know, I know: not available.

I've heard the story many times about a lack of money to fund further development/correction. Well, we could do what elementary schools do when they want to fund a new piece of equipment or something: Have a Bake Sale. I'm willing to chip in $500 to get the ball rolling if someone can demonstrate that they know WHO can and will make fixes, or create documentation and will start a fund. What about you? Are you that person?
-- Bill Jones

When you think that you have exhausted all possibilities, remember this: you haven't.
--- Thomas Edison
annise
LibriVox Admin Team
Posts: 38572
Joined: April 3rd, 2008, 3:55 am
Location: Melbourne,Australia

Post by annise »

No one here is. As you would have noticed in the API thread
My view - Librivox operates amazingly well sticking to what we do well - which is make audiobooks. We don't distribute them. And while we stick to what we do we will survive. YEs it would be nice sometime
(It doesn't apply to you I assume, but most people wanting to use the API are planning to make money using it and they legally can. But it is not part of our prime directive. And I do get the emails telling us that they need everything including reviews so they can make them)

Anne
williamjones
Posts: 2248
Joined: April 26th, 2016, 7:47 pm
Location: Florida

Post by williamjones »

I've attended group sessions whose attendants were proud of ripping off the LV data so as to make RSS feeds and "books" and such. While maybe not illegal, I see it as deplorable.

Thanks, again, for your replies.
-- Bill Jones

When you think that you have exhausted all possibilities, remember this: you haven't.
--- Thomas Edison
annise
LibriVox Admin Team
Posts: 38572
Joined: April 3rd, 2008, 3:55 am
Location: Melbourne,Australia

Post by annise »

If you just read the last couple of pages of the API thread there is mention of an API at Archive - I don't know anything about the details but you could try PMing the poster for details
ChristopherW
Posts: 33
Joined: August 4th, 2009, 1:29 pm

Post by ChristopherW »

For what it's worth, I managed to get exactly 15882 records regardless of "gulp size" using a script that iterates over limit and offset:

Code: Select all

https://librivox.org/api/feed/audiobooks/?offset=${offset}&limit=${limit}&fields={id}
When limit is 500, offset is 0, 500, 1000, ..., 15000, 15500. When limit is 2000, offset is 0, 2000, ..., 12000, 14000.

Maybe you get a different number of results when you ask for more than just the record IDs?
williamjones
Posts: 2248
Joined: April 26th, 2016, 7:47 pm
Location: Florida

Post by williamjones »

ChristopherW wrote: March 24th, 2021, 5:39 pm For what it's worth, I managed to get exactly 15882 records regardless of "gulp size" using a script that iterates over limit and offset:

Code: Select all

https://librivox.org/api/feed/audiobooks/?offset=${offset}&limit=${limit}&fields={id}
When limit is 500, offset is 0, 500, 1000, ..., 15000, 15500. When limit is 2000, offset is 0, 2000, ..., 12000, 14000.

Maybe you get a different number of results when you ask for more than just the record IDs?
I'm thinking that's the case. I was specifying these fields:
<?xml version="1.0" encoding="utf-8"?>
<xml><books><book><id>8758</id><title>Ko Te Rongo Pai A Matiu Upoko 1 to 9</title><url_project/><url_librivox>https://librivox.org/ko-te-rongo-pai-a-matiu-upoko-1-to-9/</url_librivox><authors><author><id>950</id><first_name/><last_name>King James Version (KJV)</last_name><dob/><dod/></author></authors></book>

When I asked for ALL fields, the returned XML was ill-formed. I'm thinking that some internal buffer was being overrun.

Thanks for commenting.
-- Bill Jones

When you think that you have exhausted all possibilities, remember this: you haven't.
--- Thomas Edison
Availle
LibriVox Admin Team
Posts: 22428
Joined: August 1st, 2009, 11:30 pm
Contact:

Post by Availle »

Could you please have the actual discussion about actual solution over there in the designated thread:

viewtopic.php?f=24&t=44129

There's no use splitting up discussions so they will be harder to find.
I'm locking this thread to prevent this from happening.
Cheers, Ava.
Resident witch of LibriVox, channelling
Granny Weatherwax: "I ain't Nice."

--
AvailleAudio.com
Locked