Another LibriVox Catalog

[ScottLawton]

Until Aug/2013 Forum URLs and Forum names are interfaced through API.

That might be useful prior to release but I personally doubt the forum link adds much once the book is done. (Others may have a different view!) Still, it's not that difficult to get the information from the forum itself -- we don't have to depend on the API.

Readers' forum names are also shown. I'm familiar to Forum Names rather than Catalog Names.

Again, this info is available on the Web. I expect to have the data in convenient form by the end of the year and will be happy to send you a copy. (If you have *old* data from the previous API, I could use that as a starting point.)

Cheers,

Scott

[ekzemplaro]

Hello,

Thank you for your messages.
I've installed nginx,MariaDB,phpMyAdmin on my Arch Linux.
I created the database using librivox_catalog_new.sql.
Now I'm ready to import data when export-file or dump-file is given.

Admins do not have access to root

My example was not adequate. Root can do it, but you don't need to be root.
The following also works if you know username & password.

mysqldump -uscott -ptiger librivox_catalog_new > librivox_nov15_2014.sql

Again, this info is available on the Web.

Yes, you are right. All information is available on the Web. But it's difficult to parse Web using current technology.
archive.org is doing a good job at this point. All information on the Web is available in JSON.

(If you have *old* data from the previous API, I could use that as a starting point.)

Here,
http://ekzemplaro.org/librivox/api/data_previous_api.tar.bz2
It's in XML. I converted them into JSON and merged them.
The merged file is http://ekzemplaro.org/librivox/statistics_classic/latest_on_going.json

Looks like they are using http://www.codeigniter.com/
I learned this in librivox-public/system/database/DB.php

Cheers,
Masa

[ScottLawton]

Masa san,

I could be wrong, but I doubt that our friendly admins here can 'ssh' into the server and run a command -- or even have a login for phpMyAdmin.

However, I confess I don't understand why they can't get less than an hour of help from archive.org -- a non-profit fully committed to the public domain. If the tech staff sets up a 'cron' job then the db dump will be automatic every day or week or whatever they specify.

Meanwhile, there are lots of libraries to parse HTML and help extract info. For Python: BeautifulSoup. For PHP: http://php.net/manual/en/domdocument.loadhtml.php or http://code.google.com/p/phpquery/ or http://simplehtmldom.sourceforge.net/ and probably others. (I only use Python so haven't tried these.)

In any case, I plan to share the information you're looking for.

Scott

[RuthieG]

You're quite right, Scott. We cannot just ssh into the server. Without going into details, we have had unfortunate occurrences in the past which mean that we have to be scrupulous about security.

We are always chasing our tails. As you know, we are all (including our sysadmin) volunteers, and juggling many balls in the air in our lives, including LibriVox. The software rewrite still has many flaws that need fixing, and there are other behind-the-scenes issues that are demanding our attention. Some of these are quite high priority. We know that the API needs attention, and we know that third parties would like more information provided. An sqldump may be the solution, but we ask for your patience until other issues have been attended to.

In the meantime, we continue to provide access to an ever-growing number of free public domain audiobooks, and this is our primary goal.

Ruth

[ScottLawton]

In the meantime, we continue to provide access to an ever-growing number of free public domain audiobooks, and this is our primary goal.

I fully agree! LibriVox is a remarkable ongoing achievement. And, though I'm not in a position to volunteer for the 'usual' important tasks, I am in the midst of a project which will yield data for ekzemplaro and others.

[ekzemplaro]

Hello Scott san,

LibriVox is a wonderful project. I just like to make it more wonderful.

or even have a login for phpMyAdmin.

Even though they can't use ssh, they can use phpMyAdmin or an equivalent tool.
As you know, they have a write privilage to the database. They can even delete a project.
Why can't do make a dump or export file? The logical conclusion is that they don't wish it.

I don't understand why they can't get less than an hour of help from archive.org

This is because of the same reason. Then don't wish it.

We got a funding from Mellon funding with a condition that the result becomes as open-source.
If I were a board member of the Melon funding, I expect that after finishing the funding, LibriVox adopts open-source
method for the further development.
But LibriVox selected extra funding rather than open-source method.
This is a very disappointing decision.

Lots of readers come and contribute several sections to the collection. And they know their recordings are not searchable by Appli. And they stop contribution. System development is important to boost recordings.

Ruth san,

and there are other behind-the-scenes issues that are demanding our attention. Some of these are quite high priority.

These should be fixed and tested on the mirror site. Open-source is the solution.

Cheers,
Masa

[annise]

Masa -
You continually say They won't do this, They won't do that. Maybe in Japanese it works better - but to my mind the word should be we. We are all part of a fantastic group effort. Things are accepted within the constraints of the possibilities - and we are continually trying to improve things , not because we think they are awful but because we always want it to be better.
We have a prime directive - to produce in audio form all books that are public domain - and we are making a good start. Anything that helps us do that is always top priority. All our projects are downloadable , all our projects are found by the search
I know the search is not perfect in the way the results are presented - but they are there.
We can't delete projects from the database - I don't know where you got the idea we could ?
We met the Mellon requirements, the coding was released as per their requirements.

We don't provide what other people want to enable different ways of downloading and that is a pity and some of us are more interested in working out ways it could be happening than moaning about it all the time. Meanwhile we just keep on with our prime directive - producing audiobooks. Not apps , not alternative catalogues, not anything else that our email continually suggests we should do
We produce audiobooks, Internet Archive hosts them for free and I'm proud to be part of us.

Anne

[TriciaG]

As you know, they have a write privilage to the database. They can even delete a project.
Why can't do make a dump or export file? The logical conclusion is that they don't wish it.

You are wrong. We don't have the functionality to do that. We don't have permissions, or a means of doing it.

If I were a board member of the Melon funding, I expect that after finishing the funding, LibriVox adopts open-source
method for the further development.
But LibriVox selected extra funding rather than open-source method.
This is a very disappointing decision.

I have no idea what you're talking about - we selected extra funding rather than being open source? No, that is incorrect.

We did provide an open source version of the back-end software. The Mellon project did NOT require us to open up our detailed programming to anyone and everyone to play with and modify. That would be irresponsible - an exploiter's dream.

[Darvinia]

I think perhaps the problem lies in a misunderstanding by Masa of the term "open-source"?

Open-source means that we make the source code available free of charge to the public to use, copy, modify, sublicense, or distribute for their own projects. As Tricia just said we did provide the code as open-source on github as per the Mellon grant requirement.

Open-source does NOT mean that we open our own project for examination or manipulation. Again, as Tricia stated, the Mellon grant did not require us to do this.

[ekzemplaro]

Hello Bev san,

Thank you for your message. I just consulted Wikipedia.
http://en.wikipedia.org/wiki/Open_source

In production and development, open source as a development model promotes a universal access via a free license to a product's design or blueprint, and universal redistribution of that design or blueprint, including subsequent improvements to it by anyone.

http://en.wikipedia.org/wiki/Open-source_software

Open-source software (OSS) is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change and distribute the software to anyone and for any purpose.[1] Open-source software is very often developed in a public, collaborative manner.

Cheers,
Masa

[Availle]

Masa san,

I still have the impression you don't fully understand as you are highlighting parts of the quotes that are not relevant to us...

In production and development, open source as a development model promotes a universal access via a free license to a product's design or blueprint, and universal redistribution of that design or blueprint, including subsequent improvements to it by anyone.

The important part here is universal access ... to a product's design or blueprint and universial redistribution (rights) ... to anyone.
We have done this. The blueprints/designs are available on github for everybody to do with them as they wish and see fit, including making changes and redistributing the work they built on ours.

Open source does NOT mean you can waltz into the factory and mess with the machines or the current production; or, in our case: with our (running) system.

Open-source software (OSS) is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change and distribute the software to anyone and for any purpose.[1] Open-source software is very often developed in a public, collaborative manner.

Our code is available and you have the right to study, change and redistribute the software.

In your highlighted part it says very often and not always. At the moment, our systems have not been developed in a public, collaborative manner (there was a single programmer doing this), and for the time being we will not incorporate any changes from outside people to our code.

The reason for this is that any changes to our system will have to be thoroughly tested on OUR site to make sure they do not interfere with any other parts of the system.

And, as has been explained above, for the time being, we do not have the resources to do so, neither technical (a server set aside for development and testing), nor personnel-wise (a dedicated person who thoroughly knows our systems and code to do the testing or futher development or even bugfixing), nor financial (to hire such a person, or even re-hire the developer who produced the code in the first place).


To give an example:
Firefox is open source.
You can go ahead and do whatever you want with the code. You can add an instant Japanese-English translation to it and distribute it as "Masafox" and it would be great. I think you could even sell it under that name.
However, if you wanted Firefox to include your new feature, there would probably first be a committee to decide whether this idea of yours is really that great, and then there would be a thorough testing of Firefox under various environments to see if it's still working before a new version of firefox is released.
Still, Firefox would remain open source even so.