Library Project: Corpus and Eureka

Hello lainons. I've been collecting PDFs. I have a lot of papers and books. I did not read all of them, they are there, ready to be consumed if needed. They are mostly good books and papers, cherry-picked by me or other people I esteem. Around 2019, after seeing one of the book threads, I wanted to create something that could help me, my friends and other lainons sharing our own personal book collection. Back then I was thinking about a protocol to sync catalogs. Time passed and I recently realized I can't organize a catalog. I can't find a suitable way to split my catalog in categories or sort it by topic. Maybe you found a way. But I may also be able to find a counter example that wouldn't fit my needs, more or less like one finds irrational numbers in the rational field. Aside me digressing, I started building a simple search engine. It turns out to be a good solution for my arrangement problem. I will discuss about this later on. What do you lainon use to solve this? Is there already a solution I don't know? How do you lookup for papers you don't know about? I'm aware of Google dorks, Google Scholar, Arxiv, eprint.iacr and if you want to suggest something else you are welcome. I always find crap in betweens or even duplicates, how do you remove those from your search results? Going back to the original idea, I think there are two really important things in this system. We already share documents and files, somehow. We already have ways to update each other, whether it is irc or a simple rss feed. So we don't need that. The key elements are two: a list of books and an index of those books. The latter is enough to lookup in someone else's library for some book we don't have on a specific topic we are researching. The former will allow us to find such book either buying a copy or by other means. The former will also allow us to insert tags to extend the information about a book. In this system I can search in someone else's index. I may even download the index for offline search, mirroring or even merging. Up here you are already seeing a data structure where we have two tables, one for the list and another for the index. So how do you search? There are simple and yet effective methods: tfidf (term frequency-inverse document frequency) and vsm (vector space model). You compare the documents vectors with the query vector to find the most similar (I won't explain here how this works, but you can search the cosine theorem). I find this method to be the most effective in terms of size and time. I must admit that creating the index is rather slow for now and maybe using tries (special trees) or DAFSAs (deterministic acyclic finite state automatons) could help me counting the frequencies faster. Do you think there are better ways of counting and comparing? Would caching intermediate results be efficient? Now that we saw the engine underneath, we shall discuss about the two vehicles. Yes, two. We should have a personal tractor for editing such database and a car for getting where we want really fast. I wish to call the former corpus and the latter eureka. Corpus will take care or importing documents, adding and removing tags and other information like multiple ISBN and links, and even merging and exporting databases. It will also be able to perform searches, but it is meant for back office only. Eureka will be the front facing service so you and other can easily search. I don't know how complicated this can become. It can be as simple as a command line tool, up to a web page with users and whatever users find useful. But it must allow for both searching and downloading the index. I already have implemented a version of corpus. It uses S-expression as index, just because it works. I tried SQLite but I wasn't able to get the desired performance. After some tweaks I can search five terms in less than one second in 114 documents. The total pool of documents is 706MB and the index is 9.7MB. I have not refined the data structure enough to be extensible and I don't have a good tool to do it yet, so for now I can only incrementally import documents. Here you can find the current public state of my code: https://github.com/1dotd4/corpus To get a stream of words out of a PDF I chose pdftotext. In the end, what I hope it will happen is that anybody having a curated list of papers and books will actually index such list and share it so everybody can search.

Systems like Duey Decimal and Library of Congress are fairly popular for a valid reason. While they do not make the best use of the computer, they're fairly effective and (in my experience) highly browseable, particularly if you're familiar with them. Where a book lives on the Duey Decimal Scale tells you a lot about the book. What is the scope of the documents you wish to collect and index ? What is the interface by which you want to search them like ? These are important questions. An archive of material all dedicated to a similar topic and in a couple of types can be organized much differently than a large and general library. I assume that your collection is very general, and you'd like a system that will give you shakespear when you want him, and high quality technical papers on optics when you want those. Is this correct ? The Duey Decimal system puts these both in their places, and a savvy user of a well organized library can walk directly to either of them. I suppose, perhaps your system could be an autosorter, an effort to place given texts in their correct spot on a scale like Duey's without human intervention. I wonder what you envision a user's search terms to be like, and what sorts of fruits you would like your engine to bear. I wonder if you are aware of Xapian, a fine and easy search engine tool with a couple of language interfaces, though I think not scheme.

>>31788 >What is the scope of the documents you wish to collect and index ? Currently my scope is collecting high quality books and papers. Later, when facing a topic, I would have a good collection where to find answers I'm looking for. I also want to share the collection in a useful way, not just a list and I hope I'm not the only one. My collection is broad between mathematics and some computer science. I think it would be between 510 and 519. But in those ranges there are various decimals that one may use, but sometimes titles aren't that good in explaining how good a book is for an introduction or for a deep study. Yes, my system could be an autosorter if I could separate with a line the various topics, but sometimes I can't. Sometimes there's something that's in between. Other times some novelty. But as you can see from my excuses, I didn't study Dewey Decimal Classification nor Library of Congress Classification that far to know how to solve those cases, yet. I've built this so I don't have to think about it: I see a good paper, I just drop it into my folder and when I'll lookup for something the document can explain, I'll find it. >What is the interface by which you want to search them like ? Given the nature of my archive, it would be a plain HTML form, but that doesn't mean one can't query with a terminal and receive the resulting list. >I wonder if you are aware of [...] This project is just my own bird-feeder at this point. I appreciated your critics and those will make corpus something better. Note to self and curious: this is what I found about those two systems. Maybe I'll come back later and use one of those instead. https://www.oclc.org/content/dam/oclc/webdewey/help/500.pdf https://udcsummary.info/php/index.php http://bpeck.com/references/ddc/ddc.htm#table500 https://www.loc.gov/catdir/cpso/lcco/ https://www.loc.gov/aba/publications/FreeLCC/LCC_Q2022TEXT.pdf (under QA)

I too have a similar problem. I started compulsively collecting ebooks in 2008 or so and never stopped. I have several multi-terabyte external disks full of pdfs. For searching I use recoll (its on Linux, idk if on other OSes too) Its not the best but it works. For organizing: Consider extracting the ISBN using pdf2text and a regex and then making API-calls to amazon or other online book databases and get either the description or the tags or categories the book is in. It won't be perfect, but with a graph algorithm you might be able to calculate a nearness factor and group books that are "similar" into one cluster. You might have to tell your DB what synonyms are so that it does not place a book that talks about "networks" too far from one that talks about the "internet".