FOSS, distributed, low tech crowd sourced Search Engine

Has anyone ever thought about the possibility of a free, open source, distributed, serverless search engine? The principle idea is to leverage the data of users who regularly download and archive online content locally and give them the ability to effortlessly index and share these local databases online. When the user downloads content, they give it descriptor tags (which can be automated if desired). From these tags, an inverted index is generated and upload to a platform like pastebin. Pastebin hosts these flat database files which are then called by a search algorithm written in javascript. The user downloads a website to his local machine. This website is just the front- and backend of the search engine application. The user navigates to the local index.html in their browser and can start using the app. The application has two functions: 1. to tag/archive and generate index files for their local database and 2. to search the index files created by other users as well as himself. The idea is that that anyone can upload a pastebin flat database file and share the links. Every user can collect these links, link them to their search engine and start searching all of these databases. The search algorithm returns the corresponding URLs that are then rendered appropriately in the browser. The question is: how would one go about designing this when the goal is to search large amounts of distributed data performantly? There are some interesting serverless JS libraries out there. But most of them are focused on performing queries on localstorage in the browser. I think that for this particular project, it could be useful to generate a master index of all the linked indexes owned by other users, and use localstorage/cache queries for quick searches and auto-completion. But if one wishes to search the entire collection of databases, as the project increases in size, it wouldn't be possible to put all this data into the user's localstorage. So I was wondering, is there anyone who thinks they know how to go about tackling this project? Any libraries worth looking at?

I feel like it would be a good idea to either strip the archives sites of js or making the users block it somehow. If all users can dhare their archived versions of websites there's nothin stopping them from sharing self contained htmls with malicious js, for example. It would also help reduce wait times in places with poor internet speeds

>>40710 it's not meant to archive websites; it's meant to index URLs that point to media files, existing sites, text files, ... The search engine is not concerned with protecting the identity of the user, and naturally it is at the user's own discretion to visit the links returned by the search engine. It's also up to the user to make a decision as to "trust" someone else's database. If a user gets a bad reputation, the word will spread and people will avoid them. There could be forums dedicated to vetting certain archives. But in the end the choice is always up to the user. That's why it's different from Google. Google takes away your choice whereas this search engine does not. Naturally, there are potential risks involved with that. Ultimately, the goal is to create a search engine that is the exact inverse of what Google et al. is. Everything Google suppresses can go here in stead.

>>40710 >archiving websites stripping JS and/or maybe using something like the reader mode functionality some browsers have.

YaCy https://github.com/yacy/yacy_search_server Seeks https://github.com/beniz/seeks

>>40716 yacy is something completely different; it aims to do the same thing as a normal search engine but on a p2p network. Seeks seems to be pretty much dead and I don't even know what it does because the website is dead. This project is different because one the one hand it is a tool to index your own local archive(s) and on the other hand, you can share it by uploading the index files as static pages on free hosting providers. Users do not see each other's IPs. You just host your archive on third party hosts of your own choosing. Of course, if links go down, you have to maintain your archive yourself if you want it to be available to other people.

>>40708 >The principle idea is to leverage the data of users who regularly download and archive online content locally and give them the ability to effortlessly index and share these local databases online. That sounded a lot like Soulseek.

>>40764 a little but afaik, soulseek runs centralised servers for the indexes. The music data itself is transferred via p2p. The goal here is to try to decentralise the indexes. The first problem I'd like to solve is personal information management: how come there are these amazing online search tools but if you want to search your own data/archive on your local machine, there is literally nothing that helps you do it. Currently the best search tools we have are literally, (rip)grep and fzf on unix systems. That's it. But that doesn't guarantee accurate results unless the documents themselves are properly indexed according to a decent standard. There are currently no tools to help users with that afaik. One possible solution might be to create a watchfolder where you can dump your downloads, document exports, etc.. This watchfolder then triggers a script that pops up a window so that the user can index that document: >timestamp >location (usually where the document was published/created; could be a publisher or a website, or forum, or whatever) >author >title >subject/keywords/tags/... I'm still trying to figure out a "good enough" universal indexing language/ontology. On the one hand it has to support reasonably accurate query results but on the other hand, it also has to be simple and straightforward. Indexing a document, shouldn't take more than 10 - 30 seconds. But that small amount of time should yield significant search capability. The second problem deals with making one's personal archive(s) easily shareable with and searchable by others.

>>40785 >The second problem deals with making one's personal archive(s) easily shareable with and searchable by others. and the reason for doing this is because Google and other commercial search engine use robots to index information. They do use human intervention however to block out "unwanted" information. In the end, the companies that invest in SEO the most and suck Google's dick and are politically correct, and manage to keep users on their page for longer are the ones that are rewarded by the ranking system and get better index scores. I think that's simply unethical: you literally BUY your ranking. Information indexing and cataloging should be a much more democratic and transparent process. And having robots determine the value and meaning of an informational document is just stupid. And the power of indexing should not be in the hands of the few: everyone should have access to indexing and then share that index with people who might be interested in using that index.

The main problem with sharing "indexes" like this is that a sufficiently detailed index of a web page is just a larger denormalized representation of the source page [1]. So you'd be better off sharing the raw web crawl data and indexing it locally, or possibly sharing some sort of cleaned up plaintext representation of the page. There are other advantages to this sort of system compared to distributing posting lists, for instance individual users can freely experiment with any indexing and retrieval system they want instead of being constrained by the indexing file format. But of course the problem is that sharing large plaintext corpora of website scrapes is a pretty clear copyright violation if you're not a FAGMAN megacorp, so the whole thing would have to be pretty underground by its very nature, and that just raises the barrier to entry sky high. An alternative model might keep the source data and all indexes on a specific host and just allow others to query it. But if a bunch of hosts each indexed only a small fraction of the internet, then every user has to query every host in parallel for every query. And to the extent that this one host has indexed a significant fraction of the internet, they've just built their own centralized search engine. [1] If you want to be able to search for a multiple-word phrase and score contiguous matches more highly than pages that just happen to have all the words somewhere, you need to include positional information for each word.

>>42767 >So you'd be better off sharing the raw web crawl data and indexing it locally, or possibly sharing some sort of cleaned up plaintext representation of the page. i made something like this with BERT and selenium here >>42388 every new query is first checked for similarity against previous (potentially shared) queries and if it's more than 20% different it is treated as unique - the crawler stores a mapping of queries and visited links with like 500 first characters this way the amount and variance of queries and crawled data directly correlates to result correctness and most importantly you can always start from scratch and get usable results relatively quickly; all automated and all off of live internet data - no middlemen