lainchan / λ
Curated, P2P Search Engine
Lukasiewicz · 2024-03-31 01:43
Searching for high-quality information and websites, has become progressively more difficult. There's way too much conflict of interest. I'm interested in creating a search engine that's curated and fault-tolerant.
The usage would look like this: a curator selects domains to crawl and builds an index. Users would both query this index, and contribute to its storage by replicating parts of it. Users would be able to query over multiple curators' index. Any user could also be a curator.
The idea is for storage to be decentralized, so as to not have one point of failure. Writing though would be restricted to the original curator. Users would pick which curators to use and get aggregate results from their index, from anybody who contributes to its storage.
Right now, I'm trying to figure out how to implement the storage. Yacy uses a DHT. The problem with it though, is that it's slow and can hardly handle queries longer than two words. It's been over 20 years since it was first released, so there has to be a better approach now.
https://linuxreviews.org/YaCy
I'm not sure full text search can really work with DHTs, so I've been considering a distributed database like CockroachDB. I don't know if those can be used in a P2P context though, with highly unreliable nodes. This blog post is about P2P databases, which sound like a good solution, but it's entirely theoretical.
https://blog.mauve.moe/posts/peer-to-peer-databases
https://blog.mauve.moe/posts/peer-to-peer-databases#cross-organizational-search
Anonymous · 2024-03-31 04:52
Had the same idea once but been too dumb/busy/depressed to implement any of that so I hope you succeed instead where I failed.
Don't get hung up on the choice of DB though. In fact make a point about not being reliant on it nor make this network gnostic of dependencies like that. Instead approach it like a protocol, which is what real p2p at its core is. Especially if it's a project of spite you want to allow anybody and anything to hook into it.
That leads to the next point: think, and I mean really think about possible attack vectors against such a network. What are all the possible ways to mess with it or even endanger its users? Can someone upload cp or return it as their response to another user's search query and making them liable? What could prevent that? Or if g--gle launches a bunch of malicious nodes which aim to render that system unusable?
Regarding p2p compatibility of any framework or library, remember it's just one machine sending a request to another with a static ip which in turn just does another hop to another static ip if there's a need to do that so REST or basic sockets with a shared interface or API should suffice, but may need to be optimized for throughput. Nothing even i2p related. But do look into flood search and node discovery and the problems with them.
Then also think about what the incentive could be for anyone joining that network just to host some web indexes. What would lure them? Sadly the crypto train has turned out to be less helpful than I hoped but then can they put up ads in their search responses, possibly from local businesses and in general act like a network point for a very short geographical radius? There needs to be motivation in order for a network to actually become one. But hey, I've got an idea how digital vouchers could work! Not sure if that's applicaple though.
I probably should've kept better notes on my ideas but you seem to got some good ideas thus far. Try to flesh it out some more and try to have something running and testable in the wild asap, be it with some dump python or js server. But key is to iron out the part of how the nodes communicate with each other. And hey, if things run well, maybe it will turn into its own internet some day!
Hope this helps and maybe I'll remember some more later. God bless.
Just another Random Lainon · 2024-03-31 06:35
Personally I would implement search by first converting each unique word into an unique integer, storing a copy of pages as arrays of integers, then searching would just be looking through the array for matching integers.
Additionally each page would have a bitflag array of if a category of a word appears in the page.
Unique integers representing words are lossily compressed into a single index in the bitflag array.
That way its possible to do a cheap precheck before a full check.
Pages would be preprocessed in this way when they are added to the search cache.
Donna Hawthorne · 2024-03-31 11:32
>>38683
>Instead approach it like a protocol, which is what real p2p at its core is. Especially if it's a project of spite you want to allow anybody and anything to hook into it.
I'd rather build on-top of a preexisting protocol than try creating my own. I'd hate to make a crappy protocol in my inexperience and ignorance, while a good solution to the problem already exists.
>if it's a project of spite you want to allow anybody and anything to hook into it
I'd rather it be good and have stringent requirements than bad, but universally accessible. The use case I have in mind is very specific, so making my own internet isn't a goal.
>Can someone upload cp or return it as their response to another user's search query and making them liable?
PGP signatures seem to have this problem solved.
>but then can they put up ads in their search responses
Avoiding advertising is one of the main goals though. As for providing an incentive, I think access to the index is good enough. However, being a node shouldn't be too resource demanding either.
sizeof · 2024-03-31 12:58
>>38682
> The usage would look like this: a curator selects domains to crawl and builds an index.
This part was the idea behind SearchHut. However Drew gave up on the project after someone linked posted a link to it on Hacker News.
It's at least worth taking a look if you are interested in search engines.
The guy behind marginalia.nu also posts technical details about his work if you are interested.
Jens Hansen · 2024-03-31 13:03
check out xapian, its alright as a search backend.
note that yacy uses some apache product
don't underestimate how big the internet is. I have tens of millions of websites crawled and I still miss common & useful things despite picking I think sensible and relevant start positions.
null · 2024-03-31 13:17
If I give up on the index itself being distributed, this is the model I've come up with:
The search engine has a standard schema which all curators must follow.
Curators transmit SQL scripts to update this database. Users choose which curators to accept updates from.
Curators publish a public key, and a "get-up-to-date" script.
Every update of theirs comes with a PGP digital signature created from the hash of the update script.
When users receive this update script, they hash it and use the curator's public key to compare it with the accompanying signature.
Records are uniquely identified by URL. The script inserts or updates records based on if they already exists, and if the new timestamp is more recent than the current one.
There are two types of records, websites and images.
Websites consist of: its url, the latest GET response, the GET response upon it first being crawled, the most recent crawl timestamp.
Images consist of: its url, a hash of the image, a perceptual hash of it, a textual description and or extracted features of it.
Queries are performed using SQLite's(or whatever DB is chosen) built-in FTS. OR, I'm pretty sure this update scheme could work with a separate search engine, since everybody would have a complete copy of the index.
>>38686
Will do.
>>38687
If I decide to use a separate engine for searching, I'll probably go with Bleve since I'm most comfortable with Go. As for blindspots, I'm hoping multiple curators will alleviate the issue.
Moore · 2024-04-04 07:44
>>38684
you are thinking of "hash tables".
a hash is a function f ( string) -> int
with good functions spreading the output uniform on the available range which is important as it would be disastrous is every output landed in same bin.
>>38682
just use existing torrent systems as the deployment framework and build a small app that uses torrent files for updating a local search engine db.
There is a smart publication on search that merges words for a faster search. For example:
Gain
Rain
Train
result: [G], [nil/[T] +R] + ain
So if you type its not comparing to 3 words but comparing against +ain and if it fits then trying out the 2 known combos G and R and if R fits its sub section nothing and T. I dont remember the exact implementation but i remember being really impressed because they did it performant to search entire books merging all words.
Hong, Guil Dong · 2024-04-04 07:52
>>38700
should mention this works for sentences not just single words since for the algo its all just strings. And since you already doing a closest neighbor comparison you get the mistype also covered without extra effort.
Eldon Tyrell · 2024-04-04 08:24
>>38688
>im hoping multiple curators will alleviate the issue
I hope so too, just saying I think you're gonna need a yuuge number of sites, and a lot of crawls, if you want to be good and not overly specialized. Just my take from having attempted something vaguely similar. Product was usable but massive blindspots even in topical areas I tried to crawl through fairly thoroughly. I did find some neat sites with it.
How do you deal with conflicts ? Two curators crawling same URL, just use whichever is newer ? Maybe have some ranking for curators that the user can adjust on his own instance ?
Ivan Horvat · 2024-04-04 09:35
>>38700
>just use existing torrent systems as the deployment framework
I've decided on IPFS because IPNS allows updates to be in the "same location", hosted by multiple people, and guarantees the authenticity of files and their publisher. Right now, I'm figuring out the search engine part.
>smart publication on search
I couldn't find more information on this. Could you provide a link?
>>38702
>How do you deal with conflicts ?... just use whichever is newer ?
Yes, that's my plan. The idea is that users trust whoever is a curator and benefits from having multiple of them. Allowing redundancy in the data and granulating the curators into search parameters, feels like it would undermine that benefit.
James K · 2024-04-05 04:41
>>38682
The problem you've stumbled upon is not an easy one to solve. The real issue is hierarchical organisation of content, rather than actually creating a P2P search engine. There are many existing approaches to this. BlueSky's AT Proto may be of interest to you, as BlueSky is essentially an attempt to build a second internet on top of the first one. That said, it's mechanisms seem not to be beyond manipulation. Google's Page Rank is obviously flawed, but provides at least a good basis which you could adjust by user scores. Reddit makes a key realisation, that users will express that they either "like" or "dislike" content. If you try to give them any other metric, for instance "usefulness", that will inevitably just boil down to weather they liked it or not.
None of these approaches prevent manipulation of the system. Likes can be botted on Reddit, and Page Rank is all too easy to game with link farms. So Facebook, YouTube, and similar sites realise that users can be sorted into categories based on behaviour. Fake/malicious users will behave differently than ordinary users, so their inputs into the system (likes, dislikes, views, etc.) are deprioritised when ranking content for non-fake users. This manifests as the eponymous Algorithm, which goes further than simply removing botted content and separates users into many small bubbles who all perceive the web in different ways. This seems like a bad idea for a search engine.
To put it simply, we have yet to find an algorithm for truth.
A strategy where users individually maintain lists of websites they find useful, and you can then subscribe to another user's list and receive recommendations from it, may seem appealing. Manual curation, as you would see in something like Wikipedia, with the additional step that you can choose who is curating your content. This is, in essence, how all search engines used to work. Some group of people compiles a curated list of links and let's you search over it. This will still have the issue of segmenting users into groups, but to a lesser extent and users will be more explicitly aware of it.
Unfortunately, this system has already failed. It requires much more input from the user, and yields "worse" results. As much as people complain about the Algorithm, they secretly like it. It shows them what they want to see better than any other option. As much as we'd like to pretend otherwise, truth stems from authority. In a distributed system, there is no authority, and so there can be no truth. In science, authority lies in journals who enforce standards of publishing, and in respected names who have knowledge of a field. In Wikipedia, authority is deferred to references and trusted editors. You can see that both of these systems are still open to abuse and manipulation wherever there is some exterior motive driving people to a conclusion.
So I think this problem may be intractable. Certainly, any solution would be of great interest to many people. It would likely bring fame and fortune beyond your wildest dreams. And so, conversely, it is unlikely for you to find it.
Cereal Killer · 2024-04-05 04:51
>>38706
The last time google showed what you want was maybe a decade ago. I remember how good it was and how soykaf it is now.
Its easy to beat the existing systems by simply avoiding marketer manipulation. Especially since alternative existing search engines seem to only copypaste google results.
James K · 2024-04-05 07:37
>>38707
>by simply avoiding marketer manipulation
The only way to avoid such manipulation is to be too small to be worth bothering with. Both in terms of the network as a whole, and in terms of curators. What's to stop ad companies from offering a small payout for putting their links in your curation over a competitor? The only way to avoid this is to have an exceedingly large number of curators relative to consumers, at which point everyone is essentially maintaining their own list and very little is gained from it being P2P.
Tomohisa "Tom" Kanezaki · 2024-04-05 07:59
>>38707
>marketer manipulation
SEO is also imho a big problem. by being weird (and too small for anyone to care about) you *mostly* avoid it, because its not aimed at you, but its still there and will probably get picked up in your wordsearches and stuff. If you get big enough to care about, then have fun.
>>38706
I think you're undervaluing the fact that I'm *searching* for something, potentially obscure. Im not OP, but from what I can tell this isnt meant to be the social media suggested posts system that you seem to be discussing, nor a 1984 factchequer computer, its ideally like a breddy gud index for looking up specific, perhaps obscure, things.
Personally I want to be able to type in a rare or unique sentence from a book or blog or something and find the thing just like that, maybe with another keyword or two to help out. Search engines used to be better at this it seems like. Have they gotten worse ? Maybe, maybe the internet has gotten worse too, hard to say. But there's still a good amount of good stuff out there and one can find it.
Also, when it comes to search engines I don't really care what my 'friends' are linking, they can send me those links on IRC just fine. I just want to find the answer to some question, or more information about some topic, or some blog I remember finding years ago that seemed good and I want to check if its been updated. That sort of thing.
>truth stems from authority
well you can take it on my authority that you're an enormous glitterboy
Spooky · 2024-04-05 08:16
>>38706
I think you've made a few oversights. Google, Facebook, etc. 1) try to be the end all be all in their stated purpose 2) exist to make money. This gives them disadvantages. I'm not trying to create the one search engine to rule them all, something that's suited for everybody and every purpose. It's okay if it's niche, as long as SOME people find it genuinely useful.
Because I'm not worried about making money, or appeasing any governments, this will be able to provide results Google, or any other profit-driven/centralized search engine, will never be capable of providing. That alone provides utility to SOME people.
>This seems like a bad idea for a search engine
Those examples are automated systems, which again, try to appeal to the lowest common denominator.
>authority lies in journals who enforce standards of publishing, and in respected names who have knowledge of a field. In Wikipedia, authority is deferred to references and trusted editors.
Guess what? You can crawl those too. The internet is much more structured than when Google was in its infancy. That structure and "authority", can benefit anything which crawls it.
>>38709
>What's to stop ad companies from offering a small payout for putting their links in your curation over a competitor?
If a curator puts ads in their results, users will take notice and stop using their index. The way I'm conceiving of it, users would also easily be able to fork an existing index. When somebody becomes a curator, everything they already have will be in their "catch-up" file.
Zhora · 2024-04-05 11:34
>>38706
the power in wikipedia is not in the links. Its in the article maintainer controlling what is linked. You can find supporting articles for any stance and the maintainer controls what stance gets supported with links.
>>38709
The point of p2p would be sharing lists.
You can use concepts like web of thrust that simply assign thrust scores to users. The relationships are then crawled applying the trust score to their output.
Another option would be blacklists. Now we know from adblock addons that the maintainers get paid by advertisers to get unblocked but despite such concerns they remain an improvement over using none.
James K · 2024-04-05 16:42
>>38711
>I just want to find the answer to some question
So you want the truth essentially. How do you determine that? For instance, a common question: which of these products is best? Now you can see there is a pretty vested interest for displaying certain answers to that question over others. In the end, you need some way to differentiate good from bad content. If you are just "searching for something", you'll end up with a bunch of content-farm listicles stuffed to the wahzoo with every keyword imaginable.
>I don't really care what my 'friends' are linking
You realise that linking is essentially how google works, right? You need some way of determining which websites are "better" than others, given the massive number of websites in operation, and the huge incentive to appear useful when you aren't. Without a central aggregator, the only way to do this is to ask what your friends are looking at.
>well you can take it on my authority that you're an enormous glitterboy
True, but I don't like your tone.
I think that you maybe haven't thought about this issue enough. At the most basic level, social media and search engines both solve the same problem: present users with relevant content, hide content the user doesn't want to see. This is why approaches in one field are highly related to those in another. Reddit is essentially just a search engine with a comment section. Other forms of social media obscure this relationship more, but it is still present in all of them.
>>38712
>users will take notice and stop using their index
How would they notice? Ads in this case would just be linking one site over another. Not necessarily easy to notice. Like Google's degeneration, it would happen gradually as any list grows beyond a certain number of users.
>You can crawl those too
You can't algorithmically determine the authority from which a truth stems. In the journal example, the only way to "crawl" a journal's publishing standards is to have a human go in and verify that articles adhere to those standards.
>It's okay if it's niche, as long as SOME people find it genuinely useful
"Search engine" might be the wrong term for what you are trying to create. The idea of a search engine is one big box that you can look things up in, but I think what you describe is closer to a set of content directories. It would make more sense to say users compile "journals" on a given topic (a niche) that can be searched and distributed freely. I think framing it in this "journal-oriented" way makes it clearer that an individual or organisation is publishing a set of links, and that those links should be tied to the authority of the publisher. It also makes more sense from a categorisation perspective, you know about fishing so you publish a fishing journal, and you can then maybe categorise content further within that journal and make it easier to search. I think this is the most similar to the Wikipedia model of content organisation, which has an excellent track record for factual accuracy and usefulness, but in this case a greater focus is put on the references rather than summarising them.
Given this, a primary challenge would be that of content submission. This can be relatively distributed, but the experienced would likely be improved if there was some way of submitting your own links to a given "journal", rather than relying on the journal author encountering those links in the wild. Making submission easier is also very good for growing the network.
Wintermute · 2024-04-05 17:25
>>38716
i like the ease of use point.
Just have stupid simple txt lists of urls you like and want to share.
That's a level anyone can participate on with minimal friction. You can make tagging a secondary step.
list · 2024-04-05 17:41
>>38716
>How would they notice?
Unlike Google, there's transparency. You would know exactly what domains have been indexed, which curator crawled them, and when they did so. People could communicate with each other about curators too.
>You can't algorithmically determine the authority from which a truth stems.
If a curator has reason to believe a source is trust-worthy, that's good enough. If people disagree, they don't have to use that curator's data.
>The idea of a search engine is one big box that you can look things up in, but I think what you describe is closer to a set of content directories... I think framing it in this "journal-oriented" way makes it clearer that an individual or organisation is publishing a set of links
This feels pedantic. The front-end will look like a search-engine and people know what that is.Needing to educate prospective-users in a set of project-specific terminology, would only diminish interest.
>This can be relatively distributed, but the experienced would likely be improved if there was some way of submitting your own links to a given "journal"
Just share the link with a curator if you don't want to crawl it yourself. That's a surrounding infrastructure thing which can be developed independently.
James K · 2024-04-06 02:51
>>38718
>The front-end will look like a search-engine
If different users have different curated lists of links, how would you decide which set of links are used by default?
>crawl
I am confused by your usage of this word. If the lists of links are curated, I don't see where crawling comes into this. A person can't "crawl" a given domain because crawling by it's very nature traverses the whole of the accessible web.
>project-specific terminology
Using the word "journal" or "directory" to refer to things which are journals and directories is not project specific terminology. It is just using words.
>You would know exactly what domains have been indexed
And how could you tell from a list of domains whether one has been sponsored? If, for instance, the list contains burgerking.com but not mcdonalds.com, how could you tell if that list has been sponsored or if that person just prefers Burger King?
St. IGNUcius · 2024-04-06 11:49
>>38720
not him but webcrawlers can crawl a single page. All it means is you have a program that tries to extract valuable information from the site in an automated process. For that purpose it might crawl all the pages of the website. The goal after all is to index all the content a site contains and you dont know that unless you crawl all the pages of a site.
James K · 2024-04-06 16:41
>>38726
>The goal after all is to index all the content a site contains and you dont know that unless you crawl all the pages of a site
I'm not sure on the practicality of this then. If the idea of these indexes is curation, then including all pages in a given domain is opposed to that. My own website is quite barren, yet it has perhaps 50 pages at the bare minimum. I'm not sure how curated an index would be that included all of them.
Hououin Kyouma · 2024-04-06 17:05
>>38732
>how would you decide which set of links are used by default?
There are no "default links". Read >>38688 and
https://docs.fileformat.com/database/sql/
>I am confused by your usage of this word.
You give a crawler a list of starting points. It does a depth-first traversal up to a certain depth(nodes are connected by links). Along the way, it scrapes page content and indexes it.
>how could you tell if that list has been sponsored or if that person just prefers Burger King?
I believe users are capable of making those judgment calls. If you disagree, it sounds like this project simply isn't for you.
James K · 2024-04-07 02:18
>>38734
>If you disagree, it sounds like this project simply isn't for you.
This is a very strange way to deny the truth, which is that adverts can hide themselves from people. Why do you think it is the law in most places that adverts must be labelled? It is because otherwise you wouldn't be able to tell they are ads. An advertiser may spend weeks creating an advert that you will look at for seconds, he is almost always going to outsmart you in this setup. Hiding adverts from people is so easy it had to be made illegal.
>Read >>38688
I don't see how this is curated. It seems that anyone can upload links to the network you describe. Also that you are creating a kind of distributed SQL injection attack. What is to stop me, a Burger King employee, from writing a script of the form:
-- 7000 lines of code
DELETE FROM Websites WHERE url = 'macdonalds.com';
-- 7000 lines of code It would make much more sense for users to just host a git repository of their links, or something similar, rather than an executable diff-based format.
>There are no "default links"
You said the interface would be just like a search engine, which means users would not have to initially select curators to use links from. Hence there must be some default providers of links. Otherwise the interface will first ask them to select link providers, which is not just like a search engine. And then you are back to step one. How do you search link providers? You would need some kind of search engine.
Sabo Engle · 2024-04-07 07:41
>>38739
>This is a very strange way to deny the truth
I don't want to reply to you anymore, but I'll be courteous enough to tell you this is the last time.
>It seems that anyone can upload links to the network you describe
Yes, anybody can be a curator. You have to opt in though to use those curator's updates. I've said that many times.
>DELETE FROM
Not part of the program's standard. The only commands accepted from curators would be insertions and updates. That's simply a matter of implementation. If a curator you chose is a bad actor, that means you picked a bad curator. Ideally, people would talk about this.
>which is not just like a search engine
An initial set-up, does not make it a completely different thing.
>How do you search link providers?
Word of mouth. Additionally, bootstrapping it with existing search engines, is also acceptable.
James K · 2024-04-07 08:00
>>38743
>does not make it a completely different thing
I think it does make it a completely different thing. A version of google that's initially empty, and requires you to look around for people to give you an index of some sites, would be completely different to the current google. The core feature of google is that it is just a search bar. If you put stuff in the way of the search bar, and especially a set of steps as complicated as this, it ceases to be google.
Nobody special · 2024-04-11 22:55
I made something bare-bones that partially crawled lainchan's webring, along with a few other sites. It crawls seed url's to a depth of 3, and does a single-page scrape for cross-domain links. I underestimated how slow and computationally expensive web crawling is, but the result seems somewhat promising.
https://codeberg.org/vodahn/Tektite
Church · 2024-04-12 06:54
>>38804
Mine I made multithreaded, made a big boost on pages per hour.
Technician · 2024-04-12 12:17
>>38805
Crawling begins on multiple seed-URLs at once, but I don't want to overload a single domain, so each crawler is single-threaded and there's a delay of a few seconds between each request to a domain. The sheer number of links involved makes it a lengthy process.
Motoko Kusanagi · 2024-04-12 12:44
>>38810
eventually you will get to where you have thousands upon thousands upon thousands of uncrawled links just sitting around, and can generate shuffled lists that don't overload any given site too heavily, just because there's so many sites. Alternately have one patient thread for every individual site crawler but crawl a bunch of sites at once.
Yeah its still kinda slow though. My internet isn't great but I used to just setup curated lists to crawl and let it run for a week or more.
J.C. Denton · 2024-04-12 13:36
>>38811
I've added a persistent queue in the case of crawling being interrupted. Maybe the queue could be shuffled.
⍕ · 2024-04-12 14:46
I've seen site block you if you crawl to fast so i think the slow 1 thread / site approach is best.
That aside i kinda wonder now with sites using crap like cloudflare they could obviously notice you doing many connections in parallel so they could kick you for being a bot. Would be interesting to see if its a thing.
cdr · 2024-04-14 01:43
If you have an ip6 address, here's an instance you can try out.
http://[2a07:e03:3:d3::1]:1025/
It's usable for independent use, so figuring out index sharing comes next.
car · 2024-04-14 08:29
>>38821
cute asci art and i also found a nice new page 10/10
Nemissa · 2024-04-14 20:41
When it comes to crawl time, I think I've come up with a good solution. Every link on a Seed URL will be visited and every page will have its number of links counted. On pages deeper than the seed urls, the chance of a link being followed will be 1 / (referrer's number of links * depth). This way, pages with more links will have a shallower crawl and vice versa, and crawl time per seed url should be reasonable.
Debbie Charteris · 2024-04-15 07:13
>>38831
> Every link on a Seed URL will be visited and every page will have its number of links counted. On pages deeper than the seed urls, the chance of a link being followed will be 1 / (referrer's number of links * depth). This way, pages with more links will have a shallower crawl and vice versa,
If i understand it right that would lead to crawling an entire web novel (only 1 link: next chapter) while ignoring the initial selection of 100 novels (too many links -> most dropped)
Here's how i would do it:
Simply use an array that you fill with links as you go. Arraysize sets the link limit and because you go in order of insertion you automatically follow the depth order of the site structure.
This is simply a copy of how binary trees are stored and traversed. A binary tree like
0
1 2
34 56
is stored in an array as [0,1,2,3,4,5,6] so doing a level order traversal is just a matter of sequentially reading the array. Inserting links into a list as you go will create a similar structure.
Step 2 would be adding link frequency - its unreliable cross site but still a valid metric on a site linking to itself - simple implementation is add "int counter" and "bool visited" to "string link". During insertion increment counters then sort list before selecting next page to visit and set the visited flag for the selection.
Undisclosed · 2024-04-15 08:39
idk isnt this sort of precise decision making about crawls part of the curators responsibility ?
imho there's no especially great way to do this that isn't just scan everything. One way is to look not at how many links from a page but how many links to a page, index of a blog is probably linked from every page of the blog, and quite probably more important than any individual post. Search priorities can also include number of links to website, as done in classic pagerank.
Maybe good to also have just specific types of crawls. Deep vs shallow. Sometimes I'd like to get every single page from one good site indexed, sometimes I just want to fan out broad as possible. Some logic handling whether links are internal or external could also be used for this.
Im sure you already have enough pages that you could just extract ten thousand links to get without necessarily doing any sort of link crawling. generate a big list, crawl in batches, reprocess and generate another big list. This probably limits individual site load.
Hagbard Celine · 2024-04-15 09:56
Anyone should be able to use this link: fukumen.mooo.com:1025/
>>38836
Sites to be crawled are currently stored on a queue that's in an sqlite file. That's the best option that the crawling package I'm using(colly) supports out of the box, because a crawl can be resumed after interruption, and storing everything in memory quickly bloats the ram usage. Your solution would reduce the memory usage by a lot though, and the the need for being able to resume.
The thing is, I want every link on a seed-url to be visited; the user explicitly chose that url to be crawled, so if only the first x links are followed, that might be annoying. I could set the array size based on how many links are on the first page, but that would be pretty much the same thing as my idea except with a bias towards links earlier in a page.
>add "int counter" and "bool visited" to "string link". During insertion increment counters then sort list before selecting next page to visit and set the visited flag for the selection.
So a heap with the most linked-to page at the top? That's an interesting idea.
>>38837
>idk isnt this sort of precise decision making about crawls part of the curators responsibility ?
I want to keep the barrier to entry low, so sane defaults are a must. Your idea is the same as the other anon's so I'll definitely look into that and pagerank.
>Im sure you already have enough pages
I'll probably start a new index from scratch for a number of reasons. There's a lot of junk php pages, some were mangled by my regex during different encoding handling, and I'm interested in snowball stemming.
Six · 2024-04-15 11:25
>>38837
>Search priorities can also include number of links to website, as done in classic pagerank.
The reason this metric faded out is blackhat seo learned to spam links to rank artificially high
It might work for fringe content nobody runs adds on but im not sure if it will since there are automated text generating clickfarms covering seemingly any topic.
Marly Krushkhova · 2024-05-25 02:30
So OP gave up already ? Too bad I found this project interesting
Lady Jessica · 2024-05-25 03:16
>>39184
Mwmbl is closest I can find at the moment.
https://mwmbl.org/
https://github.com/mwmbl
Donna Hawthorne · 2024-07-01 20:13
>>39184
Sorry, I got side-tracked by other things. Taking months long breaks on things isn't unusual for me though.