Web crawling thread

My boss gave me excel file with 10000 different websites. For now i have to crawl all of them - so for now just retrieve all subpages of each one of them and save HTML to db. Using Python Scrapy library for it. First problem i got - some websites are impossible to crawl - it only returns URL to main page, i thought its because of robot.txt but then i read robot.txt is just "suggestion" and crawler may decide it to ignore it and they cant do soykaf with it. But i ordered crawler to ignore it and its still not working. Then i googled script to crawl this certain webpage and i copypasted some headers from this script to mine - like 'Accept', 'Refferrer' - ok bit better, but still not crawls everything? It just retrieves all language version of main page but what about "About" page for example which i see clearly in "View Source" Html of home page? Its part of static page, its not dynamic, so why Scrapy has problem with retrieving link to this "About" page under the button? Another problem is AJAX calls or dynamic pages with pagination that you have to click "see more" button -etc. Scrapy does not handle AJAX calls well right? So how to deal with this? I dont want to use custom selectors, too many websites anyway. Also idea of adding custom headers to each website jsut to circumvent its protections - also not sure if its good idea. Do you guys know some "tips&tricks" for scraping, how to easily deal with most stubborn of websites? Easy way to automate this? If its impossible i need to at least save some statistics, like what percentage of pages were impossible to fully scrap, but again, how? I saw example in scrapy of collecting statistics using HTTP status codes, but what if all received status codes are 200, no errors in logs, nothing, all went fine, except we skipped 90% of given website?

Welcome to the modern web. Your options are to run javascript or roll everything by hand. Most python people will tell you to use Selenium but Playwright is best in class. You're going to run into CAPTCHAs and the best way to overcome this is a mix of residential IP proxies and CAPTCHA solving services. It gets complicated. Good luck. Literal pro tip: just because someone gives you a list of 10000 things doesn't mean you need to deliver 10000 things. You can deliver part of it and discuss the rest.

god i would love a job like this whats your job title?

>>36482 >Do you guys know some "tips&tricks" for scraping, use the sitemap which every website should have. you can always get that with curl since its just an xml file. then use puppeteer to crawl the pages you need. >like what percentage of pages were impossible to fully scrap none are. puppeteer will get everything. its a headless google chrome instance. make sure your code keeps it running instead of launching it for every individual url which is inefficient.

i had a similar problem and i think you should either use puppeteer or what i did was i made an extension which took all the sites from one file reached them and extracted the data (i know its a few extra steps but some dom elements are too hard to extract)

>>36485 >>36486 yeah i know about Puppetter, its based but i read its slower than Scrapy and anyway there is same problem with some websites blocking rquests, you can see in logs something like "Request failed, retrying" - always for same set of websites. I would however consider using it instead of Scrapy but its not my decision, boss says Scrapy is faster. Thank you for idea with sitemap page. >>36486 what do you mean "extension"? Retrieving HTML itself is not a problem i think, problem is reaching all subpages and you know, how to validate/test that everything was retrieved. Next step was using some ML model to summarize what is on the page, eventually what has changed since last scraping or something. We have to create our own model i think because when i calculate cost using GPT API - that may be way too expensive, for example one HTML page takes 12000 tokens and you need to split it into smaller requests to GPT and also how do you know which part of website is relevant? To strip it from irrelevant things and only feed model with text that relevant? Thinking more about it, it may be stupid idea to save entire HTML instead of just text page content - how to train NLP model on HTML? >>36484 just software engineer, its one of projects in company i was involved in, some managers create some weird innitiatives like these that probably wont even work and wont generate any money just to get recognition, get promotion etc

If you want to avoid CAPTCHAs et al but don't want to use external services, https://github.com/ultrafunkamsterdam/undetected-chromedriver works amazingly

>>36487 >logs something like "Request failed, retrying" in a list of ~1000 websites, some of them are doomed to be dead links already. have you checked if you can even reach that site normally? it's probably just down.

welcome to the world, whre every soykafty website uses a bloated js framework to render something you could have wrote in 5 lines of html... use a headless browser instead of directly making HTTP requests to a server

i always wondered what'd it be like to turn the internet on its head instead of a large company indexing everything it can and ranking it, what if users created/rented/sold "agents" which could be told what the user was interested and they would go fetch it so over some period you would have "threads" where your crawlers would bring you links to stuff that is relevant to your query it wouldn't be instant but it's much better than a search engine and much more decentralized by its nature, what's more actual websites that people are interested in would profit more than SEO trash because you can't game a horde of constantly evolving bots like you can game company/large organization

>>36907 How would you trust an "agent" without a centralized reputation system? It seems like an easy system to abuse. Bittorrent works because content checksums are distributed on a sidechannel so a user can verify content chunks and ban malicious peers while he is downloading it.

>>36908 i mean an "agent" is another word for a crawler script a sequence of standardized operations ran by e.g. selenium with some custom parameters and conditions so an "agent" i write is like a cucumber or even python/selenium script and i host it someplace and let you run it with your parameters and you'd have a webpage where the agent would - whenever it finds something - post new data it finds according to your parameters like smart contracts but as a substitute for general-purpose search engines

>>36908 >>36907 >>37186 so instead of paginated results pages you'd get something like a bunch of forum threads (each for their own "subject", the search query) where - as they find results - these agents would create something like posts to paint a clearer picture, imagine a forum where you could start a thread and ask for links relating to "restaurants in paris" (just like would query a seach engine) and other users would reply with appropriate links now imagine that same forum but instead of other users it's crawler scripts

I just got a job in a trading card game store. My job is to automate the process of getting updated price of cards on websites and on our competitor. They did all of this soykaf manually, one by one on the spreadsheet. The problem is that this is my first job. Everyone is tech illiterate (my boss was impressed when they saw a html file open), so I have no one to guide me through the best practices of development this kind of programs. I am using BeautifulSoup and manage to get the names and prices of a couple cards. Because I know that I can get derezzed if I send a bunch of requests I decided to just save manually the html pages and put then into a folder. I am planning of implements a rotating proxy using some library that I found on github that checks free proxies. The things I need some direction is: 1- I need to make the program user friendly. They never saw a terminal before. I am planning on using streamlit for the interface and use pyinstaller to make some executable for Mac OS 2- I will use the the google spreadsheet api to convert the data into a csv so then use something they are familiar with 3- I am opening a file with the 'with open' syntax and putting everything inside of that block, and everytime I get a new price I insert in the file. Is this the best practice or should I just put everything into a list and write into a file at the end? My background is purely academic, so I have no idea how to structure those things. Usually in my research, and I guess to some extend all the scientific field, just slap some code without worrying about the maintainability. If anyone could offer some source where I could learn the best practices or any tips for my problems I would be very grateful.

>>37904 >I will use the the google spreadsheet api to convert the data into a csv Is there a reason to do this instead of just using a python library or even string concatenation for putting together spreadsheets? Are your coworkers already using Google Sheets for this or something? Also, if this is a recurring chore where someone would go through and update all the prices that have changed, is there any real need for an interface or adjustable options? Why not just have this be a batch script that can be clicked on to run the python script that updates the csv? Not specific to your task but I have a technique I want to share: in order to automatically support multiple websites and to be resistant to website changes breaking your scraping script, find information without relying on id/classname. For instance if you are looking to collect information on each of a list of items, sort elements by child count (maybe also with a filter for minimum text content length) to find the one where everything branches out, then iterate the child elements applying regex to their textContent, looking at the tag attributes, stuff like that which does not depend on the variable names the devs or javascript compilers have chosen.

>>37906 Currently I am writing in a txt file. Which will be imported to Google sheets by dragging and dropping the file in the browser. I just wanted to make it appear in a folder in their drive. The idea is that they give the set of cards that they want information, so maybe it's the new set of pokemon cards, which is described by a code, (ABC4). So I need an interface so they can insert the set code. I will try to implement your technique. I am relying on the id/class so I guess it's a matter of time until something breaks. Is there any other general tip for projects like this? At the very least I am using a version control software. But besides I am just winging it.

So any advice for a newbie who's interested in bots, webscraping and automated systems? The only experience i have rn is from making a video game bot with pyautogui, and a simple site scraper with scrapy. Honestly im tired of being stuck in the tutorial and beginners guide purgatory and want to learn to actually improve. Also how the hell do you people focus on your textbooks and stuff without getting distracted, ive been trying to read C- A modern Approach for months and i havent learnt seemingly anything of value.

>>40432 I am only mildly experienced in web scraping so none of my advice is going to be technical. Some people say "Knowledge retention is a meme" and while I wouldn't go that far you need to ask yourself if you are learning the right stuff. Low-level (low level of abstraction- aka detailed) programming is not going to help you write a scraper in a high-level programming language like python. You should just try to do stuff that should be possible and if you can't do it, then research until you figure out how. No time spent learning things that you don't need to know, a lot of time spent learning what things exist, finding new keywords from old ones. A lot of semi-serendipity, like finding blackhatworld so you can understand the inputs and outputs people work with. (captcha solving, sms verification, accounts, reputation, etc) I've read some "web scraping" books, and they were almost always useless. Fast way to read them and get just as much value: read the table of contents, write down the names of any tools. Close book. Find documentation for every tool online.

>>40435 What would you suggest i try out, scrapy seems very limited in what it can do and i really couldnt understand how to setup selenium

>>40441 i would suggest you figure out how to set up selenium. What are you using? windows?

>>40442 Arch btw, though i was using nix back then, so i suppose i should try again. I guess trying to read the offical documentation is best

>>40444 oh, same. not sure what there is to set up. i just did ```pip install --break-system-packages selenium ``` and used it.

>>37904 >I just got a job in a trading card game store. My job is to automate the process of getting updated price of cards on websites and on our competitor. Mate... I'm so jealous. I'd do this for minimum wage. Hell, I already do this for fun. How do you find a job like this? I'd love to know. Were you just referred by a friend or something?

#!/usr/bin/env perl use strict; use warnings; use Data::Dump 'pp'; use Path::Tiny; use Time::Piece; use Try::Tiny; use URI; use Web::Scraper; use JSON::PP; our $J = JSON::PP->new->utf8->indent->indent_length(2); # Save data as JSON to a file. sub save_json { my $path = shift; my $data = shift; my $json = $J->encode($data); path($path)->spew($json); } sub neighborhood { my $text = shift; $text =~ /in\s+([A-Z][a-z]+)/; my $neighborhood = $1; } sub district { my $text = shift; $text =~ /in\s+([A-Z][a-z]+)([A-Z][a-z].+),/; my $district = $2; } sub city { my $text = shift; my @parts = split(/,/, $text); my $city = pop @parts; $city =~ s/^\s*//; $city; } sub number { my $text = shift; $text =~ /([0123456789,]+)/; my $num = $1; $num =~ s/,//g; $num + 0; } sub date { my $text = shift; $text =~ s/\s*Available From\s*//; $text =~ s/\s*$//; my $d = try { Time::Piece->strptime($text, '%b %d, %Y'); } catch { warn "$_"; return undef; }; if ($d) { $d->date; } else { $text; } } my $apartments = scraper { process 'div.property-listing', 'apartments[]' => scraper { process 'div.listing-title a', 'id' => ['@href', \&number]; process 'div.listing-title a', 'link' => ['@href', sub { "https://realestate.co.jp$_[0]" }]; process 'div.listing-title span.text-semi-strong', 'type' => 'TEXT'; process 'div.listing-title br + span', addr_neighborhood => ['TEXT', \&neighborhood]; process 'div.listing-title br + span', addr_district => ['TEXT', \&district]; process 'div.listing-title br + span', addr_city => ['TEXT', \&city]; process 'div.listing-right-col div.listing-item + div.listing-item', monthly_cost => ['TEXT', \&number]; process 'div.listing-right-col div.listing-item + div.listing-item + div.listing-item', available_from => ['TEXT', \&date]; }; }; sub main { my $url_template = "https://realestate.co.jp/en/rent?prefecture=&city=&district=&min_price=&max_price=&min_meter=&rooms=&distance_station=&agent_id=2930&building_type=&building_age=&updated_within=&transaction_type=&short_term_ok=1&order=&page=%d&search=Search"; my @list; for my $page (1..32) { printf("## page %02d\n", $page); my $url = sprintf($url_template, $page); my $html = `curl --silent '$url'`; my $res = $apartments->scrape($html); for my $item (@{$res->{apartments}}) { printf("%16s %15s %15s\n", $item->{addr_city}, $item->{type}, sprintf("¥%d", $item->{monthly_cost})); } # post-process $res # push on to list push @list, @{$res->{apartments}}; } my $now = localtime(); my $filename = sprintf("apartments.%s.json", $now->date); save_json($filename, \@list); } unless (caller) { main(@ARGV) } - Yesterday, I did a little bit of casual scraping, and I felt like sharing the results. - I was frustrated with how slow it was to browse the search results at https://realestate.co.jp/en/rent . - I used Web::Scraper to turn multiple pages of search results into a JSON document. - https://metacpan.org/pod/Web::Scraper - I hard-coded the search url and the number of pages, but it could be generalized with a little effort. - In another post in another thread, I'll show how I used org-mode and jq to look through the data locally.

>>41961 >- In another post in another thread, I'll show how I used org-mode and jq to look through the data locally. Org + jq: >>41967 Updated Script: https://pastebin.com/NA3xqdK1

>>37187 >>37186 >>36907 i actually built this and it works! i've throttled it a lot here (pic rel) so it's very slow i gave it a primitive storage to keep pages it visits and to keep previous queries so it doesn't start from scratch every time; both meaning the more of the web it crawls the more precise it'll end up i want to make the UI a textboard next, and write a specification document i wanna make this a standard of some kind

>>42388 source code https://github.com/xapoc/wwwps nb

>>36484 You want to be a paid wage slave? It's easy, just move to India or some third world country and sign a contract with a company that'll give you a role like "social media manager/specialist". They'll look for people with backgrounds in marketing, programming and data science. They make web crawlers to see on which websites a business has been mentioned and with the help of AI classify the mentions into positive, negative or neutral. Once negative content is detected, they make the Pakis or a bot create an account, engage with other content for some time to sound reputable and then change the narrative.

Stop scraping sites. fuarrrking leeches. There's too many jobs where brainless monkeys bang rocks together until something happens. You're a software dev? I doubt it... are all software devs this useless? Whatever your working on for your "boss" is going to make the net and the world a worst place no doubt.

from my experience, many websites simply cannot be crawled in a systematic manner depending on how specific your resolution requirements are. however, if you're looking just for subpages, it shouldddddd hopefully be a traversal for urls. however urls are kinda fake so idk. when it comes to returning URL to main page there are ways to get around this IIRC. For example there's usually a JSON or other API backing up the page, sometimes some content negotiation step is required, sometimes there's XML sitemaps. There's usually a way to get around it. scrapy should be enough, I've never had to use beautifulsoup for such a purpose. Selenium also exists as a driver for some cases IIRC? Source: I used to do a lot of scraping as a junior yonks ago.

>>42418 9 bazillion line unorganised singular file. The true power of a Python dev

>>42568 >unorganized do better if you can do better it's a 700 line python module, consisting of 1 class, 1 function that uses said class in a loop, and 3 Flask endpoints (to start, stop, get results) nb