(untitled)

Web Scraping, Data Mining, and Data Science, general automation and bots thread. What are you scraping? What are you analyzing? What are you automating? What are you training?

old backburner project but trying to get my own useable websearch engine going. Especially with duckduckgo recently pledging to break itself, not that it hasn't been going downhill in recent years anyway, seems like a nice thing to have.

>>8463 It's really sad that there are only a handful of independent search engines. Yes there is a number of so-called indie or privacy-focused search engines, but they all just leech off the big 2 (Google and Bing).* DuckDuckGo is just a proxy for Bing. We need independent crawlers, because even without all of Google's and Bing's soykafty practices, a search engine is the window to the web and today we have a severe lack of diversity of perspectives. I don't think it's possible to compete against Google at a general search engine, but who needs a general search engine that second-guesses you all the time because it's so general? I want specialized search engines. A toolbox of them. Wiby.me is cool. Search.marginalia.nu is genuinely useful. Mojeek is nice as well even if the results kinda suck. * https://searchenginemap.com

last time I needed to use some wikipedia data (wiktionary to be more specific) I had this fuarrrking soykaf file in a weird format they have, I regexed the whole soykaf I wish I had more skills with scraping (any recs?) but I did work 6 month into RPA which kinda does the same thing, and it was enjoyable. Captchas were my biggest enemy with RPA though, and with the few scraping softwares I use, it seems Cloudflare is another one >>8464 kinda off-topic but the recent Brave search engine manages to have a very impressive accuracy (to the point where it compares to google imo). Just use searx and aggregate whatever you want

>>8462 I found a book that I liked: https://automatetheboringstuff.com/ That book is a good starting point.

>>8462 I came to check on the state of this: >>8244 I want to be able to read Discord chats, without making an account. This seems promising: >>8362 There has been no new work on this lately, however. >>8464 https://search.marginalia.nu

>>8501 I think DiscordChatExporter is the only reliable scraper for discord.

>>8503 Sorry I didn't mean "the only". It's the only I've tried.

>>8464 Extensive, updated examination: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html

Currently learning reverse engineering so I can retrieve data directly from applications without limit.

I've been using OpenAI's whisper to transcribe a podcast and make it greppable.

>>8462 Trying to learn Tableau & Power BI but can't find Power BI and the tutorial data is not working so I will work on getting better at Excel for now. I am also scraping data from Yahoo finance for stock picking and some... other stuff. We shall see how this goes.

>>8462 ticket master, working on designing a system that will assist with purchasing high demand tickets. nothing illegal though, the system will only enable a desirable queue position. anyone have resources or experience in leveraging AI in this field? one possibility that comes to mind would be an AI that can dynamically determine selectors for you -- on the fly. but im sure there are lots of use cases in this space.

>>8503 Does it scrape pictures by any chance?