rgrep url regex

this is the second time i am ending up in this rabbit hole. all the searching suggest that this is a complex topic, yet i find it puzzling why there is not a standard solution in a time webdev dominates computing. is there a package, function, a phrase, an operator [[:url:]], something obvious i keep missing? do you have that magic string i can feed into "rgrep"? thank you!

I do not know if that is so easy to capture, accurately, with a regexp. I am not even sure the RFC for _all_ uniform resource locators are even regular! Email addresses aren't. Can you not just search for `https?://`?

thank you! i am trying to extract all the urls in a directory. it is a large dataset and urls comes in all forms.

Maybe see what Emacs uses for this: * `thingatpt.el`: variables `thing-at-point-beginning-of-url-regexp`, `thing-at-point-url-path-regexp`, `thing-at-point-short-url-regexp`, `thing-at-point-uri-schemes`, `thing-at-point-markedup-url-regexp`. And see the functions, starting with `thing-at-point-url-at-point`. * `ffap.el`: Variable `ffap-url-regexp` and code that uses it. * `url-util.el`: Function `url-get-url-at-point`. * ... (there may be more/different)

One approach is to create a script that extracts the URLs in a sufficiently robust way (using something like BeautifulSoup or whatever works for your used case) and produces an output in the same format as `grep -Hn`. You can then use this from `M-x grep`: M-x grep myurlgrepscript ... RET I did something similar to plug recoll into pdfgrep.

If you do end up needing to build one yourself, check out the `rx` macro built into Emacs. http://howardism.org/Technical/Emacs/eshell-why.html has a section about halfway down showing how to convert emacs-compatible rx's to PCRE which is what most greps use. You can build a url regexp that way.

eshell is a beauty, i would like to get rid of that second workspace in i3 that i reserved for screen/tmux and completely switch to eshell, but there are always a few showstoppers. that looks like a great alternative if we can't overload regexp operators, say binding [[:url:]] to a function called string-url-p

thank you! they all pass my test urls. i checked all -url- related functions with M-x but not the elisp functions via M-:

> PCRE which is what most greps use Standard (POSIX) `grep` utilities use BRE regexps by default, with ERE as an option. I think it would be *highly* unusual for any system to have a `grep` which deviated from that. Any PCRE support would be a relatively recent development (or certainly for GNU grep, which calls the option "experimental" in version 3.1, but mostly stable by the sounds of things in the latest version 3.8). BRE is similar to (but also not the same as) Elisp regexps.