reddit / emacs

notmuch like mode to search through pdfs

StillerHarpo · 2022-01-29 07:10
I have a bunch of searchable pdfs. Is there something which works simular to notmuch (especially automatic tagging) where I can search through all my pdfs in emacs.
ftrx · 2022-01-29 09:37
If you means tags you add, searching them *and them only* no full-text, you can org-attach them => the pdf title is the heading title, you add any tag(s) you like OR for a more finer/less "live" but more effective search you add a drawer and access results with org-ql. The heading title can be itself a link to the pdf so clicking on it gives you the pdf view, org-mode there act just as a kind of "access-middleware". If you mean full-text search, well there personal tags can be a bit of an issue, AFAIK you have two option: - Recoll (a small wrapper with various UI around solr), used interactively in Emacs with `counsel-recoll` in the same way of `counsel-notmuch` if you use it; - rga (RipGrep-all) with rg package for the Emacs part. Recoll demand an index so you have to setup it up-front, occupy a bit of disk space depending on the size of pdfs library size, rga demand storage I/O and CPU, again depending on how frequently you search through the pdf lib and how big it is and your hw spec you might prefer one or the other. Recoll do offer a query language other than mere full-text search, but that's a bit like google search operators, something you rarely use, rg offer regexp witch can be used to elicit similar outcome. A third option, I do not suggest much due to the software involved is using Zotero, it's FLOSS but a monster embedding Firefox and planning to switch to Electron in an unspecified future. Zotero support collecting pdfs, indexing them, tagging them etc and can be used via Emacs through zotxt package, it's limited, not anything is exposed, but does work. The downside it the wight and nature of Zotero + the need to have it running in background, it's a GUI, not a daemon like Recoll... BTW welcome to the club of people who look for search/narrow access UIs instead of fixed hierarchies and their relevant UI, personally it's few years I say that model is the future but seems to get very little traction even if the number of similar request goes up a bit :-)
_viz_ · 2022-01-29 13:17
Apart from the well detailed answer by ftrx, there's also pdf-tool's pdf-occur which is powerful when you combine it with virtual PDFs. I recently used that combo to search for a dialogue in a series. I typed M-x pdf-virtual-buffer-create RET C-c C-c M-s o <string> in the directory where I store all the volumes, and within a few minutes, I found what I needed. BTW, pdf-occur uses PCRE by default (if that matters to you).
EndlessApoptosis · 2022-01-29 23:29
If you use embark and vertico, match some regexp in a directory root full of pdfs (I do this for exam facsimile, all in their folder) export to a custom fired buffer, mark them all and call ```pdf-occur-dired-do-search```. Emacs is going to show an occur buffer (with the usual occur keybindings), but it's going to search into the requested documents only.
_viz_ · 2022-01-30 03:55
Sure, that works as well. I prefer virtual PDFs because you can also select the range of pages you search in.
StillerHarpo · 2022-01-30 06:50
I'm not suere what you mean by org-attach the tags. Do you mean org-attach the tags and/or the pdf? I only cant org-attach to an org file not an pdf, right?
ftrx · 2022-01-30 10:28
The pdf, you can org-attach anything you can store to your filesystem: org-attach just create a cache-like tree formed by at minimum two level, i.e. * An example heading :ATTACH: :PROPERTIES: :ID: f6a89f37-2cf7-4926-9449-2eae38f83ba5 :END: Some contents result in something like `/your/attach/root/f6/a89f37-2cf7-4926-9449-2eae38f83ba5/...` where ... is your content, any file or ALSO tree, files, directories, symlinks etc The idea is: do forget to maintain a file taxonomy yourself, just use org-mode as a file-manager, with the access layer made of org-mode heading and perhaps org-roam-node-fine as the search&narrow UI, doing so means that your files can be accessed like a generic graph instead of a hierarchy. The benefit of small attached tree happen when you need your files outside org-mode, like a git repo, a series of docs you want to have in the same place for same usage etc. Also, while I do not consider such approach, you can org-attach by symlink (if your fs support them) or by copy, so you still have your files in your home taxonomy. Honestly in the recent past I hesitate much destroying my decade-old super-curated home taxonomy, I start "a bit at a time" with some mistakes (no global attach dir from start, anyway easy to solve since ID are unique and no link break) and some "foot in two shoes" with just a bunch of low-importance files org-attached, but after a bit of time I've jumped the ship and now almost all my files (just some profile dirs for firefox and few others are not org-attached) are handled like that. When you have files attached and accessed via org-roam you do not need to tag the pdf, you can select your files directly via org-roam-node-find, like any other heading, or query them via org-ql eliciting various collection of files together. The org-ql issue is that's no full-text search in pdfs, you have to design a meaningful personal library catalog and write your query on top of it, and of course be consistent in labeling or your queries will be ineffective. Essentially is the same thing Zotero do: internally Zotero copy files in a cache-like tree under a common root. All files by default are presented as a list in a global library than you can create "virtual" directories and sort them in one or many, they are not really moved, directories in Zotero are just like GMail labels or org-mode org-attach links copied around. org-mode tags are Zotero tags. org-ql is Zotero search tool with small differences: org-ql offer better sophistication in queries BUT do not also offer full text search in file contents. Recoll/rga do. And nothing stop you to choose both: you have all your pdfs org-attached and sorted/accessed via org-mode/org-roam, when needed you have counsel-recoll for full text search them, or rga at your option. The key to visualize such system is decoupling the visual representation of files remembering that on-disk there are no files and folders but just files and links, some files are directories or "special files that link other files/directories" and the representation in GUI env. like suspended folders and paper sheets is just because when invented files were sheets in people's mind and folders a way to archive/access them. Actually on-disk they are not. On disk they are just storage space structured in "content" (files), "content metadata" (files metadata), "structure metadata and format" (on-disk fs structure, directory concept). In visual terms we have no reasons to keep imaging physical sheets of paper organized in suspended folders, drawers etc. Search-based *information* access is a thing since decades, it's most well know systems is named the web, you can do the same locally with org-mode instead of html pages (org-mode notes) and WebVM [1] and as the web now is not only text with at maximum some images org-mode is not limited to org-mode itself. That's a kind-of storage middlewire, a DBMS if you prefer. When you visualize that and do some test like create few notes attach some files and start looking for them via org-roam, find them linked in the heading if will be clear :-) Oh, remember that links in org-mode can be many things, non just links to other notes, to files, to specific kind of files with extras (like a page or anchor in a pdf), to mail (via notmuch/mu4e/gnus) etc but also elisp sexps to be run on click, a trivial example can be `[[elisp:(start-process-shell-command "firefox" nil "firefox -no-remote ~/org/data/fa/4c52b6-72f2-4f8b-8989-b815059f4945/ArchivedWePage.html")][open with firefox]]` and many more, for instance a link to create a new (sub)heading, one to compile a LaTeX document, one to start an impromptu backup, one to modify Emacs windows/frame for instance during a live presentation. Unfortunately those things are mostly ignored because people coming to Emacs from modern software that can't do anything like that so many, even skilled devs, do not even think about such usages. A nice demo, while light and generic is https://youtu.be/B6jfrrwR10k another a bit strange https://youtu.be/5ViUBaarsbw but the key is trying and make your environment, you can't really imaging it before coming from modern tools :D [1] modern monsters improperly named browsers (like file managers) for legacy reasons...