Recommended way to search / replace based on surrounding context in *large* buffer?

Hello All. I've been using emacs (doom and evil mode) for about a year now, and am just getting to the point where I feel like I'm beginning to understand (enough) of it to be useful. I suspect I can do what I want, but haven't found the magic combination yet. I'm hoping that someone more experienced can point me in the right direction. My current problem: I have a large text buffer (Beancount text based accounting file -- 3mb+), where I wish to change \*some\* entries to re-categorize expenses, based on the contents of the previous line or two. I need to change anywhere from 1 up to maybe 200 entries at a time (out of maybe 30,000 entries.) Usually too many to do manually, but there are enough variations in the transactions that I'm matching against that blindly doing a global regexp/replace leads to some accidental changes too. What I want is something like a multi-line occur buffer showing the matching transactions and several lines of context, so I can do a global search and replace on just the matching transactions after making the occur buffer writable, and see them all in once place without scanning through 60,000+ lines to make sure everything looks good. Unfortunately, I can't seem to make this work using occur-edit mode unless the occur buffer consists of single line matches. I've tried both prefixing the "occur" command to get N lines of context, as well as using a multiline regex for the occur match to only match the needed 2-3 line segments I wish edit. This gets me the desired occur buffer with enough context to make the needed changes, but switching to occur-edit mode doesn't actually edit the source buffer unless I'm only changing the first line of each match. (Which I will probably never be doing.) Changes to lines 2 or 3 of each match are ignored. I don't know if this is just a bug with my particular configuration, or if occur-edit buffers don't handle multiple lines by design. Other things I've tried: "query-replace" -- can't seem to get it to handle newlines. Always comes back with "Replaced 0 occurrences". Even if it worked, I don't really relish the idea of having to cycle through several hundred matches to check each one. "replace-regexp" does work using the same regexp that failed using "query-replace". With newlines and matching groups I can do the needed context based replacements, but it's pretty clunky to edit each regexp, and it's blindly global, so if I'm overly broad on the match, I may introduce errors that are hard to catch I've seen references to either helm-swoop, or icicles being able to do some sort of context based narrowing of scope, but I haven't seen any good examples, and would rather not install and learn something new if I've already got default functions that will work. I haven't tried any sort of dired or projectile based searches with wgrep buffers, as it appears they're all geared to multiple file searches, and I only want to work on the one buffer. (I'm sure I could work around that if necessary, but that didn't seem like a good fit.) Thanks for reading so far. Any suggestions?

I don’t know of anything that exists, but it wouldn’t be too many lines of elisp to do it. You could programmatically narrow to a region that you move ahead or regexp-replace with a boundary that you move along with goto-char. You would like a screenful at a time?

Use a keyboard macro. 1. Start recording a macro with `F3`. 2. Use `C-s` to search to the next thing you want to edit "near". Let's call it `"foobar"`. 3. Type `C-x q`. This inserts a kmacro marker that stops execution when you run it and asks if you wish to proceed when you finally run the macro. 4. Make whatever programmatic edits you wish to make, as though that entry met whatever you criteria you have for changing stuff. 5. Stop macro recording with `F4`. 6. Run the macro with `C-x e`. It'll jump to the next match (`"foobar"`) and give you a prompt. Pressing `N` stops the macro dead and skips anything after the macro marker leaving `"foobar"` untouched. Press `Y` to make the changes. Press `C-r` to enter recursive edit, if you like, to make manual edits. `C-M-c` returns you to the macro. 7. Regardless of the option, `e` repeats the macro. Eventually you'll work your way through your list. That should do it.

> there are enough variations in the transactions that I'm matching against that blindly doing a global regexp/replace leads to some accidental changes too. We don't really "blindly match" regexes, but rather exactly. Your problem is that your context is spread over multiple lines. Matching patterns across several lines is a bit more involved in Emacs. You can read more on [Emacs Wiki](https://www.emacswiki.org/emacs/MultilineRegexp) or [on SX](https://emacs.stackexchange.com/questions/44624/emacs-regex-with-multiple-match-for-text-in-multi-line-buffer), maybe [this one](https://emacs.stackexchange.com/questions/15061/can-occur-match-phrases-that-stretch-across-multiple-lines) could be what you need? I think at least org-mode has some multiline regexes you could take a peek at (sources are included with your Emacs). > Even if it worked, I don't really relish the idea of having to cycle through several hundred matches to check each one. > Any suggestions? I personally would write a simple defun instead of trying to figure out which package does it for me. Either construct a regex that will match over multiple lines and use groups to find which case you have matched. I think multiline regex with groups can get quite ugly and tedious, so I would instead write a defun that will match in stages with few simpler regexes. First match a line one, and then if that is matched, a line two, and then finally the thing itself, since you wrote that it will depend on the previous line or two. You would be matching from "out" to "inside" if that says anything. Exactly how you would do it, depends on how your file is structured. Let say you are matching your item based on *one* line before, here is how it *might* look, just as an illustration: (defun my-replace-fn (new-text) (while (re-search-forward "my-first-line-regex" nil t) (forward-line) (when (re-search-forward "my-item-regex" (line-end-position) t) (replace-match new-text)))) The structure of your file is different, so you might need something more elaborate, but it shouldn't be much more complicated. You would run it with M-: instead of M-x.

> based on the contents of the previous line or two You want to search only certain contexts of your buffer, right? And you can specify those contexts pretty easily, e.g., using a regexp? If so, here are two ways to do that: 1. You can use [Icicles search](https://www.emacswiki.org/emacs/Icicles_-_Search_Commands%2c_Overview) to do exactly that. You give it a regexp that defines the contexts to search, and then you search within those. 2. You can use [Isearch+](https://www.emacswiki.org/emacs/IsearchPlus) command `isearchp-regexp-context-search`, available from library [`isearch-prop.el`](https://www.emacswiki.org/emacs/download/isearch-prop.el). Again, you provide a context-defining regexp, and then you search (Isearch) only within those contexts. For the Isearch approach you need only download the one file. Here's the `isearchp-regexp-context-search`doc string: > `isearchp-regexp-context-search` is an interactive compiled Lisp function in [`isearch-prop.el`](https://www.emacswiki.org/emacs/download/isearch-prop.el). > > `(isearchp-regexp-context-search REUSE BEG END IGNORED REGEXP &optional PREDICATE ACTION)` > > Search within search contexts that are defined by a regexp. > > If `isearchp-complement-domain-p` is non-`nil` then search **outside** the contexts defined by the regexp. (Use `M-= ~` during Isearch to toggle this variable. > > Search is limited to the region if it is active. > > If you do not use a prefix arg (or non-interactively if argument `REUSE` is `nil`), or you have not previously used this command or similar commands, then you are prompted for the following: > > * the context-defining regexp > * a predicate (Boolean function) that takes these arguments: > * the search-hit string (what matches the regexp or chosen subgroup) > * a marker at the end of the search-context > > Only the search hits for which the predicate is satisfied are retained. > > If the regexp has subgroups then you are prompted for the subgroup to use to define the contexts. Subgroup 0 means use the entire regexp match as a context. Subgroup 1 means use the first regexp group match as a context. And so on. > > The search contexts are marked in the buffer using a text property. You can also search them using `isearch-property-*` commands. The text property used is the symbol whose name is the regexp. The property value is a cons whose car is the regexp (a string) and whose cdr is `PREDICATE`. > > If you use a prefix arg (or non-interactively if argument `REUSE` is non-`nil`), and you have previously used this command or similar commands, then this command acts like `isearchp-property-forward` with a plain prefix arg. That is, it searches for the text property or overlay property that you last used in an Isearch command that searches or applies such properties. See `isearchp-property-forward` for more information. > > Non-interactively, `_IGNORED` is ignored, and the other arguments are as in `isearchp-add-regexp-as-property`. To help you better see the contexts, the text outside the contexts is dimmed slightly, by default (option `isearchp-dim-outside-search-area-flag` controls this). If you instead want to search the *complement* of the defined contexts, that is, the text outside them, they just use `M-= ~` any time during Isearch (it's a toggle). (That also flips the dimming, i.e., it dims the original contexts defined by your regexp).

> This gets me the desired occur buffer with enough context to make the needed changes, but switching to occur-edit mode doesn't actually edit the source buffer unless I'm only changing the first line of each match ... Changes to lines 2 or 3 of each match are ignored ... I don't know if this is just a bug with my particular configuration, or if occur-edit buffers don't handle multiple lines by design. Please `M-x report-emacs-bug` to request that `occur-edit-mode` support changes to context lines. At present `occur-after-change-function` relies upon `(get-text-property (line-beginning-position) 'occur-target)` returning a marker for the original buffer in the original buffer, but that's only true for the occur lines with line numbers; context lines do not work. (Please include this explanation in the bug report, for clarity.) I don't see why this couldn't be implemented, though.

Thank you everyone for the thoughtful suggestions. I apologize for not following up sooner (Just finished the tax stuff that prompted my initial problem / request.) I was able to make the changes I needed last week using multiline regexes and the "replace-regexp" command I mentioned in my earlier post. (It's fast, but without the query option, kinda risky, and caused a few mistakes I had to go back and fix.) I had some time last night, and looked at some of the other search options available in the current Doom emacs setup. (Vertico, consult, embark packages). I still haven't found a buffer specific search, but I did figure out how to narrow the default "Search Project" (calls ripgrep eventually via "vertico/project-search"). I could search for the initial matching transaction I wanted, and get two trailing lines of context by appending "-- -A2" and limit the scope of the search to the file by appending "-- -g*FILENAME*". (Could be combined to "-- -A2 -g*FILENAME*"). This resulted in a buffer containing what I was looking for: All of the expense categories tied to the transaction(s) I was interested in, condensed into just dozens to hundreds of lines, rather than scattered all over 60,000+ lines of the parent buffer. This buffer could then be exported to a writeable wgrep buffer by calling "embark-export-write" (or C-c C-e). From there, it's trivial to replace the categories I was interested in, see all of the changes to catch any errors, then write the changes back with a "C-c C-c" Success!