completing-read, search also in annotations

Can someone please tell me how can I force `#'completing-read` to search not only in the displayed rows, but also within annotations? Consider a trivial example such as this: (let* ((coll '(("Affenpinscher" . "Loyal and amusing") ("Akita" . "Ancient Japanese") ("Bulldog" . "Kind but courageous") ("Caucasian Shepherd" . "Serious guarding breed") ("Miniature Schnauzer" . "Long-lived and low-shedding"))) (completion-extra-properties '(:annotation-function (lambda (k) (let ((desc (alist-get k minibuffer-completion-table nil nil #'string=))) (format "\n\t%s" desc)))))) (completing-read "Select a breed: " coll)) Evaling this snippet would display names of the dog breeds, along with annotated descriptions. And if I type "sheph", it would correctly narrow the search to "Caucasian Shepherd". But I also want to be able to type "long" and to find "Miniature Schnauzer", but sadly, it doesn't work that way. How can I change that?

This is pretty much the entire reason that annotations (and the newer affixations) exist: to add something to the candidate display **without** that something being matched against input. If you're providing the completion candidates yourself, just add the annotation text to the candidate text. If it's a source you don't control, I guess you'll need advice around `completing-read` to pre-process the candidates to do this.

I'm pretty sure you're mistaken. Just think about it, If I concat the strings, then how am I gonna filter out the exact entity after the selection? e.g., a more practical example: if you have a bunch of book titles and URLs to buy them. Besides, I like how annotations are displayed differently than the main collection rows. I think I need to learn more about "Programmed Completion" described here: https://www.gnu.org/software/emacs/manual/html_node/elisp/Programmed-Completion.html

> how am I gonna filter out the exact entity after the selection You would make an alist or a hash table before completion, and afterwards look up the original string using the composite string that comes out of completion. > I like how annotations are displayed differently than the main collection rows. You can still style the substrings that came from the annotation differently than the rest of the candidate. (concat candidate " " (propertize annotation 'face 'completions-annotations))

https://emacs.stackexchange.com/q/74547

No, you're suggesting hacks for no good reason. As I correctly guessed and pointed out, it has to be done via completing-read collection function. Some basic, dirty experimentation got me to something like this: (defun dogs-filter (seq) (lambda (str pred flag) (if (string-blank-p str) (all-completions str seq) (all-completions str (lambda (s _ _) (seq-map #'car (seq-filter (lambda (x) (unless (string-blank-p str) (or (s-contains-p str (car x) :ignore-case) (s-contains-p str (cdr x) :ignore-case)))) seq))))))) (let* ((coll '(("Affenpinscher" . "Loyal and amusing") ("Akita" . "Ancient Japanese") ("Bulldog" . "Kind but courageous") ("Caucasian Shepherd" . "Serious guarding breed") ("Miniature Schnauzer" . "Long-lived and low-shedding")))) (completing-read "Select a breed: " (dogs-filter coll))) There's some weird case I can't seem to solve at the moment inside seq-filter, that's why I had to do that weird "if" check, otherwise it won't show the rows unless something typed. But it works as I wanted. I have to put back the annotations though. I just didn't want to blow up the snippet beyond the screen height, and with the completion-fn, I don't have to use `completion-extra-properties` anymore, they can be attached via metadata right in that function. Doing that way you can also attach metadata that allows you to set the category (e.g., file, url, etc.), grouping, annotations, affixation, and more. Setting a proper category, for example for URLs (if you're dealing with those) would correctly dispatch Embark actions. Unfortunately, the documentation is not very clear and is lacking examples, it's a bit tricky to get it right, but that's the gist of how it gets done. Another great example of using a collection function I've found in this blogpost: https://kisaragi-hiu.com/emacs-completion-metadata.html

Great googling skills! Good job, you've found my own post that repeats the question word-to-word, and has the same answer in both places. :) I'll let you know if I decide to start a related discussion in emacs-dev mailing list. Watch for updates at https://yhetil.org/emacs-devel, in case I forget.

u/eleven_cupfuls is right, see the \`completion-extra-properties\` docstring.

/u/eleven_cupfuls is correct. If you want to search the content you're putting in the annotations, don't use annotations; the feature isn't designed for that.

Great snarkiness - *bravo!* (It's info for others, not for you. Do let us know, if you decide to broadcast further.)

[deleted]

I'm glad you found a solution that works for you and I appreciate you sharing it, but my suggestions are not "hacks for no good reason".

I didn't mean it to sound bad or offensive. It was my poor attempt to say 'sorry for cross-posting' and I simply wanted to explain that the info in both places is the same (at least at that moment). You see, I grew up when the Internet was still small, at the peak of FidoNet, when crossposting was frowned upon. And I'm also promoting yhetil.org. That's "the economics of FOSS" - you have to advertise free tools you like; sometimes, even when it doesn't fit the context of the conversation.

> my suggestions are not "hacks for no good reason". Alright, I should've chosen my words more wisely. Let me try to explain. My original question was about filtering things based on what is **not** in the original sequence of the rows. And it seems the camps have divided into: "you shouldn't (can't) filter based on things that are not present on the display (or annotations)", and "yes, you can do that." From my perspective, there can be many situations where you'd want to filter things based on something you can't display (or in my case, annotations). Consider a case when you need to select from a bunch of log entries. And imagine you'd want to provide a mechanism when you'd type something like "last 10 m" and it would filter the log entries for the events with timestamps no older than ten minutes ago. Obviously, in a situation like that, you can't encode dynamic time information into the strings directly. So what would be the solution? To use "programmed completion". And I think, attempts to do it differently would sound exactly like "hacks for no good reason". I have a feeling that you just got irritated because instead of thanking you I decided to dig further. But honestly, do take a look at the blogpost I shared, read the documentation (I posted the link to), play with the elisp snippet I used, then maybe you can see how powerful an instrument that thing is. Even with a seemingly simple use case of my example with dog breeds, consider how difficult would be to implement "hide/show" annotations, if I'd to follow your advice of concatenating breed names and descriptions. But I do appreciate your help and suggestions, I'm not saying they are without merits. Come to think of it, sometimes, doing exactly the way you suggested works better. One example is, completion frameworks (Vertico, Ivy, Helm) would highlight the occurrences within the strings, and they don't highlight them in the annotations. Still, programmed completions is very flexible and powerful tool, I wish the documentation had more practical examples and explained it better.

The annotations themselves may not be exactly "designed for that", nevertheless, Emacs provides a very powerful (yet poorly understood) mechanism for flexibly filtering things inside completing-read. See my [comment in the adjacent thread](https://www.reddit.com/r/emacs/comments/ytlodv/completingread_search_also_in_annotations/iw8iinb/)

I think you're missing the whole point. See my [comment in the adjacent thread](https://www.reddit.com/r/emacs/comments/ytlodv/completingread_search_also_in_annotations/iw8iinb/).

OIC; I think focusing on "annotation" confused the issue then. The example you posted, using dynamic completion, doesn't display annotations, but certainly does get what you want in terms of filtering.

I see you've discovered the dynamic completions; but this is a hack because depending on the users completion style it will throw errors. Let me demonstrate, if you have vanilla emacs 28 and use `basic` completion as the style, `eval-buffer` and then typing `baz` then `<TAB>` will throw an internal error of of "foo doesn't match `'baz" ``` (defun bad-completion (str pred action) (pcase action ('t (list "foo" "bar")))) (completing-read "Test: " #'bad-completion) ``` This is _why_ packages like [consult](https://github.com/minad/consult) split their completion into two steps - one to narrow the completion and the second to respect the completion-style of the user. Of course, if you don't care about other users then there are completion-styles and frameworks which won't care about the returned results. The technically correct approach is what consult or embark do; narrow the candidates before passing to the `completing-read` functions or have a recursive minibuffer/command to narrow the candidates.

Yes, turns out, complex completions are tricky. I'm now down this rabbit hole and the documentation is very terse, I don't understand half of it. I wish someone made a detailed explanation tutorial of some sort. I tried looking through various packages using non-trivial completions and still can't figure out when and how to use things like `completion-table-dynamic`, `completion-table-with-cache`, `completion-table-with-predicate`, `completion-table-case-fold`, et al.; when to use `all-completions` and `test-completion`; how `completion-boundaries` work, etc. On top of that, there are things like different completion types, `orderless`, and other things. I can't get my head around all that. Thank you for this comment, I will try to understand how `consult` works, although it would probably take me a couple of decades to comprehend it all.

The documentation is extremely sparse unfortunately and most of the information is across mailing lists and random GitHub issues. On top of that, I found no completion framework works the same (vertico is extremely close, but there are still a few differences between it and vanilla Emacs). The way I understand it is this; completion within Emacs is in 2 (rough) parts. The first is the completing-read function which is meant to return a list of candidates and other metadata to the completion style of the user. This is what the completing-table- functions do. The second part is each "type" of thing can have a completion style (set in the completion style alist and you need to say what you're completing in the metadata) falling back to a user specified default. There are special parts here and there with 'try-completion' and boundaries but you can ignore them for now unless you have more complex completions. The way to think about the completion style, is given a list of candidates, how do you want to narrow them. For example, 'basic' is a strict prefix match which is useful for file stems etc... Most people like using a fuzzy completion which is like 'orderless' and don't bother customizing the completion style per item. To get back to the problem at hand, you can possibly create a completion style specific for your item which searches elsewhere (say the string properties) as well or do what consult does. I haven't tried it myself (and YMMV depending on the completion framework you are using).

consult uses the two step only for functions where "narrowing" cannot possibly respect the completion style, like external grep commands. For most things, consult just shows the full candidate set.

The entire completion system has access *only* to the list of completions. Annotation and prefix/suffix are just window dressing added at the end for display. So if you want to search the window dressing, you have to alter that candidate list to include the annotation material, then strip it back out after selection. Non-trivial. What I do in such cases is embark-snapshot (maybe after I already narrowed the candidate list some), then use consult-focus-lines to show only full matching lines, annotation info included.

>(lambda (s \_ \_)(seq-map#'car(seq-filter(lambda (x)(unless (string-blank-p str)(or(s-contains-p str (car x) :ignore-case)(s-contains-p str (cdr x) :ignore-case))))seq))))))) This is a malformed collection table function which will fail for many completion styles. Add a message statement and print all 3 arguments and you'll see how it gets called in various contexts, with other completion styles like partial-completion or substring. Table functions are asked for many things, keyed to the action (3rd argument). See the info node *Programmed Completion* for more info. An example of searchable annotation data: [citar](https://github.com/emacs-citar/citar/blob/c37a4827d019eac76a297ebd2d6af04ffce42afb/citar.el#L521) aggregates bib info into a single string, stores it in a hash, then reverses the mapping after selection ends.

An informative note from the consult [command wishlist](https://github.com/minad/consult/issues/6): >Alternative to consult-help - implement generic function which allows to search through marginalia annotations, basically transforming candidates such that they also include the annotations and make them searchable. **not implemented, use apropos-documentation or embark collect instead for a searchable help**

I was suggesting to put some metadata as a text property rather than changing the text content; Emacs doesn't strip those out until the very end (though you can keep them, sort of, with minibuffer-allow-text-properties) and you can make use of text properties from the candidate list in all the metadata functions. I haven't personally written a completion style so I don't know if the completion style function gets the candidate list with the text properties for sure, but I'd be surprised if it didn't.

Sadly text properties are not preserved by completing-read. This is why tools like consult embed non-printable unicode "tofu" characters and use that as a form of metadata. The basic issue is that "completion", which was originally conceived just as it sounds: to add additional characters based on the in-buffer text and context, has been overloaded to "selection", using the same API.

They are preserved; but it is not straight-forward. If you enable `mini-buffer-allow-text-properties`, then the completion itself inherits the text properties from the candidate, _during_ the completion. Take the snippet below, if you use vanilla emacs and completion, typing 'f', '<TAB>' then `<Enter>` will return you a string where the _text properties are preserved where you started text completion_. Basically, 'f' has no text properties but `oo` will have the required text properties. You _can_ couple that with `require-match` as a function to make sure the user has tabbed completion by examining the properties starting from the end of the text. ``` (let ((minibuffer-allow-text-properties t)) (completing-read "Test: " (list (propertize "foo" 'prop "hello") (propertize "bar" 'prop "baz")))) ``` Of course, the `affixation-function`, `grouping-function` and `annotation-function` have full access to the properties from the candidates without restriction. I agree with you overall that `completing-read` is overloaded for both selection and completion and that's what makes the behaviour so hard to deal with.

>mini-buffer-allow-text-properties Yes this is new in 28.1. See [minads thoughts on it](https://github.com/minad/consult/pull/669#issuecomment-1280002450).

Yes, generally I agree with the post; that text properties are a poor way to implement an index i.e. given the \_return\_ value of \`completing-read\` for all the reasons listed. I wanted to give a more thorough explanation of how text properties behave in a \`completing-read\`. However, I don't believe it has much to do with the original post because we're talking about narrowing the candidates, which \_do\_ have access to the properties and duplicates are not important. That is, \`completion-basic-all-completions\` or other completion styles have a \`table\` argument which \_will\_ have all the metadata which you can narrow on. You can \`debug-on-entry\` to take a look.

Sorry it took me so long, I got swamped with too many other things and didn't get a chance to hack on the thing I was experimenting with. I owe an apology. A sincere and profound. Turns out you were right from the get-go, and I was wrong in my assumptions. Turns out, it really is incredibly difficult to make the completing-read to search for things that do not exist in the main string, and all my attempts to do so without employing clever hacks have failed. But you gave me a great insight about propertizing the text, and I did just that, except I used `invisible t` to hide what I don't want to display but still want to be searched upon. Once again, I apologize for acting like an arrogant, pigheaded smartass. For some reason, I thought you were missing the point of what I'm trying to achieve, but I haven't considered certain limitations built in Emacs. `completing-read`, as it turns out never meant to be used "for searching things". And when breaking boundaries, sometimes you have to do things that seemingly have little sense. Thank you for your suggestions and your help.

🍻 May your future hacks have more success