Help with creating async Consult command with pipes

Here's what I'm trying to accomplish, in case there's a better way than my current plan: I want to search the full text of the files for **all** the terms of the input (terms separated by spaces), no matter where in the file they are or in what order. So, if I search for "apple banana", I want to find a file where "banana" is on line 1 and "apple" is on line 15. Since pretty much all search tools are line-oriented and/or keep the order of the terms and/or use 'OR' matching instead of 'AND' matching, I came up with a way to do what I want via a series of pipes. Something like: rg --ignore-case --files-with-matches --color=never --null . -e 'apple' | xargs -0 rg --ignore-case --files-with-matches --color=never -e 'banana' I have a builder function that will output the correct command according to the input, but for the sake of example let's say it's a static function like so: (defun my-builder (input) '("rg" "--ignore-case" "--files-with-matches" "--color=never" "--null" "." "-e" "'apple'" "|" "xargs" "-0" "rg" "--ignore-case" "--files-with-matches" "--color=never" "-e" "'banana'")) The string version of this command works perfectly in the command line. But when I try to use the builder with a Consult command: (defun my-consult (&optional dir initial) (interactive "P") (find-file (consult--read (consult--async-command #'my-builder (consult--async-map (lambda (x) (string-remove-prefix "./" x)))) :prompt "Hey: " :sort nil :require-match t :initial (consult--async-split-initial initial) :add-history (consult--async-split-thingatpt 'filename) :category 'file :history '(:input consult--find-history)))) ... It fails. The log reads: consult--async-process sentinel: event=exited abnormally with code 2 lines=0 >>>>> stderr >>>>> |: No such file or directory (os error 2) xargs: No such file or directory (os error 2) rg: No such file or directory (os error 2) <<<<< stderr <<<<< It seems `make-process`, which Consult uses, doesn't like to have a complex line with pipes in it. (The documentation says the first item of the command list is the program name, and the rest are args -- but of course, "|" and the rest are not args for rg. I *think* that's where the problem is, but I'm not really sure.) So. Does anyone know of a workaround? Or is there a better way to accomplish what I'm after? Any help appreciated. Thanks!

Pipe is a shell construct so either use it, or see if rg already supports what you need (I, for one, for would be surprised if it didn't already). EDIT: But from the reddit comments I read, you are doing something that is unneeded. Nothing is stopping you from further filtering the matches using your completion mechanism

Ah, you're right, of course... Counsel, for instance, uses `start-file-process-shell-command` instead of `make-process`, which actually delegates to a shell and allows for the pipe. As far as I can tell, ripgrep unfortunately does not support this natively. It does have multiline support, but it's only for *patterns* that are multiline. > Nothing is stopping you from further filtering the matches using your completion mechanism Do you have any idea how to do this further filtering? Is there a way to easily chain filters? (I'm using `fido-vertical-mode`, which is really `icomplete`.) Just asking in case you have some experience in this area already, but I'll dig further. Thanks for the help!

> Do you have any idea how to do this further filtering? Is there a way to easily chain filters? (I'm using fido-vertical-mode, which is really icomplete.) Just asking in case you have some experience in this area already, but I'll dig further. No idea, sorry. I don't use consult and friends.

Thanks to /u/\_viz\_'s comment, it became clear that if I wanted to execute a complex shell command then, well, I would need to use a shell. So I patched Consult's `consult--async-process` to do just that when needed. It's a very hacky thing and I'm not even sure if it'll behave correctly in all cases, BUT it works for for me! Here's the patched function (using `el-patch`). It's a long function, so search for "THE PATCH" to get to the good stuff: (el-patch-feature consult) (el-patch-defun consult--async-process (async cmd &rest props) "Create process source async function. ASYNC is the async function which receives the candidates. CMD is the command line builder function. PROPS are optional properties passed to `make-process'." (let (proc proc-buf last-args indicator count) (lambda (action) (pcase action ("" ;; If no input is provided kill current process (when proc (delete-process proc) (kill-buffer proc-buf) (setq proc nil proc-buf nil)) (setq last-args nil)) ((pred stringp) (funcall async action) (let* ((args (funcall cmd action)) (flush t) (rest "") (proc-filter (lambda (_ out) (when flush (setq flush nil) (funcall async 'flush)) (let ((lines (split-string out "[\r\n]+"))) (if (not (cdr lines)) (setq rest (concat rest (car lines))) (setcar lines (concat rest (car lines))) (let* ((len (length lines)) (last (nthcdr (- len 2) lines))) (setq rest (cadr last) count (+ count len -1)) (setcdr last nil) (funcall async lines)))))) (proc-sentinel (lambda (_ event) (when flush (setq flush nil) (funcall async 'flush)) (overlay-put indicator 'display (consult--process-indicator event)) (when (and (string-prefix-p "finished" event) (not (string= rest ""))) (setq count (+ count 1)) (funcall async (list rest))) (consult--async-log "consult--async-process sentinel: event=%s lines=%d\n" (string-trim event) count) (with-current-buffer (get-buffer-create consult--async-log) (goto-char (point-max)) (insert ">>>>> stderr >>>>>\n") (insert-buffer-substring proc-buf) (insert "<<<<< stderr <<<<<\n"))))) (unless (equal args last-args) (setq last-args args) (when proc (delete-process proc) (kill-buffer proc-buf) (setq proc nil proc-buf nil)) (when args (overlay-put indicator 'display #("*" 0 1 (face consult-async-running))) (consult--async-log "consult--async-process started %S\n" args) ;; HERE'S THE PATCH: (el-patch-swap ;; OLD FORM: (setq count 0 proc-buf (generate-new-buffer " *consult-async-stderr*") proc (apply #'make-process `(,@props :connection-type pipe :name ,(car args) ;;; XXX tramp bug, the stderr buffer must be empty :stderr ,proc-buf :noquery t :command ,args :filter ,proc-filter :sentinel ,proc-sentinel))) ;; NEW FORM: (progn (setq count 0 proc-buf (generate-new-buffer " *consult-async-stderr*")) (if (string= (car (last args)) "@@@") (progn (setq proc (start-file-process-shell-command (car args) nil (string-join (butlast args) " "))) (set-process-sentinel proc proc-sentinel) (set-process-filter proc proc-filter)) (setq proc (apply #'make-process `(,@props :connection-type pipe :name ,(car args) ;;; XXX tramp bug, the stderr buffer must be empty :stderr ,proc-buf :noquery t :command ,args :filter ,proc-filter :sentinel ,proc-sentinel))))))))) nil) ('destroy (when proc (delete-process proc) (kill-buffer proc-buf) (setq proc nil proc-buf nil)) (delete-overlay indicator) (funcall async 'destroy)) ('setup (setq indicator (make-overlay (- (minibuffer-prompt-end) 2) (- (minibuffer-prompt-end) 1))) (funcall async 'setup)) (_ (funcall async action)))))) The patched function recognizes if the last element of the command list is "@@@". If so, it'll concat it into a string and pass it to `start-file-process-shell-command`, setting up the same sentinel and filter. (I kinda ignored the other options that Consult passes to `make-process` and hoped they wouldn't matter too much. It worked, so...) My builder function just has to append "@@@" to the command list and now I can use my pipes! Here's the real builder function: (defun my-builder (input) (let* ((common (split-string "rg --ignore-case --files-with-matches --color=never --max-depth=1")) (query (split-string input)) (cmd (if (= (length query) 1) (append common '("." "-e") query) (flatten-tree (list common "--null" "." "-e" (car query) (mapcar (lambda (term) (list "|" "xargs" "-0" common "--null" "-e" term)) (butlast (cdr query))) "|" "xargs" "-0" common "-e" (last query)))))) (append cmd '("@@@")))) As I said, this is very hacky, but it's a very small change to Consult that does what I need. So I'm happy.