Emacs on Windows, grep and escaping characters

Whenever I use rgrep and enter eg. umlauts, Emacs creates the following command line: find \[...\] -exec grep --color=always -i -nH --null gef\\ördert \\{\\} \\; Grep returns with no results found. The problem is that the umlaut ö get's prepended with an escaping sequence. If I run the same command on the commandline without the backslash, grep succeeds. Why is Emacs escaping certain characters anyhow, especially utf-8 valid characters? Using bash as the shell. file encoding is utf-8, I set LC\_ALL in .profile to en\_US.utf8, LANG=C, SHEL=/bin/bash. My Windows is German though.

I think your problem is exactly that you use Bash as the shell. That is not how the Windows port of Emacs is supposed to be used. Also, don't set your locale encoding to UTF-8, it will cause you even more grief. What happens if you do the same from `emacs -Q` -- does it work then?

No, it's still escaped and doesn't work. I am using git bash as the Windows provided find doesn't work at all in rgrep.

You can find a native Windows port of GNU Find on the ezwinports site. It works for me in rgrep.

* I unset SHELL=bash and got rid of utf-8 locales. My default shell is now cmd.exe again, good. * I donwloaded ezwinports grep and find. When I use that in `setq find-program` it can't "find" certain paths containing blanks * So back to the git-bash provided find and grep. No issues of not finding paths. * When I rgrep for a string where the containing line contains umlauts, the strings are printed to the compile buffer as unicode escapes * When I run `C-x RET c utf-8 rgrep ... ` the umlauts are correctly printed again. I guess this was caused by no longer setting an UTF-8 locale. * When I search for an string containing an umlaut, still nothing found. My suspicion is that the string provided to find-grep are not unicode encoded? `w32-system-coding-system` says cp1252. The text I input in emacs buffers IS unicode encoded saved. This is all seriously confusing.

> it can't "find" certain paths containing blanks Details, please. What exactly did you try to "find", and what do you mean by "find certain paths containing blanks"? > So back to the git-bash provided find and grep IMO, bad idea. That version of `find` is an MSYS2 executable, which is subtly incompatible with the native Windows build of Emacs you are using. If you keep using it, all bets are off, especially for non-ASCII search strings: MSYS2 programs expect UTF-8 encoded non-ASCII arguments, whereas the native Windows build of Emacs doesn't support that. > When I rgrep for a string where the containing line contains umlauts, the strings are printed to the compile buffer as unicode escapes This is because when your shell is Bash, Emacs quotes strings differently. If the non-ASCII characters are from your system codepage, you don't want that and don't need them to be escaped. But if you use Bash, they *will* be escaped. Which is why I advised not to use the Git Bash stuff. > When I search for an string containing an umlaut, still nothing found Because the native Windows build of Emacs doesn't support UTF-8 encoded command-line arguments for child programs, and you ask it to do precisely that. The way to have this work for you is to use a native Windows port of `find`, and invoke `rgrep` with `C-x RET c cp1252 RET M-x rgrep ...`. > This is all seriously confusing. Yes, it is. I'm trying to save you all that confusion, but you need to listen to my advice, not disregard it.

Thank you for your in-depth explanation. So I am no longer fighting against your highly-appreciated advice. * I extracted [https://sourceforge.net/projects/ezwinports/files/findutils-4.2.30-5-w64-bin.zip/download](https://sourceforge.net/projects/ezwinports/files/findutils-4.2.30-5-w64-bin.zip/download) and [https://sourceforge.net/projects/ezwinports/files/grep-2.10-w32-bin.zip/download](https://sourceforge.net/projects/ezwinports/files/grep-2.10-w32-bin.zip/download) into Emacs/usr (as the windows port prepends emacs/usr/bin to PATH) * I run rgrep with an Umlaut in it .... it works! The only "problem" I have now is that the result is now strangely displayed, instead of e.g. `gefördert` I get `gefördert` which is mildly annoying but alas I find what I ask for. BTW. I discovered this "native" nature of this Emacs windows build as `M-! echo "ö" > out.txt` produces utf16-encoded output?, the "problem" behind grep not finding matches in utf8-encoded files. To your question > Details, please. What exactly did you try to "find", and what do you mean by "find certain paths containing blanks"? I think it's not blanks but maybe "erroneous" encoding of file names like in ./Projekte/WKO/BQM/DigitalePrüfungsabwicklung/Präsentationen_Anbieter/KnowledgeCheckR/KnowledgeCheckR_Pru¨fungen.pdf: No such file or directory I guess the "problem" lies in the file name KnowledgeCheckR_Pru¨fungen.pdf which Windows Explorer as well as dired displays as KnowledgeCheckR_Prüfungen.pdf yet "ez-find" prints it as `...u"fungen.pdf` EDIT: I guess the problem is that these file names are (because of umlauts) utf8-encoded and the ezwinport expects utf16-encoding in file names?

> The only "problem" I have now is that the result is now strangely displayed, instead of e.g. gefördert I get gefördert Is the text in the file encoded in UTF-8 or is it encoded in your system's codepage? I think it's the former, in which case try setting things up so that the coding-system Emacs uses for *decoding* (but NOT for encoding!) stuff from `find` is UTF-8. > I think it's not blanks but maybe "erroneous" encoding of file names If you have files whose names cannot be expressed in your system codepage, then yes, this is expected. The port of `find` to which I pointed doesn't support non-ASCII file names with characters outside of the current system codepage. > I guess the problem is that these file names are (because of umlauts) utf8-encoded and the ezwinport expects utf16-encoding in file names? No, if the Windows Explorer displays them correctly, they are not in UTF-8. Does Dired say that "Prüfungen" has 2 characters for u+umlaut, or just 1 character? You can see that with `C-x =` in the Dired display, by looking at character positions it shows: does the position changes by 1 or by 2 when you move across ü?

> Is the text in the file encoded in UTF-8 or is it encoded in your system's codepage? I think it's the former, Yes, it's in utf-8. > in which case try setting things up so that the coding-system Emacs uses for decoding (but NOT for encoding!) Would I do that with `default-process-coding-system`, `set-buffer-process-coding-system` ? I read [Coding Systems for Interprocess Communication](https://www.gnu.org/software/emacs/manual/html_node/emacs/Communication-Coding.html) and I think I need an equivalent of `C-x RET p input-coding RET output-coding RET` but for the inter-process communication so that input is in the systems codepage yet output in utf-8. I can do it with > If you have files whose names cannot be expressed in your system codepage, then yes, this is expected. Of the many files in my workspace only three are "special" where find has some issues. I can live with that, in fact I may simply delete them / move them somewhere else and call it done ;) > Does Dired say that "Prüfungen" has 2 characters for u+umlaut, or just 1 character? dired displays two characters namely u", yet Windows explorer displays it as ü

> Would I do that with default-process-coding-system, To set up specific coding-system for a program, you need to add a cons cell of the form `("find" . (utf-8 . cp1252))` to the variable `process-coding-system-alist`. Use `add-to-list` for that. > dired displays two characters namely u", yet Windows explorer displays it as ü This is some problem with your default font: Emacs should also display these as a combined character, provided that the font supports that. But this is secondary: the main issue here is that the file name contains the u-umlaut in decomposed form -- 'u' followed by the lone umlaut. And your system codepage doesn't support the lone umlaut character, so `find` cannot access such files.