segment.el -- porting complex sentence break rules from ICU regexes to elisp [help?]

hi emacsers, i'm working on segment.el ([https://codeberg.org/martianh/segment](https://codeberg.org/martianh/segment)), which aims to port complex sentence break/no-break rules from translation software like OmegaT and the Okapi Framework into Elisp. I manually ported a set of English rules, but then I started working on conversion functions to do the rest of the languages that have rulesets programatically. i'm by no means an expert programmer, and my regex foo is middling. i was wondering if anyone might be able to take a look and suggest how to go about doing conversion reliably? one of the difficulties i have is in dealing with the different ways emacs displays strings, both for its own regexes and for the ICU regex strings that are to be converted. in the repo, segment-regexes.el contains some manually converted rules, segment-convert.el contains some very primitive/blind converting fumbling in the dark, and the ICU-formatted rules are in .srx files in subfolders. they're formatted as XML. some explanation of the unicode properties used by ICU regexes is in segment-unicode-properties-used-by-ICU-regexes.txt i started doing this to reliably segment text for translation in emacs, but then thought if it got done properly it could be useful for other packages also. the code in the repo is very green, btw. PS: let's not have a stoush about double spaces ending sentences. i have read those discussion, and i'm interested in segmenting files that i have not typed out myself.

This package *looks* like something that might be right up my alley; I am also unsatisfied with Emacs's current sentence-filling heuristics. However, I've never heard of OmegaT and Okapi before; do you think you could explain what's going on exactly so I can understand and start helping?

hey a-c-d-w, i should have known to hassle you about this. :) OmegaT is a CAT (computer assisted translation) tool. it works by splitting a given text into segments, then the user translates segment by segment, which ensures you don't miss any, and can suggest filling in segments automatically if they're identical or almost. in latin alphabet languages, segments are often sentence-based. to avoid splitting your document into crappy segments, a series of rules are provided to split or not split given certain conditions. each rule had a before-break regex and an after-break one, which is more reliable than just a single regex. i.e. if you run into X then a period, also check if it is/isn't followed by Y. OmegaT is GPL, so i looked up their default rulesets. the problem is they're written as ICU regular expressions ([https://unicode-org.github.io/icu/userguide/strings/regexp.html#regular-expression-metacharacters](https://unicode-org.github.io/icu/userguide/strings/regexp.html#regular-expression-metacharacters)), so the XML rule files are useless as is. so, either use brain to convert by hand (and in my case that probably means introducing errors everywhere), or convert the regex strings with elisp magic (which in my case also means introducing errors everywhere :P ). hence asking for help. you can take a look at the rulesets in subfolders in the repo. they're SRX files, which is just XML. i thought once a few of them were implemented/converted, then a few people might start using, then gradually we cd add more rules to hone it.

> the problem is they're written as ICU regular expressions Under the "Regular Expression Operators" section of your link there are zero-width "look ahead" and "look behind" assertion features which are not supported by Emacs regexps. In Emacs we do that kind of matching using a combination of regexps and elisp. So while the equivalent functionality could be written, there will be no direct conversion of ICU regexps to Emacs regexps if such incompatible features are used in the patterns in question.

[deleted]

I don't know, never worked much with ICU, just did some code to snitch locales from their github sources once, so I am not an expert, but what I read in the docs they use [pcres](https://unicode-org.github.io/icu/userguide/strings/regexp.html). Emacs regex are not exactly pcre compatible all the way, but they are not well too far away either? There is even a [tool that converts pcre to Emacs regexes](https://github.com/joddie/pcre2el), so you could maybe parse xml files, read regexes from corresponding tags and convert them into Emacs regexes with the tool? Shouldn't be too difficult. Emacs has support for parsing xml files already, but you could also just do your own re/strring-search for <beforebreak>, <afterbreak>, and few other things you need from the file. Maybe you can get along with [Emacs binding to libpcre](https://github.com/syohex/emacs-pcre), and don't even need to translate regexes?

Oh okay, cool -- so many times, with English at least, OmegaT would split on sentences to aid translation? This could have implications I'm directly interested in ... Of course, translation is also a great use-case for the tool! I'm moving this week but I'll keep my eye on this and see if I can help. First thing I noticed --- the ICU regex page you linked shows that ICU has the possibility for look-ahead assertions, which Emacs regexps do *not* have. So if any of the rules include look-aheads, AFAIK they won't be able to be translated to Emacs Lisp regexps.

i'm a little confused by your answer, as the regexes i need to translate are indeed in ICU. i know about pcre2el, but i don't think i can use it with ICU strings. i also looked around for an ICI <>PCRE conversion tool but found nothing. i already have functionality to parse the XML rule files too.

thanks for the heads up about those. i had already seen one other feature that isn't convertible in the simple sense of swapping characters in a given string with others, but it had an easy workaround. so far i have just collected a list of all the ICU features that i have come across as i convert by hand, and haven't seen those yet.

thanks for the heads up re look-aheads. i'll have to keep an eye out to see if they are ever used in the rules. OmegaT by default splits a whole document at every sentence break, except for any no-break rules that are specified. it also has a paragraph option for languages where that is more suitable. yeah i want to work on a basic translation mode in Emacs, but its challenging. i have also been trying out `segment.el` with `hl-checkerboard` (https://codeberg.org/martianh/hl-checkerboard) also written for translation.

> i don't think i can use it with ICU strings I am not sure if I understand the problem. What do you really mean when you say "with ICU strings"? When you read your [XML file with regexes](https://codeberg.org/martianh/segment/src/branch/main/OmegaT-defaults/defaultRules.srx) in Emacs, you are working with utf8 strings. It even says explicitly utf8 encoding on top of the file, but Emacs would probably translate it to utf8 anyway. If you are going to use their regexes to find sentence endings, you will have to match each sentence with "before" and "after" regex. You will have to create your own elisp solution for this, and you will need to translate those regexes into Emacs regexes to use them in Emacs buffers, since pcre regexes [are slightly incompatible with Emacs regexes](https://medium.com/kloeckner-i/til-differences-between-vim-regex-emacs-regex-and-pcre-e26075704088). You will have to do it manually, or with your own elisp code, or with a tool like the one I linked above, it is up to you.

>If you are going to use their regexes to find sentence endings, you will have to match each sentence with "before" and "after" regex. You will have to create your own elisp solution for this, and you will need to translate those regexes into Emacs regexes to use them in Emacs buffers yeah, of course, this is exactly what i am doing at the moment! i already have a full set of English ones working, which i re-wrote by hand as emacs regexes. but i can't use a tool such as pcre2el to do further conversion automatically, which is why your post confused me. the XML file you link to in my repo contains ICU formatted regexes, its not the encoding, it's the regex system used. they use "\\P{Lu}" and the like. i am asking for help to reliably convert such regexes into emacs ones automatically, rather than manually converting rules for 18 different languages other than english.

> but i can't use a tool such as pcre2el to do further conversion automatically > it's the regex system used. they use "\P{Lu}" and the like Why can you not? It seems to be designed exactly for the purpose you need. I am not sure what you are doing, but you have to write your own converter, you can't just pass in XML file, if that is what you are expecting? In that converter you would use pcre2el to convert regex strings from the XML files into Emacs regex strings and store them into some rule object(s), or whatever you are using to store your regexes in your lisp code. I am sorry if misunderstand you. By the way, have you tried to compile [the module](https://github.com/syohex/emacs-pcre)? In that case, you don't even need to convert regexes, you just have to snitch them out of the XML file :). Instead of re-search-forward, you would be using pcre-re-search-forward etc in your lisp code.

i know i have to write my own converter, that is what i am half-way through doing, and how to do it in a robust way is what i'm asking for help with. but the regexes i want to convert are not PCRE style, they are ICU style. so i can't see why pcre2el be able to help at all.

> the regexes i want to convert are not PCRE style, they are ICU style. I looked at the docs and interpreted the ["The regular expression patterns and behavior are based on Perl’s regular expressions"](https://unicode-org.github.io/icu/userguide/strings/regexp.html), as if they are pcres. I am sorry for the confusion. After looking more at it I see the differences. Is it not possible to modify pcre2el.el tool, call it 'icure2el.el' or something, so it recognizes the ICU stuff and converts that to Emacs re:s? Seems to me that code for pcre2el is written relatively well and clean, and it is only rule conversions you would have to modify. 3k lines of code looks intimidating, but lots of the code is 'around-stuff' for the mode itself, buffer etc, which you don't need to look at. It is quite well documented too. I think you would really be interesting in those three functions: ;; All the conversion functions take a single string argument, the regexp ;; to translate: ;; - `rxt-pcre-to-elisp' ;; - `rxt-pcre-to-rx' ;; - `rxt-pcre-to-strings' But those are just the API, 'rxt-parse-re' seems to be the driver. Things to take extra care is multiline matching which is tricky in Emacs, you can't do it easily with a single regex, and that Emacs regex search is by default case-insensitive, while ICU seem to use case-sensitive by default. They also do have quite in common, and looking through the XML file, there is a lot of "common cases", so you could still use the tool, and create a small test to probe each converted regex automatically, and spit out those that are not converted successfully and fix only those manually. That would at least reduce the number of regex that need to be converted manually.

thanks for following up. yes, i was intimidated by the pcre2el code, so it's helpful to have a pointer of what functions to take a look at. most of the SRX break rules are very simple regexes, and there's no need to try to implement conversion for the whole of ICU. i'll have a crack one day soon.

pcre2el can do many of the basic forms, but not unicode property names, i.e. those of the form "\P{Xy}", which are used extensively in the SRX rules. ` rxt-parse-braces` fails on every one of them. i guess PCRE doesn't use these at all? looks like `{}` are only used for {M,N} counting, at least going on pcre2el code.