Any experts on font-lock here?

I am writing a major mode for a lisp like-language where I have two kinds of text in the buffer: code and comments. I can parse buffer correctly, and I set up some text properties as markers for the text. So far so good, it works as intended. I need help with getting font-lock search based fortification when using the function. I am getting it to work, but not really over the entire buffer; *initially*, when font-locking for the first time when a buffer is opened, it stops after the first match is found. If I manually edit the text further in the buffer, say add and remove a space somewhere in the text, it gets font-locked correctly, but not when the text is initially font-locked. [Here is a screenshot to illustrate](https://imgur.com/a/2ha5cC3) what I am talking about. It is maybe hard to see, everything gets quite darkened when I exported to PNG, but you can see that the shade of non-code is lighter in subsequent parts later in the buffer in the first screenshot, while it is the same dark in the second screenshot. I am not sure if I am misunderstanding [the docs for the font-lock](https://www.gnu.org/software/emacs/manual/html_node/elisp/Search_002dbased-Fontification.html): > When function is called, it receives one argument, the limit of the search; it should begin searching at point, and not search beyond the limit. It should return non-nil if it succeeds, and set the match data to describe the match that was found. Returning nil indicates failure of the search. > > Fontification will call function repeatedly with the same limit, and with point where the previous invocation left it, until function fails. On failure, function need not reset point in any particular way. I am particularly worried if I set match-data incorrectly. I am not really sure what "describe the match" really means there. (defun shtml--fontify-comments (limit) "Fontify everything but code as comments." (let ((beg (if (get-text-property (point) 'shtml-cmnt) (point))) end) (when beg (setq end (next-property-change (point) (current-buffer) limit)) (set-match-data (list beg end)) t))) As seen, I simply set match-data to start and end points of the match. Below is code to set font-lock-keywords, and to call major mode, which is all quite simple, but here for reference. (defvar shtml-font-lock-keywords (list (cons #'shtml--fontify-comments 'font-lock-comment-face) `(,(concat "\\((\\)\\b\\(" (regexp-opt (html-tag-names)) "\\)[ \t\n]+") 2 'font-lock-function-name-face)) "Syntax highlight rule for HTML5 elements.") (define-derived-mode shtml-mode emacs-lisp-mode "shtml" "Major mode for writing shtml files." (shtml--parse-buffer (point-max)) (setq font-lock-defaults '(shtml-font-lock-keywords))) I certainly misunderstand something there; I am not so familiar with font-lock, this is the first time I try to do something like this with it. Calling 'font-lock-ensure' or 'font-lock-fontify-buffer' does not do anything: (define-derived-mode shtml-mode emacs-lisp-mode "shtml" "Major mode for writing shtml files." (shtml--parse-buffer (point-max)) (setq font-lock-defaults '(shtml-font-lock-keywords)) (font-lock-ensure)) Also, what I have noticed is that, when the cursor is in a comment line further down the buffer; *that line* also gets correctly font-locked, but not other lines around it. It is probably something simple I miss there, like to slay a dragon or to do this only when the Moon is in right phase with Jupiter, but I really do appreciate if anyone can hint me what I do incorrectly there. I am stuck on this. You can assume that parse-buffer function does find all bounds correctly; I have checked that all the text properties in entire buffer :). I am not sure if this is the case for [font-lock-multiline](https://www.gnu.org/software/emacs/manual/html_node/elisp/Font-Lock-Multiline.html). I have tried to add the property too, but it didn't help either.

Comments have interactions with other functions in Emacs besides fontification & indentation, I think mostly through the syntax table. Maybe you can modify the `syntax-propertize-function` to accomplish this goal, see: https://www.gnu.org/software/emacs/manual/html_node/elisp/Syntax-Properties.html Since the documentation is pretty arcane, here is an example where I implemented multi-line string literals with different start/end tokens in a lisp-like language with it: https://github.com/hylang/hy-mode/blob/master/hy-mode.el#L128. #[[This is a string, continuing the string, done]] Its been a while since I did heavy duty emacs-lisp work, so I probably won't be able to remember/follow up on the specifics. Hope this puts you on the right track though.

Thank you for the input. That looks very interesting. I have just glanced through your code. I have to say that I am aware of syntax-tables, but I have never used them. Looking the docs now, thanks for the link, I am not sure if I really understand at the moment how I would implement for it though the syntax table, but I'll try to think more about it. Currently, I have a little function that correctly parse buffer and mark parts of the buffer with text properties as either code or comments, and I use those properties for the fortification later on. Maybe I can use it as syntax-propertize-function. I don't know yet, will try, never played with syntax-tables and those parts of Emacs before :). Thank you very much for these hints and your links!

In general, https://github.com/Lindydancer/font-lock-studio (which is a font-lock debugging tool) may be able to help with font-lock problems.

[deleted]

[deleted]

Not an expert either but the fact that you are only getting comment fontifaction on the first match makes it sound like you function returns 'nil' before it should. As I understand it, font-lock will call you function until it returns nil. So you probably want to search forward for the 'beg' point instead of checking at (point).

> only getting comment fontifaction on the first match [Actually on the first match; until the first code block; and on the place where the cursor is; but not on the other places :\)](https://imgur.com/a/55TZI49). > you probably want to search forward for the 'beg' point instead of checking at (point). They are quite explicit about starting at the point and not searching beyond the limit. They say the framework will call the function repeatedly until the end of the buffer, as I understand it. I have played with it a bit today and changed it to work with just one text property but with two different values, but nope :). I can see that properties are correctly setup. Searching for properties manually from M-: gives me correct buffer positions, but the font-lock does not seem to care about it. I think goldfather8 is onto something with syntax-table, but I have to figure it out yet. Anyway, thanks.

Does this work better? ``` (defun shtml--fontify-comments (limit)    "Fontify everything but code as comments."    (let ((beg (or (if (get-text-property (point) 'shtml-cmnt) (point)) (next-single-property-change (point) 'shtml-cmnt (current-buffer) limit)) end)      (when (and beg (/= beg limit))       (setq end (next-property-change                   (point) (current-buffer) limit))       (set-match-data (list beg end))       t))) ``` Sorry for formating, coding on the phone is hard! Edit: I noticed that `next-single-peoperty-change` returns limit when reaching the limit. So we need to test that beg <> limit, I think

> So we need to test that beg <> limit, I think There is more to it, this is an overlapping intervals problem. We have to find all crossings and intervals, which is not so difficult. You were correctly about searching for the beg point in your first comment. I didn't realize it from the beginning, struck me later; but it is not just the starting point. Both starting and end point have to be potentially adjusted, and there can be additional interval in-between, so we have to look to find one or two crossings and add potentially one or two intervals according to the scheme above (if I haven't omitted some case). (defun shtml--in-cmntp (&optional p) "Are we in a comment" (eq 'cmnt (get-text-property (or p (point)) 'shtml-stx))) ;; Overlapping intervals ;; 1) B ... E ( ... ) ;; 2) B ... ( ... ) ... E ;; 3) (... B ... E ... ) ;; 4) ( ... B ... ) ... (... E ... ) ;; 5) B ... ( ... E ... ) ;; 6) ( ... B ... ) ... E (defun shtml--cross-points (beg end) (let (intervals) (if (shtml--in-cmntp beg) (push beg intervals)) (while (< beg end) (setq beg (next-single-property-change beg 'shtml-stx (current-buffer) end)) (push beg intervals)) (nreverse intervals))) (defun shtml--fontify-comments (limit) "Fontify comments." (let ((intervals (shtml--cross-points (point) limit))) (with-silent-modifications (while intervals (add-text-properties (pop intervals) (pop intervals) '(font-lock-fontified t face font-lock-comment-face)))))) And with that it works. Thanks for the help, you got me thinking in this direction.