How to properly font-lock for a custom major-mode aka how to use complex regex?

[removed]

Maybe /u/oantolin or /u/github-alphapapa will have something to say about this?

The best advice I can offer is to carefully and repeatedly study the Elisp manual section on font-lock, and to model on the source code of a similar project. The most I've done with it is in https://github.com/alphapapa/prism.el