Idiomatic way for processing large file

I have a large file of journal entries (thousands) with datestamps. Now I'd like to create individual files (one file per date). In Python, I'd process the large file line by line, check if line matches a regexp, if so, get the date from that line, create a file with that date (or open it, if it exists) and append all following journal entry lines into that file, until I encounter a new timestamp line. ## How to do this in Elisp? Now I am trying to do this in Elisp, and am looking for the most idiomatic way for doing it. I came across these concepts / functions: - `get-buffer-create` - `with-temp-buffer` - `set-visited-filename` (for later saving the buffer into a file) My idea is to process the journal file, search timestamps, if found, put following lines into a new buffer (`get-buffer-create "2020-02-03.txt"` for instance). Finally I'd like to save the temp buffers to files via `set-visited-filename` and then `save-buffer`. ### Question Is this an idiomatic way for doing it in Emacs / Elisp?

You might want to use the function ``write-region``.

If I understand this correctly it's a one-off task? If I'd do this I'd open the journal file in a window to the left and then have a dired buffer in a window to the right. Then I'd create a macro (`C-x (`) that: 1. Searches for the first date section. 2. Fetches the date of the section. 3. Moves over to the dired window and creates and opens the new file. 4. Goes back to the journal buffer and selects and cuts out the correct region. 5. Goes back to the file and pastes it and saves and kills the buffer. 6. Saves the macro (`C-x )`).

Ah, I should have added that I al trying to get better in Elisp hacking. So I‘d prefer to do this in Elisp. - But good idea! Thanks.

Sounds like a job for a loop utilising re-search-forward. Since I don't know what the file actually looks like, a very rough draft would be (with-current-buffer (find-file-noselect "filename") (goto-char (point-min)) (let ((last (point))) (while (re-search-forward regexp nil 'noerror) (write-region last (point) "newfile"))))

Your solution is excellent! As I need to append to file, I have to use `write-region` like so: (write-region <STRING> nil <FILENAME> 'append) It's a large file consisting of entries like this: ** [2013-06-11 Tue 10:07:15] (Ran 10 miles today) It was a refreshing run. Had my new shoes from New Balance on. PS: Take the other route next time. ** [2013-06-12 Wed 11:17:22] (Met Mike at Starbucks) Hadn't seen him in a while. He's been talking about his hew apartment a lot. His wife Joanna is in New York right now, will be back in a week. Writing the regex is easy: "^\\*+ \\(\\[[0-9]\\{4\\}-[0-9]\\{2\\}-[0-9]\\{2\\}\\) [A-Za-z]\\{3\\} \\([0-9]\\{2\\}:[0-9]\\{2\\}\\):[0-9]\\{2\\}\\] " Group 1 of the regex capture grabs the date, group 2 captures the time without seconds.

I like to do something like (with-temp-file "my-file" (insert-file-contents "myfile") (goto-char (point-max)) (insert "appended line\n")) with-temp-file is like with-temp-buffer, except before the buffer is killed its contents are written to the file (with write-region). If you use Helpful, you can look for references to with-temp-file, write-region, save-buffer, etc. in all code loaded in your Emacs for examples. Some notes: - insert-file-contents might run some hooks, so I've heard advices to use insert-file-contents-literally and decode the text yourself with decode-coding-region, but that seems to mean I lose the ability to make Emacs detect the coding system automatically for me. Those hooks also might be used by Tramp in some way? - find-file-noselect will run the major mode and open a buffer in the same way as find-file. If the user already opened the file, that buffer will be returned --- including possible unsaved modifications to the buffer. If you're only interested in the content and not in eg. parsing it with tools that a particular major mode provides, it's likely better to use with-temp-file + insert-file-contents. - Creating a new temporary buffer and killing it adds a little bit of overhead which adds up if it's run in a loop. In that situation it could be better to run the loop *inside* the temporary buffer. So instead of (dolist (f files) (with-temp-buffer (insert-file-contents f) (do-stuff))) you could do this to optimize it: (with-temp-buffer (dolist (f files) (erase-buffer) (insert-file-contents f) (do-stuff))) - If you want to read parts of a file, use the BEG and END arguments to insert-file-contents.

Study org-map-entries and one call to that with your custom write function should do the trick, just ensure your file is in org-mode.

Note that your working solution is setting global variables with `setq`. You should be using `let`-bound variables.

You mean define them via let and set them later via setq?

Yes, that would be better than setting global variables. I mean, it's your Emacs session, but it's generally a bad practice. And if you ever intended to publish that code for others to use, it definitely shouldn't be done. :)

Emacs has a mode where it doesn't use the file extension to enhance the editor for the file type and it has the benefit greatly improved performance. Does the way you open your file honour this method? I can't remember the function name. I think it is `insert-file-contents`