Remove first element of vector? Or: Walk a nested object (JSON)

In the process of writing a package that will import `Roam Research` JSON dumps into `org-roam`, I try to parse the large JSON dump. This JSON dump is a representation of all the notes taken with Roam Research. As you might know, Roam Research is an outliner mainly, and each line of text is an outline element, which is represented as a nested (json) structure in the dump file. Now I am able to read the json dump and get a lisp-datastructure that represents the json data. The lisp datastructure that represents it consists of vectors of alists. Example: ;; Vector holding "nodes" or lists of alists [ ;; First element of the vector. Or: First node so to speak: ( (create-time . 1629555623149) (title . "August 16th, 2021") (:create/user (:user/uid . "DELETEDDELETEDDELETED")) (:log/id . 1629555623145) (children . [...]) (uid . "08-16-2021") (edit-time . 1629883978909) (:edit/user (:user/uid . "DELETEDDELETEDDELETED") ... As you can see, the lisp representation consists of a (top level) vector holding alists with keys-and-values. To get the data out of this structure I need to walk through this nested datastructure. So I'd like to write a recursive function that looks at the current (head) element of this list, analyse it, get the relevant data out of it and see if there are child-elements (which are again vectors of alists). Now comes the recursive part: My code will look at the child elements, and if there are any, then call the recursive function on the child node, and if there are no child elements, take the rest of the initial list and apply my recursive function to the vector with the removed first element. ## Question 1. How can I remove the first element from a vector? Could not find anything on the net. 2. Are there idiomatic ways in Elisp to walk a nested structure? 3. Is recursion too slow to parse a 10mb large json structure? ## Code fragments for reading / parsing the JSON This code snippet allows me to read a json dump and return a lisp data structure that represents the json dump: (require 'json) (defvar *roam-graph-json-dump-filename* "~/.emacs.d/lisp/process-roam-json/roam-research.json" "Will hold the filename of the Roam Research json dump.") (defvar *roam-json-alist* (ugt-read-roam-json) "Read Roam Research JSON dump, save resulting vector of alists in a var.") ;; https://emacs.stackexchange.com/questions/34331/how-do-you-get-data-by-key-from-a-json-list (defun ugt-read-roam-json nil (interactive) (let* ((filepath *roam-graph-json-dump-filename*) ;; `json-read-file' ;; returns a vector (list in Python lingo.) ;; of association lists. (json (json-read-file filepath))) json)) The raw JSON data starts like this: [ { "create-time": 1629113623149, "title": "August 16th, 2021", ":create/user": { ":user/uid": "DELETEDDELETEDDELETED" }, ":log/id": 1629113623145, "children": [ { "edit-time": 1632126970878, "children": [ { "string": "Montag", "create-time": 1632126263419, "heading": 2, ":create/user": { ":user/uid": "DELETEDDELETEDDELETED" }, "uid": "2dE80Sh8T", "edit-time": 1632126267229, ":edit/user": { ":user/uid": "DELETEDDELETEDDELETED" } }, { ... The lisp datastructure that represents it, consists of vectors of alists. Example: ;; Vector holding "nodes" or lists of alists [ ;; First element of the vector. Or: First node so to speak: ( (create-time . 1629555623149) (title . "August 16th, 2021") (:create/user (:user/uid . "DELETEDDELETEDDELETED")) (:log/id . 1629555623145) (children . [...]) (uid . "08-16-2021") (edit-time . 1629883978909) (:edit/user (:user/uid . "DELETEDDELETEDDELETED") ...

1. It’s very inefficient to “remove” an element from a vector regardless of the position. Instead, just start from the index `1` when you iterate over the vector. 2. Recursion is the easiest way. Define a walker that does the recursion if you’ve found a generalizable pattern. 3. I don’t know. Try benchmarking it.

>Instead, just start from the index > >1 > >when you iterate over the vector. But when I pass the "rest" of the vector (elements 1 to end) to my recursive function as new input, do I not need to create a copy of the vector, taking 2nd to last element? Edit Alternatively I am thinking of converting vector elements to lists, as I go. Because my gut feeling tells me that lists are more suitable to recursive functions.

You’ll need to allocate a new vector of the rest of the elements if you do so, and that’s where the inefficiency is from. Vectors aren’t like lists.

I am wondering if the json package does the right thing in creating vectors of lists, instead of lists of lists. Hm.

I remembered something I did with Scheme back in those days when I worked through the book SICP. - I don't need to create copys of the original list, just pass the same vector + the incremented position of the index! That'll do the trick.

persistent data structures, when?!

1. A vector is a fixed size array, so to "remove the first element" you either set it to nil or make a new copy, with predictable performance penalties. Or as you've figured out, just remember the index instead of changing the vector. 2. You can check how other libraries do it: dash (-tree-map uses recursion), dom.el, org, etc. 3. From what I know recursion is generally slower than loops unless it's in a language that knows to compile a recursive pattern into a loop. Other than that, there's too many unknowns here, so I think you'll just have to try it out for yourself.

looking at the [manual](https://www.gnu.org/software/emacs/manual/html_node/elisp/Parsing-JSON.html) it seems that you can specify which Lisp data structure should represent an array using the `:array-type` keyword argument and `list` is also an option.

Excellent! Thank you very much! (Note to myself: RTFM! :)