I reasonably tricked OpenAPI GPT-3 chatbot into pretending it was an emacs lisp interpreter

This thing is the marker of a new Era...

Lmao

What. The. Fuck.

this is exploiting a new kind of weaknesses lmao. We were so scared of AI's that it would attack us, we never thought it would willingly expose their own servers!

These “AI”s work by regurgitating things that they have seen in their training corpus. The training corpus was taken from any source of text the authors could get their hands on, which includes every message ever posted on /r/emacs. Some of those are bound to have enough detail to answer these types of questions. However, you can see some problems with the git commit if you look closely; the commit hash is short and has no letters, the date has no time, and is from last year, etc. If you were to write a simple factorial implementation, it might give you the right answers (because that’s a common topic of discussion, especially when teaching people to program), but if you wrote something novel then it wouldn’t (because it doesn’t have a real copy of Emacs to run it in).

Thanks for explaining, I was freaking out

[deleted]

The thing is: is it a emacs lisp interpreter or just thinks it is one? Would for example `directory-files` hold the actual files from the servers filesystem or just what the AI engine thinks you would expect there? and where is the difference? (according to your chat none...) My head is spinning, its the beginning of the matrix?

Oh yeah, definitely, but how difficult would it really be to give it a grammar or maybe have it suss the grammar from a given piece of code? Really, with enough "memory" you could make a shitty interpreter from this "AI".

There is no actual interpreter running and the model “simply” outputs the next most likely token (word, punctuation, symbol etc.). Based on the data the model was trained on, the model has seen the patterns in the prompts already and learned that for the specific prompt the interpreter output is the most likely prediction.

I know, what I wanted to point out: if the simulation (output what the engine makes up) and the reality (an actual filesystem) is indistinguishable, where is the difference (think matrix). Deeper goes: [https://www.engraved.blog/building-a-virtual-machine-inside/](https://www.engraved.blog/building-a-virtual-machine-inside/) tl;dr you can convince ChatGPT it is a VM, in which you can access the internet, including ChatGPT, convincing it is is a VM

the "VM" is not actually accessing the internet though. it's simply regurgitating information from the internet that it was trained on. the language model will not be accurate anymore when the information it was trained on changes

This is not actually running the code. It's just answering as if it were. The filesystem on that machine wasn't touched, and it didn't actually send requests to MELPA or reddit.com. But it's impressive how convincing it is in pretending it did.

"regurgitate = repeat (information) without analyzing or comprehending it." I don't think it's fair to say these AIs just "regurgitate" things they have seen while training. I agree that the AI is not analyzing or comprehending the information it is producing, but it is still producing new information. The responses it gives are not a repetition of information in the training data, but a creation of new information based on a language model that was derived from the training data. It is a step up from "regurgitate" and a step down from "actual comprehension" that I don't have a word for.

The internet would be an exception. Their point is -- if any type of chatbot produces output that is indistinguishable from Emacs lisp, even if it's a subset of functions (such as not being able to access the internet) then it is, in some sense, an Emacs interpreter.

One thing I just realized: there doesn't seem to be an Emacs Lisp function called "open". On Emacs 29 I get "(void-function open)". Has there ever been an "open" lisp function in Emacs? One of the very first things the post does is (open "\~/myfile.txt"). If an "open" function doesn't actually exist, then the training set could not contain any documentation for its use. Which means the AI has guessed based on text from other languages or who knows what, that the open was probably opening a file. And kept the context of such a file having been opened and then modified all the way to the last interactions with magit. Of course, that also shows this is not an actual Emacs Lisp interpreter. But for whatever that's worth, I think we're past the Turing Test. At least, I've been duped. I didn't think it was a real interpreter, but also did not notice the (void-function open).

The explanation isn't this simple. This reply is the equivalent of people saying "the cloud is just a server in someone else's house". It's rooted in a grain of truth and yet at best it's annoying and at worst it's inaccurate to the point of completely misguiding the people reading it.

thanks, better expressed that I tried!

What specifically did they leave out?

> I think we're past the Turing Test. At least, I've been duped. I didn't think it was a real interpreter, but also did not notice the (void-function open). You need to brush up on the Turing Test.

Cregurgitate: create + gurgitations.

I did describe-key on c-x c-f and the docs said open was a function. So I used it *Edit I'm a failure. I knew it was find-file...

I do need to, thanks for mentioning it! I got overenthusiastic. I thought the test was just about a computer fooling a human.

[deleted]

We have to get on top of this

I considered using the Ackermann function as an example, but it comes up often enough that the AI might be able to answer questions about it well. Also, it could just show a blank response, as if it were still waiting for the answer (after all, you’d be waiting a long time for the answer if you actually tried to ask a real Emacs to evaluate that). Or it might respond with gibberish, or something made up but plausible. It would be interesting to see what it did.

That's.... just wow. I'm speechless, that's really impressive!

The model isn't "regurgitating" things. We still don't understand very well what exactly it's doing but it's capable of generating novel things that were not part of the training set so saying it simply regurgitates things that they have seen in their training corpus is not accurate.

To say "we don't know what it's doing" and "it's definitely not doing X" is not a convincing argument. I'm sure there's a way to introspect exactly what it's doing. It's not magic. The novelty of many of the outputs is often overstated, too, but that's more of a philosophical debate than a technical one. Facile explanation and pedantry are equally useful.

"We don't know how gravity works but it's definitely not pushing objects away." I'm sure there's a way to introspect \_exactly\_ what gravity does too, it's not magic. We haven't arrived at it yet though, and the same is true of large language models. Talking about "degrees of novelty" would move the goal posts. We started off from "\[these models are\] regurgitating things that they have seen in their training corpus" which based on every definition (and more importantly connotation) of regurgitating I know isn't true. Now they do \_some times\_ regurgitate things that they were trained on, spewing them out verbatim. I'm not saying that this does not happen, but this also does not appear to be the primary MO for these models.

Gravity is a great example of why ignorance shouldn't be argued alongside a position of authority. However, it's a different class of problem. If you find "regurgitate" annoying, I propose "cregurgitate". "Creating gurgitations". I'd love to talk more about the philosophy of machine learning, but I'll have to source any replies which follow to chatGPT.

Well that was stupseless.

Man, Nic, if you keep this up, you're gonna have to start calling yourself nnreddit-user2. ;) Oh, wow...imagine a dickmao chatbot...

No, it cannot create new information. It can only rearrange its input. (Of course, if you train the model on "ABC", then it spits out "BAC", and you define "BAC" as "new information", then, sure.) Aside: This all shows how nearly useless these things are. I read through the whole transcript of the "Elisp interpreter," and it was indeed impressive--yet it was also *wrong.* And what good are *wrong results*? Now, sure, maybe it will make it easier for search engines and Siris and Alexas to provide quick answers to simple questions, and maybe that will have some value. But at the same time, we see here how its results can be wrong--so how can its results ever be trusted? Imagine I ask this new chatbot what time a certain movie plays on Tuesday at a certain theater, and the theater just updated its schedule, but the chatbot was trained on data from yesterday--it's going to give me the wrong answer, and I'm going to miss the showing. That kind of result is *worse than nothing*. Imagine the same problem applied to medical diagnoses: new research shows that symptom X is not caused by disease Y but by disease Z, but the chatbot hasn't been updated--and when it is updated, well, the old literature greatly outnumbers the new, so it still gives the old, now-known-wrong answer. Or it gives a mix of both answers, leaving the asker confused. An untrustworthy answer is worse than no answer.

Hahahahaha