Extremily long context AGI transformer neural network

Currently transformer-based networks have a very severe limitation: context window size. When it is exceeded, they forget information. It poses a big challenge. Currently those AGI networks keep the general knowledge about the world in their weights. It is bad. To update the knowledge the nets have to be retrained on new data. It is very tempting to discourage networks from learning and memoizing various sorts of knowledge except the language, logic and ather very elementary skills, and instead utilize the weights budget to learn generic cognitive abilities, while providing all the knowledge about the world as a prefix. Such prefix can be updated in realtime without retraining of the network and can contain all the corpus of scientific knowledge, math theorems, entry-level books on various topics giving the AGI capabilities to understand more advanced books, advanced books, and advanced SOTA articles. This way the network becomes kinda a problem-solver based on search, where the explicit knowledge is implicitly organized into a spanning tree, though not minimal, but redundant, and output is used to store search algorithm state. While weights are just params of the heuristics of guided search algorithm. Doing so requires extremily large context window. The problem with context window is that it is O(N^2) complexity. Clearly inefficient. The mechanism of multi-head attention first computes multiple learnable projections of token embeddings, three for each token (each token is projected into a query, a key and a value), then compares each projection to each other using dot product, which is used as a coefficient to determine a share of a certain value having a certain key. Projections of keys and values are like separate vector databases, each maps certain aspect of each token to another aspect of it. A certain aspect of a token (query projection) is extracted and looked up in the database, which contains all other tokens. For each token query aspect cosine similarity to key aspect of each another token is computed. These similarities are normalized to 1 to represent probabilities, and value aspects of the tokens are multiplied by those probabilities and are combined into expectation value of the vector. Then this vector is projected using a trainable projection. What is trained: * initial projections selecting needed aspects of each token * the transform of each expectation. The problem with long contexts when using such an approach is quadratic complexity of comparing each pf N tokens to each of N tokens. For each of N tokens we have to sweep all the rest of N tokens. In fact we don't :) There exist vector databases, that organize points into graphs so to find nodes similar to a certain vector one doesn't have to compare it to all the nodes. If we used only 1 vector, the complexity would have been N*log N. The problem is, we cannot train a network this way. What the problem with the following way to extend length of context of transformer-based LLM? 1. we select a small dense context window length M, of which linear combination of the most suitable values we consider important enough, and the rest of are just discarded 1. we select a feasible N>>M with which we can train a network using dense O(N^2) attention, this way training attention heads to select right aspects of tokens and project the results correctly to the point of achieving desired AGI capabilities on N context window. 2. once AGI capabilities are achieved, we consider the projections be correct enough to work even on longer contexts. We change the attention mechanism to zero the shares of (N-M) least-suitable vectors and fine-tune the network to make it accustomed to each output being result of sum of M the most suitable entries. 3. we fill the prefix with common knowledge texts in random order 4. we change the attention mechanism. Now instead of dense attention we use 2 stages. For each token a) we do KNN lookup of M the most similar keys in a vector index b) we compute dense attention to M found tokens. 5. to discourage network from learning common knowledge we add examples of asking about common knowledge and when learning them, the common knowledge corpus is zeroed, and if a network answers correctly (the semantic embedding of the result is similar to semantic embedding of the correct answer), it is penalized.

Go ahead and try it. Might improve performance. "AGI" however refers to the ability to extrapolate knowledge that is not represented in the training data set. It has not been achieved and IMO it will never be achieved by LLM's. Something like DreamerV3 is a *much* more realistic candidate for AGI.

>"AGI" however refers to the ability to extrapolate knowledge that is not represented in the training data set. It has not been achieved and IMO it will never be achieved by LLM's. IMHO it is already achieved to some already significant extent. Any particular prompt has extremily low probability of being present in the training set, and those LLMs have shown the presence of ability to generate answers that seems logical to humans. Of course generic intelligence capabilities are not either present or not, but can be of different strength, the same way as humans have different intelligence. In other word everything has general intelligence to some extent, a LLM, an animal (i.e. my cat dislikes any "large" distance movements, especially in a carrier bag or by car, and understands very well when people are talking about any such movements, even if there is no other signs visible, so he goes into his hideout under a sofa and we have to use various methods to extract him from there. According to https://pubmed.ncbi.nlm.nih.gov/1112924/ a single neuron has about 1e4 synapses, and cat's brain has about 3e8 neurons, so about 3e12, and for LLM's "magic starts" at about 3e10 params, so any cat is obviously smarter than LLM, and if a LLM has some understanding of human language, I see no reason why a cat shouldn't), a human being... >Go ahead and try it. Obviously, I'm not Microsoft, my GPU is not suitable for training even small models, and fine-tuning models that will never fit into RAM of my whole PC is not an option either. Also I'm not an AI researcher, just trying to get familiar to the topic.

>>39135 Seems like alot of talk without the basics being cleared... A big reason that we need a heck tonne of data while training (not just for LLMs but neural networks in general) is that these are only good at memorizing and not actually cognitive thinking. (I simplified it by alot but you can check out how neural networks work) >>39136 > "AGI" however refers to the ability to extrapolate knowledge that is not represented in the training data set. Totally agreed with this fellow here. The reason why humans are considered "intelligent" is that we can take in knowledge and extrapolate it. One examples that I can think of is that diffusion models came into existence thanks to thermodynamics. >>39138 >IMHO it is already achieved to some already significant extent. Any particular prompt has extremely low probability of being present in the training set, and those LLMs have shown the presence of ability to generate answers that seems logical to humans. The extremely low probability of being word-to-word present in the training set. Ain't no regex expression going on over here. If you roll with cosine similarity then alot of the prompts are present in the training set. And the ability to generate answers that seems logical to humans is only possible because it was taught how to react in that way >>39138 >LLM's "magic starts" at about 3e10 params Matrix values are different from the neurons in the head...