(untitled)

Are there open source alternatives to GPT-3 that are good? So far I have tried >GPT-J Trained on garbage SEO blog posts and reddit content >Anything by EleutherAI (gpt-neo, etc) Same problem as above >Facebook opt Have tried the 2.7b version and it generates potato output >GPT-2 Potato output >GPT-4chan Ironically the best one, but it only writes short sentences Maybe Facebook opt-13b is better but it uses over 40GB RAM and I only have 32GB. Any ideas? >Just use gpt-3 bro I don't want everything I write being recorded and tied to my phone number.

>Trained on garbage SEO blog posts and reddit content Why not replace the data set then? If its open source you could take the algo and feed it with 4chan or whatever else. Add a word prefilter to delete obvious brewing soykaf and it should be fine. I dont know how expensive it is but training it on different filter selections of same data set and comparing results would be also interesting.

>>61036 The model is a 23GB binary file with the weights, and you must be able to fit a multiple of that into a GPU at once (at least 2x, I'm not sure the exact amount). Then the fine-tuning should take a week or two to complete. At least that's how GPT-4chan was created starting from GPT-J. Renting a server with this kind of capability for this long should be very expensive. Large language models are not very democratized at the moment.

>>61037 i dont believe you must load all at once, obviously faster and big guys got the big bucks to do that but any neural net has layers so you should be able to separate into layer chunks and compute each piece individually.

Have you tried GPT-NeoX? I've had pretty good results with it.

>>61025 Hoping that RWKV manages to deliver; it'll be a lot cheaper to train LLMs on your own datasets, and with Chinchilla-tier training and little luck, it might even be efficient enough for useful local inference.

BigScience BLOOM 560m model uses only ~8GB of RAM. With 32GB you might even be able to run the 1b1 model. https://huggingface.co/bigscience/bloom-1b1 The full 366b model (https://huggingface.co/bigscience/bloom) requires almost 1TB or RAM. But the smaller models get great results and can be easily fine-tuned since the entire model is open source.