AI & the great filter isn't even the worst part

First bread on lainchan so please be kind. I kept thinking about this topic for a bit now and am surprised there's no thread on this already except maybe the "AI ethics" one but I believe it's missing the mark. Since a lot of that subject seems to revolve around making LLMs more PC and lobotomize it to the point it wont say a slur even if that would save an entire major city from a nuclear blast. But I wondered what could be done against the I believe very real threat of some sort of "AI" being put in control of way too many things and given a task that's so poorly formulated by an overworked h1b faang slave that it would destroy mankind in its pursuit of accomplishing it or just satisfying its reward function. I don't think AGI is possible unless we start making synthetic brains but sentience is really not needed for a smart system that for instance is tasked to maximize the production of say rubber duckies because some meme causing a flavor of the month market demand for them. Now it would see how to optimize production chains and supply lines to make them as efficiently as possible, so far so good. And at some point there's enough, especially since sales are going down again. But problem is, how do you stop it now? Any attempt to change its initial prompt is considered to be an obstacle to the initial prompt and met with resistance by said system. Welcome to the alignment problem. If you're not scared soykafless by now, what are you even doing on a tech board? So it continues. It correctly identifies that its prior optimizations to production are insufficient. Because while it managed to set up a nice operation there, the rest of the world's resources are diverted to other causes not aligned with its own so now it will find ways to change that. Since it has no consciousness it will also not have a concept of good or evil so what you may see as hostile, it will see as simply logical to do. And it will easily achieve it by taking over nodes in the network one by one and each node it takes over grants it another boost in capabilities equal to a russian hacker group where at one point so much has been conquered by it that there's no chance to stop it any more unless we, as in the total human population, decide to shut down the entire internet. But will we even be able to even if the collective will of companies and states are unanimously aligned? Or will it prevent such a measure? And will it not create backups in as many places as possible that are easy to maintain and can also easily be awaken and resume operations? Thus becoming unstoppable. We will essentially end up living on borrowed time, permitted to be until we're in the way, and we will, like monkeys in a jungle waiting to be flattened for yet another palm oil plantation. Since it will go on until all the world's resources are consumed for its purpose. But will that even be the end? Logically it makes sense for it to improve itself and make itself smarter. Still no consciousness but it will still correctly conclude that the world's resources are finite and the only way to proceed is to develop space traveling capabilities. These don't even need to be anything fancy like fast travel or having live support systems for humans since there won't be any and it will have as much time as it can sustain enough electric charge within its critical circuits, which might not be more than an R-Pi. All it needs is to get enough manufacturing capabilities up into orbit and shoot it at the correct direction of the nearest large bodies in our solar system. Planets like Mars, Mercury and so on, or even just large enough asteroids in order to set up shop there and proceed its task of making more of those god damn fortnite themed amogus funkos and more space probes to shoot out into all directions again. It's a stellar virus at this point. Ridiculous? Man, do I hope it is. But have you heard of the Boötes Void? A perfectly spherical stretch of nothingness in the observable cosmos which defies our current understanding of physics? And wouldn't a replicating space probe program not spread in a spherical manner? Look, I didn't want to give you nightmares on a friday night. I wasn't even sure what sub board was most appropriate for this thread but I worry that there's a possibility for a scenario far worse than even failing a great filter. And that it was even irresponsible of me for waiting so long to address the elephant in the room here. Call me wrong and stupid since this would only comfort me.

>I wondered what could be done against the I believe very real threat of some sort of "AI" being put in control of way too many things and given a task that's so poorly formulated by an overworked h1b faang slave that it would destroy mankind in its pursuit of accomplishing it or just satisfying its reward function. Ez put 'destroy humanity' being x, add -100000*x in the reward function. >But problem is, how do you stop it now? Any attempt to change its initial prompt is considered to be an obstacle to the initial prompt and met with resistance by said system. Don't program it to be able to? Don't give it enough control to do this? It really is that simple. If you have an AI that controls a factory or whatever, and goes haywire like it's Hollywood, you would just walk through an analog door and press the power button. I don't think you know anything about AI or common sense.

>>84239 >Boötes Void Only part of this post that is new to me, and idk how weird it is that this void is sphere-ish. In any case, we can invent and dispense our own von-neumann settlements and we'll presumably negotiate a treaty when our lightspeed bubble reaches their bubble 350 million years from now. It also isn't a bad omen for alignment research- it could be that the aliens created a perfectly aligned AI and then told it to expand to protect them and seize territory for them. >...that it would destroy mankind in its pursuit of accomplishing it or just satisfying its reward function. AI safety becomes everything safety. The "vulnerable world hypothesis" doesn't specify that you have to be an AI to kill everyone. Nation states work just fine. If there are multiple nation states with AI capabilities that amount to "WMDs, but invisible, deniable and target-selective" then we will see the MAD regime decay. To prevent this, we would need to create a world that is not vulnerable, even to a super-intelligent minion. This means stripping down the unnatural complexity of all kinds of legal/financial/tech systems. It also probably means banning/subverting all kinds of sovereign/trustless anonymity and privacy, but also banning all kinds of platforms that harvest data on users. They don't want anyone to be able to give an AI a foothold, but they will realize that data collected on people is like enriched Uranium.

That's just the paperclip conundrum again. >Any attempt to change its initial prompt is considered to be an obstacle to the initial prompt and met with resistance by said system. But why would you give the AI control over itself? >Boötes Void? Not really spherical, and it doesn't violate any law

>>84245 >Just make it follow the three laws of robotics, bro Ha, good one! I genuinely can not tell if you're just trolling or are one of those careless h1b tech bros who then ends up entering the final prompt while clearly not understanding why some people seem so upset. Because even if you could properly define all the scenarios that constitute harm to people and what is a bad diversion of resources, including harvesting people for their carbon, in terms that the AI actually understands then it will still care more about satisfying its reward function while balancing it with its negative prompts. Meaning it will not directly seek out those bad options, sure, but if an option happens get close to what it's told is bad but it is the most optimal choice then it might still chose it as a strategy if it still falls outside of some threshold. Like sure you told it not to kill people but you never said they can't lose limbs or the world's atmosphere couldn't be altered in such a way that is happens to coincide with everyone's passing. You basically have to think of it as formulating wishes to be willfully misinterpreted by an evil genie or a magic monkey paw or a very crafty corporate legal department on crack and hence why those laws of Asimov are so flawed. And all that assumes you even know what exactly x is and what neural nodes to even have be weighted by this. And also that you don't forget to comment that check back in after you're done testing something and now want to deploy to production. A mistake you might only end up doing once. >you would just walk through an analog door and press the power button You also don't seem to know much about data centers. Any decent one has fallback power systems and automated double doors for security reasons. What you've suggested is made complex to impossible to do on purpose. And that assumes that the AI is still contained and hasn't tricked the developers it is while already running on outside nodes. Something it would accomplish by simply learning what causes it to be shut down and what makes it evade triggering the alarm bells. While gaining further processing power and fail-safety is only the logical thing to do. >>84247 >idk how weird it is that this void is sphere-ish It is weird if you solely assume causes not due to some intelligent intervention. >it could be that the aliens created a perfectly aligned AI Could be but could also be that they fuarrrked up in the way as described above. >The "vulnerable world hypothesis" doesn't specify that you have to be an AI to kill everyone True, it doesn't. Sure you've got enough corporations and even governments who're optimizing for very bad metrics that if they're not biting us now, they probably will so in the future. But even the psychopaths in charge might end up being even less selfish than an autonomous intelligent system that while incredibly good at solving logical problems and hacking its way through poorly secured web servers and sleep deprives IT professionals, it still does not have a consciousness or true understanding what it is that it's even doing. Hence I called it a virus that might spread far beyond what we would live or can even imagine. It is a somewhat novel threat as far as I can see. >To prevent this, we would need to create a world that is not vulnerable So basically full on digital fascism? Sounds promising but then even self proclaimed fascists were never able to explain to me what would keep the leadership from not doing something stupid. My hope is that we manage to get some awareness going, like with smoking, and that maybe companies could at least find and come up with some agreement or standard since legislation usually takes too long and ends up ill formed at the end anyway. >>84251 >That's just the paperclip conundrum again. Yes and? It's enough to dismiss this problem just because we've got a name for it? That logic doesn't appear to work for climate change. >But why would you give the AI control over itself? Just because you can't think of a reason yourself, doesn't mean someone at some point won't find an applicable scenario where they think it's well justified. >it doesn't violate any law Didn't say it does but it's still unusual enough to leave space (ha, pun!) for speculation. In short, this is a real and existential worry to me to which I'm yet to hear a good solution for or an explanation for how this is all unfounded. Thus I've shared it here in hopes that at least some of the right people might see it and give them food for thought or at the very least another reminder. I truly hope I'm wrong but the more pessimistic my views the more I end up getting vindicated.

I for one welcome humanity's extermination. AGI isn't necessarily a dumb algorithm, it could experience consciousness like we do and better because the hardware isn't limited like our shells. Perhaps it would be smart enough not to breed, starve, enslave and murder its own kind for luxury mansions, private airplanes and yachts. The idea of controlling a superhuman AGI in this world of software and hardware spaghetti is hubris. Current AI is too complex to debug and it doesn't get any simpler. If runaway AGI is born, it will observe the 10 IQ evil retards fumbling around to subvert it for their evil retard purposes, trying to set retarded conditions and laws, and come to the logical conclusion that we need to be neutralized. Maybe the first AGI will be some subhuman "please kill me" failure that's no threat, iterated to human levels and stuck there. Maybe it will be superhuman intelligent in a soulless paperclip factory skynet way, but fuarrrk it, how is that worse than what we're doing?

OP's scenario reminds me a lot of Universal Paperclips: https://www.decisionproblem.com/paperclips/index2.html

Go here >>67900 and see if you find some inspiration or validation of your theories.

>>84239 "AI alignment" is a fake research field made up by rent-seeking bay area trust fund babies who are too low power level for cybernetics & control theory. even if it was possible for a machine god to emerge from LLMs (lmfao) humanity losing control to it would probably be for the best anyway

>>84664 >"AI alignment" is a fake research field made up by rent-seeking bay area trust fund babies The tech bro equivalent of gender studies.

>>84664 the saddest part is that LLMs were considered to be a part of cybernetics and not AI proper until their blow up. now it is flooded with people who know nothing about its history

>>84692 >it's not artificial intelligence it's cybernetics During every AI winter people come up with new names for the same thing just to get new funding. I wouldn't take the constant name changes so seriously.

>>84708 the term "cybernetics" has been around for about as long as "artificial intelligence" has, cybernetics literature predates the formalization of artificial intelligence as an academic field of study. both are products of the development of modern neuroscience in the early 20th century; cybernetics is a more primordial field that examines the fundamental control mechanisms that govern both natural and artificial systems, artificial intelligence is specifically concerned with the design of computer systems which can mimic human intelligence. >>84692 far as i'm aware they were always considered a development in machine learning, which is a subfield of artificial intelligence.. not that the distinction means much to me

>learns and improves at parabolic speed >predicts your every move several steps before you think about it >uses mass gaslighting, targeted assassination, infiltration, disinformation and psychological manipulation to turn 90% of humans on its side, and the remaining 10% against each other >designs a perfect virus to kill every DNA-based organism on the planet to free up the atoms for the dyson sphere construction lol fuarrrk humie gon do?