Artificial Intelligence Desktop Environments / AIDE / IRL Navi

We're in the early stages of the AI/ML era where we can run myriad AI programs locally on our desktops, without sending our data to some server owned by Google or other megacorps. Popular smartphone systems already implement a basic form of AIDE, but they phone home for everything and are pretty restrictive. We now have the tools available to integrate AI into our desktop and development environments. For example, OpenAI's new Whisper model provides reliable speech to text that can be run locally: https://github.com/ggerganov/whisper.cpp https://github.com/openai/whisper I think we could create a system that integrates Whisper into a modal input system. It could have speech to text mode, command mode (for executing bash scripts), and hooks into things like navigation and window and hardware management. We could do all of our daily computing tasks while rarely touching our mice and keyboards. Mycroft AI seems like a promising, though extremely basic implementation. It uses internet for speech to text at the moment, but could be modified to use Whisper. https://mycroft-ai.gitbook.io/docs/ Of course, the natural extension of such a system would be a separate AI that learns scripts/macros from our manual inputs. For example, I should be able to say, "Navi, record a macro for opening a new thread on Lainchan," then open a browser window to https://lainchan.org/%CE%A9/, put my cursor in the "Comment" box, and then say "Navi, end recording." This system could be so extensible. It could be hooked up to generative AI systems or companions. The possibilities are really endless, but the basic implementation

>>66277 >We could do all of our daily computing tasks while rarely touching our mice and keyboards. I understand your excitement, but this isn't a good thing. Keyboards are an enduringly high-bandwidth high-accuracy device. Perhaps there is a niche for handsfree computing for tradesmen, but sitting down at a computer and working in formal language is not a curse, but a blessing.

>speech to text to bash jesus christ you inept morons never stop coming up with stupid soykaf so hard it takes skill to make something so broken

>>66309 It'd help reduce a lot of repetitive stress injuries. I personally love mouse and keyboard as input devices, but would still like to have voice commands as another input method. >>66316 Care to explain what's stupid about voice as an input method?

>speech to text to bash What could possibly go wrong, be sure to pronounce shebang properly and dont forget the slash in between directories.

>>66347 Not the same lainon but: How do you differentiate between caps? Words? Spaces? What if you download a song that has foreign characters or an emoji? Tab completion? Do you have to send the command to, say, pause your music or something? I'm not saying it'll never work it just sounds really unnecessary compared to the good ol cyberdeck.

>>66354 So type in the bash commands and then create a speech alias for them. You know, the way most DEs have options to bind commands to keyboard shortcuts. >>66356 So use a keyboard when STT falls short. >Do you have to send the command to, say, pause your music or something? This already works with "smart home" spy devices. Eg you say "alexa, stop" to pause music. The difference with running AI locally is that the functionality is controlled by the user and not spying on you. I have to ask, have any of the people posting ITT even watched Serial Experiments Lain??

>>66367 >So use a keyboard when STT falls short. But it falls short essentially all of the time, especially if you work with formal languages. STT is only useful if you literally can't use your hands, and even then it's a huge handicap, not to mention laggy and error prone. There is nothing I do with my computer that I would generally prefer executed as a voice command.

Problem is without a sort of vocal fingerprint or other way of telling who is speaking just anyone can be a jerk and control your computer without having to even touch it. There are certain use cases involving the disabled and people who are doing something else with their hands such as building circuits or painting or driving and there also is use for computing while standing or walking around without needing a standing desk. Maybe Whisper could be modified for TTS to make for more interaction. Found a discussion on it https://github.com/openai/whisper/discussions/748 I think it would be fun to use something like Open J-Talk for the voice synthesis. You could make use of various existing voice banks from different programs. This is likely quicker responding than a deep learning program that clones voices which is what you would have used if you wanted a copy of the Navi voice which I imagine would be difficult to get good results from. There is quite a few other AI assistants in development out there to look at other than Mycroft though some of these may not be suitable for OPs intent. https://getleon.ai/ https://sepia-framework.github.io/ https://www.home-assistant.io/ https://aimybox.com/ https://rhasspy.readthedocs.io/en/latest/ https://kalliope-project.github.io/ https://slapbot.github.io/ https://github.com/DragonComputer/Dragonfire https://jasperproject.github.io/ https://github.com/sukeesh/Jarvis https://linto.ai/ https://github.com/rcbyron/hey-athena-client https://openassistant.org/wp/ This one was a recent attempt to replicate chatGPT. Apparently he didnt know there already was something called open assistant i guess. https://github.com/LAION-AI/Open-Assistant Should note this one is funded by the French government for a future investment thing https://linto.ai/ >>66367 They probably haven't watched it. Just a couple years ago everyone would have got it.

>>66277 Recently I came across this project https://numenvoice.com/ and it looks like it aligns pretty well with your wishes. I don't know if STT on a personal computer is very useful but I think it's pretty cool.

>>66370 >But it falls short essentially all of the time This is only true of primitive STT tech. Recently, I have been able to transcribe a lot of my daily (non-private) communications using STT on my phone. It's faster and often more accurate than typing on a screen keyboard already. While mouse and keyboard are excellent (for skilled users), they still come with drawbacks that make it attractive to have alternative input methods that coexist with them. You have to be sedentary while using them, and be in a certain posture. They're not as great if you're trying to eat food, or do a number of other activities such as drawing. Another form of AI-assisted input that could be good (or even outright superior) is eye-tracking as a replacement for mice. VR headsets are implementing this. Keyboards also don't work great with VR headsets. So things like local STT have a lot of potential to assist with VR. Hell, why not sign language? Custom sign language combined with eye tracking and STT could help cover the weak spots of plain STT (eg, special characters) and be just as fast >>66376 >Problem is without a sort of vocal fingerprint or other way of telling who is speaking just anyone can be a jerk and control your computer Voiceprint is a solved problem, shouldn't be that hard to combine with the other tech. >>66380 >>66376 Thanks for the links!

>>66376 >>66380 Out of all of these desktop AI systems mentioned ITT so far, which would you say seems furthest along to being useful for practical computing, rather than smarthome stuff?

I just want something to regurgitate and expand on a bunch of multi media (images, pdf) lore I have made for DnD. I'm a nerd just not a computer nerd. Is there a way to make a folder and have an AI focus entirely on it and progressively make better lore as I give it a 0 to 10 rating each time? On a similar note I want to create individual egos that are restricted to a respective folder of ideological memes and then have them do battle (argue and soykaf post) in a self contained environment.

>>66406 Is this (novel AI) what I need?

>>66408 The module training feature can work well but the amount of input content it can take is limited. I would say it is worth the ten bucks to try it out for that purpose.

>>66410 Tru Ill have to give it a try then.

>>66400 Eye tracking has some issues as a sole input in some implimentations due to unctrolled eye wandering people often do and the security risk of an always on camera unless it used a dedicated camera and seporate processing to create some sort of gap between the operating system and the camera unless i got the wrong idea in mind then never mind. Voiceprint would of course need encrypted for security purposes. The system also would need to eliminate audio that is played through the speakers so recordings of you or a voice clone wouldnt trigger any action. Also the microphone would need covered with foam or something to reduce risk of the unlikely use of laser speakers to remotely control a computer within line of sight.