Neural Network / GAN Art

Has anyone here tried working with generative adversarial networks? Seems like an interesting new direction for generative/glitch art that has really started to take off after the publication of google's BigGAN paper (and associated pre-trained model). Here's a cool website someone made using BigGAN: https://ganbreeder.app/ In order to create animations like pic related I wrote a script that takes "key" latent space points from ganbreeder and then interpolates through them. You can find the code here: https://gitlab.com/Vee9ahd1/gantools/ I've also done some stuff with StyleGAN and I've seen other people get cool results from pix2pix and ProGAN. Finally, here's a link dump of papers and code I found interesting and relevant. Note that a lot of the github repos link to associated papers that are worth reading if you can understand the math. bigGAN related: https://arxiv.org/abs/1809.11096 https://tfhub.dev/s?q=biggan https://github.com/joel-simon/ganbreeder/ https://github.com/huggingface/pytorch-pretrained-BigGAN https://gist.github.com/htoyryla/bef3c9935de9dba844c6d693ecb5ed4d styleGAN related: https://github.com/NVlabs/stylegan https://github.com/Puzer/stylegan-encoder https://github.com/Aiterasu/stylegan misc: https://arxiv.org/abs/1905.08233 https://github.com/tkarras/progressive_growing_of_gans https://github.com/apple2373/pytorch-small-dataset-image-generation https://github.com/phillipi/pix2pix https://github.com/openai/gpt-2 https://open_nsfw.gitlab.io/ https://github.com/NVlabs/MUNIT https://www.gwern.net/Faces

A couple more animations made with BigGAN

This one was done with StyleGAN using nvidia's pretrained ffhq model.

I love Ganbreeder, these look amazing OP. However, when I try to run this, everything seems to work until I get these errors at the end. With this an input: 08f197b18ed8b0b4120bc1bc c4c5e7ad812909ec2a817ce0 34d8862ec05524eee0e76a4d I get this an error: ValueError: Cannot feed value of shape (1, 140) for Tensor 'z:0', which has shape '(?, 128)' And with this an input: 7095a13cdd05f6051d04ead6 89d5c1aec44e1296c93fc1ed d7b65ec5c3f2941f4830594a I get this an error: 2019-07-13 12:36:22.110862: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order. I'm on Windows 7, if it matters.

>>3513 >topological sort fail... that's a warning coming from tensorflow you can ignore it as long as the script isn't totally crashing out. just be patient the first time you run it, it has to download the model from tfhub which can take several minutes depending on your internet connection. idk where that other error is coming from. it's saying that the z vector it's getting from ganbreeder is 1x140 instead of 1x128. if you keep getting that one let me know and i'll take a closer look.

>>3514 The errors seem to the be last thing to appear, and that's after it's done downloading the model (BigGAN?) Then it just waits for a minute and says "Done". Also, the first error in that other post is probably because I tried using older ganbreeder images back when they were 256x256px. My specs are Ryzen 5 1600 / 1050Ti / 8GB of RAM.

>>3513 >>3514 i figured out what the problem is. ganbreeder used to use biggan-256 which for whatever reason has a 140 element input/z vector instead of the 128 element vector of biggan-256. you can see that here: https://ganbreeder.app/info?k=08f197b18ed8b0b4120bc1bc . as a result older ganbreeder nodes are incompatible with my script. if you'd really like to make an animation with these older (lower res) nodes you can open up biggan.py and edit the MODULE_PATH line to this: MODULE_PATH = 'https://tfhub.dev/deepmind/biggan-256/2' note: i'm guessing ganbreeder uses the biggan-256 v2 model from tfhub but if you try that and the images dont come out right you may want to try v1 or even biggan-deep-256. >>3515 it worked check the directory you ran it in you should see images. without going into too much detail those errors are coming from the tensorflow implementation of biggan. tensorflow errs on the side of posting way too much information since its really meant to be a back end or research type tool but it can recover from those errors. as long as the script executed properly and printed "Done." you should be good to go.

>>3516 128 element input vector of biggan-512* not 256

>>3516 >it worked check the directory you ran it in Ah soykaf, it all the images were being stored in system32 - I expected them to be in the python folder. This is some amazing soykaf, OP.

>>3516 Another one. Damnit OP, now I'm gonna be obsessed with this soykaf for weeks.

>>3518 >>3519 nice work! glad you like it. you can set the output folder with the `-o` flag: -o OUTPUT_DIR, --output-dir OUTPUT_DIR Directory path for output images. (default: None)

>>3516 >>3516 I just tried the 256 model and it works! Pic related was done with it. Note to anyone else doing this - the first and last frames it generates are identical, so delete one of them unless you want a stutter when it loops. You can see it stutter here: >>3518

Moar pics The source images in the second animation all had a (nearly) black background, but it forms into a rock-like texture at one point. Any idea why it sometimes chooses to generate its' own background texture?

>>3522 >Any idea why it sometimes chooses to generate its' own background texture? i'm going to give some background info before actually answering that question. you may already know this stuff but i figure it's somewhat specialized knowledge so I might as well explain for anyone who doesn't. the interpolation script isn't like a post-processing effect applied to images from ganbreeder. it uses ganbreeder only to obtain a z vector and a label vector (it simply uses the ganbreeder.app/info?k= api endpoint; the "label" vector is literally just the mix of genes you see on the website). the space of all possible such vectors is typically called the "latent space", and the keys you give it provide the script with points in the latent space, not actual images yet. it then constructs a cubic spline which passes through these "keyframe" points (in the language of splines i think they would be called "knots"). by sampling along this spline and then feeding the resulting samples into a self hosted instance of bigGAN it is able to render a relatively smooth animation. why is the animation smooth? basically, it's because the generator function is continuous (afaik there's usually some kind of lipschitz condition involved). this means that a continuous path through the input space will produce a similarly continuous path through the image space. the coolest thing about GANs is the way this image space continuity manifests itself. because the latent space is organized "semantically" interpolating between two different points will "morph" the structure of the first image into the other one rather than simply fading between them. google's paper gives the example of "puppy flowers" but im sure if you've played around with ganbreeder you know what i'm talking about. so finally, to answer your question, the background of the image is not tied as tightly to the semantic organization of the latent space as the subject is. for instance, latent vectors that produce dogs will be close to eachother in the latent space regardless of whether the dog is on grass or cement or whatever. so when you create a trajectory that passes through a bunch of "dog on grass" points, there's a pretty good chance it's going to pass through a region of "dog on cement". it just so happens that the path between your black background points ran through a region of rock background points (in my experience these backgrounds tend to come up often with the "sea creature" genes). i hope that all makes sense, but feel free to ask follow up questions if anything is unclear to you.

We gotta do this with porn. I mean it's the obvious decision. When I get home I might download this and take a whack at it

>>3524 https://open_nsfw.gitlab.io/

>>3523 I don't know what a 'Libshits condition' is, but I assumed it was because when creating inbetween frames, the program was using images from ganbreeder that were different from the ones I fed it. And since some of those inbetween images had textures that were not present in the input images, I sometimes got random background changes. Seems that was about right?

>>3526 In any case, this is fascinating - I've never noticed the connection between something like this and those trippy Google dogs. Now I can't unsee it.

god DAMN this is cool as fuarrrk

I once made a video which took all the neuralgae frames and made an animation out of it. Much less structured than what you did here.

>>3526 >the program was using images from ganbreeder that were different from the ones I fed it sort of. it doesn't just download a bunch of images from ganbreeder and then put them together if thats what you mean. the only thing it gets from ganbreeder are those initial key vectors then it renders everything locally along a smooth path it constructs through the latent space.

>>3533 So if everything is local, then the BigGAN dataset I've downloaded actually contains all the information like "sea creature = rock background"? How large is the file, actually? I can't find it. And is this the exact same dataset that Ganbreeder uses, or have they expanded it?

>>3525 Awe ye that's good soykaf. Now I gotta make some animated ones.

>>3534 The model gets stored in your tfhub cache directory (depends on your system, idk where it is on windows). Here's some information on that https://medium.com/@xianbao.qian/how-to-run-tf-hub-locally-without-internet-connection-4506b850a915 . However, it doesn't actually use the dataset directly; it uses a model which was pretrained on the dataset by google's deepmind team. The network is making an inference based on initial conditions you give it; you can think of it like an "image synthesizer" as opposed to an "image sampler" or something. Both ganbreeder and this script use the same model (specifically, this one: https://tfhub.dev/deepmind/biggan-512/2) so that's why they are in a sense compatible. The dataset in question is called ImageNet (http://www.image-net.org/) and it's a pretty standard one among machine learning researchers. One of the most useful features of BigGAN is "conditional inference". In other words, the model is able to infer an image raster from an input vector, contingent upon a "label" vector that tells it what it's supposed to be creating. The "genes" you see on ganbreeder are the standard ImageNet labels that were used in training the model. In implementation it's basically an enumerated type and you can see the whole list here: https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a . Here's an example of how this is all used, taken from tfhub. import tensorflow as tf import tensorflow-hub as hub # Load BigGAN 512 module. module = hub.Module('https://tfhub.dev/deepmind/biggan-512/2') # Sample random noise (z) and ImageNet label (y) inputs. batch_size = 8 truncation = 0.5 # scalar truncation value in [0.02, 1.0] z = truncation * tf.random.truncated_normal([batch_size, 128]) # noise sample y_index = tf.random.uniform([batch_size], maxval=1000, dtype=tf.int32) y = tf.one_hot(y_index, 1000) # one-hot ImageNet label # Call BigGAN on a dict of the inputs to generate a batch of images with shape # [8, 512, 512, 3] and range [-1, 1]. samples = module(dict(y=y, z=z, truncation=truncation)) If you change that line to y_index = 11 you will get a goldfish because the imagenet label for "goldfish" is 11. Notice how the actual input to the model is a 1000 element one-hot vector? Well, instead of just setting a single label to 1 you can define a mix of different labels, which is how ganbreeder works. I'd recommend normalizing it for more consistent results.

this might also help https://gitlab.com/snippets/1869377