My AI Enhancement workflow (Patreon)

Hi, I got a request to briefly explain how I do the enhancing of renders with AI so I thought it neat to post a few screenshots of the process for anyone who wishes to try their hand on.

First, which AI tool I use. It's called Stable Diffusion. What I like about it is that you can download the tool and the model to your PC en execute it locally. You don't need to pay for the service and you don't have to use a Discord Server. The greatest advantage in my view, is privacy.

I won't go into how to install it but here is a link to the youtube video from which I learned to do it:  https://www.youtube.com/watch?v=6MeJKnbv1ts

Ok, so you have Stable Diffusion installed and Running.

It has 2 basic modes of generating images. The first is pure text based, which means you describe with as much detail as possible what you want to see and it will try to generate it from scratch. 

The second one is the one I use and is called "Image to Image" and looks like this:

This is something I'm currently working on. An image of Helga. On the left is the original image, then there are a few sliders with parameters you play with to define just how "imaginative" it gets. Basically you can play with settings that tell SD to either to recreate the image exactly or to go wild and take many liberties. Each image has its own sweet spot and requires tweaking to get it about right. 

Img2Img also requires a prompt with text description of the image. In the prompt you describe things you want in the finished version. For example I want it to focus on her muscles, specially on her abs, and also want the face to look young.

At the start I specified that the subject is a German girl but the faces it was generating looked of a woman in her 30s or 40s and Helga is meant to be in her early 20s and possibly looking much younger, so I emphasized that I wanted it to give her a teenage face. It of course did not give it a teenage look but made her look young enough to apear to be in her 20s. Sometimes you have to exaggerate on the prompt to coax SD to get closer to what you want. It is a lot of trial and error and tweaking both the prompt and the sliders.

It is highly unlikely you'll get a perfect image right out of the box. You'll probably generate dozens of images where each is a slight variation on the original. There will be some that have her face just how you like it, others the body, etc. The idea is to look through all the ones that are being generated and select the correct pieces for the final product. I usually set the parameters in SD to restrict deviations from the original pose, so all the generated images have a similar composition and is easy to put one on top of the other and mask out the pieces you want from each image and they fit almost without problem.

I haven't gotten to the photoshop stage with the Helga piece so I'm showing the Photoshop file for the image with Sofia. This one is made up of 4 different pieces. 1 with Sofia's body, 1 with the guy's body, 1 with Sofia's face (I generated it separately by making a new image with just the face and neck and having SD generate a high resolution version of her face. When I did the full body one, the face looked weird because it was small on the picture and its not good with small faces. I don't have to do this with Helga's image because her face is large on the image and SD does just fine with it but for full body shots I make the head separate), and 1 with the man's face.

In photoshop is a matter of just stacking the images one on top of the other and masking out the parts that need to be hidden from each. Then I usually add 1 layer where I do subtle touch ups with a brush to fix minor problems in the seams or little errors on the generations. Just minor touch ups.

Fun fact. In the AI image there are a few elements left from the original image, which are: Sofia's feet and shoes, both her hands, the man's mouth, eyes, and hand. The reason is that AI image generators have problems with certain body parts like feet and hands. They get extra fingers or are deformed, so I mask that out and let the original hands and feet (which the 3D render did quite well), and the guy's mouth because it looked a little odd, and couldn't het the Awe expression that I wanted, so I preserved his mouth and eyes.

Hope this helps shine a light in how I do the AI thing with my characters. I'm sure other artists do it differently, I'm still learning and tweaking my own workflow with this tool that is still very new to me.

Cheers!

-Kurt