Stable WarpFusion v0.12 - Tiled VAE, ControlNet 1.1 changelog (Patreon)

Sort of a disclaimer: don't dive headfirst into a nightly build if you're planning to use it for your currect project, which is already past its deadline - you'll have a bad day. This is not a production-ready user-friendly software :D

Make sure you've tried awesome free solutions like Deforum or TemporalKit before subscribing and submitting yourself to the warp.

ATTENTION! 16GB VRAM NVIDIA GPU is required for local run, or colab pro with a high ram env. You may be able to run it on a smaller amount of VRAM, but I haven't tested it and can not confirm how low you'll have to go with your settings, so please join discord and ask around before subscribing.

Hello there!
I like how the new preview->nightly cycle is going and I guess I'll stick to that, because the number of XL tier suporters is much smaller than L ones, I have to do less tech support and iterate faster on the way to a nightly build.  :D

New stuff:

• tiled vae
• controlnet v1.1
• controlnet multimodel GUI
• saving controlnet predictions and rec noise

Changelog:

• add shuffle, ip2p, lineart, lineart anime controlnets
• add shuffle controlnet sources
• add tiled vae
• switch to controlnet v1.1 repo
• update controlnet model urls and filenames to v1.1 and new naming convention
• update existing controlnet modes to v1.1
• add consistency controls to video export cell
• add rec noise save\load for compatible settings
• save rec noise cache to recNoiseCache folder in project dir
• save controlnet annotations to controlnetDebug folder in project dir
• add controlnet multimodel options to gui (thanks to Gateway#5208)
• add controlnet v1.1 annotator options to gui

Fixes:

• fix controlnet depth_init for cond_video with no preprocess
• fix colormatch stylized frame not working with frame range
• tidy up the colab interface a bit
• fix dependency errors for uformer
• fix lineart/anime lineart errors
• fix local install
• bring together most of the installs (faster install, even faster restart & run all), only tested on colab
• fix zoe depth model producing black frames with autocast on

Notebook download: https://www.patreon.com/posts/82012982

Tiled VAE

Tiled VAE -> use_tiled_vae

Basically, we split the image into tiles to use less VRAM during encoding and decoding the image into and from the latent space. This is the main VRAM bottleneck, as the diffusion itself is well-optimized and is already using less VRAM thanks to xformers. You can now render 1920x1080 on 16Gb colab cards, though this will be really slow :D

Tiled VAE -> num_tiles

By default, the image is split into 4 tiles by setting num_tiles = [2,2], but you can change it to [2,1] or [1,2] for non-square images.

I'd suggest leaving other variables as is.
Has only been tested with the default v1 VAE.

Controlnet v1.1

Add controlnets from https://huggingface.co/lllyasviel/ControlNet-v1-1

control_sd15_hed -> control_sd15_softedge
control_sd15_hed is now control_sd15_softedge (has hed and pidi detectors)
To select a detector, set control_sd15_softedge_detector to 'HED' or 'PIDI'

control_sd15_normal -> control_sd15_normalbae
control_sd15_normal is now control_sd15_normalbae

control_sd15_depth
Now has midas and zoe detetors
To select a detector, set control_sd15_depth_detector to 'Zoe' or 'Midas'

control_sd15_openpose
Now has pose+hands+face mode
To enable, set control_sd15_openpose_hands_face = True

control_sd15_seg
Now has 3 detectors.
To select a detector, set control_sd15_seg_detector to 'Seg_OFCOCO', 'Seg_OFADE20K', or 'Seg_UFADE20K'

Seg_UFADE20K is the default detector from CN v1.0

control_sd15_scribble
Now has 2 detectors.
To select a detector, set control_sd15_scribble_detector to 'HED' or 'PIDI'
Predicts much wider strokes compared to control_sd15_softedge mode.

control_sd15_lineart_anime, control_sd15_lineart
New modes, for creating images from lineart. Has coarse prediction mode:
To enable, set control_sd15_lineart_anime_coarse = True or control_sd15_lineart_coarse = True, respectively.

control_sd15_inpaint
New mode, works like inpainting model (though a bit differently)
Supports a few mask sources: consistency_mask, None, cond_video
To select a mask source, set control_sd15_inpaint_mask_source to 'consistency_mask', 'cond_video' or None.

None will inpaint the whole image. consistency_mask will inpaint inconsistent  areas. cond_video will take a mask from cond_video path.

The tests have showed that it tends to overcook real fast when used solo (probably to running the unmasked part of the image through VAE over and over again), but may work well together with other contrlnets.

control_sd15_shuffle
New mode, works like image prompt, but better (as it doesn't mess with the main prompt)
Can be used to set style from a single image with a very simple promtp (haven't tested no prompt, but it may work as well)

Supports a few image sources:
color_video, init, prev_frame, first_frame

To select a source, set control_sd15_shuffle_source to 'color_video', 'init', 'prev_frame', 'first_frame'

color_video: uses color video frames (or single image) as source
init - uses current frame's init as source (stylized+warped with consistency mask and flow_blend opacity)
prev_frame - uses previously stylized frame (stylized, not warped)
first_frame - first stylized frame

You can set the 1st frame source as well. For example, if you need to geet the 1st frame style from your image, and for the consecutive frames you want to use the resulting stylized images.

To select a source, set control_sd15_shuffle_1st_source to 'color_video', 'init', 'None'
color_video: uses color video frames (or single image) as source

init - uses current frame's init as source (raw video frame)

None - skips this controlnet for the 1st frame. For example, if you like the 1st frame you're getting and want to keep its style, but don't want to use an external image as a source.

control_sd15_ip2p
New mode, works like instruct pix2pix. Enable it and write prompts like this:

'replace a man with a woman, make the coat red'

save_controlnet_annotations

You can now save controlnet annotator predictions.
To enable it, set save_controlnet_annotations = True

They will be saved to controlnetDebug folder

reconstructed noise cache

Reconstructed noise will now be cached in the recNoiseCache folder in project dir (usually ./images_out/stable_warpfusion_0.12.2/)

It can be reused if most of the settings haven't been changed (to ensure that the cached noise is compatible with current settings at least to some extent)

You can edit: prompt, neg prompt, cfg scale, steps, style strength.
Changing other settings will recreate the noise from scratch.

Controlnet Multimodel GUI

Controlnet Multimodel GUI is now available at controlnet tab. You can disable models via enable checkbox.
Controlnet multimodel settings can be loaded from settings.

Also, this can be done by setting weight to 0, or settings active range to 0-0.

Notebook download:
https://www.patreon.com/posts/82012982

Local install guide:
https://discord.com/channels/973802253204996116/1067887206125015153/1067888756910215278
https://github.com/Sxela/WarpFusion/blob/main/README.md

Quickstart guide: https://docs.google.com/document/d/1gu8qJbRN553SYrYEMScC4i_VA_oThbVZrDVvWkQu3yI

Youtube playlist with settings:
https://www.youtube.com/watch?v=wvvcWm4Snmc&list=PL2cEnissQhlCUgjnGrdvYMwUaDkGemLGq

For tech support and other questions please join our discord server.

Discord is the preferred method, because it is nearly impossible to provide any decent help or tech support via Patreon due its limited text formatting and inability to add screenshots or videos to comments or DMs.
Error reports in comments will be deleted and reposted in discord.