Stable WarpFusion v0.5.22 Daily (Patreon)

Happy holidays! Here's the next cumulative stable-build.

Notebook: https://www.patreon.com/posts/76287697

Changelog:

• add colormatch turbo frames toggle
• add colormatch before stylizing toggle
• add faster flow generation (up to x4 depending on disk bandwidth)
• add faster flow-blended video export (up to x10 depending on disk bandwidth)
• add 10 evenly spaced frames' previews for flow and consistency maps
• add warning for missing ffmpeg on windows
• add force_os for xformers setup
• add load ckpt onto gpu option
• add patchmatch inpainting for inconsistent areas
• add warp towards init (thanks to Zippika from deforum team)
• add grad with respect to denoised latent, not input (4x faster) (thanks to EnzymeZoo from deforum team)
• add init/latent scale towards real frame option (thanks to Zippika from deforum team)
• add json schedules (thanks to brbbbq)
• add settings comparison
• save output videos to a separate video folder (thanks to Colton)
• add xformers for colab
• add latent init blending
• add verbose mode
• add 4 channels (4d) support to user-defined latent stats
• add 4 channels (4d) support to fixed code mode
• track latent stats across all frames
• print latent norm average stats
• add latent warp mode
• add consistency support for latent warp mode
• add masking support for latent warp mode
• add normalize_latent modes: init_frame, init_frame_offset, stylized_frame, stylized_frame_offset
• add normalize latent offset setting

fixes

• fix consistency check failing after flow generation
• fix xformers install for win and colab torch versions 1.12/1.13
• fix installation not working after being interrupted
• fix xformers install for A*000 series cards.
• fix error during RAFT init for non 3.7 python envs
• fix settings comparing typo
• fix frame correction for non-existent reference frames
• fix init scale loss to use 1/2 sized images
• fix xformers not loading until restart
• fix inference mode bug
• fix xformers not working on colab

colormatch turbo

stage-settings -> danger zone -> colormatch_turbo
Previously, colormatch functions were applied to each frame, which could produce an overly pronounced effect in turbo mode. You can now turn it off for turbo mode only and keep on for non-turbo frames, where colormatching may actually be useful.

colormatch before stylizing

stage-settings -> danger zone -> colormatch_after

Previously, colormatch functions were applied after stylizing the frame right before saving it to disk. This made the result on disk look different than the preview and mislead you a bit.

faster flow generation

Optical map settings -> Generate optical flow and consistency maps -> threads / workers

Previously flow has been calculated for 1 frame at a time, each frame waiting for previous one to finish rendering and saving flow maps and previews to disk.

Now we don't wait for files to be saved and begin processing the next frame. Even at treads=1 \ workers=1 it may offer a slight increase in performance. The process is still run on GPU, so multithreading is only used for saving files here.

The boost depends on GPU \ disk speed ratio, so if you have a slow GPU and fast SSD, you won't notice much of a difference. For example, T4 colab pro only has 20% boost.
On machines with multiple cores and beefy GPUs you can get a nice boost in performance on high resolutions, so A100 colab pro+ will have up to x4 boost at 2k resolutions.

faster video export in flow blend mode

Create the video -> create video -> threads

Same as in the previous case, but this time it's completely multithreaded, as frames are warped on CPU, so you can get a higher boost, depending on CPU cores and disk bandwidth. A100 colab pro+ had x10 speedup compared to 1 thread.

Add previews for all the maps

(enabled by default)

Previews are now generated for frames, flow, consistency, and alpha maps. You can now check that everything is okay with your input videos faster. This may be helpful during debugging. For example, bad video quality may produce a bad alpha mask which will not detect the background correctly and instead mess up the whole render, which is hard to debug without looking at those images manually.

warning for missing ffmpeg.exe on windows

A lot of people were getting "missing file" error during video frames extraction, making them think that the video path was incorrect, when it was ffmpeg.exe that was missing. Now you'll get a missing ffmpeg.exe error.

Patchmatch inpainting

stable-settings->patchmatch_inpainting

You can now inpaint inconsistent areas via patchmatch algorithm. Really slow and haven't really impressed me, but I will leave it there for you to test.

Warp towards init

(thanks to Zippika from deforum team)

stable-settings->warp_towards_init

This options calculates flow between real frame and stylized frame and warps stylized frame back too real frame. May produce really interesting results, though the 2nd frame will be messy because of the large difference between real and stylized frames, but after it will be okay because the frames won't be that different.

grad with respect to denoised latent

(thanks to EnzymeZoo from deforum team)

stable-settings->grad_denoised

Calculates gradients with respect to denoised latent, not input latent as it was before. Works x4 faster and better.

init/latent scale towards real frame

option (thanks to Zippika from deforum team)

stable-settings->init_grad

Add option to calculate init/latent scale against real frame, not warped previous frame.

json schedules

(thanks to brbbbq)

Schedules now support json (dictionary) format, like the one used in discodiffusion

Format example:

{
 0: 1,
 10: 10,
 50: 5
}

will use value of 1 for 0-9 frames, value of 10 for 10-49 frames, and value of 5 after 50 frames.

You can enable blend_json_schedules to blend intermediate frames' values,  so that in the example above the first 10 frames will have values from 1 to 10.

You can use list schedules as well.

settings comparison

Compare settings->

Insert paths to two settings.txt files to compare values. It will print the differences and missing/new keys.

save output videos to a separate video folder

(thanks to Colton)
Output videos are now saved to /video folder insire your frames' output folder.

Fixes

• fix xformers not loading until restart

xformers for colab

Add xformers optimization fro colab (or linux).
Increases diffusion performance by about 30%, decreases memory requirements.

For example, 16bg card can now run 720x1280 with init and latent scale on. Without xformers than won't be possible.

Currently works on colab or linux machines, at it only has binaries compiled for those architectures. You can try building it on windows if you feel lucky :D

Latent init blending

stable-settings -> danger zone -> blend_latent_to_init

You can now blend the latent vector to current frame's raw latent vector. Helps stay closer to the init video, but not in a pixel-perfect way like fdecreasing flow blend does. 0.2 is a good value to begin with.

Verbose mode

stable-settings -> danger zone -> verbose

Toggles printing out various stuff during render. Turn off if you're not experiencing any errors.

4 channels (4d) support to user-defined latent stats

Frame correction -> latent_fixed_mean, latent_fixed_std

User defined latent norm stats can now be entered in 4-channel format, like [0.8, 0.7, 0.5, 0.1]

Check printed latent stats for reference values for your video during render.

Latent Warp Mode

Previously, most of the transformations were made in image (pixel, RGB, you-name-it) space. Due to the nature of stable diffusion architecture this led to feedback loops, whether it were lines (with the already derelict colormatch-first-frame on), or deep purple color palette (with colormatch off).

This happens because fpr each frame we take previously stylized frame, warp it according to optical flow maps, encode it into latent space, run diffsuion in latent space, decode back to image space, rinse and repeat. This constant encoding-decoding process introduces its own degradations to an image, because even if we just encode and instantly decode an image back, we won't be getting exactly the same image. These degradations build up each frame, shifting the image to places we don't want it to be.

Warping directly in latent space should address this issue directly. We now always keep our stylized frames in latent space, only encode raw frames to add to latents, and only decode them when saving an output image. You can still use warp, consistency, background masking and color/latent matching, but this time it will stay much longer without falling apart even with no color\latent matching.

Turbo is not working well in this mode right now, due to the low-res nature of latent vector.

The warp_mode switch can be found underneath stable-settings. use_latent is the new mode, use_image is the older mode used in all the previous notebooks.

Latent matching

New normalize latent modes:
init_frame, init_frame_offset, stylized_frame, stylized_frame_offset. normalize first frame has been removed and can be mimicked with stylized_frame + normalize latent offset 0.

Modes and offsets work the same way as in color match frame mode.

Notebook: https://www.patreon.com/posts/76287697