Stable WarpFusion v0.25 - AnimateDiff (Patreon)

Had so much fun playing with AnimateDiff that I can't help but bring it into Warp.

Local install: https://github.com/Sxela/WarpFusion#local-installation-guide-for-windows-venv

Please note that the notebook has its default values set up for AnimateDiff.
Also keep in mind that it is a bit more memory intensive with ControlNets at the moment, so 1-2 is more than enough. Also, I'd recommend disabling normalize_controlnet_weights, as it seems to be too strong. Just set 1-2 controlnets with values at 0.5 as a starting point, and disable normalize_controlnet_weights.

Changelog:

• add animatediff from https://github.com/guoyww/AnimateDiff
• add motionmodule wrapper from https://github.com/ArtVentureX/comfyui-animatediff
• add sliding context
• add context scheduler
• add batched processing
• add batch overlap/blending
• add a separate clean run loop
• add clean schedule processing
• add animatediff settings and tab to gui
• hide unused gui for animatediff mode
• lazy init lpips
• fix pytorch dependencies error
• fix zoe depth error
• move installers to github repo

add some warp features to animatediff:

• add contronets with their settings
• save exif
• add softcap, seed, offload model, static threshold
• add loras/lora parser
• add captions
• add patterns

21.10.2023, v0.25.4:

• fix ImportError: cannot import name 'CLIPVisionModelWithProjection' from 'transformers' for local install
• fix import error
• assert video length is >= context_lenght
• fix missing diffusers error (grab a fresh install.bat)

27.10.2023, v0.25.5:

• fix lpips error in Content-aware scheduling
• fix prompt length for v1 models (still need to reproduce for sdxl)
• remove Prompt length > 77 detected. Shorten your prompt or split into multiple prompts. warning

30.10.2023, v0.25.6:

• additional safeguard for prompt length in animatediff mode

31.10.2023:

• moved to L tier

23.12.2023:

• moved to M tier

Features

This is a very raw edition, some warp features like color matching or masking are not implemented yet for AnimateDiff mode, but core features are here:

• controlnets
• loras/lycoris
• embeddings
• captions
• patterns
• schedules (though they are applied on per-batch basis, not per-frame

Also cleaned up the run loop a lot. Plan to transfer that back onto the main loop.

Not implemented yet:

• any sort of masking
• multiple prompts
• color matching
• resuming
• guidance (init\latent scale)
• predicted noise

Some stuff may never be implemented for AnimateDiff mode, as it may be useless. I have a lot of experiments to run with existing warp features, and new AnimateDiff-specific ones, but this will be too much code change for a single update, need your help field testing the AnimateDiff core first.

AnimateDiff

define SD + K functions, load model -> model_version -> control_multi_animatediff

To activate it, switch model_version to control_multi_animatediff
v1_animatediff is there as well, but it's not that interesting.
It will download mm_sd_v15_v2.ckpt file and put it into {root_dir}/animatediff/models/Motion_Module/

So if you have it already on your disk, you can put it there to avoid extra downloads.

Settings

AnimateDiff renders frames in batches. The batch is rendered at once, each diffusion step being spread across all the frames. So you will have to wait until the batch finishes to get a preview and output.

batch_length

Number of frames to process in 1 batch.
No one is stopping you from putting the whole frame sequence in a single huge batch, though you can get OOM at some large values. Produces the smoothest results.

overlap_stylized

If you have more than 1 batch, you can use previously rendered frames that are overlapping across adjacent batches, as inits. So with an overlap of 4, you will have 4 stylized frames as init and the rest will be raw frames.

context_overlap

When batch_length is larger than context_length, we slide the context across all the frames. This smoothes the transition and allows processing batches larger than context_length.

context_length

AnimateDiff context length. This is one of the core settings. It determines how many frames the model is looking at at the same time. 16 is a good default value. The problem is that we can't set it too low or too high, as it will produce artifacts or intense flickering, because the model was trained on a different length. I'd suggest staying within 12-24 range.

I'd suggest leaving these as is for starters:

blend_batch_outputs

Blend overlapping frames between batches.

batch_overlap

If you have more than one batch, makes sense to overlap them to smooth out the transition

looped_noise

When doing vid2vid with style strength < 1, noise is added on top of input frames. It is random per frame, but you can loop it so it will repeat with a period of context_length

Move installers to GitHub repo

I've decided to move installer scripts to GitHub after Colab's env update borked everything twice in a row. For now, it affects versions 0.25, 0.24, 0.23, and 0.20, but I plan to convert 0.22 and 0.21 as well.
I do hope it will at least make fixes easily distributed to you, without the need to download a new version or apply manual fixes if you have unsubscribed.

Local install guide:
https://github.com/Sxela/WarpFusion/blob/main/README.md

Guides made by users:

• 05.05.2023, v0.10 Video to AI Animation Tutorial For Beginners: Stable WarpFusion + Controlnet | MDMZ
• 11.05.2023, v0.11 How to use Stable Warp Fusion
• 13.05.2023, v0.8 Warp Fusion Local Install Guide (v0.8.6) with Diffusion Demonstration
• 14.05.2023, v0.12 Warp Fusion Alpha Masking Tutorial | Covers Both Auto-Masking and Custom Masking
• 23.05.2023, v0.12 STABLE WARPFUSION TUTORIAL - Colab Pro & Local Install
• 15.06.2023, v0.13 AI Animation out of Your Video: Stable Warpfusion Guide (Google Colab & Local Intallation)
• 17.06.2023, v0.14 Stable Warpfusion Tutorial: Turn Your Video to an AI Animation
• 21.06.2023, v0.14 Avoiding Common Problems with Stable Warpfusion
• 21.06.2023, v0.15 Warp Fusion: Step by Step Tutorial
• 04.07.2023, v0.15 Intense AI Video Maker (Stable WarpFusion Tutorial)
• 15.08.2023, v0.17 BEST Laptop for AI ( SDXL & Stable Warpfusion ) ft. RTX 4090 - Make AI Art FREE and FAST!

YouTube playlist with settings:
https://www.youtube.com/watch?v=wvvcWm4Snmc&list=PL2cEnissQhlCUgjnGrdvYMwUaDkGemLGq

For tech support and other questions please join our discord server:
https://discord.gg/YrpJRgVcax

Discord is the preferred method because it is nearly impossible to provide any decent help or tech support via Patreon due to its limited text formatting and inability to add screenshots or videos to comments or DMs.
Error reports in comments will be deleted and reposted in Discord.