Jerry Davos

[Update] [v2.0] Animate ControlNet Animation - LCM [Early Access] (Patreon)

Published:

2023-12-12 13:51:01

Edited:

2024-05-03 15:37:45

Imported:

2024-05

Tags:

Animatediff controlnet animation

Downloads

Content

I'm So excited to release this new version of my workflow, It now support LCM which gives a tremendous boost in rendering and processing.

The Workflow is divided into 5 parts :

Part 1 - ControlNet Passes Export
Part 2 - Animation Raw - LCM
Part 3 - AnimateDiff Refiner - LCM
Part 4 - AnimateDiff Face Fix - LCM
Part 5 - Batch Face Swap - ReActor [Experimental]

This breakdown is for easy processing which is Ram Efficient as well as user friendly.

This workflow is version 2 of to this Main Animation workflow : https://youtu.be/HbfDjAMFi6w [Old Version]

UPDATE LOG DEC 23 :

In this New v2 Update :

LCM is now Finally Supported, This workflow renders 2x faster than the previous workflow, It can also speed up the rendering up to 3x in some cases.
Due to application of LCM, only 10 sampler steps are required to render the final image which approximately half the Ram processing which means you can input 2x more images in the batch range, also you can increase the resolution while making the Raw Pass.
This Workflow has now 2 new parts -
1) Refiner - Which enhances the details from the raw animation.
2) Face Fix - Which fixes the face using the segs nodes, Now No Need for A1111 for Face Fixing , you can do it inside comfy UI entirely.
In the V1 workflow, there was a weird "Color Bleed"/ Color cast all over the image from the prompts keywords, this is now resolved in the V2 version.
Loras which were struggling to work in AnimateDiff can be implemented to a more greater context.(Eg RPG loras).
New Motion Module which is alternative to TemporalDiff motion module has been found, which handles the motion, color bleed and details better than the regular ones.
The UI is Completely Re-Designed and all the inputs are brought into one place for intuitive workflow.
Detailed Instructions, Usage, Tips, Research, Models links and Resources are mentioned in the notes inside the workflows.

Video Demos can be found here :

Demo 1 Epic Realism - Golden Dress Animation
Demo 2 Mistoon - HipHop Mashup 2
Demo 3 Epic Realism - Loka Loka
Demo 4 Animerge - Tell Me

WORKFLOW BREAKDOWN:

[Part 1] ControlNet Passes Export

Now no need to extract frames using after effects, It can entirely be done inside comfyUI.
Control

As before in Version 1 of this script you had to manually resize, Extract the Frames using after effects and organize them inside a folder.

It Exports ControlNet Passes - Canny, Depth, Hed, DwOpenPose, LineArt, Normal and Image Source Frames which are First hand, base dough for the main Animation Workflow

This Workflow firsthand make these passes and saves them in an organized way which can be used in any animation workflow

This workflow saves 10% - 20% time and ram, used up in extracting the passes during the main render. This also enables to test and experiment Animations faster.

[Part 2] Animation Raw - LCM

Main AnimateDiff Controlnet Animation is rendered in this workflow,

All the passes which are extracted from [Part 1] will be used here in this workflow

With the help of Prompts and Loras You can make any abstract type of animation

Loras Showcase:

Prompts only - Look Showcase with Epic Realism:

[Part 3] AnimateDiff Refiner - LCM

In this Refiner workflow, all the raw images from [Part 2] gets upscaled and Refined which give more details and better proportions

You can reuse this refiner workflow for better results.

[Part 4] AnimateDiff Face Fix - LCM

In this Face Fix workflow, all the Refined images from [Part 5] gets a face fixed pass.

Only The face is affected, nothing else is changed, You can also run face fix x2 for better results

[Part 5] Batch Face Swap - ReActor [Experimental]

This is an alternative to Face Fix, you can also swap bad face with good faces, but may sometimes get blurry faces.

This is an experimental workflow and difficult to setup, you can skip this method also.

HOW TO USE

All the Instructions on how to use, is present inside every workflow in Orange Neon Box in the starting

INSTALLATION

All the Custom Nodes used can be downloaded easily from inside the manager and for some cases manager fails to install some of them so here mentioned below:

Part 1-4: