[Tutorial] AnimateDiff ControlNet Animation (Patreon)

This workflow utilized "only the ControlNet images" from external source which are already pre-rendered before hand which saves GPU's memory and skips the Loading time for controlnet (2-5 second delay for every frame)  which saves a lot of time for doing final animation.


Note : The Passes can be created for anything, humans, animals, cars, nature, abstract things and even from CG renders, This new technology opens up doorways to a whole new technique of animating.

As a 3D artist, this technique can minimize the efforts needed to achieve a quality output and in only a few amount of time.

So learning this New AI Technology is must as it will take over the future jobs of artist if we are not up to date.

EXCERCISE: 

How to know when to choose which pass ?

OpenPose is used only for human figures initially it can also sometimes work on cats, dogs.
For Face Close ups use Softedge(HED) with combination of Lineart or Canny.
For Interior, Exterior, or landscape Combination of Canny&LineArt for details and Softedge for broader silhouette is used. 

Depth Pass can be used for anything. It helps in shaping the composition. 

Homework : I have gathered some references for you to test out and Stylize them in AnimateDiff so you can develop a new vision sense for detecting passes.

1) Car (Rotate 90 degree in After Effects)
- Canny and LineArt Pass can be applied.
- Prompt should have "car" in it

2) Ocean Waves
- Softedge (HED with Safe) pass
- Prompt " Lava, glowing liquid, or anime water... think anything. 

3) Whales
-HED pass with lineart will do the trick
-Abstract the whales

4) Rocky Waterfall Mountains
-Canny and LineArt
-Prompt, be creative.

5) Abstract Waves  (Rotate and Use)
Try to test and select the passes yourself to develop a technical sense.
-For Prompt Use your imaginations.

Paste The Above link in the Instagram Downloader and save video.

1) Export as JPEG sequence from After effects in 480p
2) Make the passes in the ControlNetPasses_Export.json file
3) Organize them in separate folders
4) Follow the instructions from the above video"
5) Choose model, Set dimensions and batch range to 10 for testing, enter prompts....
6) IMPORTANT Select the proper Control Model according to the Controlnet Pass you have exported from previous json file.
7) Test and render

You can Post the renders down in the comments or Dm me on discord for feedback
Discord ID : jerrydavos
-----------------------------------------------------------------------------------------------------

Requirements for Using this Workflow in ComfyUi : 

• 1) ComfyUI Impact Pack 
• 2) ComfyUI_FizzNodes 
• 3) ComfyUI's ControlNet Auxiliary Preprocessors 
• 4) ComfyUI-Advanced-ControlNet  
• 5) ComfyUI Inspire Pack 
• 6) AnimateDiff Evolved 
• 7) ComfyUI-VideoHelperSuite 
• 8) Dynamic Thresholding   

Checkpoint

• 1) Temporaldiff-v1-animatediff
  Put in : ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\models