Stable WarpFusion v0.11 Daily - Lora, Face ControlNet - Changelog (Patreon)

Sort of a disclaimer: don't dive headfirst into a nightly build if you're planning to use it for your currect project, which is already past its deadline - you'll have a bad day. This is not a production-ready user-friendly software :D

ATTENTION! 16GB VRAM NVIDIA GPU is required for local run, or colab pro with a high ram env. You may be able to run it on a smaller amount of VRAM, but I haven't tested it and can not confirm how low you'll have to go with your settings, so please join discord and ask around before subscribing.

Happy Easter (with a delay) :D

This release features Loras support, Face controlnet, and some other quality-of-life tweaks.

Changelog:

• add lora support
• add loras schedule parsing from prompts
• add custom path for loras
• add custom path from embeddings
• add torch built-in raft implementation
• fix threads error on video export
• disable guidance for lora
• add compile option for raft @ torch v2 (a100 only)
• force torch downgrade for T4 GPU on colab
• add faces controlnet from https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace
• make gui not reset on run cell (there is still javascript delay before input is saved)
• add custom download folder for controlnets

Notebook download: https://www.patreon.com/posts/81496861

Lora Support

Added lora support.

Firstly, download them and place in a folder. When downloading, mind the base model used for loras, v1 and v2 loras may not be compatible.

Specify their folder in LORA & embedding paths cell. After you run this cell, a list of detected loras will be printed.

To use loras, add them to your main prompt (preferrably at the end, it will be detected and removed from the prompt). Use the following format: <lora:lora_name:lora_weight>

For example:

<lora:urbanSamuraiClothing_urbansamuraiV03:1> where urbanSamuraiClothing_urbansamuraiV03 is the detected lora name printed in LORA & embedding paths cell.

The full prompt may look like this: {0: ['a beautiful highly detailed cyberpunk mechanical augmented most beautiful (man) ever, cyberpunk 2077, neon, dystopian, hightech, trending on artstation, <lora:urbanSamuraiClothing_urbansamuraiV03:1>']}

Scheduling:

You may schedule loras in your prompts. (requires blend_json_schedules enabled for that, otherwise it will use the last weight value without blending)

For example:

{0: ['a prompt, <lora:urbanSamuraiClothing_urbansamuraiV03:1>'],
100: ['a prompt, <lora:urbanSamuraiClothing_urbansamuraiV03:0>']}

will gradually reduce lora weight from 1 to 0 across 100 frames.

You can use multiple loras. For example:

{0: ['a prompt, <lora:urbanSamuraiClothing_urbansamuraiV03:1> <lora:zahaHadid_v10:0>'],  
100: ['a prompt, <lora:urbanSamuraiClothing_urbansamuraiV03:0> <lora:zahaHadid_v10:1>']}

will gradually reduce urbanSamuraiClothing_urbansamuraiV03 weight from 1 to 0 across 100 frames, and increase zahaHadid_v10 weight from 0 to 1.

if you don't specify a 0-weight keframe, the lora will just pop. For example:

{0: ['a prompt'],  100: ['a prompt, <lora:urbanSamuraiClothing_urbansamuraiV03:1>']}

will have no lora for frames 0-99, and urbanSamuraiClothing_urbansamuraiV03 lora with weight 1 at frame 100.

Init scale and latent scale (guidance) are disabled  when loras are used.

Faces ControlNet

Added Faces ControlNet support. To use it set weight above 0 in the cell with other controlnets.

Settings: GUI -> controlnet - max_faces: max faces to detect.

If no faces were detected in a frame, the controlnet will not be used for that frame even if it was enabled.

QOL improvements

The GUI will now keep its values even when the cell is re-run. Keep in mind that it's a javascript app and may still lag, not saving the most recent changes if your system if under load.

Added user paths for loras, embeddings, controlnet models. You can now store all of these in a common (non-warp) folder / google drive.

Added built-in raft implementation instead of the pre-compiled one used before. It will support torch v2. To use it, uncheck use_jit_raft. You can compile built-it raft model for 30% speed-up (available only for a100 on google colab / torch v2 on local install)

Notebook download: https://www.patreon.com/posts/81496861

A reminder:

Changes in GUI will not be saved into the notebook, but if you run the diffuse! cell with new settings, they will be saved to a settings.txt file as usual.

You can load settings in misc tab.

You do not need to rerun the GUI cell after changing its settings, but you can now :D

Local install guide:
https://discord.com/channels/973802253204996116/1067887206125015153/1067888756910215278
https://github.com/Sxela/WarpFusion/blob/main/README.md

Youtube playlist with settings:
https://www.youtube.com/watch?v=wvvcWm4Snmc&list=PL2cEnissQhlCUgjnGrdvYMwUaDkGemLGq

For tech support and other questions please join our discord server.

Discord is the preferred method, because it is nearly impossible to provide any decent help or tech support via Patreon due its limited text formatting and inability to add screenshots or videos to comments or DMs.
Error reports in comments will be deleted and reposted in discord.