Stable WarpFusion v0.16 - multi prompts - nightly - Download (Patreon)

Making Temporal consistency studies took sime time, so this preview has been delayed a bit.

Sort of a disclaimer: only nvidia gpu with 8gb+ or hosted env

don't dive headfirst into a nightly build if you're planning to use it for your currect project, which is already past its deadline - you'll have a bad day. This is not a production-ready user-friendly software :D

Make sure you've tried awesome free solutions like Deforum or TemporalKit before subscribing and submitting yourself to the warp.

• add prompt weight parser/splitter
• update lora parser to support multiple prompts. if duplicate loras are used in more than 1 prompt, last prompt lora weights will be used
• unload unused loras
• add multiple prompts support
• add max batch size
• add prompt weights
• add support for different prompt number per frame
• add prompt weight blending between frames
• bring back protobuf install
• add universal frame loader

v0.16.4-5:

• add masked prompts support for controlnet_multi-internal mode
• add controlnet low vram mode

v0.16.6:

• add masked prompts for other modes
• fix undefined mask error

v0.16.8

• fix consistency error between identical frames
• add ffmpeg deflicker option to video export (dfl postifx)
• export video with inv postfix for inverted mask video
• add sd_batch_size, normalize_prompt_weights, mask_paths, deflicker_scale, deflicker_latent_scale to gui/saved settings
• fix compare settings not working in new run

v0.16.9

• fix reference controlnet not working with multiprompt

v0.16.10

• disable ffmpeg deflicker for local install

30.06.2023:

• moved to nightly

5.7.2023

• fix torchmetrics version thx to tomatoslasher

28.07.2023:

• fix pillow error

v0.16.13, 24.08.2023:

• fix safetensors error

Multiple prompts

You can now use multiple prompts per frame. Just like this:

{0:['a cat','a dog']}

In this case with no weights specify it will give each prompt a weight of 1.

You can speciffy weights like this: {0:['a cat:2','a dog:0.5']}

The weights should be at the end of the prompt.

normalize_prompt_weights: enable to normalize weights to add up to 1.

For example this prompt {0:['a cat:2','a dog:0.5']} with normalize_prompt_weights on will effectively have weights {0:['a cat:0.8','a dog:0.2']}

Prompt weights can be animated, but the weight will be applied to the specific prompt number, not exact text. So {0:['prompt1:1', 'prompt2:0'], 5: ['prompt1:0', 'prompt3:1']} will blend the weights but not the prompts, so you will have prompt1 until frame5, then it will be replaced with prompt3, but the weights will be animated, so that a prompt for a frame between 0 and 5 will look like ['prompt1:0.5', 'prompt2:0.5']

You can have different number of prompts per frame, but the weights for prompts missing in a frame will be set to 0

For example, if you have:

{0:['a cat:1', 'a landscape:0'], 5: ['a cat:0', 'a landscape:1'], 10:['a cat:0', 'a landscape:1', 'a galaxy:1']}

'a galaxy' prompt will have 0 weight for aall frames where it's missing, and will have weight 1 at frame 10

Each additional prompt adds +50% to vram usage and diffusion render times.

Masked prompts

You can now use masks for your prompts. The logic is a bit complicated, but I hope you'll get the idea.

You can use masks if you have more than one prompt.

The first prompt is always the background prompt, you don't need a mask for it.
If you decide to use masks, you will need to provide them for every other prompt other than the 1st one. Each next prompt+mask will be placed on top of the previous, only white areas of the mask will be preserved. For example, if your 2nd prompt mask is completely covering the 1st prompt mask, you will not see the 1st prompt in the output as it will be covered by the 2nd prompt mask completely.

You need to specify path to your mask frames/video in the mask_paths variable. for 2 prompts you will need 1 mask, for 3 prompt - 2 masks, etc.

Leave mask_paths=[] to disable prompt masks. Enabling prompt masks will effectively disable prompt weights.

Max_batch_size

By default oyur image is diffused with a batch = 2, consisting of conditioned and unconditione images (positive and negative prompt). When we add more prompts, we to diffuse more images, one extra image per extra prompt.

Depending on your gpu vram, you can decide to increase batch size to process more than 2 prompts at a time.

You can set batch size to 1 to reduce VRAM usage even with 1 prompt.

Controlnet low vram fix

Enable to juggle controlnets from cpu to gpu each call. Is very slow, but saves a lot of vram. Right now all controlnets are offloaded and loaded to gpu ram once per frame, so that they are only kept on GPU during diffusion.

With controlnet_low_vram=True all controlnets will stay offloaded to cpu and only be loaded to gpu when being called during diffusion, then offloaded back to cpu, each diffsuion step.

Fixes

Unused loras should now be kept unloaded in a new run that doesn't use loras.

Local install guide:
https://discord.com/channels/973802253204996116/1067887206125015153/1067888756910215278
https://github.com/Sxela/WarpFusion/blob/main/README.md

Tutorials:
https://youtu.be/HkM-7wxtkGA
https://www.youtube.com/watch?v=FxRTEILPCQQ
https://www.youtube.com/watch?v=wqXy_r_9qw8
https://www.youtube.com/watch?v=VMF7L0czyIg
https://www.youtube.com/watch?v=m8xaPnaooyg

Youtube playlist with settings:
https://www.youtube.com/watch?v=wvvcWm4Snmc&list=PL2cEnissQhlCUgjnGrdvYMwUaDkGemLGq

For tech support and other questions please join our discord server:
https://discord.gg/YrpJRgVcax

Discord is the preferred method, because it is nearly impossible to provide any decent help or tech support via Patreon due its limited text formatting and inability to add screenshots or videos to comments or DMs.
Error reports in comments will be deleted and reposted in discord.