Stable WarpFusion v0.5.28 Daily - changelog (Patreon)

Quickstart guide if you're new to google colab notebooks: https://docs.google.com/document/d/1gu8qJbRN553SYrYEMScC4i_VA_oThbVZrDVvWkQu3yI

Local+colab install guide: https://github.com/Sxela/WarpFusion/blob/main/README.md

Notebook download: https://www.patreon.com/posts/79222171

Changelog:

New Stuff:

• add conditioning source video for depth, inpainting models
• add conditioning source to gui
• add automatic caption generation
• add caption syntax to prompts
• add separate color video / image as colormatch source
• add color video to gui options
• save settings to settings folder
• add embeddings, prompt attention weights from AUTOMATIC1111 webui repo
• bring back colab gui for compatibility
• load default settings option
• add frame difference manual override for content-aware scheduling
• add stable diffusion inpainting model support
• add inpainting mask source switchadd inpainting mask strength slider
• add invert inpainting mask option
• move use-mask to run after colormatching to avoid flickering
• save model_path to settings.txt
• add samplers
• add v2-768-v support
• add mask clipping
• add tooltips to gui settings
• add first beta gui
• remove settings included in gui from notebook
• add fix for loading pkl models on a100 that were saved not on a100

Fixes:

• fix blip config path
• shift caption frame number
• convert loaded setting keys to int ("0" -> 0)
• fix batchnum not inferred correctly due to moved settings files
• fix init frame / stylized frame colormatch with offset 0 error
• fix ANSI encoding error
• fix videoFramesCaptions error when captions were off
• fix caption keyframes "nonetype is not iterable" error
• change default colormatch mode to stylized frame to avoid filenotfound error
• fix colormatch first frame error
• fix raft_model not found error when generate flow cell is run after the cell with warp_towards_init
• fix blip_model not found error
• fix settings not loading with -1 and empty batch folder
• fix blip generationg captions for n-th frame even with a different setting
• fix load settings not working for filepath
• fix norm colormatch error
• fix warp latent mode error
• fix description_tooltip=turbo_frame_skips_steps error
• fix -1 settings not loading in empty folder
• fix -1 settings error
• fix colormatch offset mode first frame error
• fix PDF color transfer being used by default with LAB selected
• fix content-aware schedules
• print settings on error while saving settings :D
• reload embeddings in Diffuse! cell (you can add new without reloading the whole model)
• fix pkl not saveable after loading embeddings
• fix xformers install (no need to downgrade torch thanks to TheFastBen)
• add ffmpeg instal for windows
• add torch install/reinstall options for windows
• add xformers install/reinstall for windows
• disable shutdown runtime for non-colab env
• fix error saving settings with content aware schedules
• fix legacy normalize latent = first latent error
• fix xformers install in colab A100
• fix 'load default settings' not working in GUI cell
• fix mask_clip not loading from settings
• fix error with loaded settings' schedules
• fix another error with loaded schedules ("0" vs 0)
• fix xformers for a100
• remove PatchMatch to fix jacinle/travis error on colab
• fix indentation error
• fix schedules error when loading saved settings
• add safetensors support
• fix missing safetensors import
• fix consistency map generation error (thanks to maurerflower)
• fix colab crashing on low vram env during model loading
• fix xformers install on colab (and maybe locally)
• fix v2-768-v
• fix xformers (again) and again :D
• fix mask clip save settings crash
• add xformers for 1.13
• fix v2-768-v blue output

Image conditioning source video

Video Input Settings -> Image Conditioning Video Source

This is a special video source for image-conditioned models, such as inpainting and depth.

You can now specify a separate video source for those models. For example, you can use alpha mask as depth source to create a cardboard-like animation

To acctivate conditioning source video, select
GUI -> depth_source / inpainting_mask_source -> color_video option, depending on the model used. This setting will have no efffect with non-depth and non-inpainting models.

Caption generation

Generate frame captions -> Generate captions for keyframes

In order to further extend the content-aware automatic settings feature, I've added caption generation. You can now caption frames based on various keyframe settings.
Captions are saved in videoFramesCaptions folder inside your output folder, so you can verify and correct captions manually.

make_captions:

Enable caption generation during run-all-cells

keyframe_source:

Content-aware scheduling keyframes
- captions frames at keyframes, using frame difference detected in the Content-aware scheduling cell

User-defined keyframe list
- you can specify a list of frame numbers manually

Every n-th frame
- generate caption for every n-th frame, starting with the 1st frame.

user_defined_keyframes:

Used only with keyframe_source = User-defined keyframe list.
Specify a list of keyframes to generate captions for.

diff_thresh:

Used only with keyframe_source = Content-aware scheduling keyframes.
Defines threshold difference above which will trigger caption generation.

nth_frame:

Used only with keyframe_source = Every n-th frame.
Generates caption for every n-th frame, starting with the 1st frame.

offset_mode:

Fixed
- Fixed offset. If you have keyframes at frames 0 and 10, fixed_offset 1, caption wil be generated for frames 1 and 11.

Between Keyframes
- Captions are generated between keyframes. If you have keyframes at frames 0 and 10, caption wil be generated for frame 5.

None
- No offset

fixed_offset:

Used only with offset_mode = fixed. Shifts captioned frame number with an offset.

Using Captions in Prompts

You can call caption in you prompt by using the {caption} keyword. It will read currect frame's caption, or the latest caption if current frame doesn't have one.
You can mix prompt schedule with dynamic captions, for example have multiple caption with one prompt or vice versa.

External colormatch / norm source

Video Input Settings -> Colormatch -> Colormatching Video Source

Added external colormatching / norm source video / frame option.
You can specify a video or a single image as color palette source.

For this setting to work well, also enable color_match in consistency tab.

To use colormatching video enable either or both of those:
GUI -> colormatch_frame -> color_video & color_video_offset
GUI -> normalize_latent -> color_video & color_video_offset

Enabling colormatch_after can also be useful.

Save settings to settings folder

Settings' files are now saved to settings subfolder. Please run this notebook with a new batchname so that run numbers will be detected correctly.

Embeddings

Added embeddings support from the awesome AUTOMATIC1111 repo.

Place embeddings into embeddings folder that should be ccreated in your warp folder. Embeddings are loaded right after the model itself, so if you add some more files mid-run, make sure to reload the notebook.

You can use embeddings by specifying their filename in the prompt. For example, if you have some_embedding.pt, add some_embedding to your prompt as text to call it.

Prompt attention weights

Added keyword weights support from AUTOMATIC1111 repo.
You can weight keywords and embeddings as well with the following syntax:
(someword:1.1) - increases the weight of that word by 10%,
(someword:0.9) - decreases the weight by 10%.
You can also use () without a number to increase the weight by 10%, or use [] without a number to decrease the weight by 10%

Colab gui is back

Brough back the old colab gui (by popular demand). The new gui is still there, but now its defaults are taken from the old gui, so you can see what defaults you're gonna get.

Load default setings

GUI -> default_settings_path

You can now specify the default settings file that will be loaded. It will override the old-fashioned gui settings.

Frame difference manual override

Content-aware scheduling -> Create schedules from frame difference -> diff_override

You can override the keyframes that were inffered during frame difference analysis with your own list of keyframes, for example obtained from audio to keyframe script.

Fix default color transfer
The default color transfer is now LAB, as is stated in the interface (was PDF even if LAB was selected)

Fix content-aware schedules
Content-aware schedules were applied before the GUI and were effectively overridden by the GUI.

Inpainting model

define SD + K functions, load model -> model_version -> v1_inpainting

You can now use runwayml stable diffusion inpainting model. just select v1_inpainting from the dropdown menu when loading the model, and specify the path to its checkpoint.

Inpainting mask source

gui -> diffusion -> mask_source

You can either use consistency mask as inpainting mask, or a full white mask, if you select "none"

Inpainting mask strength

gui -> diffusion -> inpainting_mask_weight

This parameter controls inpainting mask conditioning strength.

Inverse inpainting mask

gui -> diffusion -> inverse_inpainting_mask

Inverts inpainting mask

Mask result

gui -> warp -> mask_result

This setting has been around even before stable diffusion came to warp.
Turning it on will keep all consistent areas untouched, only changing (stylizing) the inconsistent parts.

Samplers

Added most if not all the samplers from k-diffusion repo.
For samplers ending with "K" turn Karras noise on. (the noise may behave differently than in Automatic, for example, needs more testing)

Tooltips

Added tooltips to each one of the GUI setting that supported it :D Just hover your mouse over its description.

GUI

The interface contains only the most frequently used settings.

It is located in "stable-settings - gui" tab, right before the "Diffuse!" tab.
Running locally via VSCode is still a pain in most delicate places, but I do plan to bring all the user-defined variables to the neighbouring cells.

Note that changes in GUI will not be saved into the notebook, but if you run it with those, they will be saved to a settings.txt file as usual.

You can load settings in misc tab.

You do not need to rerun the GUI cell after changing its settings.

A reminder:

Changes in GUI will not be saved into the notebook, but if you run it with new settings, they will be saved to a settings.txt file as usual.

You can load settings in misc tab.

You do not need to rerun the GUI cell after changing its settings.

Local install guide:
https://discord.com/channels/973802253204996116/1067887206125015153/1067888756910215278
https://github.com/Sxela/WarpFusion/blob/main/README.md

Youtube playlist with settings:
https://www.youtube.com/watch?v=wvvcWm4Snmc&list=PL2cEnissQhlCUgjnGrdvYMwUaDkGemLGq

Notebook download: https://www.patreon.com/posts/79222171