sxela

Stable WarpFusion v0.5.23 Nightly v2 / depth / patchwarp (Patreon)

Published:

2022-12-10 17:25:36

Imported:

2023-09

Content

Notebook: https://www.patreon.com/posts/75724464

Changes:

add v2-depth support
add v2 support
add v1 backwards compatibility
add model selector
add depth source option: prev frame or raw frame
add fix for TIFF bug
add torch downgrade for colab xformers
add forward patch-warping option (beta, no consistency support yet)
add *.mov video export thanks to cerspense#3301

Added support for v2 models: v2-512 and v2-depth

Load up a stable -> define SD + K functions, load model -> model_version

Added model selector, you can now choose between v1, v2-512, v2-depth models. You still need to specify path to checkpoint.

Add depth source option

stable-settings -> depth_source

v2-depth model uses depth map, predicted from input image. You can use previous stylized frame or raw init video frame as the source of this map.

Add forward patch-warping

stable-settings -> depth_source

A test feature. Uses forward flow to move large clusters of pixels, grouped together by motion direction and speed

warp_num_k - number of those clusters per frame

warp_forward - enable\disable this feature

Add *.mov video export thanks to cerspense#3301

Videos can now be exported as quicktime *.mov in nearly lossless quality

Xformers install fix for colab

Important!

As colab env has been updated, the prior xformers build has stopped working.
In order to make it work we need to downgrade pytorch to v1.12.1.

This is done automatically in the notebook, but the env needs to be restarted after that, otherwise xformers will not install correctly. So the Install xformers cell will run, downgrade pytorch and kill the env, so you'll have to click run all to continue.

Notebook: https://www.patreon.com/posts/75724464

Files

Stable WarpFusion v0.5.23 Nightly v2 / depth / patchwarp

Notebook: https://www.patreon.com/posts/75724464 Settings: { "text_prompts": { "0": [ "a beautiful highly detailed helena bonham carter as corpse bride directed by tim burton, in the style of alphonse mucha and ilya kuvshinov and peter mohrbacker, ross tran rossdraws, watercolor, featured on artstation, 70mm, rendered in octane" ], "1000": [ "a highly detailed matte painting, No Man's Sky Screenshot, tall grass field, broken machinery, Simon Stalenhag , featured on Artstation." ] }, "user_comment": "testing cc layers", "image_prompts": {}, "range_scale": 0, "sat_scale": 0, "max_frames": 136, "interp_spline": "Linear", "init_image": "", "clamp_grad": true, "clamp_max": 2, "seed": 4275770367, "fuzzy_prompt": false, "rand_mag": 0.05, "eta": 0.55, "width": 1920, "height": 1024, "diffusion_model": "stable_diffusion", "diffusion_steps": 1000, "video_init_path": "/content/wednesday dance.mp4", "extract_nth_frame": 1, "flow_video_init_path": null, "flow_extract_nth_frame": 2, "video_init_seed_continuity": false, "turbo_mode": false, "turbo_steps": "3", "turbo_preroll": 1, "flow_warp": true, "check_consistency": true, "turbo_frame_skips_steps": null, "forward_weights_clip": 0, "forward_weights_clip_turbo_step": 0, "padding_ratio": 0.2, "padding_mode": "reflect", "consistency_blur": 1, "inpaint_blend": 0, "match_color_strength": 0, "high_brightness_threshold": 180, "high_brightness_adjust_ratio": 0.97, "low_brightness_threshold": 40, "low_brightness_adjust_ratio": 1.03, "stop_early": 0, "high_brightness_adjust_fix_amount": 2, "low_brightness_adjust_fix_amount": 2, "max_brightness_threshold": 254, "min_brightness_threshold": 1, "enable_adjust_brightness": false, "dynamic_thresh": 30, "warp_interp": 1, "fixed_code": false, "blend_code": 0.1, "normalize_code": true, "mask_result": false, "reverse_cc_order": true, "flow_lq": true, "use_predicted_noise": false, "clip_guidance_scale": 0, "clip_type": "ViT-H-14", "clip_pretrain": "laion2b_s32b_b79k", "missed_consistency_weight": 1, "overshoot_consistency_weight": 1, "edges_consistency_weight": 1, "style_strength_schedule": [ 0.4, 0.2 ], "flow_blend_schedule": [ 0.8 ], "steps_schedule": { "0": 50, "1": 50 }, "init_scale_schedule": [ 0, 2000 ], "latent_scale_schedule": [ 0, 0 ], "latent_scale_template": "", "init_scale_template": "", "steps_template": "", "style_strength_template": [ 0.2, 0.5, 0.55, 3 ], "flow_blend_template": [ 0.75, 0.9, 0.55, 3 ], "make_schedules": false, "normalize_latent": "off", "normalize_latent_offset": 0, "colormatch_frame": "stylized_frame", "use_karras_noise": false, "end_karras_ramp_early": false, "use_background_mask": false, "apply_mask_after_warp": true, "background": "init_video", "background_source": "red", "mask_source": "init_video", "extract_background_mask": true, "mask_video_path": "", "negative_prompts": { "0": [ "text, naked, nude, logo, cropped, two heads, four arms, lazy eye, blurry, unfocused" ] }, "invert_mask": false, "warp_strength": 1, "flow_override_map": [], "cfg_scale_schedule": [ 15 ], "respect_sched": true, "color_match_frame_str": 0.2, "colormatch_offset": 0, "latent_fixed_mean": 0.0, "latent_fixed_std": 0.9, "colormatch_method": "LAB", "colormatch_regrain": false, "warp_mode": "use_image", "use_patchmatch_inpaiting": 0, "blend_latent_to_init": 0, "warp_towards_init": "off", "init_grad": false, "grad_denoised": true, "colormatch_after": false, "colormatch_turbo": false, "model_version": "v2_depth", "depth_source": "init", "warp_num_k": 128, "warp_forward": false }