sxela

Stable WarpFusion v0.5.19 Nightly - roto, background mask, alt frame matching (Patreon)

Published:

2022-10-22 20:51:05

Edited:

2022-10-26 15:38:44

Imported:

2024-08

Tags:

colab notebook stable diffusion text2video warp warpfusion

Downloads

Content

Whew, these last 30 days have been the busiest in my life: leaving my motherland, adapting, moving a lot and, of course, warping around.

Anyway, behold: here's the new nightly warp!

The new features and changes looks like this:

- add background masks for both render and video export
- add automatic mask extraction from video
- add raw video input as a mask
- add background input options (flat color, still image, init video)
- unpack video frames folder into separate folders
- add match first frame color toggle (was always on before)
- add match first frame latent option
- add karras noise support and ramp option

Detailed changes:

Video Masking

mask_source: init_video, mask_video :
Generate (extract) background mask from your init video or any other video, or use a raw video as a mask

extract_background_mask: Check to rotoscope the video and create a mask from it. If unchecked, the raw monochrome video will be used as a mask.

mask_video_path: Specify path to a mask source video for mask_video mode.

Video masking (render-time)

use_background_mask:
Check to enable background masking during render. Use with care, because model will react to new injected background, possibly producing unexpected results, better use masking when creating the output video for more control and faster testing.

apply_mask_after_warp:
Apply mask right before feeding init image to the model. Unchecking will only mask current raw init frame before warping and blending with previous frame.

background:
Choose background source to paste masked stylized image onto: image, color, init video.

background_source:
Specify the init image path or color depending on your background source choice.

Frame correction

normalize_first_latent: match the range of latent vector towards the 1st frame range. Doesn't restrict colors, but may limit contrast.

colormatch_first_frame:
color match frame towards the 1st stylized frame. Helps prevent images going deep purple. As a drawback, locks colors to the 1st frame. Turn this one to reproduce previous notebooks.

use_karras_noise:
Should work better with current sample, needs more testing.

end_karras_ramp_early: controls karras noise schedule

upd:

add negative prompts
move google drive init to the top

Files

Stable WarpFusion v0.5.19 Nightly - background masking

Stable WarpFusion v0.5.19 Nightly. Mask is applied during video output render Settings: "user_comment": "testing cc layers", "image_prompts": {}, "range_scale": 0, "sat_scale": 0, "max_frames": 151, "interp_spline": "Linear", "init_image": "", "clamp_grad": true, "clamp_max": 0.7, "seed": 4275770367, "fuzzy_prompt": false, "rand_mag": 0.05, "eta": 0.55, "width": 768, "height": 512, "diffusion_model": "stable_diffusion", "diffusion_steps": 1000, "video_init_path": "/content/drive/MyDrive/Footage/shakira.mp4", "extract_nth_frame": 1, "flow_video_init_path": null, "flow_extract_nth_frame": 2, "video_init_seed_continuity": false, "turbo_mode": false, "turbo_steps": "3", "turbo_preroll": 1, "flow_warp": true, "check_consistency": true, "turbo_frame_skips_steps": null, "forward_weights_clip": 0, "forward_weights_clip_turbo_step": 0, "padding_ratio": 0.2, "padding_mode": "reflect", "consistency_blur": 2, "inpaint_blend": 0, "match_color_strength": 0, "high_brightness_threshold": 180, "high_brightness_adjust_ratio": 0.97, "low_brightness_threshold": 40, "low_brightness_adjust_ratio": 1.03, "stop_early": 0, "high_brightness_adjust_fix_amount": 2, "low_brightness_adjust_fix_amount": 2, "max_brightness_threshold": 254, "min_brightness_threshold": 1, "enable_adjust_brightness": false, "cfg_scale": 15, "dynamic_thresh": 2, "warp_interp": 1, "fixed_code": false, "blend_code": 1, "normalize_code": false, "mask_result": false, "reverse_cc_order": true, "flow_lq": true, "use_predicted_noise": false, "clip_guidance_scale": 0, "clip_type": "ViT-H-14", "clip_pretrain": "laion2b_s32b_b79k", "missed_consistency_weight": 1, "overshoot_consistency_weight": 1, "edges_consistency_weight": 1, "style_strength_schedule": [ 0.5, 0.2 ], "flow_blend_schedule": [ 0.999, 0.999, 0.9 ], "steps_schedule": [ 50, 60 ], "init_scale_schedule": [ 0, 0 ], "latent_scale_schedule": [ 0, 0 ], "latent_scale_template": "", "init_scale_template": "", "steps_template": "", "style_strength_template": [ 0.2, 0.5, 0.55, 3 ], "flow_blend_template": [ 0.999, 0.0, 0.55, 3 ], "make_schedules": false, "normalize_first_latent": true, "colormatch_first_frame": false, "use_karras_noise": false, "end_karras_ramp_early": false }