Introduces Nth-frame full face detection combined with KCF bounding
box tracking and Lucas-Kanade (LK) optical flow for keypoint (KPS)
tracking on intermediate frames. This is primarily for single-face
webcam mode to improve performance while maintaining per-frame swaps.
Key Changes:
- Modified `face_swapper.py` (`process_frame`):
- Full `insightface.FaceAnalysis` runs every N frames (default 5)
or if tracking is lost.
- KCF tracker updates bounding box on intermediate frames.
- Optical flow (`cv2.calcOpticalFlowPyrLK`) tracks the 5 keypoints
from the previous frame to the current intermediate frame.
- A `Face` object is constructed with tracked bbox and KPS for
swapping on intermediate frames (detailed landmarks like
`landmark_2d_106` are None for these).
- Experimental similar logic added to `_process_live_target_v2`
for `map_faces=True` live mode (non-many_faces path).
- Robustness:
- Mouth masking and face mask creation functions in `face_swapper.py`
now handle cases where `landmark_2d_106` is `None` (e.g., by
skipping mouth mask or using bbox for face mask).
- Added division-by-zero check in `apply_color_transfer`.
- State Management:
- Introduced `reset_tracker_state()` in `face_swapper.py` to clear
all tracking-related global variables.
- `ui.py` now calls `reset_tracker_state()` at appropriate points
(webcam start, mode changes, new source image selection) to ensure
clean tracking for new sessions.
- `DETECTION_INTERVAL` in `face_swapper.py` increased to 5.
This aims to provide you with a smoother face swap experience with better FPS
by reducing the frequency of expensive full face analysis, while the
actual swap operation continues on every frame using tracked data.
I've completely removed the Nth frame processing logic (frame counter,
interval, and conditional execution) from the `create_webcam_preview`
function in `modules/ui.py`. The frame processing block has been
unindented to ensure it runs on every frame.
This resolves an `UnboundLocalError` for 'detection_frame_counter'
that occurred because the variable was being used after its
initialization was removed in a previous attempt to revert this logic.
The webcam preview will now process every frame as it did before the
Nth frame optimization was introduced.
Removes the leftover `detection_frame_counter += 1` line from the
`create_webcam_preview` function in modules/ui.py. This line was
erroneously kept after the rest of the Nth frame processing logic
was reverted, causing an UnboundLocalError as the counter was no
longer initialized.
This fix ensures the webcam preview can start correctly without this error.
Reverts the Nth frame processing logic previously introduced in
modules/ui.py (create_webcam_preview function). Webcam frames
will now be processed by the full pipeline on every frame,
instead of skipping frames.
This change is based on your feedback requesting to focus on
optimizing the per-frame performance rather than using frame
skipping techniques at this stage.
Optimizes webcam performance by running full face detection and
frame processing (face swap, enhancers) only every N frames
(currently N=3) in modules/ui.py (create_webcam_preview function).
For intermediate frames, the raw (but mirrored/resized) webcam
feed is displayed. This aims to improve UI responsiveness and reduce
overall CPU/GPU load during live webcam sessions, particularly when
resource-intensive operations like hair swapping or face enhancement
are active.
The actual swap/effect will appear at a reduced frame rate (FPS/N),
but the UI should remain smoother.
Removes a duplicated 'if not modules.globals.map_faces:' line within the create_webcam_preview function. This error was causing a syntax issue and preventing the application from running correctly, particularly in webcam mode.
This commit introduces shell scripts to automate the setup process and provide convenient ways to run the application on macOS.
New files added:
- setup_mac.sh: Checks for Python 3.9+ and ffmpeg, creates a virtual environment, installs pip dependencies from requirements.txt.
- run_mac.sh: Runs the application with the CPU execution provider by default.
- run_mac_cpu.sh: Explicitly runs with the CPU execution provider.
- run_mac_coreml.sh: Runs with the CoreML execution provider.
- run_mac_mps.sh: Runs with the MPS execution provider.
The README.md has also been updated with a new section detailing how to use these scripts for macOS users.
These scripts aim to simplify the initial setup and execution of the project on macOS, similar to the .bat files available for Windows.
This commit introduces the capability to swap hair along with the face from a source image to a target image/video or live webcam feed.
Key changes include:
1. **Hair Segmentation:**
- Integrated the `isjackwild/segformer-b0-finetuned-segments-skin-hair-clothing` model from Hugging Face using the `transformers` library.
- Added `modules/hair_segmenter.py` with a `segment_hair` function to produce a binary hair mask from an image.
- Updated `requirements.txt` with `transformers`.
2. **Combined Face-Hair Mask:**
- Implemented `create_face_and_hair_mask` in `modules/processors/frame/face_swapper.py` to generate a unified mask for both face (from landmarks) and segmented hair from the source image.
3. **Enhanced Swapping Logic:**
- Modified `swap_face` and related processing functions (`process_frame`, `process_frame_v2`, `process_frames`, `process_image`) to utilize the full source image (`source_frame_full`).
- The `swap_face` function now performs the standard face swap and then:
- Segments hair from the `source_frame_full`.
- Warps the hair and its mask to the target face's position using an affine transformation estimated from facial landmarks.
- Applies color correction (`apply_color_transfer`) to the warped hair.
- Blends the hair onto the target frame, preferably using `cv2.seamlessClone` for improved realism.
- Existing mouth mask logic is preserved and applied to the final composited frame.
4. **Webcam Integration:**
- Updated the webcam processing loop in `modules/ui.py` (`create_webcam_preview`) to correctly load and pass the `source_frame_full` to the frame processors.
- This enables hair swapping in live webcam mode.
- Added error handling for source image loading in webcam mode.
This set of changes addresses your request for more realistic face swaps that include hair. Further testing and refinement of blending parameters may be beneficial for optimal results across all scenarios.
Added:
- try-finally Block: This makes sure the camera.release() is called no matter how the while loops end.
- Resource Cleanup: The finally block takes care of cleaning up resources to keep the application stable.
The following changes have been implemented:
-A "clear" button has been incorporated.
-The Source x Target Mapper window has been retained following the submission of data via the "submit" button.
google-labs-jules[bot]
Gordon Böer
Kenneth Estanislao
Soul Lee
Nimish Gåtam
qitian
Makaru
KRSHH
theogbob