Skip to content

Latest commit

 

History

History
110 lines (79 loc) · 7.75 KB

README.md

File metadata and controls

110 lines (79 loc) · 7.75 KB

ComfyUI Nodes

After installation, you can find the three nodes provided by this repository in the Add Node - Ruyi menu, as shown in the image below:

The following sections will introduce the functions and parameters of each node.

Note: The new version of ComfyUI nodes is displayed in the NODE LIBRARY on the left side of the interface.

Load Model

The Load Model node is used to load the model from disk. It also provides the functionality for automatic model downloading (auto_download parameter).

  • model: Select which model to use. Currently, Ruyi-Mini-7B is the only option.
  • auto_download: Whether to automatically download. Defaults to yes. If the model is detected as missing or incomplete, it will automatically download the model to the ComfyUI/models/Ruyi path.
  • auto_update: Whether to automatically check for and update the current model. Defaults to yes. When auto_download is enabled, the system will automatically check for updates to the model and download any updates to the ComfyUI/models/Ruyi directory. Please note that this feature relies on the caching mechanism of huggingface_hub, so do not delete the .cache folder in the model directory to ensure a smooth update process.

Load LoRA

The Load LoRA node is used to load LoRA models, which need to be placed in the ComfyUI/models/loras path.

  • lora_name: The LoRA to be loaded; it will automatically search and display all model files in the ComfyUI/models/loras path.
  • strength_model: The degree of influence of the LoRA, typically set between 1.0 and 1.4 for optimal results based on experience.

Sampler for Image to Video

The Sampler for Image to Video node is used to generate videos based on input images. The starting frame image (start_img) is a required input, while the ending frame image (end_img) is optional. This node also supports camera control (camera_direction parameter) and motion amplitude control of the video subject (motion parameter).

  • start_img: The starting frame image.
  • end_img: The ending frame image, optional input.
  • video_length: The length of the video, which must be divisible by 8, with a maximum of 120 frames.
  • base_resolution: The video resolution, such as 512, indicates that the generated video will have pixel dimensions close to 512 x 512. The model will automatically select the closest output video aspect ratio based on the input image.
  • seed: A random number; different random numbers usually generate different videos. If the generated video does not meet requirements, this value can be adjusted to try other generation possibilities.
  • control_after_generate: The method of changing the random number after each generation.
    • Fixed indicates the seed is fixed.
    • Increment indicates the seed is increased by one each time.
    • Decrement indicates the seed is decreased by one each time.
    • Randomize indicates the seed is randomly set each time.
  • steps: The number of iterations for video generation. More iterations require more time. Typically, 25 iterations yield good results.
  • cfg: The guidance of instructions (such as input images). A higher value indicates better guidance, with values between 7 and 10 usually achieving better generation results.
  • motion: Controls the motion amplitude of the video subject.
    • 1 indicates minimal motion, which means the video subject is nearly static.
    • 2 indicates normal motion, which could be used in most case.
    • 3 indicates significant motion. The video subject is trying to move as much as possible.
    • 4 indicates a very large motion. Sometimes the video subject may move out of the camera frame.
    • Auto indicates the motion is automatically determined by the model.
  • camera_direction: Camera movement.
    • Static indicates a stationary camera.
    • Left indicates the camera moves left.
    • Right indicates the camera moves right.
    • Up indicates the camera moves up.
    • Down indicates the camera moves down.
    • Auto indicates automatic determination.
  • GPU_memory_mode: Determines how GPU memory is utilized.
    • normal_mode is the default mode, using more GPU memory and generating faster.
    • low_memory_mode is the low memory mode, which significantly reduces GPU memory usage but severely impacts generation speed.
  • GPU_offload_steps: Used to optimize GPU memory usage by moving some temporary variables from GPU memory to RAM, which increases memory usage and decreases generation speed.
    • 0 indicates no optimization.
    • 1 - 10, where 1 has the least GPU memory usage and the slowest generation speed; 10 has the most GPU memory usage (less than the non-optimized case) and the fastest generation speed.
    • Generally, with 24GB of GPU memory, you can use 7 to generate a 512 resolution, 120 frame video. For more detailed data, please refer to the following.

Workflow Example

This section presents an example workflow for generating videos from images. You can import the workflow using the Load button in the (bottom right) menu. Note that the new version of ComfyUI move the menu to the top left, and allows you to load workflows through the Workflow - Open option.

The workflows are located in the comfyui/workflows/ directory, while the assets can be found in the assets/ directory.

After importing the workflow, you need to manually re-specify the input image for the LoadImage input node. Since the workflow file can only record the names of input files, it does require manual configuration.

Image to Video (Starting Frame)

The workflow corresponds to the workflow-ruyi-i2v-start-frame.json file. For users with larger GPU memory, you can also use workflow-ruyi-i2v-start-frame-80g.json to enhance the generation speed.

Image to Video (Starting and Ending Frames)

The workflow corresponds to the workflow-ruyi-i2v-start-end-frames.json file. For users with larger GPU memory, you can also use workflow-ruyi-i2v-start-end-frames-80g.json to enhance the generation speed.

Frequently Asked Questions

Model loading error: LoadModel: ConnectionError (MaxRetryError)

This is usually caused by network issues leading to a failure in downloading from huggingface_hub. If the network is functioning properly, simply rerunning the LoadModel node should resolve the issue.

Video generation speed is slow and far below expectations

  • First, check if the low_memory_mode in the Load Model node is enabled. This mode significantly reduces video generation speed.
  • Second, verify the version of PyTorch. PyTorch version 2.2 supports FlashAttention-2 (link), which can greatly enhance computational efficiency. Installing the latest version of PyTorch can effectively improve generation speed.