ComfyUI: Node-Based Stable Diffusion GUI Guide

Introduction

ComfyUI is a powerful, modular node-based interface for Stable Diffusion and other generative AI models. Unlike traditional GUI applications, ComfyUI allows you to construct complex image generation workflows by connecting individual processing nodes. This provides unparalleled flexibility and control over the generation pipeline.

Key Features

• Node-Based Architecture: Build custom workflows by connecting processing nodes
• Highly Modular: Supports extensions for various AI models and techniques
• Cross-Platform: Runs on Windows, macOS, and Linux
• GPU Acceleration: Optimized for NVIDIA GPUs with CUDA support
• Extensible: Large ecosystem of custom nodes and integrations
• Workflow Sharing: Save and share complete workflows as JSON files or embedded in images

ComfyUI vs Other Interfaces

Installation

System Requirements

• GPU: NVIDIA GPU with CUDA support (recommended)
• RAM: 8GB+ (16GB+ recommended)
• Storage: 10GB+ free space for models
• Python: 3.10+ (3.11 recommended for best compatibility)

Windows Installation

Portable Version (Recommended)

1. Download the portable version from ComfyUI releases

   • Choose ComfyUI_windows_portable_nvidia.7z for NVIDIA GPUs
   • Choose ComfyUI_windows_portable.7z for CPU-only

2. Extract to your desired location (e.g., C:ComfyUI)

3. Install Git (required for extensions):

   		# Download from https://git-scm.com/downloads
   # Install with default settings
   	

4. Run ComfyUI:

   		# Double-click run_nvidia_gpu.bat (for NVIDIA)
   # or run_cpu.bat (for CPU)
   	

Manual Installation

1. Install Python 3.11:

   		# Download from python.org
   # Ensure pip is installed
   	

2. Clone repository:

   		git clone https://github.com/comfyanonymous/ComfyUI.git
   cd ComfyUI
   	

3. Create virtual environment:

   		python -m venv venv
   venvScriptsactivate
   	

4. Install dependencies:

   		pip install -r requirements.txt
   pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
   	

5. Run ComfyUI:

   		python main.py
   	

Linux Installation

Arch Linux (Pacman)

		# Install Python 3.11 (avoids sentencepiece issues)
paru -S python311
cd ~/Programs
 
# Clone ComfyUI
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
 
# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate.fish
 
# Install requirements
pip install -r requirements.txt
 
# For GPU support (NVIDIA)
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
	

Ubuntu/Debian

		# Install Python and dependencies
sudo apt update
sudo apt install python3.11 python3.11-venv git
 
# Clone ComfyUI
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
 
# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate
 
# Install requirements
pip install -r requirements.txt
 
# For GPU support
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
	

macOS Installation

1. Install Python 3.11 from python.org or via Homebrew:

   		brew install python@3.11
   	

2. Clone and setup:

   		git clone https://github.com/comfyanonymous/ComfyUI.git
   cd ComfyUI
   python3.11 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   	

3. For Apple Silicon (M1/M2):

   		pip install torch torchvision torchaudio
   	

Getting Started

First Launch

1. Run ComfyUI using the appropriate script for your platform
2. Open your web browser to http://127.0.0.1:8188 (default)
3. You should see the default text-to-image workflow

Interface Overview

• Canvas: Main workspace where you build workflows
• Sidebar: Queue, node library, and workflow management
• Menu Bar: Manager button (if installed), settings
• Bottom Panel: Zoom controls, fit view, execution status

Basic Controls

• Zoom: Mouse wheel or pinch gesture
• Pan: Middle mouse button or right-click drag
• Select Nodes: Left-click and drag to select multiple
• Connect Nodes: Drag from output dot to input dot
• Context Menu: Right-click for node options

Core Concepts

Nodes and Workflows

Each node represents a processing step:

• Inputs: Data coming into the node (left side)
• Outputs: Data produced by the node (right side)
• Parameters: Configurable settings (center)

Workflows are created by connecting nodes in a processing pipeline.

Essential Node Types

Loaders

• Load Checkpoint: Load Stable Diffusion model
• Load LoRA: Load LoRA weights
• Load VAE: Load VAE model
• Load Upscale Model: Load AI upscaler

Conditioning

• CLIP Text Encode: Convert text prompts to embeddings
• CLIP Vision: Process images for conditioning

Latent Space

• Empty Latent Image: Create initial noise
• Latent Upscale: Increase latent resolution
• VAE Encode: Image to latent
• VAE Decode: Latent to image

Sampling

• KSampler: Main diffusion sampler
• SamplerCustom: Advanced sampling options

Image Processing

• Save Image: Save final results
• Preview Image: Display intermediate results
• Image Scale: Resize images

Basic Workflows

Text-to-Image

1. Load Checkpoint: Select your model
2. CLIP Text Encode: Input positive prompt
3. CLIP Text Encode: Input negative prompt
4. Empty Latent Image: Set dimensions (512x512 default)
5. KSampler: Configure sampling parameters
6. VAE Decode: Convert latent to image
7. Save Image: Output final result

Key Parameters:

• Steps: 20-50 (higher = better quality, slower)
• CFG Scale: 7-12 (higher = closer to prompt)
• Sampler: Euler a, DPM++ 2M Karras (good defaults)
• Scheduler: Karras (recommended)

Image-to-Image (Img2Img)

Add these nodes to the text-to-image workflow:

1. Load Image: Input source image
2. VAE Encode: Convert to latent space
3. Set Latent Noise Mask: Define areas to modify (optional)

Key Parameter:

• Denoise: 0.3-0.7 (lower = more similar to input, higher = more change)

Inpainting

1. Load Image: Source image
2. VAE Encode: Convert to latent
3. Create Inpaint Mask: Define area to modify
4. Set Latent Noise Mask: Apply mask
5. KSampler: Generate with mask
6. VAE Decode: Convert back to image

Advanced Workflows

Hi-Res Fix (Two-Stage Generation)

1. Generate base image at low resolution
2. Upscale latent space
3. Apply additional sampling steps
4. Decode to high resolution

Benefits: Better quality than direct high-res generation, faster than upscaling final image.

SDXL Workflow

SDXL requires different conditioning:

• CLIP Text Encode (Prompt): Use SDXL-specific encoder
• CLIP Text Encode (Prompt): Separate for positive/negative
• Empty Latent Image: 1024x1024 recommended
• Refiner Checkpoint: Optional second model for refinement

LoRA Integration

Add LoRA nodes after checkpoint loading:

1. Load LoRA: Select LoRA file
2. Strength: 0.5-1.0 (influence level)
3. Model: Connect to KSampler model input
4. CLIP: Connect to text encoders

ControlNet

1. Load ControlNet Model: Select control model
2. ControlNet Apply: Process control image
3. Strength: 0.5-1.0
4. Start/End: Control application range

ComfyUI Manager

Installation

For portable version:

		# Download install script
# Right-click save as: scripts/install-manager-for-portable-version.bat
# Run the downloaded script
	

For manual installation:

		cd ComfyUI/custom_nodes
git clone https://github.com/ltdrdata/ComfyUI-Manager.git
# Restart ComfyUI
	

Features

• Install Missing Nodes: Automatically detect and install required custom nodes
• Install Models: Download checkpoints, LoRAs, VAEs, etc.
• Update All: Update ComfyUI and all extensions
• Custom Node Management: Install/uninstall specific nodes

Model Management

Directory Structure

		ComfyUI/
├── models/
│   ├── checkpoints/     # Main SD models (.ckpt, .safetensors)
│   ├── vae/            # VAE models
│   ├── lora/           # LoRA weights
│   ├── embeddings/     # Textual inversions
│   ├── upscale_models/ # AI upscalers
│   ├── controlnet/     # ControlNet models
│   └── clip_vision/    # CLIP vision models
├── custom_nodes/       # Extensions
├── input/             # Input images
├── output/            # Generated images
└── temp/              # Temporary files

	

Model Sources

• Civitai: Community models (checkpoints, LoRAs)
• Hugging Face: Official and community models
• Stability AI: Official SD models
• Runway ML: Stable Diffusion 1.5 base

Model Recommendations

Beginners:

• Checkpoint: Realistic Vision v5.1
• VAE: vae-ft-mse-840000-ema-pruned

Advanced:

• SDXL: SDXL 1.0 Base + Refiner
• Specialized: Anything V5, Deliberate v4, etc.

Custom Nodes and Extensions

Essential Custom Nodes

ComfyUI Impact Pack

		# Install via Manager
# Includes: Face Detailer, Upscale, Segmentation
	

Features:

• Face restoration with GFPGAN/CodeFormer
• Detail enhancement
• Mask generation
• Segmentation tools

Ultimate SD Upscale

		cd ComfyUI/custom_nodes
git clone https://github.com/ssitu/ComfyUI_UltimateSDUpscale.git --recursive
	

Features:

• High-quality upscaling with SD
• ControlNet tile support
• Seamless texture generation

ControlNet Auxiliary Preprocessors

		# Install via Manager
# Provides preprocessing for ControlNet
	

AnimateDiff

		cd ComfyUI/custom_nodes
git clone https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved.git
git clone https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite.git
	

Features:

• Video generation from images
• Motion LoRAs
• Camera control

Workflow Optimization

Efficiency Nodes

		cd ComfyUI/custom_nodes
git clone https://github.com/jags111/efficiency-nodes-comfyui.git
	

Features:

• Batch processing
• XY Plot generation
• High-res fix presets
• Noise injection tools

Custom Scripts

		cd ComfyUI/custom_nodes
git clone https://github.com/pythongosssss/ComfyUI-Custom-Scripts.git
	

Features:

• Node auto-arrangement
• Prompt autocompletion
• Image feed viewer
• Mathematical expressions

Performance Optimization

GPU Memory Management

		# In ComfyUI arguments
--gpu-only          # Force GPU usage
--highvram          # High VRAM mode
--lowvram           # Low VRAM mode
--novram            # No VRAM mode (slow)
--disable-smart-memory  # Force memory unloading
	

Batch Processing

• Use Rebatch Latents for large batches
• VAE Tiling for high-resolution images
• Model Offloading to manage GPU memory

Caching and Reuse

• ComfyUI caches model weights between runs
• Reuse loaded models across workflows
• Use Cache Node for expensive operations

Troubleshooting

Common Issues

"Model not found" Error

• Check model file location
• Refresh ComfyUI (press R)
• Verify file format (.ckpt or .safetensors)

Out of Memory (OOM)

• Reduce batch size
• Lower resolution
• Use --lowvram flag
• Enable model offloading

Nodes not appearing

• Restart ComfyUI after installing custom nodes
• Check console for errors
• Verify dependencies are installed

Generation fails silently

• Check queue for error messages
• Verify all nodes are connected
• Ensure model files are not corrupted

Debug Mode

Run with verbose output:

		python main.py --verbose
	

Logs

Check logs in:

• Console output
• ComfyUI/temp/ directory
• Browser developer console

Advanced Features

API Usage

ComfyUI provides a REST API for automation:

		import requests
import json
 
# Queue a workflow
response = requests.post("http://127.0.0.1:8188/prompt", json=workflow_data)
	

Custom Node Development

Create your own nodes:

		class MyCustomNode:
    @classmethod
    def INPUT_TYPES(s):
        return {"required": {"image": ("IMAGE",)}}
 
    RETURN_TYPES = ("IMAGE",)
    FUNCTION = "process"
 
    def process(self, image):
        # Your processing logic here
        return (processed_image,)
	

Workflow Templates

• Save workflows as JSON
• Embed workflows in PNG metadata
• Share workflows via GitHub Gist

Resources and Community

Official Resources

• ComfyUI GitHub
• ComfyUI Examples
• ComfyUI Documentation

Learning Resources

• Stable Diffusion Art ComfyUI Guide
• ComfyUI Academy
• YouTube: Matthew's ComfyUI

Community

• Reddit r/ComfyUI
• ComfyUI Discord
• Civitai ComfyUI Workflows

Model Sources

• Civitai - Community models
• Hugging Face - Official models
• Stability AI - Base models

Tips and Best Practices

Workflow Design

• Keep workflows modular and reusable
• Use clear naming for nodes
• Group related nodes together
• Document complex workflows

Performance

• Use appropriate resolutions for your GPU
• Batch similar operations
• Cache frequently used models
• Monitor VRAM usage

Organization

• Save workflows with descriptive names
• Use folders for different project types
• Backup important workflows
• Version control your custom setups

Learning Path

1. Master basic text-to-image
2. Learn img2img and inpainting
3. Experiment with ControlNet
4. Try advanced upscaling techniques
5. Create custom workflows for specific needs
6. Develop your own custom nodes

Conclusion

ComfyUI represents the cutting edge of Stable Diffusion interfaces, offering unparalleled flexibility for creative and technical users alike. While it has a steeper learning curve than simpler interfaces, the control and customization possibilities make it invaluable for serious AI image generation work.

Start with the basics, experiment fearlessly, and gradually build your understanding of the node-based paradigm. The ComfyUI community is active and helpful, with extensive documentation and shared workflows to accelerate your learning.

Remember: every complex workflow started as a simple connection of basic nodes. Happy generating!