[MacOS] Instructions for installing Stable Diffusion WebUI and ComfyUI for Macbook

As a complete newbie, I’ve only been learning about Stable Diffusion for a short time, so the content is inevitable. errors, the purpose of writing this article is to support those who use Mac but still like to tinker with and run light models like SD1.5. For SDXL and Flux, you should rent a server or use Google Colab.

Before entering the settings, I will cite some basic definitions and types of Stable Diffusion interfaces available at the present time. If you are new to learning, you should read through them to understand better.

I. What is Stable Diffusion?

Stable Diffusion is a deep learning model developed to generate images based on text. It belongs to the family of diffusion models, a class of machine learning models that can learn and reconstruct data by gradually removing noise from image or video data to restore the original image or create a new image from the noise.

The Stable Diffusion model is famous for its ability to generate high-quality images from descriptive text (text-to-image generation). It is widely used in applications such as:

• Generate images from text: Users just need to enter a description, the model will generate images based on that information.
• Modify or improve the image: For example, change the style, or add specific details to the image.
• Generate random images based on short description or any keywords.

Stable Diffusion was developed by the Stability AI team in collaboration with several other organizations, including EleutherAI and LAION. The first version was released in 2022, and it quickly gained traction in the community for its feasibility and the quality of images it can produce.

Some outstanding features:

• Extendability and Customization: Users can tweak the model or add features to meet specific needs.
• Open Source: Stable Diffusion is open source, allowing the community to use and develop more applications based on this platform.
• Various applications: It is used in the fields of digital art, creative design, and artificial intelligence research.

II. Types of Stable Diffusion graphical interfaces

1. WebUI (Stable Diffusion WebUI)

WebUI is a popular graphical interface for interacting with the Stable Diffusion model. Instead of having to use command line or technical methods to work with AI imaging models, WebUI provides an easy environment for users to upload text descriptions and receive corresponding images generated by the model.

Some main features:

• Easy to use: Users do not need to have deep technical knowledge, just enter a text description (prompt) and get back an image from Stable Diffusion.
• Customize Parameters: The interface allows users to customize various parameters such as description length, image resolution, and art style for optimal results.
• Plugins and Integrations: Many plugins and tools are built-in to extend functionality such as adjusting color, lighting, or adding special filters.

WebUI has many versions through development periods such as: A1111, Forge and the latest is Forge v2.

2. ComfyUI

ComfyUI is a GUI based control interface for Stable Diffusion, but with a flexible approach and a focus on powerful customization. Instead of just entering a description and getting an image like WebUI, ComfyUI allows users to control the image creation process step by step through a node-based system.

Some highlights:

• Node-based workflow: ComfyUI works like a node-based photo editing software. Each node represents a specific action in the image creation process (e.g. adding noise, diffusing, combining elements, or refining). Users can freely connect nodes to create their own workflow, allowing them to customize the image creation process in detail.
• Model customization and fusion: ComfyUI allows for the combination of different models and methods, including combining pretrained models to create unique results.
• Pipeline Tuning: Users can tune the pipeline of the imaging process, from selecting different datasets to tuning each step of the diffusion model.

Must-know terms in ComfyUI:

A. Workflow

• Workflow in ComfyUI is the process of creating an image through a set of nodes. These nodes represent processing steps such as pre- process images, combine AI models, adjust parameters, and export the final result. The workflow as a project can be saved and uploaded for reuse.

B. Node

• Nodes are the basic units in ComfyUI that perform certain tasks in the workflow. Each node can receive input data and process it to produce output data for other nodes. There are many different types of nodes, from image processing nodes, AI model related nodes, to parameter tuning nodes.

C. Model

Model in ComfyUI involves using artificial intelligence models to generate images, edit images, or control the image creation process. Popular models include:

• Checkpoint: This is the main model used to generate images from input. Checkpoint stores the state of the deep neural network after training and is used to initialize images from the description or input image.
• ControlNet: Is a model that helps control the image generation process, usually by adding constraints or guidance information to the original model. For example, you can use ControlNet to impose conditions such as the structure of objects in an image.
• LoRA (Low-Rank Adaptation): A technique for fine-tuning a model by updating only small components of the main model, instead of retraining the entire model. LoRA is often used to increase the ability to generate images of a specific style or theme.
• VAE (Variational Autoencoder): A type of autoencoder used to encode images into latent space and vice versa. In ComfyUI, VAE can be used to compress and decode images, support high-quality image generation, and control the color or detail of images.

Understanding these terms will help you create and customize workflows in ComfyUI more efficiently

III. Instructions for installing Stable Diffusion ComfyUI for Macbook

There are 2 main ways to install Stable Diffusion for Macbook: directly through Terminal and indirectly through a 3rd party application called Stability Matrix.

1. Install directly via Terminal

This method currently only has instructions for installing ComfyUI but it’s a bit complicated, only for those who like to tinker. For simplicity, see method 2 below, install ComfyUI and WebUI with just 1 click.

• Create a folder anywhere with the name Stable Diffusion

• Open the Terminal app, type cd[space] and drag the newly created folder into it and hit enter.

• Go to https://brew.sh/ to copy the command and paste it into Terminal to install Brew. Or use the line below:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• Terminal will ask for a password with a key icon, enter your mac password and enter. Note that nothing will happen when entering. Continue to Enter again to start.

• Wait a moment for Brew to complete the installation process. Continue running the following code:

eval "$(/opt/homebrew/bin/brew shellenv)"

• Check if python is installed with the command python3 –version. If not, go to google to download the latest version or you can also install version 3.11 with the command :

brew install python@3.11

• Wait for python to finish installing, continue running the command:

pip3 install --upgrade pip setuptools

• The following command will take quite a while, please be patient

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu

• After running, inside the initially created Stable Diffusion folder, a ComfyUI folder will appear. Type in Terminal cd[space] and drag the ComfyUI folder in, then enter.
• Continue running the following command to install all required extensions.

pip3 install -r requirements.txt

• While Terminal is still in the ComfyUI folder, run the following command to start ComfyUI.

python3 main.py

• Continue to install ComfyUI-Manager, this node helps you manage and download other missing nodes when you use other people’s shared workflow.

With this method, when you want to open ComfyUi, you have to go to command, type cd[space] drag the Stable Diffusion folder that was installed initially and enter, then run the command python3 main.py. It’s a bit complicated, so for convenience, I will guide you through the second method.

2. Using Stability Matrix to Install WebUI and ComfyUI

Go to StabilityMatrix, scroll down and there will be Download buttons, choose the correct operating system you need, here I use MacOS (Apple Silicon).

After downloading, open the file and drag the application to the Application folder, open it and select Add Package, here you can install any favorite interface.