🎨 Diffusion Digest: Open Model Initiative Takes Shape, PixArt Joins Forces with NVIDIA, SD3 License Complexities (June 30, 2024)

🎨 Welcome to DiffusionDigest for the week of June 30, 2024! In this issue, we explore the launch of the Open Model Initiative, the PixArt team's move to NVIDIA, and the legal hurdles surrounding Stable Diffusion 3's license. So grab a cup of coffee, settle in, and let's go!

🚀 Open Model Initiative Launch

TL;DR: The new Open Model Initiative aims to develop open AI models for generating media, but the community has concerns about potential censorship, synthetic data quality, and resource requirements.

The Open Model Initiative, a community-driven effort, has recently launched with the goal of advancing the development and adoption of openly licensed AI models for generating images, video, and audio. The initial members of the Open Model Initiative include Invoke, ComfyOrg (the ComfyUI team), LAION, and Civitai. Their plans involve establishing governance, surveying the community, creating interoperability standards, supporting high-quality open model development, and hosting events.

The announcement of the Open Model Initiative generated substantial discussion online, with community members expressing various concerns and opinions. Some worried that an overemphasis on "safety" and "ethics" could lead to heavily censored and limited models, hampering artistic expression. Many called for the initiative to prioritize model capability and avoid compromising on content filtering. Others expressed skepticism about the feasibility of training a competitive model using primarily AI-generated or synthetic data, fearing that it could amplify flaws and artifacts. However, some noted that synthetic data is easier to train on and that Civitai has a strong repository.

There was also uncertainty regarding the immense compute resources and funding needed to deliver a high-quality open model in a reasonable timeframe, with estimates of training costs ranging from $50K to $300K+. Some community members called for the initiative to avoid introducing arbitrary restrictions around "consent" for training data, arguing that permission requirements for learning from public data would make open models nonviable. Despite these concerns, the effort represents an important step in keeping generative AI accessible as the technology advances rapidly.

Source.

🤝 PixArt Team Joins NVIDIA

TL;DR: The PixArt team's move to NVIDIA has raised concerns about licensing, censorship, and corporate influence on open-source AI models, but also presents opportunities for growth and improvement. The community remains divided on the implications for the future of open-source AI.

The recent announcement of the PixArt team, creators of an open-source alternative to Stable Diffusion 3 (SD3), joining NVIDIA has sparked a lively discussion within the AI community. The news has raised questions about the future of the PixArt project and the implications for open-source AI models in general.

Licensing and Censorship Concerns

One of the primary concerns expressed by the community is the potential impact of NVIDIA's involvement on the licensing and censorship of the PixArt model. Some users worry that NVIDIA's influence may lead to more restrictive licensing terms and increased censorship. However, others point out that NVIDIA benefits from open-source models driving demand for their GPUs, which could incentivize them to continue supporting the project.

NVIDIA's Open-Source Track Record

NVIDIA has a mixed history when it comes to open-source initiatives. While the company has released some models and tools under permissive licenses, it has also faced criticism for its proprietary CUDA ecosystem and lack of open driver documentation.

Potential for Growth and Improvement

The PixArt team aims to leverage NVIDIA's resources to make the model "more efficient and stronger." Some users speculate that NVIDIA's involvement could help establish PixArt as a new center for open-source image generation, especially given the uncertainties surrounding Stability AI and the mixed reception of SD3.

Reservations About Corporate Involvement

Not all reactions to the news have been positive. Some users express skepticism about big tech companies releasing open-source image generation models, citing examples of Meta, Google, and others releasing AI models while restricting the image generation capabilities. There are also concerns that NVIDIA's involvement could lead to the "consumption" of the open-source project, although it's noted that PixArt is a collaboration between Huawei Lab and academic institutions, which may limit NVIDIA's control.

Source.

Source.

⚖️ Legal Review Raises Concerns Over Stability AI's Stable Diffusion 3 License

TL;DR: Civitai has banned Stable Diffusion 3 (SD3) due to problematic terms in its Creator License Agreement, which could impact a wide range of AI models and require users to have a membership agreement with Stability AI. The decision aims to protect the community from potential legal risks until further clarification is provided.

Civitai, a prominent AI art community platform, has recently banned Stable Diffusion 3 (SD3) from its site due to licensing concerns. The decision came after Civitai's legal team reviewed the Creator License Agreement (CLA) for SD3 and found several problematic terms that could have far-reaching consequences for users and derivative works. According to the legal review, the CLA's definition of "Derivative Work" extends beyond fine-tuned models based on SD3. It also includes any model trained using outputs from SD3 or SD3-based models, even if not directly based on SD3. This broad interpretation could potentially impact a wide range of AI models in the community, not just direct SD3 derivatives.

Furthermore, the CLA requires users who download SD3 or any derivative works to have a membership agreement with Stability AI. This means that making a copy of SD3 or derivative works for use within Civitai's platform would be considered a "download" requiring a license, even if users don't take possession of the software. The legal review also highlighted that if notified by Stability AI, users must cease use of SD3 and derivative works that allegedly infringe on any third party's rights, including removing models from Civitai's platform. Additionally, the CLA is silent on who owns derivative works, and users would need sufficient rights from Stability AI in the original SD3 model that derivatives are based on.

Another concern raised was Stability AI's unilateral ability to modify the agreement at any time, with changes effective for the next 30-day renewal term. Given Stability AI's recent financial struggles, this provision has caused apprehension among the community. In response to these findings, Civitai has decided not to support SD3 on its platform until Stability AI provides further clarification on the license terms. The decision aims to protect Civitai and the broader stable diffusion community from potential legal risks stemming from the SD3 license.

Source.

🆕 Pushing the Boundaries: Latest Developments in AI-Powered Creative Tools

• SDXL ControlNets: Xinsir released new ControlNets for Stable Diffusion XL (SDXL), including models for depth and tile, joining their existing canny, scribble, and openpose ControlNets. The tile ControlNet generates excitement for its potential to improve SDXL's image upscaling capabilities compared to previous options.

• ControlNet Support Added for SD3: A new update to the Comfy UI app introduces support for using ControlNet models with Stable Diffusion 3 (SD3). SD3's transformer architecture deeply integrates ControlNet at each diffusion step, unlike convolutional ControlNets used with SDXL, potentially enabling more nuanced control. Early tests indicate ControlNet usage may reduce SD3 output quality.

• Diffutoon: a new AI technique using diffusion models for high-resolution, editable toon shading in animation, with a Jupyter notebook available to try. While impressive, the technology still has some flaws and inconsistencies.

• AuraSR Upscaler: a new open-source super-resolution upscaler based on GigaGAN, was recently released on GitHub. However, initial testing by the community found that AuraSR produces poor quality results compared to existing upscalers like SUPIR and StableSR, with common issues including artifacts, oversharpening, noise, and sensitivity to compression.

What did you think of this week’s issue? We love hearing from you!