What is Compositionality?

At the core of this pioneering approach lies the concept of deconstructing an image into representative layers—such as text descriptions, depth maps, styles, semantics, and color palettes. These layers can then be recombined or modified to create entirely new images.

“Isn’t this just an endless combination of layers?”

Precisely! This opens up a vast and unprecedented creative landscape for generating custom content. The researchers demonstrated the capabilities using eight representational layers, though the potential extends far beyond that. Even with these eight layers, “Composer” presents astonishing possibilities! Let’s explore further.

Changing Individual Representational Layers

First, let’s consider the following layers: sketch, segmentation, color palette, and text description. By either creating or modifying an image's sketch layer and recomposing it, you can achieve incredible transformations, such as adding clouds or altering the position of limbs:

Alternatively, by adjusting the segmentation layer, you can change ear placements or framing:

Moreover, if you want to rotate a specific element within an image, simply create a segmentation layer that indicates the desired translation:

What about altering color palettes? Yes, you can extract or create new palettes and recombine them with images as you see fit:

Changing the text description layer operates similarly to traditional AI image generation tools (img2img), but with enhanced preservation of image quality and composition:

Combining Layers with Masking

If you thought working with individual layers was impressive, wait until you see the effects of combining them with masking techniques:

In these examples, the masked areas are altered without compromising the overall image quality. The first row illustrates changes to the “text description” layer, while the second row highlights modifications to the “color palette” layer.

Combining Multiple Representational Layers

The potential becomes truly extraordinary when combining multiple layers. For instance, merging text with a color palette yields astonishing results:

Combining depth maps or sketches with text can also lead to incredible visuals:

When combining depth maps, sketches, images, and palettes, the results can be truly mesmerizing:

A combination of segmentation, sketches, images, and palettes showcases the potential of this framework:

And finally, the remarkable fusion of distinct layers—image, palette, depth map, sketch, and segmentation—leaves us in awe of the possibilities:

To stay informed about these advancements, consider following my updates on Twitter or Medium. If you appreciate this content, a “clap” at the end would help others discover it too.