Transferscope: Using Gen AI and XIAO RP2040 to Transform Real-World Images

Generative AI is an incredible tool with seemingly unlimited potential. In many cases, the primary bottleneck to unlocking this potential is human creativity. While prompting these models with natural language is intuitive, it often poses a challenge as it requires users to construct and envision complex ideas solely in their minds without a tangible reference.

Yet, fret no more. Gone are the days of having to meticulously envision and subsequently write prompts for DALL-E, Stable Diffusion, and MidJourney. Now, all you need is your surroundings. Transferscope, developed by Christopher Pietsch, is a XIAO RP2040 and Raspberry Pi powered tool that allows users to directly use their physical environments, bypassing the need for predefined prompts or extensive digital input and forming a prompt that captures textures, patterns, and concepts from real-world objects and applying them to other surfaces. This lowers barriers to creative expression and allows for spontaneous, inspired creations.

Transferscope ready to apply the textures and patterns
from the three yellow tulips (left) to a new scene

Credit:  Christopher Pietsch

The Genesis of Transferscope

Christopher Pietsch, a creative technologist at the HfG Schwäbisch Gmünd, developed Transferscope during his residency at the AI+D Lab. The project was born out of a desire to explore generative image synthesis using Stable Diffusion in a more intuitive and tactile way. As Pietsch explains:

“I wanted to know what it would feel like to use a physical tool to instruct AI – using your surroundings instead of typing prompts.”

This vision led to the creation of a device that allows users to capture any object or concept with a simple click and seamlessly blend it onto any scene, creating new and imaginative realities.

Transferscope result with two sets of images

Credit:  Christopher Pietsch

How Transferscope Works

Transferscope is a marvel of compact engineering. Powered by a Raspberry Pi Zero 2 and encased in a custom 3D-printed housing, the device features a 720×720 pixel resolution HyperPixel screen and integrated Li-Ion batteries. The hardware components include:

• Screen: HyperPixel 4.0 square from Pimoroni
• Computing: Raspberry Pi Zero 2
• Camera: Raspberry Pi Camera Module 3
• I2C Button: the super mini MCU Seeed XIAO RP2040
• Battery: 18650 Li-Ion with DFRobot Power Booster

Credit:  Christopher Pietsch

The device runs on Debian with a custom Python script for image capture and processing. It performs edge detection on-device before sending the data to a server equipped with an RTX 4080 GPU and Intel i7-12700K processor for image interpretation and generation.

Christopher used the XIAO RP2040 in his project because the Raspberry Pi’s GPIO pins were fully occupied by the HyperPixel display. The XIAO allowed him to add a button and potentially more components by connecting to the remaining I2C port on the back of the display. He chose it for its ability to handle I2C communication, its compact size, and the potential for future expandability.

Credit:  Christopher Pietsch

The server runs an optimized pipeline that combines two key functionalities: Stable Diffusion for image generation and Kosmos 2 by Microsoft for image interpretation. This setup allows Transferscope to transform captured images into new textures and patterns and apply them to a second scene almost instantly (in less than a second), creating a seamless experience for the user.

The User Experience

Despite the complex technology behind it, Transferscope offers a remarkably simple and intuitive user experience. Pietsch emphasizes this balance between advanced technology and user-friendly design, saying it is what makes the product “fun and engaging to use.”

The XIAO RP2040-powered single-button interface, combined with clear screen feedback, allows users to focus on creativity without getting bogged down in technical details. This simplicity, enabled by the XIAO’s compact design and I2C capabilities, makes Transferscope easy to use in various environments, encouraging spontaneous creativity. As Pietsch notes, “It’s not just technology deciding the result: the user’s guidance directs the transformation.”

The Development Process

Creating Transferscope was an iterative process that involved exploring both the technology and the form of the device. Pietsch and his team went through multiple prototypes, 3D printing, testing, and refining their designs. The project evolved from two separate devices for sampling and transferring to a single, more streamlined tool.

Interestingly, the development of Transferscope coincided with rapid advancements in AI and image synthesis technologies. This made it a “living project” that evolved alongside these technological developments, constantly incorporating new capabilities and refinements.

Public Reception

The public’s reaction to Transferscope has been overwhelmingly positive. Pietsch shared an amusing anecdote about its reception at an exhibition:

“I put it in an exhibition, and it’s interesting how people react to it and use it. I use it mainly to try that it works, but they started to make face swapping. They point it at somebody they know and then at themselves, and they have a combination of both people and styles. It was funny. They were all laughing and making jokes.”

Transferscope making its public debut

Credit:  Christopher Pietsch

Future Directions

While Transferscope is already an impressive achievement, Pietsch sees room for further development. He’s considering upgrades to the hardware, such as using a compute module for increased processing power, and adding features like an accelerometer, vibration feedback, or NeoPixel lighting. There’s also the possibility of creating a more robust version with an aluminum casing.

Pietsch is open to various possibilities for the future of Transferscope, including potential commercialization through avenues like Kickstarter. However, he emphasizes that these are still exploratory ideas.

Transferscope creator, Christopher Pietsch, showing what his product can do

Credit:  Christopher Pietsch

As a whole, Transferscope represents a significant leap in AI-driven creative processes. By providing an intuitive, tangible interface for image synthesis, it opens new avenues for artistic expression and exploration of our visual world. By seamlessly integrating the XIAO RP2040 to handle crucial I/O functions, Transferscope demonstrates the potential of combining open-source tools, interdisciplinary collaboration, and human creativity with AI. As we navigate the evolving landscape of technology and creativity, projects like Transferscope serve as important milestones that challenge us to reimagine our relationship with AI and our environment, blurring the lines between reality and imagination.