RunwayML Experiment - Case Study with Ceramics
PART I
THOUGHTS ABOUT CERAMICS, MACHINE LEARNING, AND THE FUTURE OF NON-ESSENTIAL WORK.
For the past few weeks, I have been isolated in my apartment with my husband. We moved from Santiago, Chile to New York three years ago, and we never had clear plans about how long we would stay here. Due to this uncertainty and for financial security, I decided that my pottery business would be based on a made-to-order model and that I would carry a small stock. I also chose to rent a space in a collaborative studio for ceramic artists instead of having my own studio. Today, I realize that everything that seemed to suit my lifestyle has turned out to be somewhat of an obstacle. Nevertheless, I feel that pottery develops resilience, and I also believe that limitations can be beneficial for creation, for rethinking one's work, and for pushing creativity into new territories.
I have been contemplating how the crisis we are facing highlights our dependence on supply chains, materials, and workspaces. Along with these thoughts, I have been trying to focus on what I have available at this moment: a computer, my phone, internet access, and a vast archive of pictures of my work. Can these assets become my new tools for work?
Perhaps the way we have been creating, producing, making, and shipping products and art will turn out to be outdated. Today, we are being forced to lead more virtual lives and move towards a more digital existence. Maybe we are on the brink of a new creative revolution, and now is the time to discover and explore the capabilities of digital tools. If we revert to the way things were, at least we will know that there is an alternative for staying creative.
Digital tools such as machine learning, 3D modeling and printing, augmented reality, and virtual reality have made exciting progress in recent years, especially for designers, artists, and creators. Until now, I have never paid attention to any of them, mainly because I am passionate about handmade crafts and lack a background in computer science. However, my partner has been developing software to make machine learning easy and accessible for the past year. In this coronavirus world, I have become curious about its functionality.
What is Machine Learning or ML (from a ceramicist’s point of view)?
ML, or machine learning, is a process in which a computer utilizes pattern recognition and algorithms to perform and execute specific tasks without relying on explicit instructions. It possesses the ability to comprehend, predict, and define future trends or aesthetics. In addition to the features mentioned below, ML has the capacity to generate outputs resembling its training data and discover unforeseen combinations that can prove beneficial to you.
Using Machine Learning
RunwayML is a software that is accessible to those of us who don't know how to code but want to explore the capabilities of Machine Learning. The application has pre-trained machine learning models that can assist you in tasks such as colorizing and restoring old images, generating paragraphs of text from a prompt, and stylizing images in the style of famous paintings, among many others. The models are fascinating to experiment with, but they have limited customization options if you're seeking more flexibility.
Colorize and Restore Old Images Model. Source: www.runwayml.com > ML Lab
Generate paragraphs of text from prompt “NASA discovered a new type of clay on Mars”. Source: www.runwayml.com ML Labs
Style Transfer. A Mug + Matisse
Browsing RunwayML Models
Test 1
I trained a custom model using images of my mugs and cups. The first step to training a custom model is to gather the dataset (a collection of similar images) and then preprocess them. In this case, the preprocessing step involved standardizing the background, removing undesired elements, and cropping them into a square format. It's important to consider that for better training results, your dataset should consist of no fewer than 500 images.
Mugs & Cups dataset for my custom model and Test 1.
After I had the dataset ready, I uploaded the files to the software. I selected the generative model StyleGAN, set up the training steps to 3,000, and waited for 3 hours until the training process was finished. Once it was ready, I could explore the model through the latent space interface. The latent space consists of 512 dimensions, each offering infinite possibilities for synthetic images.
Setting up the training experiment.
The results were not what I expected since the synthetic images were very similar to my work. This detail made me realize the limitations and biases of the hand-building technique I use and that the mugs and cups dataset wasn't diverse enough. As you can see from the pictures below, all the mugs and cups have a similar cylinder-like appearance. So, for the second test, I wanted to explore if machine learning could help me envision a different version of my work, one in which the objects look more organic. I wondered what might happen if I mixed the mugs and cups dataset with another dataset.
Navigating the latent space of Test 1. The result of this training process is a generative model. As it names says, a generative model is capable of generating images or outputs based on the dataset that was fed in the first step. In Test 1 my dataset was the mugs.
Test 2
I searched the internet for "organic" shaped objects that could be related to what I was looking for, and I came across a fantastic collection of Open Access* vases from the MET Museum. The catalog featured over 28,000 images, from which I handpicked 400. This process was reminiscent of searching for inspiration and creating a mood board.
Vases dataset for my custom model and Test 2.
In the digital space, there is still an aspect of copyright that remains unclear to me. Can a mood board be subject to copyright infringement? What is considered public domain? I believe this is a topic that should be thoroughly examined in the future.
Dataset preview of Mugs & Cups + Vases for Test 2.
I followed the same steps as in Test 1 and waited for the model to be ready. However, this time, the outputs generated in the latent space dimensions were unusual and unexpected. The model was able to recognize the handles and manipulate them. The color palette and the overall aesthetic were also intriguing.
Latent walk video.
I hand-picked 124 generative “synthetic ceramics” that I organized into collections. None of the objects above are real. They have all been generated via ML.
During the process of this experiment with ML, I felt like a curator. I could select the dataset by manually choosing the images that I found interesting. While browsing the outputs generated by the model, I was once again the one who identified which of them were intriguing and meaningful to me. After spending a few hours exploring the latent space of the custom model, I hand-picked 124 generative images and organized them into collections which you can check below.
Thoughts and Next Step
My vision for machine learning (ML) is not one where technology comes and takes over everything. Instead, it is one where ML can become your creative partner and assistant. A new kind of collaborator that can surprise you, provide unseen details of your work, tune in with your aesthetic quests, and create new opportunities for creative collaborations. Now that I have explored some capabilities of this technology, I am wondering how I can translate the outcomes into final products or experiences. The paths I am thinking of are:
Human-built by modeling with clay.
Machine-built by printing clay with 3D.
Virtual built by mapping and creating a 3D virtual version of the objects. Can you imagine a world in which you only own "essential" products, and everything that is considered "non-essential" is available in digital format? Is this going to be the real sustainable way of creating in a post-COVID-19 world facing the climate change threat?
Handles
When I found these synthetic creations I was fascinated. The combinations and playfulness of the handles makes me wonder if ML can be creative.
Color Palettes
Mugs created by ML. Color palette is soft and neutral.
Mugs created by ML. Color palette is soft and neutral.
Shapes
My Final Selection
UPDATE: Here is a basic demo of how to use RunwayML.
PART II
FROM IDEA TO “REALITY”
It is interesting to see some of the questions I had last year.
Can you imagine a world in which you only own “essential” products, and everything that is considered “non-essential” is available in digital format?
At one moment, I truly believed it was the end of tangible objects and struggled to imagine how I could create digital versions of them and sell them as collectibles. I read some articles that discussed how the fashion industry was beginning to create digital clothes, but I couldn't muster any real enthusiasm for the idea.
The landscape has changed significantly in 2021, thanks to the evolution and widespread adoption of NFTs. However, I am not fond of how NFTs are employed in the art world. Perhaps there is a valid use case for real estate and legal documents, but when it comes to art, I still can't find genuine value and meaning.
From ML to hand-built, to 3D modeling, to 3D Printing
The option of hand-building one of the vases was the most attractive but I had to consider that I wouldn’t have a ceramic kiln available so I used a box of Air Dry and Self-Hardening Clay. I also realized that my tools were at the studio so I had to use a kitchen knife and cut a piece of cloth from a bedsheet. After building the vase I let it dry for a day and it was ready for display. It is important to note that this kind of material is not waterproof so it is not functional and is intended for decoration.
Later I took pictures of the piece from different angles and then used DaVinci to create a render.
Once I had the render ready I googled for a 3D printing service that could make a piece with a soft and smooth surface and came across Kwambio. I reached out to them at the beginning of the pandemic so the process was slow and it took around 1 year, and 73 emails, to finally receive the piece. Although it was slow, the service they offer is promising.
To start working they sent a guideline with instructions and explaining what is possible to 3D print, the thickness minimums and maximum requirements, specifications for firing and glazing, and production time. There was also a .pdf catalog of materials and prices. I had to adjust one of the handles because the thickness minimum was 5mm. My pieces are usually 2mm thick so it is a big difference in feeling and overall look. Anyways, I still consider the 3D printing option as a great alternative for remote creation and production.
Unfortunately, the result didn't look as I expected and it arrived broken. But it was worth the try. What do you think?
