Adversarial texture distribution learning as a tool of artistic expression

Deep learning is progressing fast these days. Despite advances that were expected to happen sooner or later (e.g. accurate face and speech recognition), there are some new developments that would have seemed like a pipe dream years ago: neural networks can now generate realistic images just by looking at few examples of their properties.

Zalando Research is currently exploring such methods and their potential to aid Zalando’s content creation, private fashion labels, and sizing recommendation teams, and offer our customers a new fashion experience. In addition, working with large image collection and generative machine learning models has great synergy with cutting edge neural network art. The tools created for fashion research purposes are also useful as tools for visual artistic creation and exploration.

Generative texture models Our research in texture generation is a good example of this. Earlier this year, we developed new deep learning generative models to learn textures from just a few sample images, and textures are key ingredients in multiple artistic techniques. Having a tool like a Periodic Spatial Generative Adversarial Networks, or PSGAN, to learn texture distributions can lead to great flexibility in choosing applications for it. Figure 1 shows what textures we can learn and sample using only a single image (Figure 2) as training material.

Figure 1. Ocean textures generated from our model, a PSGAN trained using the single image from Figure 2.

Figure 2. An example ocean texture, which is used as training material for PSGAN

Textures and mosaics Mosaics are a classical artform starting from the times of ancient Romans and going to modern texture transfer techniques. The artist Max Ernst captured textures by physically copying and using them for his painting, a technique called Frottage. In present times, selecting a texture as a type of stylization and applying it to a large image with global composition is a very popular case of ML art, as signified by the success of Neural Art Style Transfer, but also as seen in multiple advertising campaigns such as Figure 3.

Figure 3. Example outdoor advertisement billboard using mosaic techniques for stylization, seen recently on the streets of Berlin.

In follow-up work we apply generative texture synthesis to create high-resolution mosaics from input content images. It was demonstrated at the NIPS 2017 Workshop for Machine Learning and Art, a great venue to explore collaborations between machine learning researchers and artists. Figure 4a shows how the texture process learned in the previous paragraph can be conditioned and used to stylize a human face from Figure 4b.

Figure 4a.A mosaic stylisation of the image from Figure 2b)

Figure 4b. A fashion model from Zalando's catalogue. * Texture control: from noise to music* Static image stylization works by conditioning the latent factors of textures on a target content image, which is a 2D array. Textures flow into one another in space and the artist can play flexibly with our tool and condition on a moving signal in time, which would then lead to smooth transitions and animations between textures in time. Music visualization is one such application of our technique. Figure 5 shows our music video accepted at the ECCV 2018 Computer Vision for Fashion, Art and Design Workshop.

*Figure 5: water textures morphing, controlled by music audio signal. * With an appropriate audio descriptor we can map the distribution of audio samples to the distribution of textures. And since music is a smoothly varying signal, we can create an animation frame by frame of a texture process controlled by music. Selecting input images with a suitable theme (water) to train the PSGAN allows us to emphasize the artist’s vision and represents a novel form of digital synesthesia. This tool also opens a totally new pathway for collaboration between musicians and generative model visualizations. It is also a conceptually new approach to art, since we replace the source of randomness with the variation of a music piece.

We will be showcasing the video at the workshop mixer on September 12th, at the Container Collective in Munich.

Follow Nikolay Jetchev’s twitter account for the latest developments and experiments with art and generative deep learning.

We're hiring! Do you like working in an ever evolving organization such as Zalando? Consider joining our teams as a Machine Learning Engineer!