Creative Sketch Generation

Sketch-related AI so far has primarily focused on mimicking the human ability to perceive rich visual information from simple line drawings [1, 2] and to generate minimal depictions that capture the salient aspects of our visual world [3, 4]. Most existing datasets contain sketches drawn by humans to realistically mimic common objects [5, 6, 7, 8]. Surprisingly, in spite its popularity, the role of sketching or doodling as a creative exercise has been largely unexplored in AI research.

Figure 1: Cherry-picked example sketches from our proposed datasets: Creative Birds (left) and Creative Creatures (right). See random examples in Figure 2.

In this work we focus on creative sketches. AI systems that can generate and interpret creative sketches which can inspire, enhance or augment the human creative process or final artifact. Concrete scenarios include automatically generating an initial sketch that a user can build on, proposing the next set of strokes or completions based on partial sketches drawn by a user, presenting the user with possible interpretations of the sketch that may inspire further ideas, etc.

AI for creative sketches is challenging. They are diverse and complex. They are unusual depictions of visual concepts while simultaneously being recognizable. They have subjective interpretations like aesthetics and style, and are semantically rich – often conveying a story or emotions.

Figure 2: Random sketches from existing datasets (left) and our creative sketches datasets (right).

To facilitate progress in AI-assisted creative sketching, we collect two datasets – Creative Birds and Creative Creatures (Figure 1) – containing 10k creative sketches of birds and generic creatures respectively, along with part annotations (Figure 2 right columns). To engage subjects in a creative exercise during data collection, we take inspiration from a process doodling artists (e.g., Michal Levy) often follow. We setup a sketching interface where subjects are asked to draw an eye arbitrarily

∗The work was done when the first author interned at Facebook AI Research.

4th Workshop on Machine Learning for Creativity and Design at NeurIPS 2020, Vancouver, Canada.

                                   QuickDraw Bird TU-Berlin Bird Sketchy Bird Creative Birds Creative Birds Creative Creatures (Jonge-jan et al., 2016) (Eitz et al.,2012) (Sangkloy et al., 2016) (w/ part labels) (w/ part labels)

 

around a random initial stroke generated by the interface. Subjects are then asked to imagine a bird or generic creature that incorporates the eye and initial stroke, and draw it one part at a time. Figure 2 shows example sketches from our datasets. Notice the larger diversity and creativity of birds in our dataset than those from existing datasets with more canonical and mundane birds. More sketch examples can be found in Figures 6 and 7.

We focus on creative sketch generation. Generating novel artifacts is key to creativity. To this end we propose DoodlerGAN – a part-based Generative Adversarial Network (GAN) that generates novel part appearances and composes them in previously unseen configurations. During inference, the model automatically determines the appropriate order of parts to generate. This makes the model well suited for human-in-the-loop interactive interfaces where it can make suggestions based on user drawn partial sketches. A schematic of our approach is shown in Figure 3.

(a) Training Phase (b) Inference Phase

Figure 3: DoodlerGAN. (a) During training, given an input partial sketch represented as stacked part channels, a part selector is trained to predict the next part to be generated, and a part generator is trained for each part to generate that part. (b) During inference, starting from a random initial stroke, the part selector and part generators work iteratively to complete the sketch.

Figure 4: DoodlerGAN generates sketches of hybrid creatures

We do quantitative evaluation and human studies to show that our approach generates more creative and higher quality sketches than existing vector-based generation model SketchRNN [3] and raster- based generation model StyleGAN [9]. The results can be found in Table 1 and Figure 9. In fact, subjects prefer sketches generated by DoodlerGAN over human sketches from the Creative Birds dataset! We find that our model often generates hybrids of creatures (e.g., a cow and a parrot, see Figure 4) – a strong indication of creativity!

   True part Predicted part

Cross Entropy Loss

GAN Loss

      Input partial sketch

Classifier

 Part generator:

                       Input partial Encoder StyleGAN2 Discriminator sketch generator

Part selector:

   Stroke sampler

Random initial stroke

Intermediate partial sketch 1

Intermediate partial sketch 2

Intermediate partial sketch N

Part selector

Part selector

Part selector

Part selector

Eye selected

𝑧~𝑁(0,1)

Head selected

𝑧~𝑁(0,1)

Beak selected

𝑧~𝑁(0,1)

Part generator (eye)

Part generator (head)

Part generator (beak)

Final sketch

                         Completion signal

        Squirrel Sea turtle

Cow Parrot

Kangaroo Frog Parrot Penguin

Panda Raccoon Snake Owl

Parrot Penguin

Penguin Mouse

Horse Cat

Bear Sea turtle

Shark Bird

Monkey Parrot

      Finally, we convert the generated raster images of sketches to vector representation results with open source software mkbitmap and Potrace. We also intentionally introduce ar- tifacts by controlling the stoke styles and shapes. We found such artifacts to be more visually appealing and preferred by human subjects 96% of the time. Examples are shown in Figure 5. Our datasets, code, a web demo, and human evaluation protocol will be made publicly available.

Figure 5: Vectorized sketches at higher resolution.

     2

References

[1] Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. Sketch-a-Net that Beats Humans. In BMVC, 2015.

[2] Lei Li, Changqing Zou, Youyi Zheng, Qingkun Su, Hongbo Fu, and Chiew-Lan Tai. Sketch- R2CNN: An Attentive Network for Vector Sketch Recognition. arXiv:1811.08170, 2018.

[3] David Ha and Douglas Eck. A Neural Representation of Sketch Drawings. In ICLR, 2018.

[4] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image Translation

with Conditional Adversarial Networks. In CVPR, 2017.

[5] Mathias Eitz, James Hays, and Marc Alexa. How Do Humans Sketch Objects? In SIGGRAPH,

2012.

[6] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The Sketchy Database:

Learning to Retrieve Badly Drawn Bunnies. ACM Transactions on Graphics (TOG), 2016.

[7] Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. Quick,

Draw! The Data, 2016. Licensed under (CC-BY 4.0).

[8] Haohan Wang, Songwei Ge, Eric P. Xing, and Zachary C. Lipton. Learning Robust Global

Representations by Penalizing Local Predictive Power. In NeurIPS, 2019.

[9] TeroKarras,SamuliLaine,andTimoAila.AStyle-BasedGeneratorArchitectureforGenerative

Adversarial Networks. In CVPR, 2019.

[10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS, 2017.

[11] Nan Cao, Xin Yan, Yang Shi, and Chaoran Chen. AI-Sketcher: A Deep Generative Model for Producing High-quality Sketches. In AAAI, 2019.

3

A Example sketches from datasets and generations

See Figures 6 and 7 for example sketches from the Creative Birds and Creative Creatures datasets respectively. See Figure 8 for randomly sampled sketches generated by DoolderGAN.

Figure 6: Random example sketches from our Creative Birds dataset.

 4

 Figure 7: Random example sketches from our Creative Creatures dataset.

5

 (a) Creative Birds.

 (b) Creative Creatures.

Figure 8: Uncurated creative sketches generated by DoodlerGAN.

6

B Evaluation

Table 1: Quantitative evaluation of DoodlerGAN against baselines on Fréchet inception distances (FID)) [10], generation diversity [11], characteristic score (CS) and semantic diversity score (SDS). CS that checks how often a generated sketch is classified to be a bird (for Creative Birds) or creature (for Creative Creatures) by the trained Inception model. SDS measures how diverse the sketches are in terms of the different creature categories they represent.

 Methods

Training Data

SketchRNN Uncond. SketchRNN Cond. StyleGAN2 Uncond. StyleGAN2 Cond. Percentage-based

DoodlerGAN

Creative Birds Creative Creatures

FID(↓) DS(↑) CS(↑) FID(↓) DS(↑) CS(↑) SDS(↑)

   - 19.40 0.45

79.57 17.19 0.20 82.17 17.29 0.18 71.05 17.49 0.23

130.93 14.45 0.12 103.79 15.11 0.20

39.95 16.33 0.69

- 18.06 0.60 1.91 60.56 15.85 0.43 1.22

54.12 16.11 0.48 1.34 103.24 14.41 0.18 0.72 56.81 13.96 0.37 1.17 57.13 13.86 0.41 1.17

43.94 14.57 0.55 1.45

    SketchRNN Unconditional StyleGAN2 Unconditional Percentage-based Creative Datasets SketchRNN Conditional StyleGAN2 Conditional QuickDraw Birds

     80 60 40 20

                                Creative

Looks Like like bird

Human drawn

Initial stroke integrated

Creative

Looks like creature

Like

Human drawn

Initial stroke integrated

Figure 9: We ran human studies using Amazon Mechanical Turk (AMT) on Creative Birds (left) and Creative Creatures (right) for five questions: which one of two sketches (1) is more creative? (2) looks more like a bird / creature? (3) they like better? (4) is more likely to be drawn by a human? For the conditional baselines, we also ask (5) in which sketch is the initial stroke (displayed in a different color) better integrated with the rest of the sketch? We evaluated 200 random sketches from each approach. Each pair was annotated by 5 unique subjects. Higher values → DoodlerGAN is preferred over the approach more often.

7

% times DoodlerGAN wins