Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing (ECCV 2022)

Jaskirat Singh1,2, Liang Zheng1, Cameron Smith2, Jose Echevarria2,
1Australian National University, 2Adobe Research
Paper arXiv Video Demo Code Cite

We propose paint2pix which helps the user directly express his/her ideas in visual form by learning to predict user-intention from a few rudimentary brushstrokes. The proposed approach can be used for (a) synthesizing a desired image output directly from scratch wherein it allows the user to control the overall synthesis trajectory using just few coarse brushstrokes (blue arrows) at key points, or, (b) performing a diverse range of custom edits directly on real image inputs.

Abstract

Controllable image synthesis with user scribbles is a topic of keen interest in the computer vision community. In this paper, for the first time we study the problem of photorealistic image synthesis from incomplete and primitive human paintings. In particular, we propose a novel approach paint2pix, which learns to predict (and adapt) “what a user wants to draw” from rudimentary brushstroke inputs, by learning a mapping from the manifold of incomplete human paintings to their realistic renderings. When used in conjunction with recent works in autonomous painting agents, we show that paint2pix can be used for progressive image synthesis from scratch. During this process, paint2pix allows a novice user to progressively synthesize the desired image output, while requiring just few coarse user scribbles to accurately steer the trajectory of the synthesis process. Furthermore, we find that our approach also forms a surprisingly convenient approach for real image editing, and allows the user to perform a diverse range of custom fine-grained edits through the addition of only a few well-placed brushstrokes.

Some sample images and artwork progressively synthesized from scratch using paint2pix.

Progressive Image Synthesis

When used in conjunction with recent works in autonomous painting agents, we show that paint2pix can be used for progressive image synthesis from scratch. During this process, paint2pix allows a novice user to progressively synthesize the desired image output, while requiring just few coarse user scribbles to accurately steer the trajectory of the synthesis process.

Real Image Editing

In addition to being able to perform progressive synthesis from scratch, paint2pix also offers a surprisingly convenient approach for making a diverse range of custom semantic edits (e.g., add smile for faces) on real images by simply initializing the input canvas with a real image.



Here is a sample reference on using the provided demo for achieving a diverse range of real image edits.

Artistic Content Generation

While the main goal of paint2pix is to perform photorealistic image synthesis from primitive human paintings and brushstrokes, it can also be used for artistic content generation. In the following paint2pix allows a novice artist to generate different forms of professional artistic content (e.g., colored sketches, oil painting, disney cartoons, watercolor paintings, etc.) using just few rudimentary brushstroke inputs.

BibTeX

If you find our work useful in your research, please cite the following works: 
@inproceedings{singh2022paint2pix,
      title={Paint2pix: interactive painting based progressive image synthesis and editing},
      author={Singh, Jaskirat and Zheng, Liang and Smith, Cameron and Echevarria, Jose},
      booktitle={European Conference on Computer Vision},
      pages={678--695},
      year={2022},
      organization={Springer}
    }
@inproceedings{singh2022intelli,
      title={Intelli-Paint: Towards Developing More Human-Intelligible Painting Agents},
      author={Singh, Jaskirat and Smith, Cameron and Echevarria, Jose and Zheng, Liang},
      booktitle={European Conference on Computer Vision},
      pages={685--701},
      year={2022},
      organization={Springer}
    }