DemoCaricature: Democratising Caricature Generation
with a Rough Sketch

Dar-Yen Chen Ayan Kumar Bhunia Subhadeep Koley Aneeshan Sain
Pinaki Nath Chowdhury Yi-Zhe Song

SketchX, CVSSP, University of Surrey, United Kingdom

[Paper]        [Code]        [BibTeX]

Abstract

In this paper, we democratise caricature generation, empowering individuals to effortlessly craft personalised caricatures with just a photo and a conceptual sketch. Our objective is to strike a delicate balance between abstraction and identity, while preserving the creativity and subjectivity inherent in a sketch. To achieve this, we present Explicit Rank-1 Model Editing alongside single-image personalisation, selectively applying nuanced edits to cross-attention layers for a seamless merge of identity and style. Additionally, we propose Random Mask Reconstruction to enhance robustness, directing the model to focus on distinctive identity and style features. Crucially, our aim is not to replace artists but to eliminate accessibility barriers, allowing enthusiasts to engage in the artistry.

Approach

Our personalised text-to-image (T2I) framework involves fine-tuning the SD model to capture identity in the reference photo and generate the same identity in various contexts. Consequently, we leverage an off-the-shelf T2I-Sketch-Adapter to spatially condition the identity-adapted SD model. This process effectively integrates shape guidance from sketch, aligning results with the intended shape.

Gallery

Caricature of celebrities.



Caricature of renowned artworks.



Caricature of synthetic portraits.

BibTex

@article{chen2023democaricature,
  title={DemoCaricature: Democratising Caricature Generation with a Rough Sketch},
  author={Dar-Yen Chen and Subhadeep Koley and Aneeshan Sain and Pinaki Nath Chowdhury and Tao Xiang and Ayan Kumar Bhunia and Yi-Zhe Song},
  booktitle={arXiv preprint arxiv:2312.04364},
  year={2023}
}