Image generation from freehand sketches using Diffusion Models

Abstract

Significant advancements have been made in the field of image-to-image translation and image synthesis in recent years. Generation of images from sketches is a popular topic in this field. It has many use cases in day-to-day life especially for artists. One useful kind of generative model that has recently come into use for this purpose are diffusion models. In this thesis, we investigate this topic further by developing an efficient approach to generate sufficiently similar images from simple sketch inputs using diffusion models. We utilize a custom Kolmogorov Arnold Network (KAN) based model to provide guidance to a pre-trained diffusion model, so that it generates an image following the input sketch. We also compare our approach with other existing methods and also evaluate their performance. Additionally, we experiment our model with various types of sketch styles containing varying levels of details to demonstrate its robustness. The results show that our method is able to produce images from freehand sketches efficiently.