Computer vision based reuse detection in digital artworks

Abstract

Image reuse refers to the use of visual elements of existing images in order to create new ones. In this thesis, we study the automatic image reuse detection problem in digital artworks, which is a relatively under-studied problem of image retrieval. We introduce two novel image reuse datasets: an arti cial dataset that simulates di erent types of reuse systematically, and an annotated natural dataset that includes a set of digital artworks that are crawled from the web. Based on the natural dataset, we propose a taxonomy which identi es the primary types of reuse and manipulations. Then, for image reuse detection, we evaluate di erent feature extraction and classi cation methods that are commonly used for image copy detection, content-based image retrieval, and computer analysis of artworks. The features we use include, color histograms, Histogram of Oriented Gradient (HOG) descriptors, and the Scale Invariant Feature Transform (SIFT) descriptor and its color-based variants. We use the bagof- visual-words (BoW) approach with the SIFT descriptors. We also present a novel image description algorithm, called the A ne Invariant Salient Patch (AISP) descriptor, which provides a foreground sensitive description of images by tting concentric ellipses to the most salient region in an image and extracting features from each track. Our results show that the AISP method can be suitable for reuse detection with its compactness and good retrieval accuracy, especially in images with prominent foreground objects. On the other hand, the use of the SIFT descriptors in a BoW model can be more advisable in a more natural setting and for cluttered scenes.