Experiences with Content Based Retrieval of Multimedia Information

In the last four years, the Institute of Systems Science (ISS) has developed various content based retrieval engines which work on text, images, and sound. In working with these multimedia, we find they naturally divide into two kinds: encoded and unencoded. We give characteristics of these two kinds of data, and show how they differ with respect to the key issues in multimedia retrieval: feature identification, segmentation, normalization, classification, indexing, similarity measure, filtering, and retrieval. We provide concrete examples of these differences from various content based retrieval applications developed at ISS, specifically, a multilingual freetext search system, a photograph archival system, a facial image recognition system, a trademark archival and retrieval system, and a MIDI audio file retrieval system.