Juliano Augusto Carreira , Carlos Roberto Valêncio , Rogéria C. Gratão de Souza
August 2011
Fourth BCS-IRSG Symposium on Future Directions in Information Access (FDIA 2011) (FDIA)
Future Directions in Information Access (FDIA 2011)
31 August 2011
Data Cleansing, Information Retrieval, Duplicate Tuples, Knowledge Discovery in Databases
One of the significant problems and inherent to current large databases is the incidence of duplicate tuples. This problem refers to the repetition of records that, in most cases, are represented differently in databases but refer to the same real world entity, which makes the task of identifying those tuples a hard work. Considering that each language has its peculiarities, it is believed that the use of text operations techniques from the area of Information Retrieval can enrich the content of the records for a specific language and thus maximize the amount of identified duplicate tuples and/or improve the confidence level of their classification in relation to current tools. The main contribution of this paper is to provide a language independent environment able to approximate the spelling of the records in a database and thus identify duplicate tuples more efficiently than the isolated application of traditional methods. In addition to only improve database quality this tool can also improve the process of Knowledge Discovery in Databases (KDD).
This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/