Automatic Phrase Recognition and Extraction from Text

One of the problems facing researchers in the field of Information Retrieval (IR) is that the search criteria used during retrieval (the query) contains terms which are very ambiguous and common. By this we mean that terms can have multiple meanings and occur in a large percentage of the documents in a text collection. Many approaches to addressing this problem have been tried with varying degrees of success. One approach to this problem is to attempt to make the vocabulary used by the IR system less ambiguous by using terms which occur only infrequently. In our case this is achieved through an automatic process of phrase recognition and the incorporation of these phrases into the lexicon of the indexing mechanism used. Unlike previous phrase recognition approaches based on NLP, our work requires no linguistic processing of the text in order to extract phrases but is comparable to what is called ‘statistical phrases’. In this paper we describe experiments where we evaluate our phrase recognition on the TREC-4 and TREC-5 collections.