DE-NOISING OF SMS TEXT USING TERNARY TREE FOR FAQ RETRIEVAL

Abstract

Everyone in today’s world uses mobile and technology and can get any information sitting at home within no time. A lot of information is available over the web, which is easily accessible to us. But the low-cost mobile devices don’t have access to the internet. Every device ranging from inexpensive feature phone to the smartphones have the facility of Short messaging Service (SMS). So, instead of using the internet facility, most of the people prefer to use the facility of SMS to access the information. Due to the limited number of characters in SMS, we need to remove a few characters, and thus, these texts become noisy. There are many other intentional and non-intentional errors that add noise to the SMS. Intentional errors include few intentional dropping of characters, phonetic substitutions, abbreviations, etc, and non-intentional errors include typing error and errors due to the small screen, damaged display, multi-tap keyboard, QWERTY keyboard, etc. Efficient de-noising of such texts is necessary to remove noise for the correct information gain. A model is developed for de-noising the SMS text by calculating prefix, suffix, and similarity score to find the best matching word corresponding to the noisy term. Prefix and suffix scores are calculated using Ternary Tree, and the similarity score uses the Longest Common Subsequence to get the correct English word for the noisy SMS word. The overall score is calculated using the above three scores. The ternary tree helps in reducing the memory space as compared to the previous models using trie for de-noising the text. Few noisy words are generated to test the model, and the results are compared with some previous models. The proposed novel method outperforms all the previous approaches.