Abstract
Performance of standard text analytics algorithms are known to be substantially degraded on consumer generated data, which are often very noisy. These algorithms also do not work well on enterprise data which has a very different nature from News repositories, storybooks or Wikipedia data. Text cleaning is a mandatory step which aims at noise removal and correction to improve performance. However, enterprise data need special cleaning methods since it contains many domain terms which appear to be noise against a standard dictionary, but in reality are not so. In this work we present detailed analysis of characteristics of enterprise data and suggest unsupervised methods for cleaning these repositories after domain terms have been automatically segregated from true noise terms. Noise terms are thereafter corrected in a contextual fashion. The effectiveness of the method is established through careful manual evaluation of error corrections over several standard data sets, including those available for hate speech detection, where there is deliberate distortion to avoid detection. We also share results to show enhancement in classification accuracy after noise correction.- Anthology ID:
- 2020.coling-industry.18
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics: Industry Track
- Month:
- December
- Year:
- 2020
- Address:
- Online
- Editors:
- Ann Clifton, Courtney Napoles
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 190–201
- Language:
- URL:
- https://aclanthology.org/2020.coling-industry.18
- DOI:
- 10.18653/v1/2020.coling-industry.18
- Cite (ACL):
- Gargi Roy, Lipika Dey, Mohammad Shakir, and Tirthankar Dasgupta. 2020. Learning Domain Terms - Empirical Methods to Enhance Enterprise Text Analytics Performance. In Proceedings of the 28th International Conference on Computational Linguistics: Industry Track, pages 190–201, Online. International Committee on Computational Linguistics.
- Cite (Informal):
- Learning Domain Terms - Empirical Methods to Enhance Enterprise Text Analytics Performance (Roy et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/landing_page/2020.coling-industry.18.pdf
- Data
- Hate Speech