Privacy is of primary importance when it comes to the Financial Domain as the data is highly confidential and no third party can be having access to it. Natural Language Processing (NLP) techniques can be applied for text classification and entity detection purposes in financial domains like customer feedback sentiment analysis, invoice entity detection, categorisation of financial documents by type etc. Due to the sensitive nature of such data, privacy measures need to be taken for handling and training large models with such data. In this work, we propose a contextualized transformer (BERT and RoBERTa) based text classification model integrated with privacy features like Differential Privacy (DP) and Federated Learning (FL). We present how to privately train NLP models and desirable privacy utility trade-offs and evaluate it on the Financial Phrase Bank dataset.