In this paper, we present our solutions to the FinSim4 Shared Task which is co-located with the FinNLP workshop at IJCAI-2022. This new edition of FinSim4-ESG is extended to the “Environment, Social and Governance (ESG)” related issues in the financial domain. There are two sub-tasks in the FinSim4 shared task. The goal of sub-task1 is to develop a model to predict correctly a list of given terms from ESG taxonomy domain into the most relevant concepts. The aim of subtask2 is to design a system that can automatically classify the ESG Taxonomy text sentence into sustainable or unsustainable class. We have developed different classifiers to automatically classify the terms and sentences with augmented term and pre-trained language models: tf-idf vector, word2vec, Bert, Distill-Bert, Albert, Roberta. The result dashboard shows that our proposed methods yield a significant performance improvement compared to the baseline which ranked 1st in the subtask2 and 2rd of mean rank in the subtask1.