Logical Reasoning is a novel approach to deal with challenging Machine Reading Comprehension tasks by utilizing the ability to construct logical structures in natural language. However, previous promising studies struggle with the accuracy of logical unit division and the consistency of model prediction on equivalent semantics. In this paper, we propose ThinkStruct, a new method that leverages a transformer network enhanced with the information of Rhetorical Structure (RS) relations for logical reasoning. Specifically, our method uses Rhetorical Structure Theory (RST) to split natural language text into Elementary Discourse Units (EDUs) and identify the relationship among these units. Node information is then fed into the fully connected transformer network, which is enhanced with logical relationships among the extracted units via adjacency matrix. Subsequently, the features of the transformer network are integrated before being passed into the answer prediction module. In addition, we employ a contrastive learning module for improving its understanding of the relationship between Elementary Discourse Units. Our experiments on the LogiQA and Reclor datasets demonstrate that our results outperform other state-of-the-art models.