Automatic detection of fraudulent voice calls is essential for online service platforms but faces significant challenges due to the scarcity of labeled data and the continuous evolution of conversational contexts. Standard supervised methods often fail to generalize, as they tend to overfit to variable background narratives rather than capturing the core deceptive intent. In this paper, we propose a lightweight framework that anchors detection on Semantic Primitives, a set of stable, interpretable evidentiary cues derived from expert knowledge. Our approach decomposes the fraud detection task into two distinct stages: identifying the presence of these predefined semantic signals within the transcript, and deriving a final verdict through a logical combination of the detected cues. By explicitly prioritizing stable evidence over diverse conversational noise, this framework ensures that decisions are based on verifiable fraud tactics rather than spurious correlations. Experimental results demonstrate that our method achieves superior robustness and efficiency compared to traditional baselines, particularly in scenarios with shifting service contexts.