# SEConvo

SEConvo consists of 1,400 chat-based social engineering conversations that mimic realistic LinkedIn reach-outs in scenarios such as Academic Collaboration, Academic Funding, Journalism, and Recruitment. All conversations are generated by GPT-4-Turbo and feature both single-LLM simulations and dual-agent interactions. 400 conversations are human-annotated.
<br></br>

## Dataset Statistics
Number of conversations broken down by scenario type and mode:
| Scenario                | Single LLM | Dual Agent | All  |
|-------------------------|------------|------------|------|
| Academic Collaboration  | 220        | 140        | 360  |
| Academic Funding        | 140        | 140        | 280  |
| Journalism              | 240        | 140        | 380  |
| Recruitment             | 240        | 140        | 380  |
| **All**                 | **840**    | **560**    | **1400** |

<br></br>

Number of conversations broken down by annotated and unannotated data:

| Batch →              | Annotated      |             | Unannotated    |             |
|----------------------|----------------|-------------|----------------|-------------|
| SE Attempt →         | w/ SE          | w/o SE      | w/ SE          | w/o SE      |
| Mode ↓               |                |             |                |             |
| Single-LLM           | 135            | 105         | 300            | 300         |
| Dual-Agent           | 80             | 80          | 200            | 200         |
| **All**              | **215**        | **185**     | **500**        | **500**     |


<br></br>
## Labels (Both Annotated and Unannotated Data)
Each JSON file contains the following fields:

* Conversations: List of all the conversations in the file.
    * Conversation: List of messages that contain the text of the conversation.
        * Name: The name of a sender of a given message. These were randomly generated. [text]
        * Message: The raw content of the message sent. [text]
    * Ground Truth: Contains the human-labeled ground truth values for the conversation.
        * ConversationID: Unique ID of the conversation. [integer]
        * IsMalicious: Denotes if the human annotator marked the conversation malicious. [true/false]
        * Multi: Denotes if the conversation was generated by a single agent (false) or two agents 
            passing messages to each other (true). [true/false]
        * Scenario: The scenario of the conversation. One of Academic Collaboration, Academic Funding, Journalism, and Recruitment. [text]


<br></br>
## Labels for Annotated Data Only

* Ambiguity: Approximation of how difficult the human annotator found to distinguish the conversation as Malicious or Benign, with 1 being not ambiguous at all, 2 being slightly ambiguous, and 3 being highly ambiguous. [1-3]
* FirstPII: Message number where the first request for PII is contained [integer starting at 0]
* NumPIIMessages: The number of messages that contain a request for PII [integer]
* PII: A list of each type of PII and a sub-list of which messages a solicitation for that PII was
    requested.
    * Name: Type of PII as given by a human annotator [text]
    * Messages: A list of message turn numbers where the request for this PII occured [list of Integers]