In this file, we describe the files we release.

- generate_complex_sentences_given_simple: This file contains the code to generate the data and train the model we use in our "Informativeness" reward model.  There is a main function, which can be used to parse the Reddit data, generate the more informative focus areas, train the regression model on it, and save the model/data.
- llm_prompts: This features helper functions to prompt LLM_task. Make sure to set or pass in the OpenAI URL and keys, or the local url if running LLama, appropriately. 
- llm_reward_fns: This file provides code for all our reward functions. It is used by rl4lms_custom_functions. 
- reddit_data.py: This file provides an example for how we created the reddit dataset that we release in the code. We use the Convokit library. In particular, we provide a helper function that shows how we prompt a LLM_Task model to create the "gold" focus areas that we used in the supervised training stage of our model.
- rl4lms_custom_functions.py -> This function has our custom training and reward functions to train our models. It works in combination with the RL4LMs library, and for more details on how custom reward functions are implemented in that library, we refer to their public repo. This file just has our implementation.
- train_text_generation.py -> This file is the start point for all of our training scripts. It runs supervised and RL training, using the RL4LMs library.
- RL4LMs -> This directory is a fork of the RL4LMs library. We have made some changes, such as data handling, custom evaluation, custom training procedures, etc. A diff from the RL4LMs library can provide an idea of the changes that we made.
- sh_files provides different SLURM scripts to run the code. They can also be run fairly easily as .sh scripts:
    - t5_base_train_model.sh: Shows how to train the supervised model with the gold data. Make sure to set the data_path appropriately for all of these scripts.
    - t5_base_rl: Shows how you can train the RL model when LLM_Task is a local model. Make sure to set the data_path and openai_url appropriately.
    - chatgpt_t5_base_rl: Shows how you can run the RL with ChatGPT. Make sure to set the openai parameters in llm_prompts correctly. Also, make sure to set data_path appropriately.

To run the code, use sh_files. For example, you can just do `./sh_files/t5_base_train_model.sh` inorder to run the supervised Training model after you set the appropraite paths in that file and in the YAML files.