# Development of Cognitive Intelligence in Pre-trained Language Models

## Abstract
<p align="justify">
Recent studies show evidence for \emph{emergent cognitive abilities} in Large Pre-trained Language Models (PLMs). The increasing \emph{cognitive alignment} of these models has made them candidates for cognitive science theories. Prior research into the emergent cognitive abilities of PLMs has been \emph{path independent} to model training, i.e. has only looked at the final model weights and not the intermediate steps. However, building plausible models of human cognition using PLMs also requires aligning their performance during training to the developmental trajectories of children's thinking. Guided by psychometric tests of human intelligence, we choose four task categories to investigate the alignment of ten popular families of PLMs and evaluate each of their available \emph{intermediate and final training steps}: Numerical ability, Linguistic abilities, Conceptual understanding, and Fluid reasoning. We find a striking regularity: regardless of model size, the developmental trajectories of PLMs consistently exhibit a window of maximal alignment to human cognitive development. Before that window, training appears to endow ``blank slate'' models with the requisite structure to be poised to rapidly learn from experience. After that window, training appears to serve the engineering goal of reducing loss but not the scientific goal of increasing alignment with human cognition.
</p>


## A suite of psychometric intelligence tasks
We introduce a set of assessments that benchmark PLMs across four abilities of psychometric intelligence.


| Cognitive Domain         | Task                          |        Source        |    License  |
|--------------------------|-------------------------------|----------------------|-------------|
| Numeric Abilities        | Magnitude Comparison Effects  | Shah et al. 2023     | (cc by 4.0) |
| Lingustic Abilities      | BLiMP                         | Warstadt et al. 2020 | (cc by 4.0) |
| Conceptual Understanding | Typicality Effects            | Vemuri et al. 2024   | (cc by 4.0) |
| Fluid reasoning          | Raven's Progressive Matrices  | Hu et al. 2023       | (cc by 4.0) |

## Steps to use the code

1. Clone the Repo
2. cd into the repo in your terminal

## Dependencies
Install dependencies with the following command
pip3 install -r requirements.txt

If you wish to use Anaconda:
- To install conda visit [this](https://www.anaconda.com/products/individual) link. 
- `conda create --name devplm python=3.9`
- `conda activate devplm`
- `pip3 install -r requirements.txt`

### One time Setup
In experiments.py, set the home-directory, the secret token for access to Llamma-2 models from Huggingface (this will require you to log into huggingface).

### Models
We evaluate a large number of models, therefore we suggest setting a cache-directory for Huggingface models. The final storage requirements for all the models under consideration is around 13 TB.

### Running experiments
We strongly recommend having access to having A 100 40GB or 80GB ram GPUs for running experiments.

`python3 experiments.py` 

The file also works with any job management system like slurm.