{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "bac49923-ab1a-4c68-a9b1-4d47a5b55a21"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "5d6cee11-7261-4a08-d2f9-f0f1b44d376c"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = True"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "chapter16_1 = \"\"\"\n",
        "Agent has available actions $a$, assigns a probability $P(s)$ to state s, transition model given by $P(s'\\mid s,a)$, prob of reaching state $s'$ by taking action $a$ in state $s$. \\\n",
        "The probability of landing in a state $s'$ is: \\\n",
        "$$P(RESULT(a)=s')=\\sum_sP(s)P(s'\\mid s,a)$$\n",
        "\n",
        "**Decision theory** deals with choosing among actions based on the desirability of their *immediate* outcomes (episodic environment). \\\n",
        "Agent's preferences captured by a **utility function** $U(s)$ which assigns a single number to express the desirability of a state. The **expected utility** of an action given evidence is: \\\n",
        "$$EU(a) = \\sum_{s'}P(RESULT(a) = s')U(s')$$\n",
        "\n",
        "The **principle of maximum utility** says that a rational agent should choose the action that maximizes the agent's expected utility (prescription for intelligent behavior); formalizes \\\n",
        "the goal, but does not operationalize it. Estimating $P(RESULT(a)=s')$ and utility function can be hard.\n",
        "\n",
        "If the agent acts to maximize utility function that correctly reflects the performance measure, then they will achieve the highest possible average performance score \\\n",
        "(utility function applies only to next state, performance measure gives a score for a history; sequence of states).\n",
        "\"\"\"\n",
        "chapter16_2 =  \"\"\"\n",
        "Notation: $A\\succ B$ (agent prefers A over B), $A\\sim B$ (agent is indifferent between A and B), $A\\succcurlyeq B$ (agent prefers A over B or is indifferent). A and B are lotteries. \\\n",
        "A lottery $L$ with possible outcomes $S_1,...,S_n$ that occur with probabilities $p_1,...,p_n$ is written $L = [p_1,S_1;...;p_n,S_n]$. Each outcome $S_i$ can be an atomic state or another lottery. \n",
        "\n",
        "Six axioms of utility theory:\n",
        "1. **Orderability**: given any two lotteries, an agent must prefer one or rate them as equal (exactly one of $A\\succ B$, $B\\succ A$, or $A\\sim B$ holds).\n",
        "2. **Transitivity**: $(A\\succ B)\\land(B\\succ C)\\implies(A\\succ C)$\n",
        "3. **Continuity**: $A\\succ B\\succ C\\implies\\exists p : [p,A;(1-p),C]~B$\n",
        "4. **Substitutability**: $A\\sim B\\implies[p,A;(1-p),C]\\sim [p,B;(1-p),C]$\n",
        "5. **Monotonicity**: $A\\succ B\\implies (p>q \\iff [p,A;(1-p),B]\\succ[q,A;(1-q),B])$\n",
        "6. **Decomposibility** (no fun in gambling rule): $[p,A;(1-p),[q,B;(1-q),C]]\\sim[p,A;(1-p)q,B;(1-p)(1-q),C]$\n",
        "\n",
        "Each axiom can be motivated by showing that an agent that violates it will exhibit obviously irrational behavior in some situations.\n",
        "\n",
        "Can derive the following from the axioms of utility:\n",
        "1. **Existence of a utility function**: if an agent's preferences obey axioms of utilty, $\\exists U: U(A) > U(B) \\iff A\\succ B, U(A)=U(B) \\iff A\\sim B$\n",
        "2. **Expected utility of a lottery**: $U([p_1,S_1;...;p_n,S_n]) = \\sum_ip_iU(S_i)$\n",
        "\n",
        "The outcome of nondeterministic action is a lottery, so an agent can act rationally (consistently with its preferences) only by following MEU principle.\n",
        "\n",
        "Agent's behavior would not change if utility function were transformed according to $U'(S) := aU(S) + b$, where $a$ and $b$ are constants and $a>0$ (positive affine transformation). \\\n",
        "In a deterministic environment an agent needs only a preference ranking on states (the numbers don't matter). This is called a **value function** or **ordinal utility function**.\n",
        "\"\"\"\n",
        "chapter16_3 =  \"\"\"\n",
        "Determining an agent's utility function is called **preference elicitation** (present choices and use observed preferences to determine underlying utility function).\n",
        "\n",
        "Typically we fix worst outcome at $u_{bot}$ and best outcome at $u_{top}$ (**normalized utilities** use $u_{bot}=0, u_{top}=1$). Can assess utility of prize $S$ by asking agent to \\\n",
        "choose between $S$ and a **standard lottery** $[p,u_{top}; (1-p),u_{bot}]$ (binary search with $p$ to find utility of $S$).\n",
        "\n",
        "Govt agencies use **value of a statistical life**. A **micromort** is value of a 1-in-million chance of death. Another measure is quality-adjusted life-year (**QALY**).\n",
        "\n",
        "Money may or may not work as an accurate utility function (could have **monotonic preference** but violate characteristics of utility function in lotteries). \\\n",
        "Can compute **expected monetary value** (EMV) of a lottery. Utility function is often roughly logarithm of money (diminishing marginal utility). \\\n",
        "Diminishing marginal utility equivalent to **risk aversion** (utility of lottery less than EMV), increasing marginal utility equivalent to **risk seeking**, linear equivalent to **risk neutral**. \\\n",
        "The value an agent will accept in lieu of a lottery is called the **certainty equivalent** of the lottery. Difference between EMV and certainty equivalent is called the **insurance premium** \\\n",
        "(risk aversion basis for insurance industry). \\\n",
        "\n",
        "Agent may have to work with estimate $\\hat{EU}(a)$ of true EU function due to practical limitations. We assume that the estimates are **unbiased** ($E(\\hat{EU}(a)-EU(a))=0$). Still, \\\n",
        "expected utility of best choice tends to be an overestimate (**optimizer's curse**), assuming gaussian EU errors. Can be avoided with a Bayesian approach that uses a model $P(\\hat{EU}\\mid EU)$ of the error \\\n",
        "in the utility estimates. Can treat utility estimate as evidence and compute the posterior using Bayes' rule.\n",
        "\n",
        "Decision theory is a **normative theory** (describes how a rational agent should act). A **descriptive theory** describes how actual agents (e.g. humans) act. Humans are predictibly irrational \\\n",
        "(e.g. **certainty effect**: people attracted to certain gains, **ambiguity aversion**, **framing effect**: wording of problem can influence decision, **anchoring effect**: people prefer making \\\n",
        "relative utility decisions over absolute ones).\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\"\n",
        "Sequential decision problem for fully observable, stochastic environment with Markovian transition model (probability of reaching $s'$ from $s$ depends only on $s$) and additive rewards is called a \\\n",
        "**Markov decision process**, or **MDP**. Consists of set of states (initial $s_0$), set $ACTIONS(s)$ of actions in each state, transition model $P(s'\\mid s,a)$, and a reward function \\\n",
        "$R(s,a,s') \\in [-R_{max}, R_{max}]$ (aka $T(s, a, s')$). Solutions usually involve **dynamic programming**.\n",
        "\n",
        "Solution is a **policy** $\\pi$: $\\pi(s)$ is recommended action at state $s$. Evaluated by *expected* utility of environment history. Policy-following agent is simple reflex agent, computed \\\n",
        "from info used for utility-based agent.\n",
        "\n",
        "Is there a **finite horizon** (*fixed* time $N$ after which nothing matters; $U_h(s_0,a_0,...,s_{N+k})=U_h(s_0,a_0,...,s_N)$ for any $k>0$)) or **infinite horizon** for decision making? \\\n",
        "With finite horizon, optimal policy may depend on time left (**nonstationary**, as opposed to **stationary**).\n",
        "\n",
        "How to calculate utility of state sequences? We use **additive discounted rewards**: $U_h(s_0,a_0,...)=R(s_0,a_0,s_1)+\\gamma R(s_1,a_1,s_2)+\\gamma^2...$, where the \\\n",
        "**discount factor** $\\gamma \\in [0, 1]$ . Special case $\\gamma=1$ purely **additive rewards**. Discounted rewards might reflect empirical human behavior, \\\n",
        "time value of money, uncertainty, **stationarity** of preferences between state sequences (if two histories start the same, they should be preference ordered \\\n",
        "according to the rest of the histories, as if they were to start today), and conveniently promotes convergence (if $\\gamma<1$), utilities bounded in $[-R_{max}/(1-\\gamma), R_{max}/(1-\\gamma)]$ geo series. \\\n",
        "If environment contains terminal states and agent is guaranteed to get to one (**proper policy**), we will never need to compare infinite sequences; can use $\\gamma=1$ here. \\\n",
        "Infinite sequences can alternatively be compared by **average reward** per timestep.\n",
        "\n",
        "Expected utility obtained by executing $\\pi$ starting at $s$, reaching state $S_t$ at time $t$:\n",
        "$$U^\\pi(s)=E(\\sum_{t=0}^\\infty\\gamma^tR(S_t,\\pi(S_t),S_{t+1}))$$\n",
        "Optimal policy $\\pi^*_s=argmax_\\pi U^\\pi(s)$. If using discounted utilities with infinite horizons, optimal policy is independent of starting state (can write $\\pi^*$). \\\n",
        "True utility of a state is $U^{\\pi^*}(s)=:U(s)$. The **action-utility function** or **Q-function**: $Q(s,a)$ is expected utility of taking action $a$ in state $s$.\n",
        "\n",
        "**Bellman equations**:\n",
        "$$Q(s,a):=\\sum_{a'}P(s'\\mid s,a)[R(s,a,s')+\\gamma \\max_{a'}Q(s',a')]$$\n",
        "$$U(s)=\\max_a Q(s,a)$$\n",
        "$$\\pi^*(s)=argmax_a Q(s,a)$$\n",
        "\n",
        "Let $\\Phi(s)$ be *any* function of state $s$. Then, according to **shaping theorem**, the following transformation does not change optimal policy:\n",
        "$$R'(s,a,s')=R(s,a,s')+\\gamma \\Phi(s')-\\Phi(s)$$\n",
        "Can set $\\Phi(s)=U(s)$ so greedy policy $\\pi_G$ w.r.t. $R'$ is the optimal policy ($\\pi_G(s)=argmax_a\\sum_{s'}P(s'\\mid s,a)R'(s,a,s')$). \\\n",
        "Like providing treat to animal at each step in target sequence.\n",
        "\"\"\"\n",
        "chapter17_2 =  \"\"\"\n",
        "**Value iteration**: $n$ equations, $n$ unknowns (non-linear because of \"max\" operator). Can calculate utilities with iterative approach until convergence. \\\n",
        "$U_i(s)$ utility at $i$th iteration. Apply update simultaneously to all states at each iteration:\n",
        "$$U_{i+1}(s)-\\max_{a\\in A(s)}Q_i(s,a)$$\n",
        "will converge to optimal policy, unique solution for $U$. Can stop once $\\delta:=$ max change in $U(s)$, $\\delta\\leq\\epsilon(1-\\gamma)/\\gamma$ ($\\epsilon:=$ maximum error allowed in utility of any state). \\\n",
        "Value iteration propagates information through state space via local updates.\n",
        "\n",
        "Let Bellman update be operator $B$ applied simultaneously to utility of every state: $U_{i+1}:=BU_i$. **Utility vector** has components $U(s)$ for each $s$, length $||U||=\\max_s |U(s)|$ (**max norm**). \\\n",
        "We have $||BU_i - BU_i'||\\leq\\gamma||U_i-U_i'||$ (Bellman update is a contraction by a factor of $\\gamma$ on the \\\n",
        "space of utility vectors). Replace $U_i'$ with true utility $U$: error reduced by factor of at least $\\gamma$ at each iteration.\n",
        "$$N=ceil(\\log(2R_{max}/((1-\\gamma)\\epsilon))/\\log(1/\\gamma))$$\n",
        "If $||U_{i+1}-U_i||<\\epsilon(1-\\gamma)/\\gamma$ then $||U_{i+1}-U||<\\epsilon$ (hence termination condition in value iteration algo).\n",
        "\n",
        "**Policy loss** $||U^{\\pi_i}-U||$ is the most the agent can lose by executing $\\pi_i$ instead of the optimal policy $\\pi^*$;\n",
        "$$|U_i-U||<\\epsilon \\implies ||U^{\\pi_i}-U||<2\\epsilon$$\n",
        "Often $\\pi_i$ becomes optimal before $U_i$ has converged.\n",
        "\n",
        "**Policy iteration** algorithm, starting from initial policy $\\pi_0$:\n",
        "- **Policy evaluation**: given a policy $\\pi_i$, calculate $U_i=U^{\\pi_i}$ (vector), using $U_i(s)=Q_i(s,\\pi_i(s))$\n",
        "- **Policy improvement**: calculate a new MEU policy $\\pi_{i+1}$ from $U_i$\n",
        "Terminate when policy improvement step yields no change to policy ($\\pi_i$ optimal).\n",
        "\n",
        "Policy evaluation has $n$ linear equations (no \"max\"), can be solved in $O(n^3)$ time with linal methods. If $n$ large, \\\n",
        "do a small number of simple value iteration steps (no change to policy) to approximate utilities (**modified policy iteration**).\n",
        "\n",
        "Can also pick subsets of states and apply either kind of updating (policy improvement or simplified value iteration) to that subset (**asynchronous policy iteration**). \\\n",
        "Can use heuristic to only update states most likely to be reached by good policy.\n",
        "\n",
        "**Linear programming** (LP) can be used to solve MDPs; minimize $U(s)$ for each $s$, s.t. $U(s)\\geq Q(s,a)$ $\\forall s,a$. DP usually more efficient.\n",
        "\n",
        "Online approx algo for large MDPs: expectimax tree. The $\\epsilon$-horizon is the tree depth $H$ s.t. sum of rewards beyond any leaf is less than $\\epsilon$ (use heuristic to estimate beyond leaf nodes). \\\n",
        "Depth of $H=ceil(\\log_\\gamma(\\epsilon(1-\\gamma)/R_{max}))$ suffices (useful if $\\gamma$ not close to 1). If branching factor large, \\\n",
        "can generate $N$ samples and use the sample mean (will prob focus on the most likely outcomes). General approach: **real-time dynamic programming (RTDP)**. Can also apply Monte Carlo methods (UCT).\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "  \"principle of maximum expected utility\" : \"a rational agent should act to maximize expected utility; $action=\\argmax_a EU(a)$\",\n",
        "  \"axioms of utility theory\" : \"observability, transitivity, continuity, substitutability, monotonicity, decomposability\",\n",
        "  \"preference elicitation\" : \"present choices and use observed preferences to determine underlying utility function\",\n",
        "  \"micromort\" : \"value of a 1-in-million chance of death\",\n",
        "  \"certainty equivalent\": \"the value an agent will accept in lieu of a lottery\",\n",
        "  \"insurance premium\" : \"difference between expected monetary value and certainty equivalent\",   \n",
        "  \"unbiased expected utility estimate\" : \"$E(\\hat{EU}(a)-EU(a))=0$\",\n",
        "  \"optimizer's curse\" : \"expected utility of best choice tends to be an overestimate\",\n",
        "  \"normative theory\" : \"describes how a rational agent should act\",\n",
        "  \"descriptive theory\" : \"describes how actual agents (e.g. humans) act\",\n",
        "  \"objectivist/frequentist interpretation\" : \"averages over repeated experiments\",\n",
        "  \"subjectivist / Bayesian interpretation\" : \"degrees of belief about unobserved variables\",\n",
        "  \"\": \"\",\n",
        "  \"an attribute of a history\": \"a transition $s_t to_{a_t}s_{t+1}$\",\n",
        "  \"\": \"\",\n",
        "  \"contraction\": \"one-argument function, brings any 2-points closer (e.g. divby2); has only one fixed point, moves any point towards FP\"\n",
        "}\n",
        "\n"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2760
        },
        "outputId": "d8b70d21-4095-46e7-f515-cc0beb9e7cbd"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nAgent has available actions $a$, assigns a probability $P(s)$ to state s, transition model given by $P(s'\\mid s,a)$, prob of reaching state $s'$ by taking action $a$ in state $s$. The probability of landing in a state $s'$ is: $$P(RESULT(a)=s')=\\sum_sP(s)P(s'\\mid s,a)$$\n\n**Decision theory** deals with choosing among actions based on the desirability of their *immediate* outcomes (episodic environment). Agent's preferences captured by a **utility function** $U(s)$ which assigns a single number to express the desirability of a state. The **expected utility** of an action given evidence is: $$EU(a) = \\sum_{s'}P(RESULT(a) = s')U(s')$$\n\nThe **principle of maximum utility** says that a rational agent should choose the action that maximizes the agent's expected utility (prescription for intelligent behavior); formalizes the goal, but does not operationalize it. Estimating $P(RESULT(a)=s')$ and utility function can be hard.\n\nIf the agent acts to maximize utility function that correctly reflects the performance measure, then they will achieve the highest possible average performance score (utility function applies only to next state, performance measure gives a score for a history; sequence of states).\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nNotation: $A\\succ B$ (agent prefers A over B), $A\\sim B$ (agent is indifferent between A and B), $A\\succcurlyeq B$ (agent prefers A over B or is indifferent). A and B are lotteries. A lottery $L$ with possible outcomes $S_1,...,S_n$ that occur with probabilities $p_1,...,p_n$ is written $L = [p_1,S_1;...;p_n,S_n]$. Each outcome $S_i$ can be an atomic state or another lottery. \n\nSix axioms of utility theory:\n1. **Orderability**: given any two lotteries, an agent must prefer one or rate them as equal (exactly one of $A\\succ B$, $B\\succ A$, or $A\\sim B$ holds).\n2. **Transitivity**: $(A\\succ B)\\land(B\\succ C)\\implies(A\\succ C)$\n3. **Continuity**: $A\\succ B\\succ C\\implies\\exists p : [p,A;(1-p),C]~B$\n4. **Substitutability**: $A\\sim B\\implies[p,A;(1-p),C]\\sim [p,B;(1-p),C]$\n5. **Monotonicity**: $A\\succ B\\implies (p>q \\iff [p,A;(1-p),B]\\succ[q,A;(1-q),B])$\n6. **Decomposibility** (no fun in gambling rule): $[p,A;(1-p),[q,B;(1-q),C]]\\sim[p,A;(1-p)q,B;(1-p)(1-q),C]$\n\nEach axiom can be motivated by showing that an agent that violates it will exhibit obviously irrational behavior in some situations.\n\nCan derive the following from the axioms of utility:\n1. **Existence of a utility function**: if an agent's preferences obey axioms of utilty, $\\exists U: U(A) > U(B) \\iff A\\succ B, U(A)=U(B) \\iff A\\sim B$\n2. **Expected utility of a lottery**: $U([p_1,S_1;...;p_n,S_n]) = \\sum_ip_iU(S_i)$\n\nThe outcome of nondeterministic action is a lottery, so an agent can act rationally (consistently with its preferences) only by following MEU principle.\n\nAgent's behavior would not change if utility function were transformed according to $U'(S) := aU(S) + b$, where $a$ and $b$ are constants and $a>0$ (positive affine transformation). In a deterministic environment an agent needs only a preference ranking on states (the numbers don't matter). This is called a **value function** or **ordinal utility function**.\n\n\n## AIMA Chapter 16.3 Utility Functions\n\nDetermining an agent's utility function is called **preference elicitation** (present choices and use observed preferences to determine underlying utility function).\n\nTypically we fix worst outcome at $u_{bot}$ and best outcome at $u_{top}$ (**normalized utilities** use $u_{bot}=0, u_{top}=1$). Can assess utility of prize $S$ by asking agent to choose between $S$ and a **standard lottery** $[p,u_{top}; (1-p),u_{bot}]$ (binary search with $p$ to find utility of $S$).\n\nGovt agencies use **value of a statistical life**. A **micromort** is value of a 1-in-million chance of death. Another measure is quality-adjusted life-year (**QALY**).\n\nMoney may or may not work as an accurate utility function (could have **monotonic preference** but violate characteristics of utility function in lotteries). Can compute **expected monetary value** (EMV) of a lottery. Utility function is often roughly logarithm of money (diminishing marginal utility). Diminishing marginal utility equivalent to **risk aversion** (utility of lottery less than EMV), increasing marginal utility equivalent to **risk seeking**, linear equivalent to **risk neutral**. The value an agent will accept in lieu of a lottery is called the **certainty equivalent** of the lottery. Difference between EMV and certainty equivalent is called the **insurance premium** (risk aversion basis for insurance industry). \nAgent may have to work with estimate $\\hat{EU}(a)$ of true EU function due to practical limitations. We assume that the estimates are **unbiased** ($E(\\hat{EU}(a)-EU(a))=0$). Still, expected utility of best choice tends to be an overestimate (**optimizer's curse**), assuming gaussian EU errors. Can be avoided with a Bayesian approach that uses a model $P(\\hat{EU}\\mid EU)$ of the error in the utility estimates. Can treat utility estimate as evidence and compute the posterior using Bayes' rule.\n\nDecision theory is a **normative theory** (describes how a rational agent should act). A **descriptive theory** describes how actual agents (e.g. humans) act. Humans are predictibly irrational (e.g. **certainty effect**: people attracted to certain gains, **ambiguity aversion**, **framing effect**: wording of problem can influence decision, **anchoring effect**: people prefer making relative utility decisions over absolute ones).\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\nSequential decision problem for fully observable, stochastic environment with Markovian transition model (probability of reaching $s'$ from $s$ depends only on $s$) and additive rewards is called a **Markov decision process**, or **MDP**. Consists of set of states (initial $s_0$), set $ACTIONS(s)$ of actions in each state, transition model $P(s'\\mid s,a)$, and a reward function $R(s,a,s') \\in [-R_{max}, R_{max}]$ (aka $T(s, a, s')$). Solutions usually involve **dynamic programming**.\n\nSolution is a **policy** $\\pi$: $\\pi(s)$ is recommended action at state $s$. Evaluated by *expected* utility of environment history. Policy-following agent is simple reflex agent, computed from info used for utility-based agent.\n\nIs there a **finite horizon** (*fixed* time $N$ after which nothing matters; $U_h(s_0,a_0,...,s_{N+k})=U_h(s_0,a_0,...,s_N)$ for any $k>0$)) or **infinite horizon** for decision making? With finite horizon, optimal policy may depend on time left (**nonstationary**, as opposed to **stationary**).\n\nHow to calculate utility of state sequences? We use **additive discounted rewards**: $U_h(s_0,a_0,...)=R(s_0,a_0,s_1)+\\gamma R(s_1,a_1,s_2)+\\gamma^2...$, where the **discount factor** $\\gamma \\in [0, 1]$ . Special case $\\gamma=1$ purely **additive rewards**. Discounted rewards might reflect empirical human behavior, time value of money, uncertainty, **stationarity** of preferences between state sequences (if two histories start the same, they should be preference ordered according to the rest of the histories, as if they were to start today), and conveniently promotes convergence (if $\\gamma<1$), utilities bounded in $[-R_{max}/(1-\\gamma), R_{max}/(1-\\gamma)]$ geo series. If environment contains terminal states and agent is guaranteed to get to one (**proper policy**), we will never need to compare infinite sequences; can use $\\gamma=1$ here. Infinite sequences can alternatively be compared by **average reward** per timestep.\n\nExpected utility obtained by executing $\\pi$ starting at $s$, reaching state $S_t$ at time $t$:\n$$U^\\pi(s)=E(\\sum_{t=0}^\\infty\\gamma^tR(S_t,\\pi(S_t),S_{t+1}))$$\nOptimal policy $\\pi^*_s=argmax_\\pi U^\\pi(s)$. If using discounted utilities with infinite horizons, optimal policy is independent of starting state (can write $\\pi^*$). True utility of a state is $U^{\\pi^*}(s)=:U(s)$. The **action-utility function** or **Q-function**: $Q(s,a)$ is expected utility of taking action $a$ in state $s$.\n\n**Bellman equations**:\n$$Q(s,a):=\\sum_{a'}P(s'\\mid s,a)[R(s,a,s')+\\gamma \\max_{a'}Q(s',a')]$$\n$$U(s)=\\max_a Q(s,a)$$\n$$\\pi^*(s)=argmax_a Q(s,a)$$\n\nLet $\\Phi(s)$ be *any* function of state $s$. Then, according to **shaping theorem**, the following transformation does not change optimal policy:\n$$R'(s,a,s')=R(s,a,s')+\\gamma \\Phi(s')-\\Phi(s)$$\nCan set $\\Phi(s)=U(s)$ so greedy policy $\\pi_G$ w.r.t. $R'$ is the optimal policy ($\\pi_G(s)=argmax_a\\sum_{s'}P(s'\\mid s,a)R'(s,a,s')$). Like providing treat to animal at each step in target sequence.\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\n**Value iteration**: $n$ equations, $n$ unknowns (non-linear because of \"max\" operator). Can calculate utilities with iterative approach until convergence. $U_i(s)$ utility at $i$th iteration. Apply update simultaneously to all states at each iteration:\n$$U_{i+1}(s)-\\max_{a\\in A(s)}Q_i(s,a)$$\nwill converge to optimal policy, unique solution for $U$. Can stop once $\\delta:=$ max change in $U(s)$, $\\delta\\leq\\epsilon(1-\\gamma)/\\gamma$ ($\\epsilon:=$ maximum error allowed in utility of any state). Value iteration propagates information through state space via local updates.\n\nLet Bellman update be operator $B$ applied simultaneously to utility of every state: $U_{i+1}:=BU_i$. **Utility vector** has components $U(s)$ for each $s$, length $||U||=\\max_s |U(s)|$ (**max norm**). We have $||BU_i - BU_i'||\\leq\\gamma||U_i-U_i'||$ (Bellman update is a contraction by a factor of $\\gamma$ on the space of utility vectors). Replace $U_i'$ with true utility $U$: error reduced by factor of at least $\\gamma$ at each iteration.\n$$N=ceil(\\log(2R_{max}/((1-\\gamma)\\epsilon))/\\log(1/\\gamma))$$\nIf $||U_{i+1}-U_i||<\\epsilon(1-\\gamma)/\\gamma$ then $||U_{i+1}-U||<\\epsilon$ (hence termination condition in value iteration algo).\n\n**Policy loss** $||U^{\\pi_i}-U||$ is the most the agent can lose by executing $\\pi_i$ instead of the optimal policy $\\pi^*$;\n$$|U_i-U||<\\epsilon \\implies ||U^{\\pi_i}-U||<2\\epsilon$$\nOften $\\pi_i$ becomes optimal before $U_i$ has converged.\n\n**Policy iteration** algorithm, starting from initial policy $\\pi_0$:\n- **Policy evaluation**: given a policy $\\pi_i$, calculate $U_i=U^{\\pi_i}$ (vector), using $U_i(s)=Q_i(s,\\pi_i(s))$\n- **Policy improvement**: calculate a new MEU policy $\\pi_{i+1}$ from $U_i$\nTerminate when policy improvement step yields no change to policy ($\\pi_i$ optimal).\n\nPolicy evaluation has $n$ linear equations (no \"max\"), can be solved in $O(n^3)$ time with linal methods. If $n$ large, do a small number of simple value iteration steps (no change to policy) to approximate utilities (**modified policy iteration**).\n\nCan also pick subsets of states and apply either kind of updating (policy improvement or simplified value iteration) to that subset (**asynchronous policy iteration**). Can use heuristic to only update states most likely to be reached by good policy.\n\n**Linear programming** (LP) can be used to solve MDPs; minimize $U(s)$ for each $s$, s.t. $U(s)\\geq Q(s,a)$ $\forall s,a$. DP usually more efficient.\n\nOnline approx algo for large MDPs: expectimax tree. The $\\epsilon$-horizon is the tree depth $H$ s.t. sum of rewards beyond any leaf is less than $\\epsilon$ (use heuristic to estimate beyond leaf nodes). Depth of $H=ceil(\\log_\\gamma(\\epsilon(1-\\gamma)/R_{max}))$ suffices (useful if $\\gamma$ not close to 1). If branching factor large, can generate $N$ samples and use the sample mean (will prob focus on the most likely outcomes). General approach: **real-time dynamic programming (RTDP)**. Can also apply Monte Carlo methods (UCT).\n\n\n## Module 8 Lecture Notes\n Your summary here \n\n## Module 8 Definitions\n* **Principle of maximum expected utility** - a rational agent should act to maximize expected utility; $action=\u0007rgmax_a EU(a)$\n* **Axioms of utility theory** - observability, transitivity, continuity, substitutability, monotonicity, decomposability\n* **Preference elicitation** - present choices and use observed preferences to determine underlying utility function\n* **Micromort** - value of a 1-in-million chance of death\n* **Certainty equivalent** - the value an agent will accept in lieu of a lottery\n* **Insurance premium** - difference between expected monetary value and certainty equivalent\n* **Unbiased expected utility estimate** - $E(\\hat{EU}(a)-EU(a))=0$\n* **Optimizer's curse** - expected utility of best choice tends to be an overestimate\n* **Normative theory** - describes how a rational agent should act\n* **Descriptive theory** - describes how actual agents (e.g. humans) act\n* **Objectivist/frequentist interpretation** - averages over repeated experiments\n* **Subjectivist / bayesian interpretation** - degrees of belief about unobserved variables\n* **** - \n* **An attribute of a history** - a transition $s_t to_{a_t}s_{t+1}$\n* **Contraction** - one-argument function, brings any 2-points closer (e.g. divby2); has only one fixed point, moves any point towards FP\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\"\n",
        "An **$n$-armed bandit** has $n$ levers, each yielding fixed/unknown probability distribution of winnings. Must choose levers successively.\n",
        "\n",
        "Each arm $M_i$ is an MRP. Overall Bandit problem is an MDP (Cartesian state space, actions $a_1,...,a_n$).\n",
        "\n",
        "One-armed bandit $M$ yields sequence $R_0,R_1,...$, $M_\\lambda$ yields \\\n",
        "$\\lambda,\\lambda,...$ (cost of arm pull). Number of times you pull $M$ before switching to $M_\\lambda$ is **stopping time** $T$ (first pull of $M_1$ yields no extra info so stay forever). **Gittins index** of $M$ (value of $\\lambda$ for which agent is indifferent btwn playing $M$->$M_\\lambda$ game vs just taking $M_\\lambda$):\n",
        "$$\\lambda = \\max_{T>0} E[\\sum_{t=0}^{T-1}\\gamma^tR_t]/E[\\sum_{t=0}^{T-1}\\gamma^t]$$\n",
        "Optimal policy: pull the arm that has the highest Gittins index, then update the Gittins index of the arm you pulled.\n",
        "Compute Gittins index for arm $M$, construct MDP with additional action at each state: quit and receive infinite sequence of $\\lambda$-rewards; optimal policy value is $\\lambda/(1-\\gamma)$. \\\n",
        "Equivalently, construct **restart MDP** $M^s$; add new action to each state with same rewards/probs/transitions as initial state $s$ (replace infinite path of $\\lambda$ with restarting, since \\\n",
        "agent is indifferent b.d.o. Gittins index). Gittins index for arm $M$ in state $s$ is equal to $(1-\\gamma)$ times \\\n",
        "value of optimal policy for restart MDP $M^s$ (just solve MDP).\n",
        "\n",
        "**Bernoulli bandit**: each arm $M_i$ has count of 1s $s_i$, count of 0s $f_i$ (both initialized to 1), next outcome is 1 with probability $s_i/(s_i+f_i)$, 0 with probability $f_i/(s_i+f_i)$. \\\n",
        "Infinite states (can't solve with Gitten). Accurate appx: truncated MDP with states up to $s_i+f_i=100$ and $\\gamma=0.9$. Arms with higher payoff probability preferred, but exists an \\\n",
        "**exploration bonus** with rarely-tried arms.\n",
        "\n",
        "Appx algos (no Gittins comp):\n",
        "\n",
        "**Upper confidence bound** or UCB heuristic calculates **confidence interval** \\\n",
        "for the value of the arm from samples, choose arm with highest upper bound. Approximate index value for arm $M_i$:\n",
        "$$UCB(M_i) = \\hat{\\mu} + g(N)/\\sqrt{N_i}$$\n",
        "$N:=$ total num of samples drawn from all arms. Definition of $g$ determines **regret** relative to perfect policy. For undiscounted case, \\\n",
        "no algorithm can have regret that grows slower than $O(\\log N)$. USB policy can achieve this bound with $g(N)=\\sqrt{2\\log(1+N\\log^2(N))}$.\n",
        "\n",
        "**Bandit superprocess (BSP)**: each arm is a full MDP, e.g. multitasking, leads to greedier behavior in each MDP. Can solve as MDP with Cartesian state space (exponential, impractical). \\\n",
        "Better approach: consider **opportunity cost** in each MDP. A **dominating policy** ignores opportunity cost (optimally). Adding actions \\\n",
        "to states can create a relaxed MDP with a dominating policy; upper bound on value of arm. Lower bound: solve arm individually. If lower bound of one is above upper bound \\\n",
        "of others, solved. Otherwise, look-ahead search and recomputation of bounds -> optimal solution.\n",
        "\"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\"\n",
        "**Reinforcement learning (RL)**: agent interacts with world and periodically receives **rewards** or **reinforcements**. Same goal as MDPs, but not given details of MDP.\n",
        "\n",
        "Providing reward signal easier than providing labeled training examples (efficient, doesn't require domain knowledge e.g. chess).\n",
        "\n",
        "**Sparse rewards*: just reward for end goal. Better: inject expertise / provide intermediate rewards to help learn good strategies.\n",
        "\n",
        "**Model-based reinforcement learning**: agent uses transition model of environment to help interpret reward signals and make decisions. Model might be unknown/learned \\\n",
        "or provided (e.g. chess rules). Model-based RL systems try to learn utility function $U(s)$ for each state.\n",
        "\n",
        "**Model-free RL**: learns more direct representation of how to behave (no model):\n",
        "- **Action-Utility Learning** (**Q-learning**), agent learns $Q$ function\n",
        "- **Policy Search**: agent learns policy $\\pi(s)$ that maps states to actions (**reflex agent**)\n",
        "\"\"\"\n",
        "chapter22_2 =  \"\"\"\n",
        "**Passive learning agent** executes set of **trials** in environment using policy $\\pi$.\n",
        "\n",
        "**Direct utility estimation**: the utility of a state is the expected reward-to-go; each trial provides *sample* of this quantity for each state visited. \\\n",
        "Calculate observed reward-to-go for each state and update estimated utility (running average). Infinite trials -> converge. Ignores Bellman equations / connections between states. \\\n",
        "Searches for $U$ in a hypothesis space much larger than necessary, converges slowly.\n",
        "\n",
        "**Adaptive dynamic programming (ADP)** uses constraints among utilities of states; learns transition model that connects them, then solves MDP using DP. \\\n",
        "For passive RL agent: plug learned transition models and rewards into linal solver (linear equations). Can also use **modified policy iteration** (see mod8) to update utility estimates \\\n",
        "after each change to learned model. Learning transition model easy (fully observable environment; keep track of transition probability averages). Intractable for large state spaces \\\n",
        "(cannot learn full transition model or solve so many equations).\n",
        "\n",
        "**Temporal-difference learning**: update $U^\\pi(s)$ after transition from $s$ to $'$ via $\\pi(s)$:\n",
        "$$U^\\pi(s)+=(alpha)[R(s,\\pi(s),s')+\\gamma U^\\pi(s')-U^\\pi(s)]$$\n",
        "where alpha is the **learning rate** (**temporal-difference** (TD) equations). The *average value* of $U^\\pi(s)$ will converge to the correct quantity in the limit, \\\n",
        "even if the value itself continues to fluctuate. Also if we decrease alpha as number of visits to $s$ increases, then $U^\\pi(s)$ will converge correctly. \\\n",
        "TD does not need a transition model to perform its updates. The environment provides the connection to neighboring states in the form of observed transitions. \\\n",
        "Crude but efficient first approximation to ADP. Each ADP adjustment is **pseudoexperience** (from TD pov) from simulating current transition model. Can extend TD to use transition model \\\n",
        "to generate pseudoexperiences (more closely approximate ADP, more comp. time).\n",
        "\n",
        "Can make ADP more efficient by appx algos for value/policy iteration. One idea: bound number of adjustments \\\n",
        "made after each observed transition. Alternatively: use heuristic to rank adjustments and only do most significant (**prioritized sweeping** heuristic, adjust states \\\n",
        "whose likely successors have just undergone large adjustment to utility estimates). Appx algos can converge nearly as fast, and do much less comp. Also, appx algo can use min adjust size \\\n",
        "that decreases as transition model becomes more accurate.\n",
        "\"\"\"\n",
        "chapter22_3 =  \"\"\"\n",
        "Actions provide *rewards* and *information*. Must tradeoff **exploitation** for short-term rewards and \\\n",
        "**exploration** to improve policy.\n",
        "\n",
        "Any optimal exploration scheme should be \"greedy in the limit of infinite exploration\" (**GLIE**). Must try each action in each state an unbounded number of times to \\\n",
        "avoid possibility that optimal action is missed. An ADP agent using such a scheme will eventually learn true transition model and can then exploit.\n",
        "\n",
        "Simplest GLIE scheme: choose random action at time step $t$ with prob $1/t$ and follow greedy policy otherwise. Converges slow. Better to give more weight to unexplored actions while \\\n",
        "disliking low-utility actions (like with MCTS). Alter utility equation to give higher estimate to less explored state-action pairs. Let $U^+(s)$ be optimistic utility estimate, \\\n",
        "$N(a,s)$ number of times $a$ tried at state $s$. Update equation:\n",
        "$$U^+(s) = \\max_s f(\\sum_{s'}P(s'\\mid s,a)[R(s,a,s')+\\gamma U^+(s)], N(s,a))$$\n",
        "where $f(u,n)$ is the **exploration function**: how greed (like high $u$) weighed against curiosity (like unexplored actions). Should increase in $u$ and decrease in $n$. \\\n",
        "Simple choice: $f(u,n)=u$ if $n\\geq N_e$ else $R^+$ (an optimistic estimate of best possible reward), where $N_e$ is fixed. Agent will try each state at least $N_e$ times.\n",
        "\n",
        "Many actions are **irreversible**, agent may enter **absorbing state** (no actions have any effect, no rewards, e.g. death). Better idea: choose policy that works reasonably well for whole range \\\n",
        "of possible / probable models. **Bayesian reinforcement learning** assumes prior prob $P(h)$ over hypothesis $h$ of true model; posterior $P(h\\mid e)$ obtained by Bayes' rule given observations. Then\n",
        "$$\\pi^* = argmax_\\pi\\sum_hP(h\\mid e)U_h^*$$\n",
        "Agent must consider effects of future observations. Problem becomes an **exploration poMDP**, belief states are distributions over models. Solving poMDP is usually intractable.\n",
        "\n",
        "Alternative: maintain set of possible models $H$ without considering probabilities, optimize for *worst case* over $H$:\n",
        "$$\\pi^* = argmax_\\pi \\min_h U_h^\\pi$$\n",
        "formulate as adversarial search problem. Problem: overly conservative; real life is full of risk-reward tradeoffs.\n",
        "\n",
        "Human intervention can help algo learn safely (record initial series of actions, write constraints on behavior).\n",
        "\n",
        "Active **Q-learning**: learn action-utility $Q$ values instead of utility function $U$. Agent can then choose $argmax_a f(Q(s,a), N_{sa}[s,a])$. Q-learning TD update:\n",
        "$$Q(s,a)' = Q(s,a) + (alpha)[R(s,a,s') + \\gamma\\max_{a'}Q(s',a')-Q(s,a)]$$\n",
        "does not need transition model for learning or action selection. \n",
        "\n",
        "**SARSA** (state, action, reward, state, action) like Q-learning but SARSA updates with the Q-value of the action actually taken (no max). SARSA penalizes exploration to negative reward state.\n",
        "\n",
        "Q-learning is an **off-policy** learning algo; learns values for policy it isn't using, SARSA is an **on-policy** algo (learns Q-values for its own policy).\n",
        "\"\"\"\n",
        "chapter22_4 =  \"\"\"\n",
        "Constructing compact appx of true utility or Q-function: **function approximation**. Might appx utility using weighted linear combo of features:\n",
        "$$\\hat{U}_{theta}(s) = theta_1f_1(s)+...+theta_nf_n(s)$$\n",
        "just learn parameters theta. Sometimes combine with lookahead search to produce more accurate decisions.\n",
        "\n",
        "Function approximation allows for inductive generalizations about Q-function (can use experience from similar states).\n",
        "\n",
        "Approximating direct utility estimation: each state and sum of rewards received after constitute training example for a **supervised learning** algorithm. \\\n",
        "For RL makes sense to use an *online* learning algorithm that updates params after each trial. If $u_j(s)$ is observed total reward from $s$ onward in $j$th trial, error\n",
        "$$E_j(s)=(\\hat{U}_{theta}(s)-u_j(s))^2/2$$\n",
        "parameter update\n",
        "$$theta_i -= (alpha) (\\delta/\\delta theta_i)E_j(s)$$\n",
        "(the **Widrow-Hoff rule**, or the **delta rule** for online least-squares).\n",
        "\n",
        "Function appx will make agent learn faster if hypothesis space is not too large and includes some functions that are a good fit to true utility function.\n",
        "\n",
        "Approximating TD and Q-learning:\n",
        "$$theta_i+=(alpha)[R(s,a,s')+\\gamma \\hat{U}_{theta}(s')-\\hat{U}_{theta}(s)](\\delta/\\delta theta_i)\\hat{U}_{theta}(s)$$\n",
        "$$theta_i+=(alpha)[R(s,a,s')+\\gamma \\max_{a'}\\hat{Q}_{theta}(s',a')-\\hat{Q}_{theta}(s,a)](\\delta/\\delta theta_i)\\hat{Q}_{theta}(s,a)$$\n",
        "for passive TD learning, update rule converges to closest possible appx when function approximator is linear in the features. For nonlinear functions, no guarantees. Some other algos avoid these issues.\n",
        "\n",
        "With a learned model, agent can do look-ahead search to improve decisions and carry out internal simulations \\\n",
        "to improve approximate representations of $U$ or $Q$ rather than requiring slow/expensive real-world experience. For a *partially observable environment*, learning more difficult because next percept is not \\\n",
        "a label for the state prediction problem anymore.\n",
        "\n",
        "Deep neural networks popular for function approximation in RL (deep RL). Parameter update via backpropagation.\n",
        "\n",
        "**Credit assignment** problem: figure out what elements of strategy to blame/praise for failure or success. Supply agent with additional **pseudorewards** for \"making progress\" (e.g. 17.1 \\\n",
        "$\\Phi$ transformation).\n",
        "\n",
        "**Heirarchical reinforcement learning (HRL)**: break long action of sequences into smaller pieces to make learning easier. \\\n",
        "HRL agent begins with **partial program** outlining heirarchical structure for agent's behavior. Adds primitives for unspecified choices to be filled in by learning. \\\n",
        "RL is special case of HRL. Higher-level program can decide on higher-level action, then each action has its own program, etc down to fundamentals. \\\n",
        "HRL has **joint state space** (each state $(s,m)$ has physical state and machine state). Heirarchical structure of behavior provides natural **additive decomposition** \\\n",
        "of overall utility function. Learning each term is easier than learning whole function.\n",
        "\"\"\"\n",
        "chapter22_5 =  \"\"\"\n",
        "**Policy search**: keep updating policy until performance improves, then stop. Interested in *paramaterized* representation of $\\pi$, fewer parameters than states. \\\n",
        "Could represent by collection of paramaterized Q-functions (one for each action), and then take\n",
        "$$\\pi(s) = argmax_a \\hat{Q}_{theta}(s,a)$$\n",
        "Policy search will adjust the parameters theta to improve the policy. Not the same as Q-learning; find value of theta that results in good performance. \\\n",
        "Problem: policy is a discontinuous function of the parameters when actions are discrete. Makes gradient-based search difficult. Instead: use \\\n",
        "**stochastic policy** representation $\\pi_{theta}(s,a)$ which specifies the probability of selecting $a$ in state $s$, e.g. softmax function:\n",
        "$$\\pi_{theta}(s,a) = \\exp(b \\hat{Q}_{theta}(s,a)) /\\sum_{a'}\\exp(b \\hat{Q}_{theta}(s,a')) $$\n",
        "the param $b>0$ (beta) modulates the softness of the softmax (large $b$ -> hardmax).\n",
        "\n",
        "Improving deterministic policy in determ environment: let $rho(theta)$ be the **policy value** (expected reward-to-go when $\\pi_{theta}$ executed). If \\\n",
        "we can formulate $rho(theta)$ and calculate the **policy gradient** vector $del_{theta}rho(theta)$ we can use standard optimization techniques. Otherwise, \\\n",
        "evaluate $\\pi_{theta}$ by executing it and observing accumulated reward. Follow the **empirical gradient** by hill climbing (evaluating change in policy value for small \\\n",
        "changes in each parameter).\n",
        "\n",
        "In a nondeterm environment, more difficult; trying to compute $rho(theta)$ and $rho(theta+\\Delta theta)$ for some small $\\Delta theta$, but total reward for each trial may \\\n",
        "vary widely, so estimates of the policy value from a small number of trials will be unreliable. Comparing two estimates will be more unreliable. One solution: run lots of trials \\\n",
        "(but impractical, takes too much time, too many mistakes). Instead we can obtain an unbiased estimate of the gradient at $theta$, $del_{theta} rho(theta)$ from results of trials executed at $theta$. \\\n",
        "For episodic environment where each action $a$ receives reward $R(s_0, a, s_0)$ and environ restarts in $s_0$ (for simplicity):\n",
        "$$del_{theta} rho(theta) = del_{theta} \\sum_a R(s_0,a,s_0)\\pi_{theta}(s_0,a) = \\sum_a R(s_0,a,s_0) del_{theta}\\pi_{theta}(s_0,a)$$\n",
        "now simple trick to approximate summation with samples from prob distribution defined by $\\pi_{theta}(s_0,a)$: ($N$ total trials, $a_j$ taken on $j$th trial, $u_j(s)$ total reward from \\\n",
        "state $s$ onward in $j$th trial):\n",
        "$$del_{theta} rho(theta) =approx N^{-1}\\sum_{j=1}^N (u_j(s) del_{theta} \\pi_{theta}(s_0,a_j)) / \\pi_{theta}(s_0,a_j)$$\n",
        "*Reinforce* algorithm, usually much more effective than hill climbing, but still slower than necessary.\n",
        "\n",
        "Better approach: **correlated sampling**: compare policies with environment simulator in which teh random-number sequences can be repeated. \\\n",
        "Number of random sequences required to ensure that every policy is well estimated depends only on the complexity of the policy space, not complexity of underlying domain.\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "  \"Markov reward process (MRP)\": \"MDP with only 1 action $a_i$\",\n",
        "  \"Selection problem\": \"(e.g. hiring): no index function exists (optimal policy switches preferences for two arms when third arm is added).\",\n",
        "  \"metalevel decision problem\": \"(what computations to make during a game-tree search before moving) is also a selection problem (not a bandit problem)\",\n",
        "  \"Thompson sampling\": \"chooses an arm randomly according to the probability the arm is optimal, given samples so far ($O(\\log(n))$ regret)\",\n",
        "  \"bandit superprocess (BSP)\": \"each arm is a full MDP, e.g. multitasking, leads to greedier behavior in each MDP\",\n",
        "  \"passive reinforcement learning\": \"agent’s policy is fixed and the task is to learn the utilities of states (or of state–action pairs), and/or model\",\n",
        "  \"active reinforcement learning\": \"passive RL tasks + also figure out what to do (issue: exploration)\",\n",
        "  \"passive learning agent\": \"has fixed policy $\\pi(s)$ determining actions, trying to learn $U^\\pi(s)$ (similar to policy evaluation, but does not know model)\",\n",
        "  \"expected reward-to-go\": \"expected total reward from that state onward\",\n",
        "  \"greedy agent\": \"does no exploration - sometimes pays off and converges to optimal policy, but often doesn't due to not discovering better strategy\",\n",
        "  \"evaluation function\": \"a compact measure of desirability for potentially vast state spaces (approximate utility function)\",\n",
        "  \"catastrophic forgetting\": \"learned strategy is not 'subgame perfect', works in certain equilibrium but if deviated from these conditions, might act catastrophically suboptimally\",\n",
        "  \"experience replay\": \"(solution to catastrophic forgetting) make agent keep reliving youthful mistakes at regular intervals\"\n",
        "}"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3340
        },
        "outputId": "8d6a94b5-492c-4307-9334-8b1fb4d4f824"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"AIMA Chapter 22.5  Policy Search\", chapter22_5),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n\nAn **$n$-armed bandit** has $n$ levers, each yielding fixed/unknown probability distribution of winnings. Must choose levers successively.\n\nEach arm $M_i$ is an MRP. Overall Bandit problem is an MDP (Cartesian state space, actions $a_1,...,a_n$).\n\nOne-armed bandit $M$ yields sequence $R_0,R_1,...$, $M_\\lambda$ yields $\\lambda,\\lambda,...$ (cost of arm pull). Number of times you pull $M$ before switching to $M_\\lambda$ is **stopping time** $T$ (first pull of $M_1$ yields no extra info so stay forever). **Gittins index** of $M$ (value of $\\lambda$ for which agent is indifferent btwn playing $M$->$M_\\lambda$ game vs just taking $M_\\lambda$):\n$$\\lambda = \\max_{T>0} E[\\sum_{t=0}^{T-1}\\gamma^tR_t]/E[\\sum_{t=0}^{T-1}\\gamma^t]$$\nOptimal policy: pull the arm that has the highest Gittins index, then update the Gittins index of the arm you pulled.\nCompute Gittins index for arm $M$, construct MDP with additional action at each state: quit and receive infinite sequence of $\\lambda$-rewards; optimal policy value is $\\lambda/(1-\\gamma)$. Equivalently, construct **restart MDP** $M^s$; add new action to each state with same rewards/probs/transitions as initial state $s$ (replace infinite path of $\\lambda$ with restarting, since agent is indifferent b.d.o. Gittins index). Gittins index for arm $M$ in state $s$ is equal to $(1-\\gamma)$ times value of optimal policy for restart MDP $M^s$ (just solve MDP).\n\n**Bernoulli bandit**: each arm $M_i$ has count of 1s $s_i$, count of 0s $f_i$ (both initialized to 1), next outcome is 1 with probability $s_i/(s_i+f_i)$, 0 with probability $f_i/(s_i+f_i)$. Infinite states (can't solve with Gitten). Accurate appx: truncated MDP with states up to $s_i+f_i=100$ and $\\gamma=0.9$. Arms with higher payoff probability preferred, but exists an **exploration bonus** with rarely-tried arms.\n\nAppx algos (no Gittins comp):\n\n**Upper confidence bound** or UCB heuristic calculates **confidence interval** for the value of the arm from samples, choose arm with highest upper bound. Approximate index value for arm $M_i$:\n$$UCB(M_i) = \\hat{\\mu} + g(N)/\\sqrt{N_i}$$\n$N:=$ total num of samples drawn from all arms. Definition of $g$ determines **regret** relative to perfect policy. For undiscounted case, no algorithm can have regret that grows slower than $O(\\log N)$. USB policy can achieve this bound with $g(N)=\\sqrt{2\\log(1+N\\log^2(N))}$.\n\n**Bandit superprocess (BSP)**: each arm is a full MDP, e.g. multitasking, leads to greedier behavior in each MDP. Can solve as MDP with Cartesian state space (exponential, impractical). Better approach: consider **opportunity cost** in each MDP. A **dominating policy** ignores opportunity cost (optimally). Adding actions to states can create a relaxed MDP with a dominating policy; upper bound on value of arm. Lower bound: solve arm individually. If lower bound of one is above upper bound of others, solved. Otherwise, look-ahead search and recomputation of bounds -> optimal solution.\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n\n**Reinforcement learning (RL)**: agent interacts with world and periodically receives **rewards** or **reinforcements**. Same goal as MDPs, but not given details of MDP.\n\nProviding reward signal easier than providing labeled training examples (efficient, doesn't require domain knowledge e.g. chess).\n\n**Sparse rewards*: just reward for end goal. Better: inject expertise / provide intermediate rewards to help learn good strategies.\n\n**Model-based reinforcement learning**: agent uses transition model of environment to help interpret reward signals and make decisions. Model might be unknown/learned or provided (e.g. chess rules). Model-based RL systems try to learn utility function $U(s)$ for each state.\n\n**Model-free RL**: learns more direct representation of how to behave (no model):\n- **Action-Utility Learning** (**Q-learning**), agent learns $Q$ function\n- **Policy Search**: agent learns policy $\\pi(s)$ that maps states to actions (**reflex agent**)\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\n**Passive learning agent** executes set of **trials** in environment using policy $\\pi$.\n\n**Direct utility estimation**: the utility of a state is the expected reward-to-go; each trial provides *sample* of this quantity for each state visited. Calculate observed reward-to-go for each state and update estimated utility (running average). Infinite trials -> converge. Ignores Bellman equations / connections between states. Searches for $U$ in a hypothesis space much larger than necessary, converges slowly.\n\n**Adaptive dynamic programming (ADP)** uses constraints among utilities of states; learns transition model that connects them, then solves MDP using DP. For passive RL agent: plug learned transition models and rewards into linal solver (linear equations). Can also use **modified policy iteration** (see mod8) to update utility estimates after each change to learned model. Learning transition model easy (fully observable environment; keep track of transition probability averages). Intractable for large state spaces (cannot learn full transition model or solve so many equations).\n\n**Temporal-difference learning**: update $U^\\pi(s)$ after transition from $s$ to $'$ via $\\pi(s)$:\n$$U^\\pi(s)+=(alpha)[R(s,\\pi(s),s')+\\gamma U^\\pi(s')-U^\\pi(s)]$$\nwhere alpha is the **learning rate** (**temporal-difference** (TD) equations). The *average value* of $U^\\pi(s)$ will converge to the correct quantity in the limit, even if the value itself continues to fluctuate. Also if we decrease alpha as number of visits to $s$ increases, then $U^\\pi(s)$ will converge correctly. TD does not need a transition model to perform its updates. The environment provides the connection to neighboring states in the form of observed transitions. Crude but efficient first approximation to ADP. Each ADP adjustment is **pseudoexperience** (from TD pov) from simulating current transition model. Can extend TD to use transition model to generate pseudoexperiences (more closely approximate ADP, more comp. time).\n\nCan make ADP more efficient by appx algos for value/policy iteration. One idea: bound number of adjustments made after each observed transition. Alternatively: use heuristic to rank adjustments and only do most significant (**prioritized sweeping** heuristic, adjust states whose likely successors have just undergone large adjustment to utility estimates). Appx algos can converge nearly as fast, and do much less comp. Also, appx algo can use min adjust size that decreases as transition model becomes more accurate.\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\nActions provide *rewards* and *information*. Must tradeoff **exploitation** for short-term rewards and **exploration** to improve policy.\n\nAny optimal exploration scheme should be \"greedy in the limit of infinite exploration\" (**GLIE**). Must try each action in each state an unbounded number of times to avoid possibility that optimal action is missed. An ADP agent using such a scheme will eventually learn true transition model and can then exploit.\n\nSimplest GLIE scheme: choose random action at time step $t$ with prob $1/t$ and follow greedy policy otherwise. Converges slow. Better to give more weight to unexplored actions while disliking low-utility actions (like with MCTS). Alter utility equation to give higher estimate to less explored state-action pairs. Let $U^+(s)$ be optimistic utility estimate, $N(a,s)$ number of times $a$ tried at state $s$. Update equation:\n$$U^+(s) = \\max_s f(\\sum_{s'}P(s'\\mid s,a)[R(s,a,s')+\\gamma U^+(s)], N(s,a))$$\nwhere $f(u,n)$ is the **exploration function**: how greed (like high $u$) weighed against curiosity (like unexplored actions). Should increase in $u$ and decrease in $n$. Simple choice: $f(u,n)=u$ if $n\\geq N_e$ else $R^+$ (an optimistic estimate of best possible reward), where $N_e$ is fixed. Agent will try each state at least $N_e$ times.\n\nMany actions are **irreversible**, agent may enter **absorbing state** (no actions have any effect, no rewards, e.g. death). Better idea: choose policy that works reasonably well for whole range of possible / probable models. **Bayesian reinforcement learning** assumes prior prob $P(h)$ over hypothesis $h$ of true model; posterior $P(h\\mid e)$ obtained by Bayes' rule given observations. Then\n$$\\pi^* = argmax_\\pi\\sum_hP(h\\mid e)U_h^*$$\nAgent must consider effects of future observations. Problem becomes an **exploration poMDP**, belief states are distributions over models. Solving poMDP is usually intractable.\n\nAlternative: maintain set of possible models $H$ without considering probabilities, optimize for *worst case* over $H$:\n$$\\pi^* = argmax_\\pi \\min_h U_h^\\pi$$\nformulate as adversarial search problem. Problem: overly conservative; real life is full of risk-reward tradeoffs.\n\nHuman intervention can help algo learn safely (record initial series of actions, write constraints on behavior).\n\nActive **Q-learning**: learn action-utility $Q$ values instead of utility function $U$. Agent can then choose $argmax_a f(Q(s,a), N_{sa}[s,a])$. Q-learning TD update:\n$$Q(s,a)' = Q(s,a) + (alpha)[R(s,a,s') + \\gamma\\max_{a'}Q(s',a')-Q(s,a)]$$\ndoes not need transition model for learning or action selection. \n\n**SARSA** (state, action, reward, state, action) like Q-learning but SARSA updates with the Q-value of the action actually taken (no max). SARSA penalizes exploration to negative reward state.\n\nQ-learning is an **off-policy** learning algo; learns values for policy it isn't using, SARSA is an **on-policy** algo (learns Q-values for its own policy).\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\nConstructing compact appx of true utility or Q-function: **function approximation**. Might appx utility using weighted linear combo of features:\n$$\\hat{U}_{theta}(s) = theta_1f_1(s)+...+theta_nf_n(s)$$\njust learn parameters theta. Sometimes combine with lookahead search to produce more accurate decisions.\n\nFunction approximation allows for inductive generalizations about Q-function (can use experience from similar states).\n\nApproximating direct utility estimation: each state and sum of rewards received after constitute training example for a **supervised learning** algorithm. For RL makes sense to use an *online* learning algorithm that updates params after each trial. If $u_j(s)$ is observed total reward from $s$ onward in $j$th trial, error\n$$E_j(s)=(\\hat{U}_{theta}(s)-u_j(s))^2/2$$\nparameter update\n$$theta_i -= (alpha) (\\delta/\\delta theta_i)E_j(s)$$\n(the **Widrow-Hoff rule**, or the **delta rule** for online least-squares).\n\nFunction appx will make agent learn faster if hypothesis space is not too large and includes some functions that are a good fit to true utility function.\n\nApproximating TD and Q-learning:\n$$theta_i+=(alpha)[R(s,a,s')+\\gamma \\hat{U}_{theta}(s')-\\hat{U}_{theta}(s)](\\delta/\\delta theta_i)\\hat{U}_{theta}(s)$$\n$$theta_i+=(alpha)[R(s,a,s')+\\gamma \\max_{a'}\\hat{Q}_{theta}(s',a')-\\hat{Q}_{theta}(s,a)](\\delta/\\delta theta_i)\\hat{Q}_{theta}(s,a)$$\nfor passive TD learning, update rule converges to closest possible appx when function approximator is linear in the features. For nonlinear functions, no guarantees. Some other algos avoid these issues.\n\nWith a learned model, agent can do look-ahead search to improve decisions and carry out internal simulations to improve approximate representations of $U$ or $Q$ rather than requiring slow/expensive real-world experience. For a *partially observable environment*, learning more difficult because next percept is not a label for the state prediction problem anymore.\n\nDeep neural networks popular for function approximation in RL (deep RL). Parameter update via backpropagation.\n\n**Credit assignment** problem: figure out what elements of strategy to blame/praise for failure or success. Supply agent with additional **pseudorewards** for \"making progress\" (e.g. 17.1 $\\Phi$ transformation).\n\n**Heirarchical reinforcement learning (HRL)**: break long action of sequences into smaller pieces to make learning easier. HRL agent begins with **partial program** outlining heirarchical structure for agent's behavior. Adds primitives for unspecified choices to be filled in by learning. RL is special case of HRL. Higher-level program can decide on higher-level action, then each action has its own program, etc down to fundamentals. HRL has **joint state space** (each state $(s,m)$ has physical state and machine state). Heirarchical structure of behavior provides natural **additive decomposition** of overall utility function. Learning each term is easier than learning whole function.\n\n\n## AIMA Chapter 22.5  Policy Search\n\n**Policy search**: keep updating policy until performance improves, then stop. Interested in *paramaterized* representation of $\\pi$, fewer parameters than states. Could represent by collection of paramaterized Q-functions (one for each action), and then take\n$$\\pi(s) = argmax_a \\hat{Q}_{theta}(s,a)$$\nPolicy search will adjust the parameters theta to improve the policy. Not the same as Q-learning; find value of theta that results in good performance. Problem: policy is a discontinuous function of the parameters when actions are discrete. Makes gradient-based search difficult. Instead: use **stochastic policy** representation $\\pi_{theta}(s,a)$ which specifies the probability of selecting $a$ in state $s$, e.g. softmax function:\n$$\\pi_{theta}(s,a) = \\exp(b \\hat{Q}_{theta}(s,a)) /\\sum_{a'}\\exp(b \\hat{Q}_{theta}(s,a')) $$\nthe param $b>0$ (beta) modulates the softness of the softmax (large $b$ -> hardmax).\n\nImproving deterministic policy in determ environment: let $rho(theta)$ be the **policy value** (expected reward-to-go when $\\pi_{theta}$ executed). If we can formulate $rho(theta)$ and calculate the **policy gradient** vector $del_{theta}rho(theta)$ we can use standard optimization techniques. Otherwise, evaluate $\\pi_{theta}$ by executing it and observing accumulated reward. Follow the **empirical gradient** by hill climbing (evaluating change in policy value for small changes in each parameter).\n\nIn a nondeterm environment, more difficult; trying to compute $rho(theta)$ and $rho(theta+\\Delta theta)$ for some small $\\Delta theta$, but total reward for each trial may vary widely, so estimates of the policy value from a small number of trials will be unreliable. Comparing two estimates will be more unreliable. One solution: run lots of trials (but impractical, takes too much time, too many mistakes). Instead we can obtain an unbiased estimate of the gradient at $theta$, $del_{theta} rho(theta)$ from results of trials executed at $theta$. For episodic environment where each action $a$ receives reward $R(s_0, a, s_0)$ and environ restarts in $s_0$ (for simplicity):\n$$del_{theta} rho(theta) = del_{theta} \\sum_a R(s_0,a,s_0)\\pi_{theta}(s_0,a) = \\sum_a R(s_0,a,s_0) del_{theta}\\pi_{theta}(s_0,a)$$\nnow simple trick to approximate summation with samples from prob distribution defined by $\\pi_{theta}(s_0,a)$: ($N$ total trials, $a_j$ taken on $j$th trial, $u_j(s)$ total reward from state $s$ onward in $j$th trial):\n$$del_{theta} rho(theta) =approx N^{-1}\\sum_{j=1}^N (u_j(s) del_{theta} \\pi_{theta}(s_0,a_j)) / \\pi_{theta}(s_0,a_j)$$\n*Reinforce* algorithm, usually much more effective than hill climbing, but still slower than necessary.\n\nBetter approach: **correlated sampling**: compare policies with environment simulator in which teh random-number sequences can be repeated. Number of random sequences required to ensure that every policy is well estimated depends only on the complexity of the policy space, not complexity of underlying domain.\n\n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **Markov reward process (mrp)** - MDP with only 1 action $a_i$\n* **Selection problem** - (e.g. hiring): no index function exists (optimal policy switches preferences for two arms when third arm is added).\n* **Metalevel decision problem** - (what computations to make during a game-tree search before moving) is also a selection problem (not a bandit problem)\n* **Thompson sampling** - chooses an arm randomly according to the probability the arm is optimal, given samples so far ($O(\\log(n))$ regret)\n* **Bandit superprocess (bsp)** - each arm is a full MDP, e.g. multitasking, leads to greedier behavior in each MDP\n* **Passive reinforcement learning** - agent’s policy is fixed and the task is to learn the utilities of states (or of state–action pairs), and/or model\n* **Active reinforcement learning** - passive RL tasks + also figure out what to do (issue: exploration)\n* **Passive learning agent** - has fixed policy $\\pi(s)$ determining actions, trying to learn $U^\\pi(s)$ (similar to policy evaluation, but does not know model)\n* **Expected reward-to-go** - expected total reward from that state onward\n* **Greedy agent** - does no exploration - sometimes pays off and converges to optimal policy, but often doesn't due to not discovering better strategy\n* **Evaluation function** - a compact measure of desirability for potentially vast state spaces (approximate utility function)\n* **Catastrophic forgetting** - learned strategy is not 'subgame perfect', works in certain equilibrium but if deviated from these conditions, might act catastrophically suboptimally\n* **Experience replay** - (solution to catastrophic forgetting) make agent keep reliving youthful mistakes at regular intervals\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\"\n",
        "Real-world agents need to handle **uncertainty** (partial observability, nondeterminism, or adversaries). Agent can maintain **belief state**, repr of all possible world states \\\n",
        "that it might be in (and generate plan for every possible eventuality). Too complex, considers unlikely eventualities, must have a way to compare merits of plans that are not guaranteed.\n",
        "\n",
        "USing logic to cope with complex and uncertain domain fails because of **laziness** (too much work to enumerate all antecedents or consequents needed to form exceptionless rule, too hard to use), \\\n",
        "**theoretical ignorance** (there is no complete theory for the domain), **practical ignorance** (even if all rules are known, might be uncertain about practical conditions in this case).\n",
        "\n",
        "Agent's knowledge can sometimes be at best a **degree of belief** in the relevant sentences (can use **probability theory**).\n",
        "\n",
        "Probability statements made with respect to knowledge state, not with respect to real world.\n",
        "\n",
        "Decision theroy = probability theory + utility theory.\n",
        "\"\"\"\n",
        "chapter12_2 = \"\"\"\n",
        "Logical assertions say which possible worlds are strictly ruled out, probabilistic assertions talk about how probable the various worlds are. \\\n",
        "Prob theory: set of all possible worlds is called the **sample space** $\\Omega$ containing possible worlds $\\omega$. Possible worlds are *mutually exclusive* and *exhaustive*. \n",
        "\n",
        "A fully specified **probability model** associates a numerical prob $P(\\omega) \\in [0,1]$ with each possible world : $\\sum_\\omega P(\\omega) = 1$. \n",
        "\n",
        "In logic a set of worlds is a **proposition**, in prob theory the prob of a proposition $\\phi$ is \n",
        "$$P(\\phi)=\\sum_{\\omega\\in\\phi}P(\\omega)$$\n",
        "Raw probabilities called **unconditional** or **prior probabilities**. Usually we have some information (**evidence**) that has already been revealed; then we look for the **conditional** or \\\n",
        "**posterior** probability. We have $P(a\\mid b) = P(a\\land b) / P(b)$ when $P(b)>0$. Equivalently, $P(a\\land b) = P(a\\mid b)P(b)$ (**product rule**).\n",
        "\n",
        "Variables called **random variables** map the domain of possible worlds $\\Omega$ to some **range** (set of values it can take on); can be finite, infinite, discrete, continuous. Can combine elementary \\\n",
        "propositions by using connectives of propositional logic, e.g. $P(cavity\\mid \\lnot toothache \\land teen)$. Vector of prob of each value of random variable: **P**(X)$=[0.6,0.1,...]$. Then **P** defines a \\\n",
        "**probability distribution** for X (assignment of prob to each value in range); finite discrete prob distribution called **categorical**.\n",
        "\n",
        "Define continuous distribution by **probability density function**, some paramatrized function of $x$. PDF $P(x)$ is probability that $X$ falls within an arbitrarily small region \\\n",
        "beginning at $x$, divided by the width of the region. Can form distributions on multiple variables with commas (**joint probability distribution**).\n",
        "\n",
        "A possible world is an assignment of values to all the random variables under consideration.\n",
        "\n",
        "Probability model is completely determined by the joint distribution on all random variables (**full join probability distribution**).\n",
        "\n",
        "$P(\\lnot a) = 1-P(a)$, $P(a\\lor b) = P(a)+P(b)-P(a\\land b)$ (**inclusion exclusion principle**).\n",
        "\n",
        "If an agent has some degree of belief in a proposition $a$, it should be able to state odds at which it is indifferent to a bet for or against $a$. \\\n",
        "If one's degrees of belief do not accurately reflect the world, they would lose money to an opposing agent whose beliefs more accurately reflect the state of the world. \\\n",
        "If agent expresses beliefs that violate axioms of probability, there is a combo of bets by agent 2 that will guarantee agent 1 to lose moeny each time. Refusing to bet like refusing to let time pass \\\n",
        "(we make decisions under uncertainty at each moment).\n",
        "\"\"\"\n",
        "chapter12_3 = \"\"\"\n",
        "**Marginal probability** is the probability of one variable, extracted from a joint distribution through **marginalization** or **summing out** (sum probabilities for each possible \\\n",
        "value of other variables): $P(Y)=\\sum_z P(Y,Z=z)$.\n",
        "\n",
        "**Conditioning** rule: $P(Y)=\\sum_Z P(Y\\mid Z)P(Z)$.\n",
        "\n",
        "Inference for query with single variable $X$. $E$ is list of evidence variables, $e$ is list of observed values for them, $Y$ is remaining unobserved variables. Query is\n",
        "$$P(X\\mid e) = aP(X,e) = a\\sum_y P(X,e,y)$$\n",
        "and $P(X,e,y)$ is ubset of probabilities from the full join distribution. However, answering probabilistic queries for discrete variables does not scale well; for domain with $n$ \\\n",
        "boolean variables, it requires an input table of size $O(2^n)$ and takes $O(2^n)$ time to process the table.\n",
        "\"\"\"\n",
        "chapter12_4 = \"\"\"\n",
        "**Independence** (**marginal independence** and **absolute independence**): $P(a\\mid b) = P(a)$ or $P(b\\mid a)=P(b)$ or $P(a\\land b)=P(a)P(b)$ (all equivalent). \\\n",
        "Independence assertions usually based on knowledge of the domain. Can dramatically reduce amount of info necessary to specify the full joint distribution. If complete set of variables \\\n",
        "can be divided into independent subsets, then the full join distribution can be factored into separate joint distributions on those subsets.\n",
        "\"\"\"\n",
        "chapter12_5 = \"\"\"\n",
        "**Bayes' rule** (derived from product rule):\n",
        "$$P(b\\mid a) - P(a\\mid b)P(b)/P(a)$$\n",
        "More general version conditionalized on some background evidence $e$:\n",
        "$$P(Y\\mid X,e)=P(X\\mid Y,e)P(Y\\mid e)/P(X\\mid e)$$\n",
        "$$P(cause\\mid effect) = P(effect\\mid cause) P(cause)/P(effect)$$\n",
        "$P(effect\\mid cause)$ quantifies relationship in **causal** direction, whereas $P(cause\\mid effect)$ describes the **diagnostic** direction.\n",
        "\n",
        "Can avoid assessing prior probability of the evidence by computing posterior for each value of query variable:\n",
        "$$P(Y\\mid X) = aP(X\\mid Y)P(Y)$$\n",
        "where $a$ is the normalization constant needed to make the entries sum to 1.\n",
        "\n",
        "**conditional independence** $P(X,Y\\mid Z) = P(X\\mid Z)P(Y\\mid Z)$, and $P(X\\mid Y,Z)=P(X\\mid Z)$ and $P(Y\\mid X,Z)=P(Y\\mid Z)$. Conditional independence assertions can allow probabilistic \\\n",
        "systems to scale up, and they are much more commonly available than independence assertions.\n",
        "\"\"\"\n",
        "chapter12_6 = \"\"\"\n",
        "**Naive Bayes model**: assume independence as a simplifying assumption (**Bayesian classifier**). Observed effects $E=e$, remaining effect variables $Y$ are unobserved:\n",
        "$$P(cause\\mid e) = a\\sum_y P(cause,e,y)$$\n",
        "$$P(cause\\mid e) = a P(cause) \\prod_j P(e_j\\mid cause)$$\n",
        "\"\"\"\n",
        "chapter12_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\"\n",
        "Naive approach for language model: $P(next \\mid history) = P(history + next) / P(history)$ (compute frequency counts). Problem: requires too much data.\n",
        "\n",
        "Chain rule:\n",
        "$$P(w_{1:n}) = \\prod_{k=1}^n P(w_k\\mid w_{1:k-1})$$\n",
        "doesn't really help, still hard to estimate $P(w_k\\mid w_{1:k-1})$.\n",
        "\n",
        "Intuitino of n-gram model: approximate the history by just the last few words. Bigram model approximates $P(w_n\\mid w_{1:n-1})$ with $P(w_n\\mid w_{n-1})$. \\\n",
        "Assumption that the probability of a word depends only on previous word is called a **Markov** assumption. In general, an N-gram approximates $P(w_n\\mid w_{1:n-1})$ as $P(w_n\\mid w_{n-N+1:n-1})$.\n",
        "\n",
        "Given bigram assumption, prob of complete word sequence is $\\prod_{k=1}^n P(w_k\\mid w_{k-1})$.\n",
        "\n",
        "Make an MLE estimate for the parameters of an n-gram model by getting counts from a corpus, and **normalizing** the counts between 0 and 1. E.g. for bigram: compute count $C(xy)$ of bigram, then\n",
        "$$P(w_n\\mid w_{n-1}) = C(w_{n-1}w_n)/C(w_{n-1})$$\n",
        "or for N-gram:\n",
        "$$P(w_n\\mid w_{n-N+1:n-1}) = C(w_{n-N+1:n-1}w_n)/C(w_{n-N+1:n-1})$$\n",
        "This MLE maximizes the likelihood of the corpus given the model.\n",
        "\n",
        "Practical considerations: use pseudo-words at start of sentence and marker at end of sentence. Always represent/compute LM probabilities as **log probabilities**.\n",
        "\"\"\"\n",
        "SLP_chapter3_2 = \"\"\"\n",
        "To evaluate LM, can embed in an application and measure how much app improves: **extrinsic evaluation**. But running NLP systems end-to-end is expensive. An **intrinsic evaluation** metric is one that \\\n",
        "measures the quality of a model independent of any application. We need a **test set** corpus. Whichever model assigns a higher probability to the test set (more accurately predicts the test set) is better. \\\n",
        "Don't train on test set, this can cause innacuracy in **perplexity**. Sometimes test set is used so frequently we implicity fit to it by choosing models / parameters. Then we can use initial test set \\\n",
        "**development test set** or **devset**, and use fresh test set for final eval.\n",
        "\n",
        "Should pick smallest test set to give enough statistical power to measure stat significant difference between the two potential models. In practice 80/10/10 training/devset/test. Can take large chunks or \\\n",
        "stripes out of corpus to make dif sets.\n",
        "\n",
        "**Perplexity** (PP) of a LM on test set $W=w_1,...,w_N$ is\n",
        "$$PP(W) = P(w_1,...,w_N)^{-(1/N)} = (\\prod_{i=1}^N P(w_i\\mid w_1,...,w_{i-1}))^{-(1/N)}$$\n",
        "(estimate $P(w_i\\mid w_1,...,w_{i-1}))$ as $P(w_i\\mid w_{i-n+1:i-1})$ for n-gram model). We want to minimize perplexity.\n",
        "\n",
        "Should include begin and end of sentence markers in prob comps, also include end-of-sentence marker (but not begin-marker) in total count of word tokens $N$.\n",
        "\n",
        "Perplexity is the **weighted average branching factor** of a language. Related to concept of entropy. The perplexity of two LMs is only comparable if they use identical vocabularies. \\\n",
        "Intrinsic improvement in perplexity does not guarantee extrinsic improvement in other task. But usually correlates with such improvements, can be used as a quick check on an algorithm. \\\n",
        "Should be confirmed with final extrnsic evaluation.\n",
        "\"\"\"\n",
        "SLP_chapter3_3 = \"\"\"\n",
        "To visualize what kind of knowledge a language model embodies, can **sample** from it. \n",
        "\"\"\"\n",
        "SLP_chapter3_4 = \"\"\"\n",
        "An n-gram model can bge highly influenced by idiosyncracies and phrases in its test set. Should use a test set that has a similar **genre** to whatever task we are trying to accomplish. \\\n",
        "Also get training data in the same **dialect** or **variety**.\n",
        "\n",
        "Problem of **sparsity**: any corpus is limited, so some perfectly acceptable English word sequences are bound to be missing. We'll have zero-probability n-grams that should have nonzero prob. \\\n",
        "These *zeros* (things that occur in training set but do in test set) causes us to underestimate the prob of all sorts of words that might occur, and also if any prob of any word in the test set is 0, \\\n",
        "the entire prob of test set is 0; perplexity undefined.\n",
        "\n",
        "In a **closed vocabulary** system the test set can only contain words from lexicon of known words that can occur, so no unknown words. In other cases, must deal with words we haven't seen before, \\\n",
        "called **unknown words** or **out of vocabulary (OOV)** words. Percentage of OOV words in test set is **OOV rate**. An **open vocabulary system** is one in which we model unknown words by adding \\\n",
        "pseudo-word <UNK>. Can turn problem back into closed vocabulary by:\n",
        "1. **choosing a vocabulary** fixed in advance\n",
        "2. **converting** any OOV word in train set to unknown word token <UNK>\n",
        "3. **estimate** the prob for <UNK> from its counts just like any other regular word\n",
        "\n",
        "Alternatively, can replace by <UNK> all words that occur fewer than $n$ times in the training set (for some small number $n$) or choose vocab size $V$ in advance and populate vocab with most \\\n",
        "frequent words. Perplexity can be influenced by <UNK> model, so perplexity should only be compared for models with same vocabulary.\n",
        "\"\"\"\n",
        "SLP_chapter3_5 = \"\"\"\n",
        "How to deal with unseen word sequences? Prevent 0 prob by shaving off a bit of prob from some more frequent events and giving it to events we've never seen (called **smoothing** or **discounting**).\n",
        "\n",
        "**Laplace smoothing**: add 1 to all n-gram counts before normalizing them into probabilities. Does not perform well enough for modern n-gram models, but gives a useful baseline. \\\n",
        "Define **adjusted count**\n",
        "$$c_i^* = (c_i+1)N/(N+V)$$\n",
        "Smoothing is like **discounting** (lowering) some non-zero counts to get prob mass that will be assigned to zero counts. Can describe smoothing algorithm in terms of relative **discount** $d_c$:\n",
        "$$d_c=c^*/c$$\n",
        "\n",
        "Alternative: **add-k smoothing**; add fractional count $k$ to each count:\n",
        "$$P^*_{Add-k}(w_n\\mid w_{n-1})=(C(w_{n-1}w_n)+k)/(C(w_{n-1}+kV)$$\n",
        "choose $k$ using devset. Still doesn't work very well for LM.\n",
        "\n",
        "Alternative: **backoff** to using less context when sequence is unseen. Or use **interpolation**, always mixing prob estimates from each n-gram estimator, weighted combo. \\\n",
        "Can even compute weights $\\lambda$ by conditioning on the context. Learn weigghts from **held-out** corpus. Can use **EM** algorithm, iterative learning algo that converges on optimal $\\lambda$s.\n",
        "\n",
        "For backoff model to give correct prob distribution, must **discount** the higher-order n-grams to save some prob mass for lower-order n-grams. Need explicit discount factor and function alpha to \\\n",
        "distribute prob mass to lower-order n-grams. Backoff with discounting is called **Katz backoff** (reply on discounted prob $P^*$ if we've seen n-gram before, otherwise backoff to \\\n",
        "Katz prob for shorter history). $P_{BO}(w_n\\mid w_{n-N+1:n-1}) = $P^*(w_n\\mid w_{n-N+1:n-1})$ if $C(w_{n-N+1:n})>0$ else $(alpha)(w_{n-N+1:n-1})P_{BO}(w_{n-N+2:n-1})$. \\\n",
        "Katz backoff often combined with smoothing method called **Good-Turing**; involves detailed computation for estimating Good-Turing smoothing and the $P^*$ and alpha values.\n",
        "\"\"\"\n",
        "SLP_chapter3_7 = \"\"\"\n",
        "Scalability: store string as 64-bit hash number in memory, probabilities quantized with 4-8 bits, n-grams stored in reverse tries, shrink model by pruning n-grams with counts below threshold, \\\n",
        "use entropy to prune less important n-grams, build approximate LMs using techniques like **Bloom filters**, use sorted arrays, efficiently combine probs and backoffs in single value, \\\n",
        "merge sorts to efficiently build prob tables in minimal number of passes through large corpus.\n",
        "\n",
        "With very large models, simpler smoothing algorithm may be sufficient: **stupid backoff**; give up goal of true prob distribution; no discounting of higher-order probs. If higher-order n-gram has zero \\\n",
        "count, backoff to a lower order n-gram weighed by a fixed (context-independent) weight. Does not produce prob dist, so refer to it as $S$. \\\n",
        "$S(w_i\\mid w_{i-k+1:i-1}) = count(w_{i-k+1:i})/count(w_{i-k+1:i-1})$ if $count(w_{i-k+1:i})>0$ otherwise $\\lambda S(w_i\\mid w_{i-k+2:i-1})$. Backoff terminates in the unigram. $\\lambda=0.4$ works well.\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"language models (LMs)\" : \"models that assign probabilities to sequences of words\",\n",
        "\"$n$-gram\": \"sequence of $n$ words\",\n",
        "\"relative frequency\": \"observed frequency of a particular sentence divided by the observed frequency of a prefix\",\n",
        "\"branching factor of a language\": \"number of possible next words that can follow any word\",\n",
        "\"balanced corpora\": \"has roughly equal number of words from different genres\",\n",
        "}"
      ],
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3536
        },
        "cellView": "form",
        "outputId": "017db07f-9f61-4270-ed97-10ab3f63b127"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\nReal-world agents need to handle **uncertainty** (partial observability, nondeterminism, or adversaries). Agent can maintain **belief state**, repr of all possible world states that it might be in (and generate plan for every possible eventuality). Too complex, considers unlikely eventualities, must have a way to compare merits of plans that are not guaranteed.\n\nUSing logic to cope with complex and uncertain domain fails because of **laziness** (too much work to enumerate all antecedents or consequents needed to form exceptionless rule, too hard to use), **theoretical ignorance** (there is no complete theory for the domain), **practical ignorance** (even if all rules are known, might be uncertain about practical conditions in this case).\n\nAgent's knowledge can sometimes be at best a **degree of belief** in the relevant sentences (can use **probability theory**).\n\nProbability statements made with respect to knowledge state, not with respect to real world.\n\nDecision theroy = probability theory + utility theory.\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n\nLogical assertions say which possible worlds are strictly ruled out, probabilistic assertions talk about how probable the various worlds are. Prob theory: set of all possible worlds is called the **sample space** $\\Omega$ containing possible worlds $\\omega$. Possible worlds are *mutually exclusive* and *exhaustive*. \n\nA fully specified **probability model** associates a numerical prob $P(\\omega) \\in [0,1]$ with each possible world : $\\sum_\\omega P(\\omega) = 1$. \n\nIn logic a set of worlds is a **proposition**, in prob theory the prob of a proposition $\\phi$ is \n$$P(\\phi)=\\sum_{\\omega\\in\\phi}P(\\omega)$$\nRaw probabilities called **unconditional** or **prior probabilities**. Usually we have some information (**evidence**) that has already been revealed; then we look for the **conditional** or **posterior** probability. We have $P(a\\mid b) = P(a\\land b) / P(b)$ when $P(b)>0$. Equivalently, $P(a\\land b) = P(a\\mid b)P(b)$ (**product rule**).\n\nVariables called **random variables** map the domain of possible worlds $\\Omega$ to some **range** (set of values it can take on); can be finite, infinite, discrete, continuous. Can combine elementary propositions by using connectives of propositional logic, e.g. $P(cavity\\mid \\lnot toothache \\land teen)$. Vector of prob of each value of random variable: **P**(X)$=[0.6,0.1,...]$. Then **P** defines a **probability distribution** for X (assignment of prob to each value in range); finite discrete prob distribution called **categorical**.\n\nDefine continuous distribution by **probability density function**, some paramatrized function of $x$. PDF $P(x)$ is probability that $X$ falls within an arbitrarily small region beginning at $x$, divided by the width of the region. Can form distributions on multiple variables with commas (**joint probability distribution**).\n\nA possible world is an assignment of values to all the random variables under consideration.\n\nProbability model is completely determined by the joint distribution on all random variables (**full join probability distribution**).\n\n$P(\\lnot a) = 1-P(a)$, $P(a\\lor b) = P(a)+P(b)-P(a\\land b)$ (**inclusion exclusion principle**).\n\nIf an agent has some degree of belief in a proposition $a$, it should be able to state odds at which it is indifferent to a bet for or against $a$. If one's degrees of belief do not accurately reflect the world, they would lose money to an opposing agent whose beliefs more accurately reflect the state of the world. If agent expresses beliefs that violate axioms of probability, there is a combo of bets by agent 2 that will guarantee agent 1 to lose moeny each time. Refusing to bet like refusing to let time pass (we make decisions under uncertainty at each moment).\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n\n**Marginal probability** is the probability of one variable, extracted from a joint distribution through **marginalization** or **summing out** (sum probabilities for each possible value of other variables): $P(Y)=\\sum_z P(Y,Z=z)$.\n\n**Conditioning** rule: $P(Y)=\\sum_Z P(Y\\mid Z)P(Z)$.\n\nInference for query with single variable $X$. $E$ is list of evidence variables, $e$ is list of observed values for them, $Y$ is remaining unobserved variables. Query is\n$$P(X\\mid e) = aP(X,e) = a\\sum_y P(X,e,y)$$\nand $P(X,e,y)$ is ubset of probabilities from the full join distribution. However, answering probabilistic queries for discrete variables does not scale well; for domain with $n$ boolean variables, it requires an input table of size $O(2^n)$ and takes $O(2^n)$ time to process the table.\n\n\n## AIMA Chapter 12.4 Independence\n\n**Independence** (**marginal independence** and **absolute independence**): $P(a\\mid b) = P(a)$ or $P(b\\mid a)=P(b)$ or $P(a\\land b)=P(a)P(b)$ (all equivalent). Independence assertions usually based on knowledge of the domain. Can dramatically reduce amount of info necessary to specify the full joint distribution. If complete set of variables can be divided into independent subsets, then the full join distribution can be factored into separate joint distributions on those subsets.\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\n**Bayes' rule** (derived from product rule):\n$$P(b\\mid a) - P(a\\mid b)P(b)/P(a)$$\nMore general version conditionalized on some background evidence $e$:\n$$P(Y\\mid X,e)=P(X\\mid Y,e)P(Y\\mid e)/P(X\\mid e)$$\n$$P(cause\\mid effect) = P(effect\\mid cause) P(cause)/P(effect)$$\n$P(effect\\mid cause)$ quantifies relationship in **causal** direction, whereas $P(cause\\mid effect)$ describes the **diagnostic** direction.\n\nCan avoid assessing prior probability of the evidence by computing posterior for each value of query variable:\n$$P(Y\\mid X) = aP(X\\mid Y)P(Y)$$\nwhere $a$ is the normalization constant needed to make the entries sum to 1.\n\n**conditional independence** $P(X,Y\\mid Z) = P(X\\mid Z)P(Y\\mid Z)$, and $P(X\\mid Y,Z)=P(X\\mid Z)$ and $P(Y\\mid X,Z)=P(Y\\mid Z)$. Conditional independence assertions can allow probabilistic systems to scale up, and they are much more commonly available than independence assertions.\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\n**Naive Bayes model**: assume independence as a simplifying assumption (**Bayesian classifier**). Observed effects $E=e$, remaining effect variables $Y$ are unobserved:\n$$P(cause\\mid e) = a\\sum_y P(cause,e,y)$$\n$$P(cause\\mid e) = a P(cause) \\prod_j P(e_j\\mid cause)$$\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n\nNaive approach for language model: $P(next \\mid history) = P(history + next) / P(history)$ (compute frequency counts). Problem: requires too much data.\n\nChain rule:\n$$P(w_{1:n}) = \\prod_{k=1}^n P(w_k\\mid w_{1:k-1})$$\ndoesn't really help, still hard to estimate $P(w_k\\mid w_{1:k-1})$.\n\nIntuitino of n-gram model: approximate the history by just the last few words. Bigram model approximates $P(w_n\\mid w_{1:n-1})$ with $P(w_n\\mid w_{n-1})$. Assumption that the probability of a word depends only on previous word is called a **Markov** assumption. In general, an N-gram approximates $P(w_n\\mid w_{1:n-1})$ as $P(w_n\\mid w_{n-N+1:n-1})$.\n\nGiven bigram assumption, prob of complete word sequence is $\\prod_{k=1}^n P(w_k\\mid w_{k-1})$.\n\nMake an MLE estimate for the parameters of an n-gram model by getting counts from a corpus, and **normalizing** the counts between 0 and 1. E.g. for bigram: compute count $C(xy)$ of bigram, then\n$$P(w_n\\mid w_{n-1}) = C(w_{n-1}w_n)/C(w_{n-1})$$\nor for N-gram:\n$$P(w_n\\mid w_{n-N+1:n-1}) = C(w_{n-N+1:n-1}w_n)/C(w_{n-N+1:n-1})$$\nThis MLE maximizes the likelihood of the corpus given the model.\n\nPractical considerations: use pseudo-words at start of sentence and marker at end of sentence. Always represent/compute LM probabilities as **log probabilities**.\n\n\n## SLP Chapter 3.2 Evaluating Language Models\n\nTo evaluate LM, can embed in an application and measure how much app improves: **extrinsic evaluation**. But running NLP systems end-to-end is expensive. An **intrinsic evaluation** metric is one that measures the quality of a model independent of any application. We need a **test set** corpus. Whichever model assigns a higher probability to the test set (more accurately predicts the test set) is better. Don't train on test set, this can cause innacuracy in **perplexity**. Sometimes test set is used so frequently we implicity fit to it by choosing models / parameters. Then we can use initial test set **development test set** or **devset**, and use fresh test set for final eval.\n\nShould pick smallest test set to give enough statistical power to measure stat significant difference between the two potential models. In practice 80/10/10 training/devset/test. Can take large chunks or stripes out of corpus to make dif sets.\n\n**Perplexity** (PP) of a LM on test set $W=w_1,...,w_N$ is\n$$PP(W) = P(w_1,...,w_N)^{-(1/N)} = (\\prod_{i=1}^N P(w_i\\mid w_1,...,w_{i-1}))^{-(1/N)}$$\n(estimate $P(w_i\\mid w_1,...,w_{i-1}))$ as $P(w_i\\mid w_{i-n+1:i-1})$ for n-gram model). We want to minimize perplexity.\n\nShould include begin and end of sentence markers in prob comps, also include end-of-sentence marker (but not begin-marker) in total count of word tokens $N$.\n\nPerplexity is the **weighted average branching factor** of a language. Related to concept of entropy. The perplexity of two LMs is only comparable if they use identical vocabularies. Intrinsic improvement in perplexity does not guarantee extrinsic improvement in other task. But usually correlates with such improvements, can be used as a quick check on an algorithm. Should be confirmed with final extrnsic evaluation.\n\n\n## SLP Chapter 3.3 Sampling sentences from a language model\n\nTo visualize what kind of knowledge a language model embodies, can **sample** from it. \n\n\n## SLP Chapter 3.4 Generalization and Zeros\n\nAn n-gram model can bge highly influenced by idiosyncracies and phrases in its test set. Should use a test set that has a similar **genre** to whatever task we are trying to accomplish. Also get training data in the same **dialect** or **variety**.\n\nProblem of **sparsity**: any corpus is limited, so some perfectly acceptable English word sequences are bound to be missing. We'll have zero-probability n-grams that should have nonzero prob. These *zeros* (things that occur in training set but do in test set) causes us to underestimate the prob of all sorts of words that might occur, and also if any prob of any word in the test set is 0, the entire prob of test set is 0; perplexity undefined.\n\nIn a **closed vocabulary** system the test set can only contain words from lexicon of known words that can occur, so no unknown words. In other cases, must deal with words we haven't seen before, called **unknown words** or **out of vocabulary (OOV)** words. Percentage of OOV words in test set is **OOV rate**. An **open vocabulary system** is one in which we model unknown words by adding pseudo-word <UNK>. Can turn problem back into closed vocabulary by:\n1. **choosing a vocabulary** fixed in advance\n2. **converting** any OOV word in train set to unknown word token <UNK>\n3. **estimate** the prob for <UNK> from its counts just like any other regular word\n\nAlternatively, can replace by <UNK> all words that occur fewer than $n$ times in the training set (for some small number $n$) or choose vocab size $V$ in advance and populate vocab with most frequent words. Perplexity can be influenced by <UNK> model, so perplexity should only be compared for models with same vocabulary.\n\n\n## SLP Chapter 3.5 Smoothing\n\nHow to deal with unseen word sequences? Prevent 0 prob by shaving off a bit of prob from some more frequent events and giving it to events we've never seen (called **smoothing** or **discounting**).\n\n**Laplace smoothing**: add 1 to all n-gram counts before normalizing them into probabilities. Does not perform well enough for modern n-gram models, but gives a useful baseline. Define **adjusted count**\n$$c_i^* = (c_i+1)N/(N+V)$$\nSmoothing is like **discounting** (lowering) some non-zero counts to get prob mass that will be assigned to zero counts. Can describe smoothing algorithm in terms of relative **discount** $d_c$:\n$$d_c=c^*/c$$\n\nAlternative: **add-k smoothing**; add fractional count $k$ to each count:\n$$P^*_{Add-k}(w_n\\mid w_{n-1})=(C(w_{n-1}w_n)+k)/(C(w_{n-1}+kV)$$\nchoose $k$ using devset. Still doesn't work very well for LM.\n\nAlternative: **backoff** to using less context when sequence is unseen. Or use **interpolation**, always mixing prob estimates from each n-gram estimator, weighted combo. Can even compute weights $\\lambda$ by conditioning on the context. Learn weigghts from **held-out** corpus. Can use **EM** algorithm, iterative learning algo that converges on optimal $\\lambda$s.\n\nFor backoff model to give correct prob distribution, must **discount** the higher-order n-grams to save some prob mass for lower-order n-grams. Need explicit discount factor and function alpha to distribute prob mass to lower-order n-grams. Backoff with discounting is called **Katz backoff** (reply on discounted prob $P^*$ if we've seen n-gram before, otherwise backoff to Katz prob for shorter history). $P_{BO}(w_n\\mid w_{n-N+1:n-1}) = $P^*(w_n\\mid w_{n-N+1:n-1})$ if $C(w_{n-N+1:n})>0$ else $(alpha)(w_{n-N+1:n-1})P_{BO}(w_{n-N+2:n-1})$. Katz backoff often combined with smoothing method called **Good-Turing**; involves detailed computation for estimating Good-Turing smoothing and the $P^*$ and alpha values.\n\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n\nScalability: store string as 64-bit hash number in memory, probabilities quantized with 4-8 bits, n-grams stored in reverse tries, shrink model by pruning n-grams with counts below threshold, use entropy to prune less important n-grams, build approximate LMs using techniques like **Bloom filters**, use sorted arrays, efficiently combine probs and backoffs in single value, merge sorts to efficiently build prob tables in minimal number of passes through large corpus.\n\nWith very large models, simpler smoothing algorithm may be sufficient: **stupid backoff**; give up goal of true prob distribution; no discounting of higher-order probs. If higher-order n-gram has zero count, backoff to a lower order n-gram weighed by a fixed (context-independent) weight. Does not produce prob dist, so refer to it as $S$. $S(w_i\\mid w_{i-k+1:i-1}) = count(w_{i-k+1:i})/count(w_{i-k+1:i-1})$ if $count(w_{i-k+1:i})>0$ otherwise $\\lambda S(w_i\\mid w_{i-k+2:i-1})$. Backoff terminates in the unigram. $\\lambda=0.4$ works well.\n\n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Language models (lms)** - models that assign probabilities to sequences of words\n* **$n$-gram** - sequence of $n$ words\n* **Relative frequency** - observed frequency of a particular sentence divided by the observed frequency of a prefix\n* **Branching factor of a language** - number of possible next words that can follow any word\n* **Balanced corpora** - has roughly equal number of words from different genres\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\"\n",
        "Independence and conditional independence reduces num probabilities specified for full joint distribution.\n",
        "\n",
        "**Bayesian network** represents dependencies among variables. Can represent any full join prob distribution, often consisely. It is a DAG, each node \\\n",
        "annotated with quant probability info. Full spec:\n",
        "1. Each node represents a random variable (discrete or continuous)\n",
        "2. Directed edges connect nodes. Arrow from $X$ to $Y$ implies $X$ is a *parent* of $Y$.\n",
        "3. Each node $X_i$ has probability info $theta(X_i\\mid Parents(X_i))$ that quantifies effect of parents on the node using a finite number of **parameters**\n",
        "\n",
        "The topology of the network specifies conditional independence relationships. Arrow represents that $X$ has a *direct influence* on $Y$; \\\n",
        "causes should be parents of effects. After topology, specify local probability info for each variable (conditional distribution given parents). \\\n",
        "\n",
        "Local probability info takes the form of a **conditional probability table (CPT)** (for discrete-discrete). Each row: conditional probabilities for a \\\n",
        "**conditioning case** (possible combination of values for parent nodes; miniature possible world). A table for a boolean variable with $k$ boolean parents contains $2^k$ independently \\\n",
        "specifiable probabilities. Node with no parents has only 1 row with prior probabilities.\n",
        "\n",
        "Joint distribution defined as\n",
        "$$P(x_1,...,x_n)=\\prod_{i=1}^n theta(x_i\\mid parents(X_i))$$\n",
        "Each entry in joint distribution represented by product of appropriate elements of local conditional distributions. \\\n",
        "Parameters $theta(x_i\\mid parents(X_i)) = P(x_i\\mid parents(X_i))$ implied by the joint distribution. Hence,\n",
        "$$P(x_1,...,x_n)=\\prod_{i=1}^n P(x_i\\mid parents(X_i))$$\n",
        "when estimating values for local conditional distributions, need to be actual conditional probs for the variable given parents.\n",
        "\n",
        "How to construct Bayes net. Rewrite entries in the joint distribution in terms of conditional probs using product rule: \n",
        "$$P(x_1,...,x_n)=\\prod_{i=1}^n P(x_i\\mid x_{i-1},...,x_1)$$\n",
        "Equivalent to assertion that $P(X_i\\mid X_{i-1},...,X_1)=P(X_i\\mid Parents(X_i))$ if $Parents(X_i)\\subseteq \\{X_{i-1},...,X_1\\}$. Can satisfy by \\\n",
        "numbering nodes according to topological sort of the DAG. Bayes net is correct representation of the domain if each node is conditionally independent of other predecessors in the \\\n",
        "node ordering, given its parents. Instructions:\n",
        "1. Determine set of variables required to model domain. Order them (net will be more compact if causes precede effects). Network structure, number of parameters, and even \\\n",
        "difficulty of filling in individual CPTs, depends on order of introduction. Example of difference between causal and diagnostic models. In causal model, \\\n",
        "can specify fewer numbers and each more easy.\n",
        "2. Choose minimal set of parents for $X_i$ from $X_1,...,X_{i-1}$ so that above-specified condition is satisfied. For each parent insert an edge from the parent to $X_i$. Write down \\\n",
        "conditional probability table $P(X_i\\mid Parents(X_i))$.\n",
        "\"\"\"\n",
        "chapter13_2 = \"\"\"\n",
        "Parents of node $X_i$ should contain all nodes in $X_1,...,X_{i-1}$ that *directly influence* $X_i$.\n",
        "\n",
        "Bayes network can be specified by $2^k n$ numbers, while full joint distribution contains $2^n$ numbers.\n",
        "\n",
        "Often leave out links representing slight dependency (slight gain in accuracy not worth additional complexity in network).\n",
        "\n",
        "Variable is conditionally independent of non-descendants, given parents. Variable is conditionally independent of all other nodes in the network, given its parents, children, and children's parents \\\n",
        "(its **Markov blanket**).\n",
        "\n",
        "A set of nodes $X$ is conditionally independent of another set $Y$, given a third set $Z$, if $Z$ **d-separates** $X$ and $Y$ (otherwise, conditional independence not required):\n",
        "1. Consider just the **ancestral subgraph of $X$, $Y$, $Z$, and ancestors.\n",
        "2. Add links between any unliked pair of nodes that share a common child (now **moral graph**).\n",
        "3. Change directed links to undirected.\n",
        "4. If $Z$ blocks all path between $X$ and $Y$ in the resulting graph, then $Z$ d-separates $X$ and $Y$.\n",
        "\n",
        "Worst case: filling CPT requires $O(2^k)$ numbers and experience with all conditioning cases. Usually, relationships describable by **canonical distribution**, fits \\\n",
        "standard pattern (whole table specified just by pattern and parameters). Example **deterministic node** (value specified by parents, no uncertainty).\n",
        "\n",
        "Uncertain relationships can be characterized by **noisy logical** relation: causal relationship between parent and child can be inhibited. List all possible causes \\\n",
        "(can add **leak node** that covers misc causes) and inhibition of each parent must be independent of inhibition of any other parents. Then only need $O(k)$ parameters instead of $O(2^k)$. \n",
        "\n",
        "Can handle continuous values with discretization (divide possible values into fixed set of intervals; tradeoff loss of accuracy and size of CPT). Alternatively, define continuous variable using \\\n",
        "standard pdf family. Another solution: **nonparametric representation**. \\\n",
        "New kinds of distributions: continuous given discrete (enumerate pdf for each discrete value), \\\n",
        "continuous given continuous (common choice: **linear-Gaussian**),\n",
        "and discrete given continuous (define function mapping continuous value to probability of each value, e.g. \"soft threshold\" integral of standard normal dist aka **probit model**, \\\n",
        "or **logistic function** $1/(1+e^{-x})$ aka **expit** or **inverse logit** model; also can take linear combo of continuous parents).\n",
        "\n",
        "Network containing only continuous variables with linear-Gaussian distributions has \\\n",
        "joint distribution (and posterior given any evidence) that is a multivariate Gaussian distribution. Inference only takes $O(n^3)$ time. \\\n",
        "If discrete variables are added only as parents, network defines **conditional Gaussian** (CG) distribution: given any assignments to the discrete variables, \\\n",
        "distribution over the continuous vars is multivariate Gaussian.\n",
        "\"\"\"\n",
        "chapter13_3 = \"\"\"\n",
        "Inference: compute posterior distribution for set of **query variables** given observed **event** $e$ (e.g. assignment of values to set of **evidence variables** $E$). \\\n",
        "Can easily extend single-variable queries to multivariable queries, so only consider single-variable ($X$) inference. Hidden (nonevidence, nonquery) variables: $Y$.\n",
        "\n",
        "Inference by enumeration ($O(2^n)$ time, linear space complexity):\n",
        "$$P(X\\mid e)=(alpha)P(X,e)=(alpha)\\sum_yP(X,e,y)$$\n",
        "Query can be answered with Bayes net with sums of products of conditional probabilities from net.\n",
        "Enumerate-All$(vars, e)$ computes probability of evidence in $e$ that is also in $vars$. Vars is list of variables we have left to look at, in topological order.\n",
        "\n",
        "**Variable elimination algorithm** uses DP. Time and space requirements dominated by size of largest factor constructed (determined by order of elimination variables and structure of net). \\\n",
        "Intractable to optimally order, but can use heuristics. Greedy: eliminate whichever var minimizes size of next factor to be constructed. Optimization: we can remove any leaf node from \\\n",
        "net if it is not a query variable or evidence variable. Continuing -> every variable that is not an ancestor of a query variable or evidence variable is irrelevant to the query. Variable elimination \\\n",
        "can remove these variables before evaluating the query.\n",
        "\n",
        "For **singly connected networks** or **polytrees** (at most undirected path between any two nodes), the time and space complexity of variable elimination is \\\n",
        "linear in the size of the net (number of CPT entries) for any topological ordering of vars.\n",
        "\n",
        "For **multiply connected networks**, variable elimination has worst case exponential time/space, even when number of parents per node is bounded.\n",
        "\n",
        "Because it includes inference in propositional logic as a special case, inference in Bayes nets is NP-hard (reduce 3-SAT to it). Also #P-hard (strictly harder than NP-complete).\n",
        "\n",
        "Tricks to make CSP solving more efficient (tree width, etc) can be applied to Bayes nets.\n",
        "\n",
        "Weighted model counting (WMC; computes sum of total weight of satisfying assignments, where weight is product of conditional probabilities for each variable assignment given its parents) solver \\\n",
        "can solve translated Bayes net.\n",
        "\n",
        "For multiple variable queries, can use **clustering** (**join tree**) algorithms to get linear time in size of polytree Bayes net. For arbitrary topology, can join individual nodes such that \\\n",
        "the resulting network is a polytree. Two boolean nodes replaced with **meganode** that takes on 4 possible values. Special purpose inference algo \\\n",
        "(that handles meganodes that share vars) like constraint propagation.\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\"\n",
        "Bayes nets aka graphical models.\n",
        "\n",
        "Not every BN can represent every join distribution (topology enforces certain conditional independencies).\n",
        "\n",
        "Topology may happen to encode causal structure, really encodes conditional independence.\n",
        "\n",
        "A joint distribution over $N$ boolean variables is $2^N$ size, an N-node net with each node having up to $k$-parents is $O(N*2^{k+1})$ space.\n",
        "\n",
        "$X$ and $Y$ are conditionally independent given evidence variables $Z$ if $X$ and $Y$ are d-separated by $Z$; consider all (undirected) paths from $X$ to $Y$; \\\n",
        "no active paths = independence. A path is active if each triple is active. All it takes to block a path is a single inactive segment.\n",
        "\n",
        "There might be more independence that can't be infered from structure of BN. Adding arcs increases set of distributions, but has costs. Full conditioning can encode any distribution.\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"conditioning case\": \"a possible combination of values for the parent nodes (a miniature possible world)\",\n",
        "\"conditional probability table (CPT)\": \"each row contains conditional probability of each node value for a conditioning case\",\n",
        "\"locally structured (sparse) system\" : \"each subcomponent interacts directly with only a bounded number of other components irrespective of total size (usually ~linear complexity)\",  \n",
        "\"Context-specific independence (CSI)\": \"variable is conditionally independent of some parents given *certain values* of others (can be specified with if-else syntax)\",  \n",
        "\"nonparametric representation\": \"define conditional distribution implicitly with collection of instances (examples of parent/child values)\",\n",
        "\"hybrid Bayesian network\": \"network with both discrete and continuous variables\",\n",
        "\"linear-Gaussian\": \"conditional distribution; child Gaussian distribution mean $\\mu$ relies linearly on the value of the parent and $\\sigma$ is fixed\",\n",
        "\"hidden variables\": \"in Bayes net: neither input nor output, but keep network reasonably sparse with manageable number of parameters.\",\n",
        "\"conditional probability table (CPT)\": \"CPT\",\n",
        "\"factor (variable elimination)\": \"prob table with unnormalized entries; matrix with one dimension for each variable ($Factor[val1][val2][...]\\propto P(var1=val1,var2=val2,...)$)\",\n",
        "\"pointwise product of factors $f(...)(x)g(...)$\": \"new factor whose variables are union of variables in $f$ and $g$, elements given by product of corresponding elements\",\n",
        "\"summing a variable out of a factor\": \"sum submatrices corresponding to each value of variable\"\n",
        "}\n"
      ],
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2299
        },
        "cellView": "form",
        "outputId": "cf49d34e-a596-4344-9f17-5e053eab76fd"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n\nIndependence and conditional independence reduces num probabilities specified for full joint distribution.\n\n**Bayesian network** represents dependencies among variables. Can represent any full join prob distribution, often consisely. It is a DAG, each node annotated with quant probability info. Full spec:\n1. Each node represents a random variable (discrete or continuous)\n2. Directed edges connect nodes. Arrow from $X$ to $Y$ implies $X$ is a *parent* of $Y$.\n3. Each node $X_i$ has probability info $theta(X_i\\mid Parents(X_i))$ that quantifies effect of parents on the node using a finite number of **parameters**\n\nThe topology of the network specifies conditional independence relationships. Arrow represents that $X$ has a *direct influence* on $Y$; causes should be parents of effects. After topology, specify local probability info for each variable (conditional distribution given parents). \nLocal probability info takes the form of a **conditional probability table (CPT)** (for discrete-discrete). Each row: conditional probabilities for a **conditioning case** (possible combination of values for parent nodes; miniature possible world). A table for a boolean variable with $k$ boolean parents contains $2^k$ independently specifiable probabilities. Node with no parents has only 1 row with prior probabilities.\n\nJoint distribution defined as\n$$P(x_1,...,x_n)=\\prod_{i=1}^n theta(x_i\\mid parents(X_i))$$\nEach entry in joint distribution represented by product of appropriate elements of local conditional distributions. Parameters $theta(x_i\\mid parents(X_i)) = P(x_i\\mid parents(X_i))$ implied by the joint distribution. Hence,\n$$P(x_1,...,x_n)=\\prod_{i=1}^n P(x_i\\mid parents(X_i))$$\nwhen estimating values for local conditional distributions, need to be actual conditional probs for the variable given parents.\n\nHow to construct Bayes net. Rewrite entries in the joint distribution in terms of conditional probs using product rule: \n$$P(x_1,...,x_n)=\\prod_{i=1}^n P(x_i\\mid x_{i-1},...,x_1)$$\nEquivalent to assertion that $P(X_i\\mid X_{i-1},...,X_1)=P(X_i\\mid Parents(X_i))$ if $Parents(X_i)\\subseteq \\{X_{i-1},...,X_1\\}$. Can satisfy by numbering nodes according to topological sort of the DAG. Bayes net is correct representation of the domain if each node is conditionally independent of other predecessors in the node ordering, given its parents. Instructions:\n1. Determine set of variables required to model domain. Order them (net will be more compact if causes precede effects). Network structure, number of parameters, and even difficulty of filling in individual CPTs, depends on order of introduction. Example of difference between causal and diagnostic models. In causal model, can specify fewer numbers and each more easy.\n2. Choose minimal set of parents for $X_i$ from $X_1,...,X_{i-1}$ so that above-specified condition is satisfied. For each parent insert an edge from the parent to $X_i$. Write down conditional probability table $P(X_i\\mid Parents(X_i))$.\n\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n\nParents of node $X_i$ should contain all nodes in $X_1,...,X_{i-1}$ that *directly influence* $X_i$.\n\nBayes network can be specified by $2^k n$ numbers, while full joint distribution contains $2^n$ numbers.\n\nOften leave out links representing slight dependency (slight gain in accuracy not worth additional complexity in network).\n\nVariable is conditionally independent of non-descendants, given parents. Variable is conditionally independent of all other nodes in the network, given its parents, children, and children's parents (its **Markov blanket**).\n\nA set of nodes $X$ is conditionally independent of another set $Y$, given a third set $Z$, if $Z$ **d-separates** $X$ and $Y$ (otherwise, conditional independence not required):\n1. Consider just the **ancestral subgraph of $X$, $Y$, $Z$, and ancestors.\n2. Add links between any unliked pair of nodes that share a common child (now **moral graph**).\n3. Change directed links to undirected.\n4. If $Z$ blocks all path between $X$ and $Y$ in the resulting graph, then $Z$ d-separates $X$ and $Y$.\n\nWorst case: filling CPT requires $O(2^k)$ numbers and experience with all conditioning cases. Usually, relationships describable by **canonical distribution**, fits standard pattern (whole table specified just by pattern and parameters). Example **deterministic node** (value specified by parents, no uncertainty).\n\nUncertain relationships can be characterized by **noisy logical** relation: causal relationship between parent and child can be inhibited. List all possible causes (can add **leak node** that covers misc causes) and inhibition of each parent must be independent of inhibition of any other parents. Then only need $O(k)$ parameters instead of $O(2^k)$. \n\nCan handle continuous values with discretization (divide possible values into fixed set of intervals; tradeoff loss of accuracy and size of CPT). Alternatively, define continuous variable using standard pdf family. Another solution: **nonparametric representation**. New kinds of distributions: continuous given discrete (enumerate pdf for each discrete value), continuous given continuous (common choice: **linear-Gaussian**),\nand discrete given continuous (define function mapping continuous value to probability of each value, e.g. \"soft threshold\" integral of standard normal dist aka **probit model**, or **logistic function** $1/(1+e^{-x})$ aka **expit** or **inverse logit** model; also can take linear combo of continuous parents).\n\nNetwork containing only continuous variables with linear-Gaussian distributions has joint distribution (and posterior given any evidence) that is a multivariate Gaussian distribution. Inference only takes $O(n^3)$ time. If discrete variables are added only as parents, network defines **conditional Gaussian** (CG) distribution: given any assignments to the discrete variables, distribution over the continuous vars is multivariate Gaussian.\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n\nInference: compute posterior distribution for set of **query variables** given observed **event** $e$ (e.g. assignment of values to set of **evidence variables** $E$). Can easily extend single-variable queries to multivariable queries, so only consider single-variable ($X$) inference. Hidden (nonevidence, nonquery) variables: $Y$.\n\nInference by enumeration ($O(2^n)$ time, linear space complexity):\n$$P(X\\mid e)=(alpha)P(X,e)=(alpha)\\sum_yP(X,e,y)$$\nQuery can be answered with Bayes net with sums of products of conditional probabilities from net.\nEnumerate-All$(vars, e)$ computes probability of evidence in $e$ that is also in $vars$. Vars is list of variables we have left to look at, in topological order.\n\n**Variable elimination algorithm** uses DP. Time and space requirements dominated by size of largest factor constructed (determined by order of elimination variables and structure of net). Intractable to optimally order, but can use heuristics. Greedy: eliminate whichever var minimizes size of next factor to be constructed. Optimization: we can remove any leaf node from net if it is not a query variable or evidence variable. Continuing -> every variable that is not an ancestor of a query variable or evidence variable is irrelevant to the query. Variable elimination can remove these variables before evaluating the query.\n\nFor **singly connected networks** or **polytrees** (at most undirected path between any two nodes), the time and space complexity of variable elimination is linear in the size of the net (number of CPT entries) for any topological ordering of vars.\n\nFor **multiply connected networks**, variable elimination has worst case exponential time/space, even when number of parents per node is bounded.\n\nBecause it includes inference in propositional logic as a special case, inference in Bayes nets is NP-hard (reduce 3-SAT to it). Also #P-hard (strictly harder than NP-complete).\n\nTricks to make CSP solving more efficient (tree width, etc) can be applied to Bayes nets.\n\nWeighted model counting (WMC; computes sum of total weight of satisfying assignments, where weight is product of conditional probabilities for each variable assignment given its parents) solver can solve translated Bayes net.\n\nFor multiple variable queries, can use **clustering** (**join tree**) algorithms to get linear time in size of polytree Bayes net. For arbitrary topology, can join individual nodes such that the resulting network is a polytree. Two boolean nodes replaced with **meganode** that takes on 4 possible values. Special purpose inference algo (that handles meganodes that share vars) like constraint propagation.\n\n\n## Module 11 Lecture Notes\n\nBayes nets aka graphical models.\n\nNot every BN can represent every join distribution (topology enforces certain conditional independencies).\n\nTopology may happen to encode causal structure, really encodes conditional independence.\n\nA joint distribution over $N$ boolean variables is $2^N$ size, an N-node net with each node having up to $k$-parents is $O(N*2^{k+1})$ space.\n\n$X$ and $Y$ are conditionally independent given evidence variables $Z$ if $X$ and $Y$ are d-separated by $Z$; consider all (undirected) paths from $X$ to $Y$; no active paths = independence. A path is active if each triple is active. All it takes to block a path is a single inactive segment.\n\nThere might be more independence that can't be infered from structure of BN. Adding arcs increases set of distributions, but has costs. Full conditioning can encode any distribution.\n\n\n## Module 11 Definitions\n* **Conditioning case** - a possible combination of values for the parent nodes (a miniature possible world)\n* **Conditional probability table (cpt)** - CPT\n* **Locally structured (sparse) system** - each subcomponent interacts directly with only a bounded number of other components irrespective of total size (usually ~linear complexity)\n* **Context-specific independence (csi)** - variable is conditionally independent of some parents given *certain values* of others (can be specified with if-else syntax)\n* **Nonparametric representation** - define conditional distribution implicitly with collection of instances (examples of parent/child values)\n* **Hybrid bayesian network** - network with both discrete and continuous variables\n* **Linear-gaussian** - conditional distribution; child Gaussian distribution mean $\\mu$ relies linearly on the value of the parent and $\\sigma$ is fixed\n* **Hidden variables** - in Bayes net: neither input nor output, but keep network reasonably sparse with manageable number of parameters.\n* **Factor (variable elimination)** - prob table with unnormalized entries; matrix with one dimension for each variable ($Factor[val1][val2][...]\\propto P(var1=val1,var2=val2,...)$)\n* **Pointwise product of factors $f(...)(x)g(...)$** - new factor whose variables are union of variables in $f$ and $g$, elements given by product of corresponding elements\n* **Summing a variable out of a factor** - sum submatrices corresponding to each value of variable\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\"\n",
        "Any component of agent program can be improved with ML, depending on factors 1. which component to improve, 2. what prior knowledge the agent has, influencing the model it builds, \\\n",
        "3. what data and feedback on that data is available.\n",
        "\n",
        "Components of agent include \n",
        "1. direct mapping from conditions on current state to actions\n",
        "2. means to infer properties of world from percept sequence\n",
        "3. info about how world evolves and results of actions\n",
        "4. utility information indicating desirability of world states\n",
        "5. action value info indicating desirability of actions\n",
        "6. goals describing most desirable states\n",
        "7. problem generator, critic, and learning element enabling system to improve\n",
        "\n",
        "Several models for agent components: atomic, factored, and relational models based on logic or probability. Learning algos for all.\n",
        "\n",
        "ML assumes little **prior knowledge** of agent; starts from scratch, learns from data.\n",
        "\n",
        "Three types of **feedback** can accompany inputs, determining type of learning:\n",
        "1. **supervised learning**: agent observes input-output (output=**label**) pairs and learns a function that maps input to output\n",
        "2. **unsupervised learning**: agent learns patterns in the input without any explicit feedback (e.g. **clustering**)\n",
        "3. **reinforcement learning**: agent learns from a series of reinforcements (rewards and punishments)\n",
        "\"\"\"\n",
        "chapter19_2 = \"\"\"\n",
        "Given **training set** of $N$ example input-output pairs $\\{(x_i,y_i)\\}_{i=1}^N$ generated by $y_i=f(x_i)$, learn a function $h$ (**hypothesis**, drawn from **hypothesis space** $H$ of possible \\\n",
        "functions) that approximates $f$. \\\n",
        "With alternative vocab, we say $h$ is a **model** (or **function**) of the data, drawn from a **model class** (or **function class**) $H$. Output $y_i$ called **ground truth**.\n",
        "\n",
        "How to choose hypothesis space? Might use prior knowledge, or perform **exploratory data analysis** (examining data with statistical tests and visualizations). Or can try multiple dif hypothesis spaces \\\n",
        "and evaluate which one works best.\n",
        "\n",
        "How do we choose a good hypothesis from $H$? Could hope for **consistent hypothesis**: an $h$ such that each $x_i$ in the training set has $h(x_i)=y_i$. Unattainable with continuous-valued outputs; \\\n",
        "instead, look for **best-fit function** for which each $h(x_i)$ is close to $y_i$.\n",
        "\n",
        "Test of a hypothesis is how it handles inputs it hasn't yet seen. Evaluate test set. $h$ generalizes well if it accurately predicts the outputs of the test set.\n",
        "\n",
        "Can evaluate hypothesis space with the bias they impose (regardless of training dataset) and the variance they produce (from one training set to another).\n",
        "\n",
        "Complexity of model tough to define -- some NNs with billions of parameters can generalize well. Aim for \"appropriateness\", not \"simplicity\" in a model class\n",
        "\n",
        "Supervised learning can be done by choosing the hypothesis $h^*$ that is most probable given the data:\n",
        "$$h^* = argmax_{h\\in H} P(h\\mid data)$$\n",
        "$$h^* = argmax_{h\\in H} P(data\\mid h)P(h)$$\n",
        "(MLE except take into account prior probabilities; do we have some domain knowledge that makes some hypotheses better than others?)\n",
        "\n",
        "Tradeoff between expressiveness of hypothesis space and the computational complexity of finding a good hypothesis within that space (**expressiveness-complexity tradeoff**). \\\n",
        "But an expressive language makes it possible for a simple hypothesis to fit the data, whereas restricting expressiveness of the language might mean that any consistent hypothesis must be complex.\n",
        "\n",
        "Overfitting becomes more likely as number of attributes grows, less likely as we increase the number of training examples. Larger hypothesis spaces have more capacity both to fit and to overfit. \\\n",
        "Some model classes are more prone to overfitting than others.\n",
        "\"\"\"\n",
        "chapter19_3 = \"\"\"\n",
        "A **decision tree** maps vector of attribute values to output (\"decision\"). Reaches decision \\\n",
        "by performing sequence of tests (root to leaf). Each internal node tests an input attribute. I/O can be discrete or continuous.\n",
        "\n",
        "Any function in propositional logic can be expressed as a decision tree. Not all functions can be represented concisely.\n",
        "\n",
        "Intractable to find smallest consistent tree. With heuristics, can find one that is close to smallest. Learn-Decision-Tree algorithm: \\\n",
        "always test the most important attribute first (highest information gain), then recursively solve subproblems.\n",
        "\n",
        "Entropy (measure of uncertainty, units=bits) $H(V)$ of random variable $V$ with values $v_k$ having probability $P(v_k)$:\n",
        "$$H(V)=\\sum_k P(v_k)\\log_2(1/P(v_k)) = -\\sum_k P(v_k)\\log_2P(v_k)$$\n",
        "Define $B(q)$ as entropy of boolean random variable that is true with probability $q$:\n",
        "$$B(q)=-(q\\log_2q+(1-q)\\log_2(1-q))$$\n",
        "If a training set contains $p$ positive examples and $n$ negative examples, entropy of output on whole set is\n",
        "$$H(output) = B(p/(p+n))$$\n",
        "An attribute $A$ with $d$ distinct values divides the training set $E$ into subsets $E_1,...,E_d$, where $E_k$ has $p_k$ positive examples and $n_k$ negative examples. Randomly chosen example from \\\n",
        "training set has the $k$th value for the attribute (i.e. is in $E_k$) with prob $(p_k+n_k)/(p+n)$. So expected entropy remaining after testing $A$ is:\n",
        "$$Remainder(A) = \\sum_{k=1}^dB(p_k/(p_k+n_k))(p_k+n_k)/(p+n)$$\n",
        "**information gain** from attribute test on $A$ is expected reduction in entropy:\n",
        "$$Gain(A)=B(p/(p+n))-Remainder(A)$$\n",
        "\n",
        "**Decision tree pruning** combats overfitting, increases noise tolerance. **Chi-squared pruning**: iteratively eliminate leaf nodes with information gain below threshold, set with **significance test**. \\\n",
        "**null hypothesis**: attribute is irrelevant (zero information gain for infinitely large sample space). Calculate prob that under null hypothesis, sample size $v=n+p$ would show observed deviation by \\\n",
        "comparing $p_k$ and $n_k$ with expected numbers under null: $\\hat{p}_k = p(p_k+n_k)/(p+n)$ and $\\hat{n}_k = n(p_k+n_k)/(p+n)$. Total deviation \\\n",
        "given by:\n",
        "$$del = \\sum_{k=1}^d(p_k-\\hat{p}_k)^2/\\hat{p}_k + (n_k-\\hat{n}_k)^2/\\hat{n}_k$$\n",
        "under null hypothesis, $del$ distributed according to chi-squared distribution with $d-1$ DOF (use $\\chi^2$ to confirm or reject null hypothesis). \\\n",
        "\n",
        "Decision trees can be improved by handling: missing data (not all attribute values known), continuous and multivalued input attributes (**split point** inequality test on value of \\\n",
        "attribute; for discrete attributes with large domains, can use **information gain ratio** to avoid splitting into single-example subtrees, \\\n",
        "allow **equality test**), continuous-valued output attribute (need **regression tree**, each leaf is linear function of subset of numerical attributes, eventually stop splitting and \\\n",
        "apply linear regression on attributes).\n",
        "\"\"\"\n",
        "chapter19_4 = \"\"\"\n",
        "Goal in ML: select a hypothesis that optimally fits future examples. Assume future examples will be like the past (**stationarity** assumption): examples are IID.\n",
        "\n",
        "Finding a good hypothesis involves **model selection** (choose good hypothesis space) and **optimization** (aka **training**) finds best hypothesis within space. Model selection can involve both \\\n",
        "domain knowledge and quant comparison using validation set.\n",
        "\n",
        "As we add capacity to model class, eventually can exactly represent all training examples (model has **interpolated** the data). Model classes start to overfit as \\\n",
        "capacity approaches point of interpolation (stop remembering patterns, just memorize data).\n",
        "\n",
        "DNNs, kernel machines, random forests, and boosted ensembles have property that validation set error tends to decrease as capacity increases.\n",
        "\n",
        "Accuracy not full picture of kinds of errors. Want to minimize **loss function**, amount of utility lost by predicting $h(x)=\\hat{y}$ when correct answer is $f(x)=y$: \\\n",
        "$$L(x,y,\\hat{y}) = Utility(h(x)=y) - Utility(h(x)=\\hat{y})$$\n",
        "often simplified version specified without $x$. $L(y,y)=0$. Can create loss function with table for discrete outputs, but not real outputs. Some loss functions for real outputs:\n",
        "- Absolute-value loss: $L_1(y,\\hat{y})=|y-\\hat{y}|$\n",
        "- Squared-error loss: $L_2(y,\\hat{y})=(y-\\hat{y})^2$\n",
        "- 0/1 loss: $L_{0/1}(y,\\hat{y})=0$ if $y=\\hat{y}$ else 1\n",
        "Expected **generalization loss** for a hypothesis $h$ (where $\\epsilon$ represents set of all possible input-output examples):\n",
        "$$GenLoss_L(h)=\\sum_{(x,y)\\in \\epsilon}L(y,h(x))P(x,y)$$\n",
        "best hypothesis $h^*$ minimizes expected generalization loss. To estimate generalization loss, use **empirical loss** on set of examples $E$ of size $N$:\n",
        "$$EmpLoss_{L,E}(h)=\\sum_{(x,y)\\in E}L(y,h(x))/N$$\n",
        "estimated best hypothesis $\\hat{h}^*$ minimizes empirical loss.\n",
        "\n",
        "$\\hat{h}^*$ may differ from true function $f$ due to unrealizability, variance (->0 as number of training examples increases if problem is realizable), noise, and computational complexity \\\n",
        "(if $H$ is a complicated function in a large hypothesis space, cannot search all possibilities; cannot guarantee finding best hypothesis).\n",
        "\n",
        "Fight overfitting: can minimize weighted sum of empirical loss and complexity (\"total cost\"), with hyperparameter $\\lambda$ (if chosen well, nicely balances empirical loss of simple function \\\n",
        "with complicated function's tendency to overfit):\n",
        "$$Cost(h)=EmpLoss(h)+\\lambda Complexity(h)$$\n",
        "Regularization (penalizing complex hypotheses) involves choosing loss function and the complexity measure, aka **regularization function**. Choice of regularization function depends on hypothesis space.\n",
        "\n",
        "Simplify model by reducing dimensions that the models work with. **Feature selection**: discard irrelevant attributes.\n",
        "\"\"\"\n",
        "chapter19_5 = \"\"\"\n",
        "How many training examples needed? **Computational learning theory**: any seriously wrong hypothesis will by outed with high prob after small number of exmaples (will make incorrect prediction). \\\n",
        "Thus, any hypothesis consistent with sufficiently large set of training examples unlikely to be seriously wrong (must be **probably approximately correct (PAC)**). Learning algo that returns \\\n",
        "PAC hypotheses called a **PAC learning** algo. PAC learning motivated by stationarity assumption. Simplest deal with boolean functions ($L_{0/1}$ appropriate). **error rate** of hypothesis $h$ \\\n",
        "defined as expected generalization error for examples drawn from the stationary distribution, i.e. probability $h$ misclassifies a new example:\n",
        "$$error(h)=GenLoss_{L_{0/1}}(h)=\\sum_{x,y}L_{0/1}(y,h(x))P(x,y)$$\n",
        "Hypothesis $h$ is **approximately correct** if $error(h)<\\epsilon$ for small constant $\\epsilon$ (inside $\\epsilon$-ball around true function $f$, surrounded by $H_{bad}$ section of hypothesis space). \\\n",
        "Can find $N$ such that all consisteny hypotheses approximately correct after training on $N$ examples. Prob seriously bad hypothesis $h_b\\in H_{bad}$ is consistent with first $N$ examples:\n",
        "$$P(h_b agrees with n examples)\\leq (1-\\epsilon)^N$$\n",
        "Can keep probability of $H_{bad}$ containing at least one consistent hypothesis below $\\delta$ if $N\\geq(\\ln(1/\\epsilon)+\\ln|H|)/\\epsilon$. With this $N$, learning algorithm will return an \\\n",
        "approximately correct hypothesis with probability at least $1-\\delta$. The number of required examples is called the **sample complexity**. To obtain real generalization to unseen examples, \\\n",
        "need to restrict hypothesis space $H$ without eliminating true function: use prior knowledge, insist algorithm return *simple* consistent hypothesis, or focus on learnable subsets of entire \\\n",
        "hypothesis space (assuming restricted space contains $h$ that is close enough to $f$).\n",
        "\n",
        "**Decision list** consists of series of tests (each conjunction of literals): if test succeeds when applied to example description, DL specifies value to return. If test fails, continue to next test. \\\n",
        "Can restrict size of each test to at most $k$ literals to force generalization ($k$-DL(n); uses $n$ boolean attributes; includes $k$-DT, set of decision trees of depth at most $k$). We have\n",
        "$$|k-DL(n)|=2^{O(n^k\\log_2(n^k))}$$\n",
        "num examples needed for PAC-learning a $k-DL(n)$ function is polynomial in $n$:\n",
        "$$N\\geq(\\ln(1/\\delta)+O(n^k\\log_2(n_k)))/\\epsilon$$\n",
        "Algo to find consistent decision list: greedily find test that agrees with subset of training set (choose smallest test $t$ that matches any uniformly classified subset), \\\n",
        "then construct remaining decision list with remaining examples (until no examples left).\n",
        "\"\"\"\n",
        "chapter19_6 = \"\"\"\n",
        "**Univariate linear function** (straight line) has form $h_w(x)=w_1x+w_0$ with weight vector $w=[w_0,w_1]$. Finding best $h_w$ to fit data is **linear regression**. $L_2$ loss:\n",
        "$$Loss(h_w)=\\sum_{j=1}^N(y_j-(w_1x_j+w_0))^2$$\n",
        "Unique solution:\n",
        "$$w_1=(N(\\sum x_jy_j) - (\\sum x_j)(\\sum y_j))/(N(\\sum x_j^2)-(\\sum x_j)^2)$$\n",
        "$$w_0=(\\sum y_j - w_1(\\sum x_j))/N$$\n",
        "Plotting loss function in weight space shows loss function is **convex**. This is true for *every* linear regression problem with an $L_2$ loss function (no nonglobal local minima).\n",
        "\n",
        "We can search through a continuous weight space by incrementally modifying parameters (**hill climbing** for maximization, **gradient descent** for minimizing loss). Choose starting point in \\\n",
        "weight space, compute estimate of gradient and $(alpha)(\\delta/\\delta w_i)Loss(w)$ in steepest downhill direction, repeat until converge on point with (local) minimum loss. Parameter alpha is called \\\n",
        "**step size** or **learning rate**. Can be a fixed constant or decay over time as learning proceeds (guarantees convergence). A step that covers all the training examples is called an **epoch**. \\\n",
        "**batch gradient descent** (**deterministic gradient descent**) computes gradient over whole training set. **stochastic gradient descent** (SGD) (**online gradient descent**; useful when new data \\\n",
        "coming in one at a time) randomly selects small number of training examples at each step (traditionally just 1) and updates according to gradient on that set. **Minibatch** gradient descent chooses \\\n",
        "$m$ out of the $N$ examples. Can choose $m$ to take advantage of parallel vector operations, then $m$ is hyperparameter.\n",
        "\n",
        "**Multivariable linear regression**: each example $x_j$ is an $n$-element vector, hypothesis space is of functions of the form $h_W(x_j)=w_0+\\sum_i w_ix_{j,i}$. \\\n",
        "Invent dummy input attribute $x_{j,0}:=1$, then $h_w(x_j)=w\\cdot x_j=w^{T}x_j$. Want to minimize $L_2$ loss. Let $X$ be **data matrix** (one $n$-dimensional example per row):\n",
        "$$L(w)=||\\hat{y}-y||^2=||Xw-y||^2$$\n",
        "$$w^*=(X^TX)^{-1}X^Ty$$\n",
        "this is called the **normal equation**, and $(X^TX)^{-1}$ is called the **pseudoinverse** of the data matrix.\n",
        "\n",
        "Overfitting common on multivariable linear functions. We minimize\n",
        "$$Cost(h)=EmpLoss(h)+\\lambda Complexity(h)$$\n",
        "(equivalent to minimizing $Loss(w)$ subject to $Complexity(w)\\leq c$ for constant $c$ related to $\\lambda$).\n",
        "For linear functions, can consider family of regulization functions\n",
        "$$Complexity(h_w)=L_q(w)=\\sum_i|w_i|^q$$\n",
        "$L_1$ regularization produces **sparse model** (sets many weights to zero, like pruning). Number of examples needed to find good $h$ is logarithmic for $L_1$ and linear for $L_2$ regularization.\n",
        "\"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\"\n",
        "Text classification has many applications. Even language modeling can be seen as classification (each word is a class).\n",
        "\n",
        "Introduce **multinomial naive Bayes classifier**. We represent a text document as a **bag-of-words** (unordered multiset, no position).\n",
        "\n",
        "Choose class $\\hat{x}$ for document $d$ such that \n",
        "$$\\hat{c}=argmax_{c\\in C}P(c\\mid d)=argmax_{c\\in C}P(d\\mid c)P(c)$$\n",
        "generative because it is an assumption about how the document is generated. Can represent document $d$ as set of featrures $f_1,...,f_n$:\n",
        "$$\\hat{c}=argmax_{c\\in C}P(f_1,...,f_n\\mid c)P(c)$$\n",
        "apply **naive Bayes assumption**:\n",
        "$$\\hat{c}=argmax_{c\\in C}P(c\\mid d)=argmax_{c\\in C}P(c)\\prod_{f\\in F}P(f\\mid c)$$\n",
        "to apply to text with word positions:\n",
        "$$c_{NB}=argmax_{c\\in C}P(c)\\prod_{i\\in positions}P(w_i\\mid c)$$\n",
        "done in log space to avoid underflow and increase speed\n",
        "$$c_{NB}=argmax_{c\\in C}\\log P(c)\\sum_{i\\in positions}\\log P(w_i\\mid c)$$\n",
        "in log space, this is linear combo of input features; NB is a linear classifier.\n",
        "\"\"\"\n",
        "slp_chapter4_2 = \"\"\"\n",
        "Learn $P(c)$ and $P(f_i\\mid c)$ with MLE ($N_c$ number of documents in training data with class $c$ and $N_{doc}$ total documents):\n",
        "$$\\hat{P}(x)=N_c/N_{doc}$$\n",
        "for $P(f_i\\mid c)$, assume feature is just existence of word in document's bag of words:\n",
        "$$\\hat{P}(w_i\\mid c)=count(w_i,c)/(\\sum_{w\\in V}count(w,c))$$\n",
        "vocabulary $V$ consists of union of all word types in all classes\n",
        "\n",
        "if word does not occur, probability of feature is 0, so product is 0. Can use add-one (Laplace) smoothing, common in NB:\n",
        "$$\\hat{P}(w_i\\mid c)=(count(w_i,c)+1)/((\\sum_{w\\in V}count(w,c))+|V|)$$\n",
        "\n",
        "Laplace smoothing does poorly for $P(X\\mid Y)$ when $|X|$ is very large or $|Y|$ is very large. Another option: linear interpolation\n",
        "$$P_{LIN}(x\\mid y)=(alpha)\\hat{P}(x\\mid y)+(1-alpha)\\hat{P}(x)$$\n",
        "\n",
        "ignore **unknown words** in test data that didn't show up in training. Also can ignore **stop words** (very frequent, e.g. the and a); \\\n",
        "sort vocabulary by frequency and define top words as stop words, or use predefined stop word list. Usually removing stop words doesn't improve performance.\n",
        "\"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\"\n",
        "For sentiment analysis, helps to clip word counts in each document at 1 (**binary multinomial naive Bayes** or **binary NB**). Also, helps to \\\n",
        "deal with negation; prepend NOT_ prefix to all words affected by logical negation term.\n",
        "\n",
        "Can also derive positive and negative word features from **sentiment lexicons**, lists of words that are pre-annotated with positive or negative sentiment. \\\n",
        "Add a feature that is counted whenever a word from that lexicon occurs. Adding these features works well only if training data is sparse and non-representative of test set. \\\n",
        "Using dense lexicon features instead of sparse individual-word features may generalize better.\n",
        "\"\"\"\n",
        "slp_chapter4_5 = \"\"\"\n",
        "Spam detection: rather than using all words as individual features, predefine likely sets of words or phrases as features, combined with features that are not purely linguistic.\n",
        "\n",
        "Character n-grams or byte n-gramscan be good features for other applications.\n",
        "\"\"\"\n",
        "slp_chapter4_6 = \"\"\"\n",
        "If features are just words and we use all words in text, NB model can be viewed as a set of class-specific unigram language models, in which the model for each class \\\n",
        "instantiates a unigram language model. Model assigns a probability to each sentence $P(s\\mid c)=\\prod_{i\\in positions}P(w_i\\mid c)$.\n",
        "\"\"\"\n",
        "slp_chapter4_7 = \"\"\"\n",
        "To evaluate system for detecting things, can build a **confusion matrix** (visualizes how an algorithm performs with respect to human gold labels); dimensions are \\\n",
        "system output (positive, negative) and gold labels (positive, negative). \n",
        "\n",
        "**Accuracy**: what percentage of observations are correct: $(tp+tn)/(tp+fp+tn+fn)$. Doesn't work well when classes are unbalanced (e.g. with rare events, detection).\n",
        "\n",
        "**Precision** measures the percentage of items that the system detected that are in fact positive: $tp/(tp+fp)$\n",
        "\n",
        "**Recall** measures the percentage of items actually present in the input that were correctly identified by the system: $tp/(tp+fn)$.\n",
        "\n",
        "Precision and recall emphasize true positives.\n",
        "\n",
        "Combination of precision and recall: F-measure\n",
        "$$F_b=(b^2+1)PR/(b^2P+R)$$\n",
        "where the $b$ (beta) parameter weighs importance of precision and recall, based on needs of application. $b>1$ favor recall, $b<1$ favor precision. $b=1$ balanced, called $F_1$. \\\n",
        "Weighs lower of two numbers more heavily.\n",
        "\n",
        "Multiclass: more cells in confusion matrix. In **macroaveraging**, we compute the performance for each class, and then average over classes. In **microaveraging**, we \\\n",
        "collect the decisions for all classes into a single confusion matrix, and compute precision and recall from that table. Microaverage dominated by more frequent class, \\\n",
        "macroaverage better when performance on all the classes is equally important.\n",
        "\"\"\"\n",
        "slp_chapter4_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_9 = \"\"\"\n",
        "Let's call $M(A,x)$ the score system $A$ gets on test set $x$, and $\\delta(x)$ the performance difference between $A$ and $B$ on $x$:\n",
        "$$\\delta(x)=M(A,x)-M(B,x)$$\n",
        "$\\delta(x)$ is called the **effect size**. Would like to know if effect size is positive. Null hypothesis: effect size is non-positive. $p$-value is $P(\\delta(X)\\geq \\delta(x)\\mid H_0)$. \\\n",
        "Very small p-value means difference observed is very unlikely under $H_0$. If test passes p-value threshold, then the result \"$A$ is better than $B$\" is **statistically significant**.\n",
        "\n",
        "In NLP, common to estimate $P(\\delta(X)\\geq \\delta(x)\\mid H_0)$ using non-parametric tests based on sampling: measure $\\delta(x')$ for lots of different test sets $x'$, \\\n",
        "which gives us a distribution.\n",
        "\n",
        "Two common non-parametric tests in NLP: **approximate randomization** and the **bootstrap test**. **Paired** tests are those in which we compare two sets of observations that are aligned \\\n",
        "(each observation in one set can be paired with an observation in another).\n",
        "\n",
        "**bootstraping**: repeatedly drawing large numbers of smaller samples with replacement (called **bootstrap samples**) from an original larger sample\n",
        "\n",
        "**Paired bootstrap test**: create many virtual test sets from an observed test set by repeatedly sampling from it.\n",
        "$$p-value(x)=(\\sum_{i=1}^b int(\\delta(x^{(i)} \\geq 2\\delta(x)))/b$$\n",
        "\"\"\"\n",
        "slp_chapter4_10 = \"\"\"\n",
        "Release **model card** with info about how model was trained, motivation, intended use, users, and performance across different demographic or other groups and environments.\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\"\n",
        "For classification, classes separated by **decision boundary**, linear decision boundary is called a **linear separator** (compliant data are **linearly separable**). \\\n",
        "Write classification hypothesis $h_w(x) = 1$ if $w\\cdot x\\geq 0$ else 0. Equivalently, $h_w(x)=Threshold(w\\cdot x)$ where $Threshold(z)=int(z\\geq 0)$. \\\n",
        "Choose weights $w$ to minimize the loss. Weight update rule that converges to a solution if the data are linearly separable:\n",
        "$$w_i += (alpha)(y-h_w(x)) x_i$$\n",
        "this is called the **perceptron learning rule**, typically applied one random example at a time. \\\n",
        "If alpha decays as $O(1/t)$ with iteration number $t$, perceptron rule will converge to a minimum-error solution (NP hard) even if data is not linearly separable.\n",
        "\n",
        "Soften threshold function, use logistic function:\n",
        "$$h_w(x)=Logistic(w\\cdot x)=1/(1+\\exp(-w\\cdot x))$$\n",
        "for **logistic regression**, use gradient descent with $L_2$ loss function. For logistic function $g$,\n",
        "$h_w'(x) = h_w(x)(1-h_w(x))$$\n",
        "update rule\n",
        "$$w_i+=(alpha)(y-h_w(x))\\cdot h_w(x)(1-h_w(x))\\cdot x_i$$\n",
        "logistic regression far better when data is noisy / not perfectly linearly separable.\n",
        "\n",
        "Often useful to use naive (or strong) baseline model to compare against new model.\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"transfer learning\" : \"knowledge from one domain is transferred to a new domain, so learning can proceed faster with less data\",\n",
        "\"induction\": \"going from a specific set of observations to a general rule (may be incorrect)\",\n",
        "\"factored representation\": \"vector of attribute values\",\n",
        "\"classification\": \"output is one of finite set of values\",\n",
        "\"regression\": \"output is a number\",\n",
        "\"bias\": \"tendency of a predictive hypothesis to deviate from expected value when averaged over different training sets (low complexity-> high bias)\",\n",
        "\"hypothesis is underfitting\": \"when it fails to find a pattern in the data (too high bias)\",\n",
        "\"variance\": \"amount of change in the hypothesis due to fluctuation in the training data\",\n",
        "\"hypothesis is overfitting\": \"when it pays too much attention to particular dataset it's trained on (performs poorly on unseen data); too high variance\",\n",
        "\"bias-variance tradeoff\": \"choice between more complex, low-bias hypotheses (fit training data well) and simpler, low-variance hypotheses that generalize better\",\n",
        "\"Boolean classification\": \"have outputs that are either true (**positive example**) or false (**negative example**)\",\n",
        "\"Boolean decision tree\": \"decision tree with discrete inputs and boolean output\",\n",
        "\"learning curve\": \"used to evaluate the performance of a learning algorithm; accuracy vs training set size; usually increasing (also called **happy graphs**)\",\n",
        "\"problems with decision trees\": \"suboptimal accuracy, expensive if deep, **unstable** (adding one new example can change entire tree); **Random forest model** improves\",\n",
        "\"hyperparameters\": \"tunable parameters specifying the model class, as opposed to normal parameters (specify the model)\",\n",
        "\"k-fold cross validation\": \"split train data into $k$ pieces, do $k$ rounds of learning (a dif piece held out as validation set each round), consider average test score\",\n",
        "\"leave-one-out cross-validation (LOOCV)\": \"k-fold cross-validation with $k=n$\",\n",
        "\"learning problem is realizable\": \"if the hypothesis space $H$ actually contains the true function $f$\",\n",
        "\"small-scale learning\": \"(traditional methods) dozens to thousands of training examples; generalization loss from approximation error and estimation error not enough examples\",\n",
        "\"large-scale learning\": \"(more modern) millions of training example; generalization loss dominated by limits of computation\",\n",
        "\"minimum description length (MDL) hypothesis\": \"complexity ~num bits of Turing machine program. empirical loss: num bits to encode data (correct=0, wrong depends on error size)\",\n",
        "\"(hyperparam tuning) hand tuning\": \"manual hyperparameter tuning\",\n",
        "\"(hyperparam tuning) grid search\": \"exhaustive search of all possible combos of hyperparam values\",\n",
        "\"(hyperparam tuning) random search\": \"randomly select from set of hyperparameter values, useful if too many combos for exhaustive search (or continuous)\",\n",
        "\"(hyperparam tuning) Bayesian optimization\": \"trades off exploration and exploitation (staying near best values) UCS can be used to miminize regret\",\n",
        "\"(hyperparam tuning) Population-based training (PBT)\": \"train group of models in parallel with dif hyperparams, choose next-gen based on previous results, random mutation (like genetic algo)\",\n",
        "\"training curve\": \"measures classifier performance on a fixed training set as the learning process proceeds one update at a time\",\n",
        "\"text categorization\": \"task of assigning a label or category to entire text or document\",\n",
        "\"sentiment\": \"positive or nevative orientation that a writer expresses towards some subject\",\n",
        "\"sentiment analysis\": \"extraction of sentiment from a text\",\n",
        "\"probabilistic classifier\": \"additionally will tell us the probability of the observation being in the class\",\n",
        "\"generative classifiers\": \"(e.g. naive Bayes) build a model of how a class could generate some input data\",\n",
        "\"discriminative classifiers\": \"(e.g. logistic regression) learn what features from the input are most useful to discriminate between the different possible classes\",\n",
        "\"linear classifier\": \"classifiers that use a linear combination of the inputs to make a classification decision\",\n",
        "\"representational harms\": \"harms caused by a system that demeans a social group, e.g. by perpetuating negative stereotype\"\n",
        "}\n",
        "\n"
      ],
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 6028
        },
        "cellView": "form",
        "outputId": "8b41e225-d8b5-46bd-f1ba-47a40e324a79"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\nAny component of agent program can be improved with ML, depending on factors 1. which component to improve, 2. what prior knowledge the agent has, influencing the model it builds, 3. what data and feedback on that data is available.\n\nComponents of agent include \n1. direct mapping from conditions on current state to actions\n2. means to infer properties of world from percept sequence\n3. info about how world evolves and results of actions\n4. utility information indicating desirability of world states\n5. action value info indicating desirability of actions\n6. goals describing most desirable states\n7. problem generator, critic, and learning element enabling system to improve\n\nSeveral models for agent components: atomic, factored, and relational models based on logic or probability. Learning algos for all.\n\nML assumes little **prior knowledge** of agent; starts from scratch, learns from data.\n\nThree types of **feedback** can accompany inputs, determining type of learning:\n1. **supervised learning**: agent observes input-output (output=**label**) pairs and learns a function that maps input to output\n2. **unsupervised learning**: agent learns patterns in the input without any explicit feedback (e.g. **clustering**)\n3. **reinforcement learning**: agent learns from a series of reinforcements (rewards and punishments)\n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n\nGiven **training set** of $N$ example input-output pairs $\\{(x_i,y_i)\\}_{i=1}^N$ generated by $y_i=f(x_i)$, learn a function $h$ (**hypothesis**, drawn from **hypothesis space** $H$ of possible functions) that approximates $f$. With alternative vocab, we say $h$ is a **model** (or **function**) of the data, drawn from a **model class** (or **function class**) $H$. Output $y_i$ called **ground truth**.\n\nHow to choose hypothesis space? Might use prior knowledge, or perform **exploratory data analysis** (examining data with statistical tests and visualizations). Or can try multiple dif hypothesis spaces and evaluate which one works best.\n\nHow do we choose a good hypothesis from $H$? Could hope for **consistent hypothesis**: an $h$ such that each $x_i$ in the training set has $h(x_i)=y_i$. Unattainable with continuous-valued outputs; instead, look for **best-fit function** for which each $h(x_i)$ is close to $y_i$.\n\nTest of a hypothesis is how it handles inputs it hasn't yet seen. Evaluate test set. $h$ generalizes well if it accurately predicts the outputs of the test set.\n\nCan evaluate hypothesis space with the bias they impose (regardless of training dataset) and the variance they produce (from one training set to another).\n\nComplexity of model tough to define -- some NNs with billions of parameters can generalize well. Aim for \"appropriateness\", not \"simplicity\" in a model class\n\nSupervised learning can be done by choosing the hypothesis $h^*$ that is most probable given the data:\n$$h^* = argmax_{h\\in H} P(h\\mid data)$$\n$$h^* = argmax_{h\\in H} P(data\\mid h)P(h)$$\n(MLE except take into account prior probabilities; do we have some domain knowledge that makes some hypotheses better than others?)\n\nTradeoff between expressiveness of hypothesis space and the computational complexity of finding a good hypothesis within that space (**expressiveness-complexity tradeoff**). But an expressive language makes it possible for a simple hypothesis to fit the data, whereas restricting expressiveness of the language might mean that any consistent hypothesis must be complex.\n\nOverfitting becomes more likely as number of attributes grows, less likely as we increase the number of training examples. Larger hypothesis spaces have more capacity both to fit and to overfit. Some model classes are more prone to overfitting than others.\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n\nA **decision tree** maps vector of attribute values to output (\"decision\"). Reaches decision by performing sequence of tests (root to leaf). Each internal node tests an input attribute. I/O can be discrete or continuous.\n\nAny function in propositional logic can be expressed as a decision tree. Not all functions can be represented concisely.\n\nIntractable to find smallest consistent tree. With heuristics, can find one that is close to smallest. Learn-Decision-Tree algorithm: always test the most important attribute first (highest information gain), then recursively solve subproblems.\n\nEntropy (measure of uncertainty, units=bits) $H(V)$ of random variable $V$ with values $v_k$ having probability $P(v_k)$:\n$$H(V)=\\sum_k P(v_k)\\log_2(1/P(v_k)) = -\\sum_k P(v_k)\\log_2P(v_k)$$\nDefine $B(q)$ as entropy of boolean random variable that is true with probability $q$:\n$$B(q)=-(q\\log_2q+(1-q)\\log_2(1-q))$$\nIf a training set contains $p$ positive examples and $n$ negative examples, entropy of output on whole set is\n$$H(output) = B(p/(p+n))$$\nAn attribute $A$ with $d$ distinct values divides the training set $E$ into subsets $E_1,...,E_d$, where $E_k$ has $p_k$ positive examples and $n_k$ negative examples. Randomly chosen example from training set has the $k$th value for the attribute (i.e. is in $E_k$) with prob $(p_k+n_k)/(p+n)$. So expected entropy remaining after testing $A$ is:\n$$Remainder(A) = \\sum_{k=1}^dB(p_k/(p_k+n_k))(p_k+n_k)/(p+n)$$\n**information gain** from attribute test on $A$ is expected reduction in entropy:\n$$Gain(A)=B(p/(p+n))-Remainder(A)$$\n\n**Decision tree pruning** combats overfitting, increases noise tolerance. **Chi-squared pruning**: iteratively eliminate leaf nodes with information gain below threshold, set with **significance test**. **null hypothesis**: attribute is irrelevant (zero information gain for infinitely large sample space). Calculate prob that under null hypothesis, sample size $v=n+p$ would show observed deviation by comparing $p_k$ and $n_k$ with expected numbers under null: $\\hat{p}_k = p(p_k+n_k)/(p+n)$ and $\\hat{n}_k = n(p_k+n_k)/(p+n)$. Total deviation given by:\n$$del = \\sum_{k=1}^d(p_k-\\hat{p}_k)^2/\\hat{p}_k + (n_k-\\hat{n}_k)^2/\\hat{n}_k$$\nunder null hypothesis, $del$ distributed according to chi-squared distribution with $d-1$ DOF (use $\\chi^2$ to confirm or reject null hypothesis). \nDecision trees can be improved by handling: missing data (not all attribute values known), continuous and multivalued input attributes (**split point** inequality test on value of attribute; for discrete attributes with large domains, can use **information gain ratio** to avoid splitting into single-example subtrees, allow **equality test**), continuous-valued output attribute (need **regression tree**, each leaf is linear function of subset of numerical attributes, eventually stop splitting and apply linear regression on attributes).\n\n\n## AIMA Chapter 5.5 Stochastic Games\n\nGoal in ML: select a hypothesis that optimally fits future examples. Assume future examples will be like the past (**stationarity** assumption): examples are IID.\n\nFinding a good hypothesis involves **model selection** (choose good hypothesis space) and **optimization** (aka **training**) finds best hypothesis within space. Model selection can involve both domain knowledge and quant comparison using validation set.\n\nAs we add capacity to model class, eventually can exactly represent all training examples (model has **interpolated** the data). Model classes start to overfit as capacity approaches point of interpolation (stop remembering patterns, just memorize data).\n\nDNNs, kernel machines, random forests, and boosted ensembles have property that validation set error tends to decrease as capacity increases.\n\nAccuracy not full picture of kinds of errors. Want to minimize **loss function**, amount of utility lost by predicting $h(x)=\\hat{y}$ when correct answer is $f(x)=y$: $$L(x,y,\\hat{y}) = Utility(h(x)=y) - Utility(h(x)=\\hat{y})$$\noften simplified version specified without $x$. $L(y,y)=0$. Can create loss function with table for discrete outputs, but not real outputs. Some loss functions for real outputs:\n- Absolute-value loss: $L_1(y,\\hat{y})=|y-\\hat{y}|$\n- Squared-error loss: $L_2(y,\\hat{y})=(y-\\hat{y})^2$\n- 0/1 loss: $L_{0/1}(y,\\hat{y})=0$ if $y=\\hat{y}$ else 1\nExpected **generalization loss** for a hypothesis $h$ (where $\\epsilon$ represents set of all possible input-output examples):\n$$GenLoss_L(h)=\\sum_{(x,y)\\in \\epsilon}L(y,h(x))P(x,y)$$\nbest hypothesis $h^*$ minimizes expected generalization loss. To estimate generalization loss, use **empirical loss** on set of examples $E$ of size $N$:\n$$EmpLoss_{L,E}(h)=\\sum_{(x,y)\\in E}L(y,h(x))/N$$\nestimated best hypothesis $\\hat{h}^*$ minimizes empirical loss.\n\n$\\hat{h}^*$ may differ from true function $f$ due to unrealizability, variance (->0 as number of training examples increases if problem is realizable), noise, and computational complexity (if $H$ is a complicated function in a large hypothesis space, cannot search all possibilities; cannot guarantee finding best hypothesis).\n\nFight overfitting: can minimize weighted sum of empirical loss and complexity (\"total cost\"), with hyperparameter $\\lambda$ (if chosen well, nicely balances empirical loss of simple function with complicated function's tendency to overfit):\n$$Cost(h)=EmpLoss(h)+\\lambda Complexity(h)$$\nRegularization (penalizing complex hypotheses) involves choosing loss function and the complexity measure, aka **regularization function**. Choice of regularization function depends on hypothesis space.\n\nSimplify model by reducing dimensions that the models work with. **Feature selection**: discard irrelevant attributes.\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nHow many training examples needed? **Computational learning theory**: any seriously wrong hypothesis will by outed with high prob after small number of exmaples (will make incorrect prediction). Thus, any hypothesis consistent with sufficiently large set of training examples unlikely to be seriously wrong (must be **probably approximately correct (PAC)**). Learning algo that returns PAC hypotheses called a **PAC learning** algo. PAC learning motivated by stationarity assumption. Simplest deal with boolean functions ($L_{0/1}$ appropriate). **error rate** of hypothesis $h$ defined as expected generalization error for examples drawn from the stationary distribution, i.e. probability $h$ misclassifies a new example:\n$$error(h)=GenLoss_{L_{0/1}}(h)=\\sum_{x,y}L_{0/1}(y,h(x))P(x,y)$$\nHypothesis $h$ is **approximately correct** if $error(h)<\\epsilon$ for small constant $\\epsilon$ (inside $\\epsilon$-ball around true function $f$, surrounded by $H_{bad}$ section of hypothesis space). Can find $N$ such that all consisteny hypotheses approximately correct after training on $N$ examples. Prob seriously bad hypothesis $h_b\\in H_{bad}$ is consistent with first $N$ examples:\n$$P(h_b agrees with n examples)\\leq (1-\\epsilon)^N$$\nCan keep probability of $H_{bad}$ containing at least one consistent hypothesis below $\\delta$ if $N\\geq(\\ln(1/\\epsilon)+\\ln|H|)/\\epsilon$. With this $N$, learning algorithm will return an approximately correct hypothesis with probability at least $1-\\delta$. The number of required examples is called the **sample complexity**. To obtain real generalization to unseen examples, need to restrict hypothesis space $H$ without eliminating true function: use prior knowledge, insist algorithm return *simple* consistent hypothesis, or focus on learnable subsets of entire hypothesis space (assuming restricted space contains $h$ that is close enough to $f$).\n\n**Decision list** consists of series of tests (each conjunction of literals): if test succeeds when applied to example description, DL specifies value to return. If test fails, continue to next test. Can restrict size of each test to at most $k$ literals to force generalization ($k$-DL(n); uses $n$ boolean attributes; includes $k$-DT, set of decision trees of depth at most $k$). We have\n$$|k-DL(n)|=2^{O(n^k\\log_2(n^k))}$$\nnum examples needed for PAC-learning a $k-DL(n)$ function is polynomial in $n$:\n$$N\\geq(\\ln(1/\\delta)+O(n^k\\log_2(n_k)))/\\epsilon$$\nAlgo to find consistent decision list: greedily find test that agrees with subset of training set (choose smallest test $t$ that matches any uniformly classified subset), then construct remaining decision list with remaining examples (until no examples left).\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\n**Univariate linear function** (straight line) has form $h_w(x)=w_1x+w_0$ with weight vector $w=[w_0,w_1]$. Finding best $h_w$ to fit data is **linear regression**. $L_2$ loss:\n$$Loss(h_w)=\\sum_{j=1}^N(y_j-(w_1x_j+w_0))^2$$\nUnique solution:\n$$w_1=(N(\\sum x_jy_j) - (\\sum x_j)(\\sum y_j))/(N(\\sum x_j^2)-(\\sum x_j)^2)$$\n$$w_0=(\\sum y_j - w_1(\\sum x_j))/N$$\nPlotting loss function in weight space shows loss function is **convex**. This is true for *every* linear regression problem with an $L_2$ loss function (no nonglobal local minima).\n\nWe can search through a continuous weight space by incrementally modifying parameters (**hill climbing** for maximization, **gradient descent** for minimizing loss). Choose starting point in weight space, compute estimate of gradient and $(alpha)(\\delta/\\delta w_i)Loss(w)$ in steepest downhill direction, repeat until converge on point with (local) minimum loss. Parameter alpha is called **step size** or **learning rate**. Can be a fixed constant or decay over time as learning proceeds (guarantees convergence). A step that covers all the training examples is called an **epoch**. **batch gradient descent** (**deterministic gradient descent**) computes gradient over whole training set. **stochastic gradient descent** (SGD) (**online gradient descent**; useful when new data coming in one at a time) randomly selects small number of training examples at each step (traditionally just 1) and updates according to gradient on that set. **Minibatch** gradient descent chooses $m$ out of the $N$ examples. Can choose $m$ to take advantage of parallel vector operations, then $m$ is hyperparameter.\n\n**Multivariable linear regression**: each example $x_j$ is an $n$-element vector, hypothesis space is of functions of the form $h_W(x_j)=w_0+\\sum_i w_ix_{j,i}$. Invent dummy input attribute $x_{j,0}:=1$, then $h_w(x_j)=w\\cdot x_j=w^{T}x_j$. Want to minimize $L_2$ loss. Let $X$ be **data matrix** (one $n$-dimensional example per row):\n$$L(w)=||\\hat{y}-y||^2=||Xw-y||^2$$\n$$w^*=(X^TX)^{-1}X^Ty$$\nthis is called the **normal equation**, and $(X^TX)^{-1}$ is called the **pseudoinverse** of the data matrix.\n\nOverfitting common on multivariable linear functions. We minimize\n$$Cost(h)=EmpLoss(h)+\\lambda Complexity(h)$$\n(equivalent to minimizing $Loss(w)$ subject to $Complexity(w)\\leq c$ for constant $c$ related to $\\lambda$).\nFor linear functions, can consider family of regulization functions\n$$Complexity(h_w)=L_q(w)=\\sum_i|w_i|^q$$\n$L_1$ regularization produces **sparse model** (sets many weights to zero, like pruning). Number of examples needed to find good $h$ is logarithmic for $L_1$ and linear for $L_2$ regularization.\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\nText classification has many applications. Even language modeling can be seen as classification (each word is a class).\n\nIntroduce **multinomial naive Bayes classifier**. We represent a text document as a **bag-of-words** (unordered multiset, no position).\n\nChoose class $\\hat{x}$ for document $d$ such that \n$$\\hat{c}=argmax_{c\\in C}P(c\\mid d)=argmax_{c\\in C}P(d\\mid c)P(c)$$\ngenerative because it is an assumption about how the document is generated. Can represent document $d$ as set of featrures $f_1,...,f_n$:\n$$\\hat{c}=argmax_{c\\in C}P(f_1,...,f_n\\mid c)P(c)$$\napply **naive Bayes assumption**:\n$$\\hat{c}=argmax_{c\\in C}P(c\\mid d)=argmax_{c\\in C}P(c)\\prod_{f\\in F}P(f\\mid c)$$\nto apply to text with word positions:\n$$c_{NB}=argmax_{c\\in C}P(c)\\prod_{i\\in positions}P(w_i\\mid c)$$\ndone in log space to avoid underflow and increase speed\n$$c_{NB}=argmax_{c\\in C}\\log P(c)\\sum_{i\\in positions}\\log P(w_i\\mid c)$$\nin log space, this is linear combo of input features; NB is a linear classifier.\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n\nLearn $P(c)$ and $P(f_i\\mid c)$ with MLE ($N_c$ number of documents in training data with class $c$ and $N_{doc}$ total documents):\n$$\\hat{P}(x)=N_c/N_{doc}$$\nfor $P(f_i\\mid c)$, assume feature is just existence of word in document's bag of words:\n$$\\hat{P}(w_i\\mid c)=count(w_i,c)/(\\sum_{w\\in V}count(w,c))$$\nvocabulary $V$ consists of union of all word types in all classes\n\nif word does not occur, probability of feature is 0, so product is 0. Can use add-one (Laplace) smoothing, common in NB:\n$$\\hat{P}(w_i\\mid c)=(count(w_i,c)+1)/((\\sum_{w\\in V}count(w,c))+|V|)$$\n\nLaplace smoothing does poorly for $P(X\\mid Y)$ when $|X|$ is very large or $|Y|$ is very large. Another option: linear interpolation\n$$P_{LIN}(x\\mid y)=(alpha)\\hat{P}(x\\mid y)+(1-alpha)\\hat{P}(x)$$\n\nignore **unknown words** in test data that didn't show up in training. Also can ignore **stop words** (very frequent, e.g. the and a); sort vocabulary by frequency and define top words as stop words, or use predefined stop word list. Usually removing stop words doesn't improve performance.\n\n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n\nFor sentiment analysis, helps to clip word counts in each document at 1 (**binary multinomial naive Bayes** or **binary NB**). Also, helps to deal with negation; prepend NOT_ prefix to all words affected by logical negation term.\n\nCan also derive positive and negative word features from **sentiment lexicons**, lists of words that are pre-annotated with positive or negative sentiment. Add a feature that is counted whenever a word from that lexicon occurs. Adding these features works well only if training data is sparse and non-representative of test set. Using dense lexicon features instead of sparse individual-word features may generalize better.\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n\nSpam detection: rather than using all words as individual features, predefine likely sets of words or phrases as features, combined with features that are not purely linguistic.\n\nCharacter n-grams or byte n-gramscan be good features for other applications.\n\n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n\nIf features are just words and we use all words in text, NB model can be viewed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. Model assigns a probability to each sentence $P(s\\mid c)=\\prod_{i\\in positions}P(w_i\\mid c)$.\n\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n\nTo evaluate system for detecting things, can build a **confusion matrix** (visualizes how an algorithm performs with respect to human gold labels); dimensions are system output (positive, negative) and gold labels (positive, negative). \n\n**Accuracy**: what percentage of observations are correct: $(tp+tn)/(tp+fp+tn+fn)$. Doesn't work well when classes are unbalanced (e.g. with rare events, detection).\n\n**Precision** measures the percentage of items that the system detected that are in fact positive: $tp/(tp+fp)$\n\n**Recall** measures the percentage of items actually present in the input that were correctly identified by the system: $tp/(tp+fn)$.\n\nPrecision and recall emphasize true positives.\n\nCombination of precision and recall: F-measure\n$$F_b=(b^2+1)PR/(b^2P+R)$$\nwhere the $b$ (beta) parameter weighs importance of precision and recall, based on needs of application. $b>1$ favor recall, $b<1$ favor precision. $b=1$ balanced, called $F_1$. Weighs lower of two numbers more heavily.\n\nMulticlass: more cells in confusion matrix. In **macroaveraging**, we compute the performance for each class, and then average over classes. In **microaveraging**, we collect the decisions for all classes into a single confusion matrix, and compute precision and recall from that table. Microaverage dominated by more frequent class, macroaverage better when performance on all the classes is equally important.\n\n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n\nLet's call $M(A,x)$ the score system $A$ gets on test set $x$, and $\\delta(x)$ the performance difference between $A$ and $B$ on $x$:\n$$\\delta(x)=M(A,x)-M(B,x)$$\n$\\delta(x)$ is called the **effect size**. Would like to know if effect size is positive. Null hypothesis: effect size is non-positive. $p$-value is $P(\\delta(X)\\geq \\delta(x)\\mid H_0)$. Very small p-value means difference observed is very unlikely under $H_0$. If test passes p-value threshold, then the result \"$A$ is better than $B$\" is **statistically significant**.\n\nIn NLP, common to estimate $P(\\delta(X)\\geq \\delta(x)\\mid H_0)$ using non-parametric tests based on sampling: measure $\\delta(x')$ for lots of different test sets $x'$, which gives us a distribution.\n\nTwo common non-parametric tests in NLP: **approximate randomization** and the **bootstrap test**. **Paired** tests are those in which we compare two sets of observations that are aligned (each observation in one set can be paired with an observation in another).\n\n**bootstraping**: repeatedly drawing large numbers of smaller samples with replacement (called **bootstrap samples**) from an original larger sample\n\n**Paired bootstrap test**: create many virtual test sets from an observed test set by repeatedly sampling from it.\n$$p-value(x)=(\\sum_{i=1}^b int(\\delta(x^{(i)} \\geq 2\\delta(x)))/b$$\n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n\nRelease **model card** with info about how model was trained, motivation, intended use, users, and performance across different demographic or other groups and environments.\n\n\n## Module 12 Lecture Notes\n\nFor classification, classes separated by **decision boundary**, linear decision boundary is called a **linear separator** (compliant data are **linearly separable**). Write classification hypothesis $h_w(x) = 1$ if $w\\cdot x\\geq 0$ else 0. Equivalently, $h_w(x)=Threshold(w\\cdot x)$ where $Threshold(z)=int(z\\geq 0)$. Choose weights $w$ to minimize the loss. Weight update rule that converges to a solution if the data are linearly separable:\n$$w_i += (alpha)(y-h_w(x)) x_i$$\nthis is called the **perceptron learning rule**, typically applied one random example at a time. If alpha decays as $O(1/t)$ with iteration number $t$, perceptron rule will converge to a minimum-error solution (NP hard) even if data is not linearly separable.\n\nSoften threshold function, use logistic function:\n$$h_w(x)=Logistic(w\\cdot x)=1/(1+\\exp(-w\\cdot x))$$\nfor **logistic regression**, use gradient descent with $L_2$ loss function. For logistic function $g$,\n$h_w'(x) = h_w(x)(1-h_w(x))$$\nupdate rule\n$$w_i+=(alpha)(y-h_w(x))\\cdot h_w(x)(1-h_w(x))\\cdot x_i$$\nlogistic regression far better when data is noisy / not perfectly linearly separable.\n\nOften useful to use naive (or strong) baseline model to compare against new model.\n\n\n## Module 12 Definitions\n* **Transfer learning** - knowledge from one domain is transferred to a new domain, so learning can proceed faster with less data\n* **Induction** - going from a specific set of observations to a general rule (may be incorrect)\n* **Factored representation** - vector of attribute values\n* **Classification** - output is one of finite set of values\n* **Regression** - output is a number\n* **Bias** - tendency of a predictive hypothesis to deviate from expected value when averaged over different training sets (low complexity-> high bias)\n* **Hypothesis is underfitting** - when it fails to find a pattern in the data (too high bias)\n* **Variance** - amount of change in the hypothesis due to fluctuation in the training data\n* **Hypothesis is overfitting** - when it pays too much attention to particular dataset it's trained on (performs poorly on unseen data); too high variance\n* **Bias-variance tradeoff** - choice between more complex, low-bias hypotheses (fit training data well) and simpler, low-variance hypotheses that generalize better\n* **Boolean classification** - have outputs that are either true (**positive example**) or false (**negative example**)\n* **Boolean decision tree** - decision tree with discrete inputs and boolean output\n* **Learning curve** - used to evaluate the performance of a learning algorithm; accuracy vs training set size; usually increasing (also called **happy graphs**)\n* **Problems with decision trees** - suboptimal accuracy, expensive if deep, **unstable** (adding one new example can change entire tree); **Random forest model** improves\n* **Hyperparameters** - tunable parameters specifying the model class, as opposed to normal parameters (specify the model)\n* **K-fold cross validation** - split train data into $k$ pieces, do $k$ rounds of learning (a dif piece held out as validation set each round), consider average test score\n* **Leave-one-out cross-validation (loocv)** - k-fold cross-validation with $k=n$\n* **Learning problem is realizable** - if the hypothesis space $H$ actually contains the true function $f$\n* **Small-scale learning** - (traditional methods) dozens to thousands of training examples; generalization loss from approximation error and estimation error not enough examples\n* **Large-scale learning** - (more modern) millions of training example; generalization loss dominated by limits of computation\n* **Minimum description length (mdl) hypothesis** - complexity ~num bits of Turing machine program. empirical loss: num bits to encode data (correct=0, wrong depends on error size)\n* **(hyperparam tuning) hand tuning** - manual hyperparameter tuning\n* **(hyperparam tuning) grid search** - exhaustive search of all possible combos of hyperparam values\n* **(hyperparam tuning) random search** - randomly select from set of hyperparameter values, useful if too many combos for exhaustive search (or continuous)\n* **(hyperparam tuning) bayesian optimization** - trades off exploration and exploitation (staying near best values) UCS can be used to miminize regret\n* **(hyperparam tuning) population-based training (pbt)** - train group of models in parallel with dif hyperparams, choose next-gen based on previous results, random mutation (like genetic algo)\n* **Training curve** - measures classifier performance on a fixed training set as the learning process proceeds one update at a time\n* **Text categorization** - task of assigning a label or category to entire text or document\n* **Sentiment** - positive or nevative orientation that a writer expresses towards some subject\n* **Sentiment analysis** - extraction of sentiment from a text\n* **Probabilistic classifier** - additionally will tell us the probability of the observation being in the class\n* **Generative classifiers** - (e.g. naive Bayes) build a model of how a class could generate some input data\n* **Discriminative classifiers** - (e.g. logistic regression) learn what features from the input are most useful to discriminate between the different possible classes\n* **Linear classifier** - classifiers that use a linear combination of the inputs to make a classification decision\n* **Representational harms** - harms caused by a system that demeans a social group, e.g. by perpetuating negative stereotype\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\"\n",
        "Note sigmoid satisfies $1-\\sigma(x)=\\sigma(-x)$.\n",
        "\n",
        "Logistic regression classifier: $P(y=1)=\\sigma(w\\cdot x+b)$, $decision(x)$ = 1 if $P(y=1\\mid x) > 0.5$ else 0 (**decision boundary**).\n",
        "\n",
        "Feature engineering. For NB and logistic regression, can combine primitive features into more complex features or **feature interactions** (designed by hand). \\\n",
        "For many tasks, need large number of features, can be created automatically via **feature templates**. **Representation learning**, ways to learn features automatically \\\n",
        "in an unsupervised way.\n",
        "\n",
        "Logistic regression more accurate than NB when many correlated features (usually better for large datasets). NB can work very well on small datasets or short documents, \\\n",
        "easy to implement and fast to train.\n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\n",
        "slp_chapter5_3 = \"\"\"\n",
        "Need loss function $L(\\hat{y},y)$ to specify how much $\\hat{y}$ differs from  true $y$. We want log function that prefers correct class labels of training examples to be *more likely*. \\\n",
        "**Conditional maximum likelihood estimation**: choose parameters $w,b$ that **maximize the log probability of the true y labels in the training data** given the observations $x$. \\\n",
        "Resulting function is the log likelihood loss, aka **cross entropy loss**:\n",
        "$$L_{CE}(\\hat{y},y)=-[y\\log\\sigma(w\\cdot x+b)+(1-y)\\log(1-\\sigma(w\\cdot x+b))]$$\n",
        "**cross entropy** between true probability dist $y$ and our estimated prob dist $\\hat{y}$: $y\\log(\\hat{y})+(1-y)\\log(1-\\hat{y})$.\n",
        "\n",
        "Solve with gradient descent. Loss function is convex. Gradient of cross entropy loss function for logistic regression:\n",
        "$$(\\delta/\\delta w_j)L_{CE}(\\hat{y},y)=[\\sigma(w\\cdot x + b)-y]x_j$$\n",
        "\"\"\"\n",
        "slp_chapter5_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_6 = \"\"\"\n",
        "**Multinomial logistic regression** aka **softmax regression**, want to find $P(y=c\\mid x)$ for each $c\\in C$. Uses softmax to map vector to vector of probabilities:\n",
        "$$softmax(z_i)=\\exp(z_i)/\\sum_{j=1}^d\\exp(z_j)$$\n",
        "We have\n",
        "$$p(y=c\\mid x)=\\exp(w_c\\cdot x+b_c)/(\\sum_{j=1}^K\\exp(w_j\\cdot x+b_j))$$\n",
        "Loss\n",
        "$$L_{CE}(\\hat{y},y)=-\\sum_{k=1}^K y_k\\log \\hat{P}(y=k\\mid x)$$\n",
        "$y$ is a **one-hot vector** (only one component has a 1, rest 0). So $L_{CE}(\\hat{y},y)=-\\log \\hat{y}_k$ where $k$ is the correct class;\n",
        "$$L_{CE}(\\hat{y},y) = -\\log (\\exp(w_k\\cdot x + b_k)/\\sum_{j=1}^K\\exp(w_j\\cdot x+b_j))$$\n",
        "The gradient \n",
        "$$(\\delta/\\delta w_{k,i}) = -(int(y=k)-P(y=k\\mid x))x_i$$\n",
        "\n",
        "Can investigate which features are significant with statistical tests, or inspecting magnitude of weight.\n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\"\n",
        "Building block of neural net is **unit**, taking set of real valued numbers as input, performs some computation, then produces an output. Neural unit:\n",
        "$$z=w\\cdot x + b$$\n",
        "then apply non-linear function $f$ to $z$ to get **activation** value.\n",
        "\n",
        "Common activation functions: sigmoid, tanh, ReLU.\n",
        "\n",
        "Sigmoid and tanh have problem that values of $y$ can become **saturated** (extremely close to 1, derivative close to 0), causing **vanishing gradient**.\n",
        "\n",
        "Perceptron acts as linear **decision boundary**, and XOR is not **linearly separable**.\n",
        "\n",
        "Neural net with nonlinear activation function can model XOR (linear activation function translates to 1-layer net).\n",
        "\"\"\"\n",
        "slp_chapter7_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_3 = \"\"\"\n",
        "**Feedforward network** (**multi-layer perceptron** or MLP) is a multilayer network in which units are connected with no cycles, outputs passed strictly forward.\n",
        "\n",
        "Has input layer, hideden layers, output layer. Standard architecture: layers fully connected. Parameters for hidden layer: weight matrix $W$ and bias vector $b$ for whole layer. \\\n",
        "$W_{ji}$ represents weight of connection from $i$th input unit $x_i$ to the $j$th hidden unit $h_j$. Hidden layer:\n",
        "$$h = \\sigma(Wx+b)$$\n",
        "\n",
        "One hidden layer case:\n",
        "Input layer is layer 0, $n_0$ is number of inputs, so $x\\in R^{n_0}$, hidden layer has $h\\in R^{n_1}$. Weight matrix $W\\in R^{n_1*n_0}$. Hidden layer computes **representation** of input. \\\n",
        "Output layer has weight matrix $U$, produces intermediate output $z=Uh$, then normalize with softmax function:\" $y = softmax(z)$.\n",
        "\n",
        "General notation: superscripts in square brackets mean layer number, starting at 0 for input layer. $n_j$ means number of units at layer $j$. $a^{[i]} means output from layer $i$, \\\n",
        "and $z^{[i]}$ represents combination of weights and biases at that layer.\n",
        "\n",
        "To represent bias, add dummy node $a^{[i]}_0:=1$ so $h=\\sigma(Wx)$.\n",
        "\n",
        "In NLP: represent text inputs to NNs as word embeddings $[e_{w_1};,,,;e_{e_w_n}]$ -- **pretraining**.\n",
        "\"\"\"\n",
        "slp_chapter7_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_5 = \"\"\"\n",
        "Neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction. \\\n",
        "On the other hand, they are more complex, slower to train, and less interpretable. For many smaller tasks n-gram model can still be the right tool.\n",
        "\n",
        "Neural LM takes context as input and outputs word probabilities. Model stores patterns learned from training data, representing words as embeddings.\n",
        "\n",
        "Feedforward NLM has moving window that can see $N$ words into the past. Multiply each one-hot vector (word) by embedding weight matrix $E$ (column for each word, $d$ rows); \\\n",
        "multiplying by one-hot vector selects column vector (word embedding) for that word. Then concatenate embedding vectors to produce $e$, the embedding layer. Follow by hidden layer and an output \\\n",
        "layer whose softmax produces prob distribution.\n",
        "\"\"\"\n",
        "slp_chapter7_6 = \"\"\"\n",
        "Loss function for NN: use cross entropy loss (same as logistic regression). Train with gradient descent. Compute gradient w.r.t. parameters with **error backpropagation** or \\\n",
        "**backwards differentiation**.\n",
        "$$L_{CE}(\\hat{y},y) = -\\log \\hat{y}_i$$\n",
        "where $i$ is the correct class.\n",
        "\n",
        "Computation graph like NN. **Backwards pass** / backwards differentiation: just use chain rule.\n",
        "\n",
        "Useful derivatives:\n",
        "$$(\\delta/\\delta z)\\sigma(z) = \\sigma(z)(1-\\sigma(z))$$\n",
        "$$(\\delta/\\delta z)tanh(z) = 1-tanh^2(z)$$\n",
        "$$(\\delta/\\delta z)ReLU(z) = 0 for z<0, 1 for z\\geq 0$$\n",
        "\n",
        "Optimization in NNs is a non-convex optimization problem, more complex than for logistic regression. Should initialize weights with small random numbers. Helpful to \\\n",
        "normalize input values to have 0 mean and unit variance. Regularization technique: **dropout**, randomly dropping some units and their connections from the NN during training. \\\n",
        "Tuning hyperparameters is also important (learning rate, mini-batch size, model architecture, number of hidden nodes per layer, the choice of activation functions, how to regularize, etc). \\\n",
        "Modern NNs are optimmized for parallelization on vector-based GPUs.\n",
        "\"\"\"\n",
        "slp_chapter7_7 = \"\"\"\n",
        "OK to **freeze** embedding layer $E$ (use some pretraining algorithm, then hold $E$ constant while modifying params of NN).\n",
        "\n",
        "Can also train NN and $E$ together. Make sure weights from document window one hot vectors to embedding layer share *same* $E$ (not dif for each one hot vector). $E$ can then be used for other tasks.\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\"\n",
        "Can improve perceptrons by adjusting weight update. MIRA: choose update size that fixes current mistake but minimizes change to $w$:\n",
        "$$\\min_w (\\sum_y ||w_y-w_y'||^2)/2$$\n",
        "$$w_y^*\\cdot f(x) \\geq w_y\\cdot f(x)+1$$\n",
        "\n",
        "Another solution: cap step size with constant $C$ (optimization that assumes non-separable data); usually converges faster than perceptron. Usually better, especially on noisy data.\n",
        "\n",
        "**Support Vector Machines (SVMs)** maximize margin; only support vectors matter, other training examples are ignorable.\n",
        "\n",
        "Term document matrix: rows are words, columns are documents, cells are counts.\n",
        "\n",
        "Term-term matrix (term-context matrix): value in cell indicates how often word appears in a context window surrounding other word.\n",
        "\n",
        "Alternative to counts: TF-IDF (term frequency in a document) / (number of documents term occured in)\n",
        "\n",
        "Train classifier to predict neighboring words, and learn dense embeddings for words in the training corpus. Skip gram: positive examples from context window, negative examples (noise words) \\\n",
        "from outside window. Pick noise words according to \n",
        "$$P_a(w)=count(w)^a/\\sum_w count(w)^a$$\n",
        "$a=3/4$ works well (rare noise words slightly higher prob)\n",
        "\n",
        "Word embeddings can solve analogies, vector arithmetic, visualizations.\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {  \n",
        "    \"sigmoid function\": \"\\sigma(z)=1/(1+\\exp(-z))\",\n",
        "    \"tanh function\": \"variant of sigmoid that ranges [-1,1]: $y=(e^z-e^{-z})/(e^z+e^{-z})$\",\n",
        "    \"ReLU function\": \"simplest / most common activation function: $y = \\max(z,0)$\",\n",
        "}\n"
      ],
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2638
        },
        "cellView": "form",
        "outputId": "a141899c-fa66-4b5e-e3c2-d3720161d7db"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\nNote sigmoid satisfies $1-\\sigma(x)=\\sigma(-x)$.\n\nLogistic regression classifier: $P(y=1)=\\sigma(w\\cdot x+b)$, $decision(x)$ = 1 if $P(y=1\\mid x) > 0.5$ else 0 (**decision boundary**).\n\nFeature engineering. For NB and logistic regression, can combine primitive features into more complex features or **feature interactions** (designed by hand). For many tasks, need large number of features, can be created automatically via **feature templates**. **Representation learning**, ways to learn features automatically in an unsupervised way.\n\nLogistic regression more accurate than NB when many correlated features (usually better for large datasets). NB can work very well on small datasets or short documents, easy to implement and fast to train.\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\nNeed loss function $L(\\hat{y},y)$ to specify how much $\\hat{y}$ differs from  true $y$. We want log function that prefers correct class labels of training examples to be *more likely*. **Conditional maximum likelihood estimation**: choose parameters $w,b$ that **maximize the log probability of the true y labels in the training data** given the observations $x$. Resulting function is the log likelihood loss, aka **cross entropy loss**:\n$$L_{CE}(\\hat{y},y)=-[y\\log\\sigma(w\\cdot x+b)+(1-y)\\log(1-\\sigma(w\\cdot x+b))]$$\n**cross entropy** between true probability dist $y$ and our estimated prob dist $\\hat{y}$: $y\\log(\\hat{y})+(1-y)\\log(1-\\hat{y})$.\n\nSolve with gradient descent. Loss function is convex. Gradient of cross entropy loss function for logistic regression:\n$$(\\delta/\\delta w_j)L_{CE}(\\hat{y},y)=[\\sigma(w\\cdot x + b)-y]x_j$$\n\n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n\n**Multinomial logistic regression** aka **softmax regression**, want to find $P(y=c\\mid x)$ for each $c\\in C$. Uses softmax to map vector to vector of probabilities:\n$$softmax(z_i)=\\exp(z_i)/\\sum_{j=1}^d\\exp(z_j)$$\nWe have\n$$p(y=c\\mid x)=\\exp(w_c\\cdot x+b_c)/(\\sum_{j=1}^K\\exp(w_j\\cdot x+b_j))$$\nLoss\n$$L_{CE}(\\hat{y},y)=-\\sum_{k=1}^K y_k\\log \\hat{P}(y=k\\mid x)$$\n$y$ is a **one-hot vector** (only one component has a 1, rest 0). So $L_{CE}(\\hat{y},y)=-\\log \\hat{y}_k$ where $k$ is the correct class;\n$$L_{CE}(\\hat{y},y) = -\\log (\\exp(w_k\\cdot x + b_k)/\\sum_{j=1}^K\\exp(w_j\\cdot x+b_j))$$\nThe gradient \n$$(\\delta/\\delta w_{k,i}) = -(int(y=k)-P(y=k\\mid x))x_i$$\n\nCan investigate which features are significant with statistical tests, or inspecting magnitude of weight.\n\n\n## SLP Chapter 7.1 Units\n\nBuilding block of neural net is **unit**, taking set of real valued numbers as input, performs some computation, then produces an output. Neural unit:\n$$z=w\\cdot x + b$$\nthen apply non-linear function $f$ to $z$ to get **activation** value.\n\nCommon activation functions: sigmoid, tanh, ReLU.\n\nSigmoid and tanh have problem that values of $y$ can become **saturated** (extremely close to 1, derivative close to 0), causing **vanishing gradient**.\n\nPerceptron acts as linear **decision boundary**, and XOR is not **linearly separable**.\n\nNeural net with nonlinear activation function can model XOR (linear activation function translates to 1-layer net).\n\n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n\n**Feedforward network** (**multi-layer perceptron** or MLP) is a multilayer network in which units are connected with no cycles, outputs passed strictly forward.\n\nHas input layer, hideden layers, output layer. Standard architecture: layers fully connected. Parameters for hidden layer: weight matrix $W$ and bias vector $b$ for whole layer. $W_{ji}$ represents weight of connection from $i$th input unit $x_i$ to the $j$th hidden unit $h_j$. Hidden layer:\n$$h = \\sigma(Wx+b)$$\n\nOne hidden layer case:\nInput layer is layer 0, $n_0$ is number of inputs, so $x\\in R^{n_0}$, hidden layer has $h\\in R^{n_1}$. Weight matrix $W\\in R^{n_1*n_0}$. Hidden layer computes **representation** of input. Output layer has weight matrix $U$, produces intermediate output $z=Uh$, then normalize with softmax function:\" $y = softmax(z)$.\n\nGeneral notation: superscripts in square brackets mean layer number, starting at 0 for input layer. $n_j$ means number of units at layer $j$. $a^{[i]} means output from layer $i$, and $z^{[i]}$ represents combination of weights and biases at that layer.\n\nTo represent bias, add dummy node $a^{[i]}_0:=1$ so $h=\\sigma(Wx)$.\n\nIn NLP: represent text inputs to NNs as word embeddings $[e_{w_1};,,,;e_{e_w_n}]$ -- **pretraining**.\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n\nNeural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction. On the other hand, they are more complex, slower to train, and less interpretable. For many smaller tasks n-gram model can still be the right tool.\n\nNeural LM takes context as input and outputs word probabilities. Model stores patterns learned from training data, representing words as embeddings.\n\nFeedforward NLM has moving window that can see $N$ words into the past. Multiply each one-hot vector (word) by embedding weight matrix $E$ (column for each word, $d$ rows); multiplying by one-hot vector selects column vector (word embedding) for that word. Then concatenate embedding vectors to produce $e$, the embedding layer. Follow by hidden layer and an output layer whose softmax produces prob distribution.\n\n\n## SLP Chapter 7.6 Training Neural Nets\n\nLoss function for NN: use cross entropy loss (same as logistic regression). Train with gradient descent. Compute gradient w.r.t. parameters with **error backpropagation** or **backwards differentiation**.\n$$L_{CE}(\\hat{y},y) = -\\log \\hat{y}_i$$\nwhere $i$ is the correct class.\n\nComputation graph like NN. **Backwards pass** / backwards differentiation: just use chain rule.\n\nUseful derivatives:\n$$(\\delta/\\delta z)\\sigma(z) = \\sigma(z)(1-\\sigma(z))$$\n$$(\\delta/\\delta z)tanh(z) = 1-tanh^2(z)$$\n$$(\\delta/\\delta z)ReLU(z) = 0 for z<0, 1 for z\\geq 0$$\n\nOptimization in NNs is a non-convex optimization problem, more complex than for logistic regression. Should initialize weights with small random numbers. Helpful to normalize input values to have 0 mean and unit variance. Regularization technique: **dropout**, randomly dropping some units and their connections from the NN during training. Tuning hyperparameters is also important (learning rate, mini-batch size, model architecture, number of hidden nodes per layer, the choice of activation functions, how to regularize, etc). Modern NNs are optimmized for parallelization on vector-based GPUs.\n\n\n## SLP Chapter 7.7 Training the neural language model\n\nOK to **freeze** embedding layer $E$ (use some pretraining algorithm, then hold $E$ constant while modifying params of NN).\n\nCan also train NN and $E$ together. Make sure weights from document window one hot vectors to embedding layer share *same* $E$ (not dif for each one hot vector). $E$ can then be used for other tasks.\n\n\n## Module 13 Lecture Notes\n\nCan improve perceptrons by adjusting weight update. MIRA: choose update size that fixes current mistake but minimizes change to $w$:\n$$\\min_w (\\sum_y ||w_y-w_y'||^2)/2$$\n$$w_y^*\\cdot f(x) \\geq w_y\\cdot f(x)+1$$\n\nAnother solution: cap step size with constant $C$ (optimization that assumes non-separable data); usually converges faster than perceptron. Usually better, especially on noisy data.\n\n**Support Vector Machines (SVMs)** maximize margin; only support vectors matter, other training examples are ignorable.\n\nTerm document matrix: rows are words, columns are documents, cells are counts.\n\nTerm-term matrix (term-context matrix): value in cell indicates how often word appears in a context window surrounding other word.\n\nAlternative to counts: TF-IDF (term frequency in a document) / (number of documents term occured in)\n\nTrain classifier to predict neighboring words, and learn dense embeddings for words in the training corpus. Skip gram: positive examples from context window, negative examples (noise words) from outside window. Pick noise words according to \n$$P_a(w)=count(w)^a/\\sum_w count(w)^a$$\n$a=3/4$ works well (rare noise words slightly higher prob)\n\nWord embeddings can solve analogies, vector arithmetic, visualizations.\n\n\n## Module 13 Definitions\n* **Sigmoid function** - \\sigma(z)=1/(1+\\exp(-z))\n* **Tanh function** - variant of sigmoid that ranges [-1,1]: $y=(e^z-e^{-z})/(e^z+e^{-z})$\n* **Relu function** - simplest / most common activation function: $y = \\max(z,0)$\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_10 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_11 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1238
        },
        "outputId": "ae045420-3711-4864-800f-416dadc52c23"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\n\nCan improve perceptrons by adjusting weight update. MIRA: choose update size that fixes current mistake but minimizes change to $w$:\n$$\\min_w (\\sum_y ||w_y-w_y'||^2)/2$$\n$$w_y^*\\cdot f(x) \\geq w_y\\cdot f(x)+1$$\n\nAnother solution: cap step size with constant $C$ (optimization that assumes non-separable data); usually converges faster than perceptron. Usually better, especially on noisy data.\n\n**Support Vector Machines (SVMs)** maximize margin; only support vectors matter, other training examples are ignorable.\n\nTerm document matrix: rows are words, columns are documents, cells are counts.\n\nTerm-term matrix (term-context matrix): value in cell indicates how often word appears in a context window surrounding other word.\n\nAlternative to counts: TF-IDF (term frequency in a document) / (number of documents term occured in)\n\nTrain classifier to predict neighboring words, and learn dense embeddings for words in the training corpus. Skip gram: positive examples from context window, negative examples (noise words) from outside window. Pick noise words according to \n$$P_a(w)=count(w)^a/\\sum_w count(w)^a$$\n$a=3/4$ works well (rare noise words slightly higher prob)\n\nWord embeddings can solve analogies, vector arithmetic, visualizations.\n\n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 21701
        },
        "id": "T3cm1WV542_a",
        "outputId": "d73f75a2-1687-4d97-9510-41c830cb7816"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nAgent has available actions $a$, assigns a probability $P(s)$ to state s, transition model given by $P(s'\\mid s,a)$, prob of reaching state $s'$ by taking action $a$ in state $s$. The probability of landing in a state $s'$ is: $$P(RESULT(a)=s')=\\sum_sP(s)P(s'\\mid s,a)$$\n\n**Decision theory** deals with choosing among actions based on the desirability of their *immediate* outcomes (episodic environment). Agent's preferences captured by a **utility function** $U(s)$ which assigns a single number to express the desirability of a state. The **expected utility** of an action given evidence is: $$EU(a) = \\sum_{s'}P(RESULT(a) = s')U(s')$$\n\nThe **principle of maximum utility** says that a rational agent should choose the action that maximizes the agent's expected utility (prescription for intelligent behavior); formalizes the goal, but does not operationalize it. Estimating $P(RESULT(a)=s')$ and utility function can be hard.\n\nIf the agent acts to maximize utility function that correctly reflects the performance measure, then they will achieve the highest possible average performance score (utility function applies only to next state, performance measure gives a score for a history; sequence of states).\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nNotation: $A\\succ B$ (agent prefers A over B), $A\\sim B$ (agent is indifferent between A and B), $A\\succcurlyeq B$ (agent prefers A over B or is indifferent). A and B are lotteries. A lottery $L$ with possible outcomes $S_1,...,S_n$ that occur with probabilities $p_1,...,p_n$ is written $L = [p_1,S_1;...;p_n,S_n]$. Each outcome $S_i$ can be an atomic state or another lottery. \n\nSix axioms of utility theory:\n1. **Orderability**: given any two lotteries, an agent must prefer one or rate them as equal (exactly one of $A\\succ B$, $B\\succ A$, or $A\\sim B$ holds).\n2. **Transitivity**: $(A\\succ B)\\land(B\\succ C)\\implies(A\\succ C)$\n3. **Continuity**: $A\\succ B\\succ C\\implies\\exists p : [p,A;(1-p),C]~B$\n4. **Substitutability**: $A\\sim B\\implies[p,A;(1-p),C]\\sim [p,B;(1-p),C]$\n5. **Monotonicity**: $A\\succ B\\implies (p>q \\iff [p,A;(1-p),B]\\succ[q,A;(1-q),B])$\n6. **Decomposibility** (no fun in gambling rule): $[p,A;(1-p),[q,B;(1-q),C]]\\sim[p,A;(1-p)q,B;(1-p)(1-q),C]$\n\nEach axiom can be motivated by showing that an agent that violates it will exhibit obviously irrational behavior in some situations.\n\nCan derive the following from the axioms of utility:\n1. **Existence of a utility function**: if an agent's preferences obey axioms of utilty, $\\exists U: U(A) > U(B) \\iff A\\succ B, U(A)=U(B) \\iff A\\sim B$\n2. **Expected utility of a lottery**: $U([p_1,S_1;...;p_n,S_n]) = \\sum_ip_iU(S_i)$\n\nThe outcome of nondeterministic action is a lottery, so an agent can act rationally (consistently with its preferences) only by following MEU principle.\n\nAgent's behavior would not change if utility function were transformed according to $U'(S) := aU(S) + b$, where $a$ and $b$ are constants and $a>0$ (positive affine transformation). In a deterministic environment an agent needs only a preference ranking on states (the numbers don't matter). This is called a **value function** or **ordinal utility function**.\n\n\n## AIMA Chapter 16.3 Utility Functions\n\nDetermining an agent's utility function is called **preference elicitation** (present choices and use observed preferences to determine underlying utility function).\n\nTypically we fix worst outcome at $u_{bot}$ and best outcome at $u_{top}$ (**normalized utilities** use $u_{bot}=0, u_{top}=1$). Can assess utility of prize $S$ by asking agent to choose between $S$ and a **standard lottery** $[p,u_{top}; (1-p),u_{bot}]$ (binary search with $p$ to find utility of $S$).\n\nGovt agencies use **value of a statistical life**. A **micromort** is value of a 1-in-million chance of death. Another measure is quality-adjusted life-year (**QALY**).\n\nMoney may or may not work as an accurate utility function (could have **monotonic preference** but violate characteristics of utility function in lotteries). Can compute **expected monetary value** (EMV) of a lottery. Utility function is often roughly logarithm of money (diminishing marginal utility). Diminishing marginal utility equivalent to **risk aversion** (utility of lottery less than EMV), increasing marginal utility equivalent to **risk seeking**, linear equivalent to **risk neutral**. The value an agent will accept in lieu of a lottery is called the **certainty equivalent** of the lottery. Difference between EMV and certainty equivalent is called the **insurance premium** (risk aversion basis for insurance industry). \nAgent may have to work with estimate $\\hat{EU}(a)$ of true EU function due to practical limitations. We assume that the estimates are **unbiased** ($E(\\hat{EU}(a)-EU(a))=0$). Still, expected utility of best choice tends to be an overestimate (**optimizer's curse**), assuming gaussian EU errors. Can be avoided with a Bayesian approach that uses a model $P(\\hat{EU}\\mid EU)$ of the error in the utility estimates. Can treat utility estimate as evidence and compute the posterior using Bayes' rule.\n\nDecision theory is a **normative theory** (describes how a rational agent should act). A **descriptive theory** describes how actual agents (e.g. humans) act. Humans are predictibly irrational (e.g. **certainty effect**: people attracted to certain gains, **ambiguity aversion**, **framing effect**: wording of problem can influence decision, **anchoring effect**: people prefer making relative utility decisions over absolute ones).\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\nSequential decision problem for fully observable, stochastic environment with Markovian transition model (probability of reaching $s'$ from $s$ depends only on $s$) and additive rewards is called a **Markov decision process**, or **MDP**. Consists of set of states (initial $s_0$), set $ACTIONS(s)$ of actions in each state, transition model $P(s'\\mid s,a)$, and a reward function $R(s,a,s') \\in [-R_{max}, R_{max}]$ (aka $T(s, a, s')$). Solutions usually involve **dynamic programming**.\n\nSolution is a **policy** $\\pi$: $\\pi(s)$ is recommended action at state $s$. Evaluated by *expected* utility of environment history. Policy-following agent is simple reflex agent, computed from info used for utility-based agent.\n\nIs there a **finite horizon** (*fixed* time $N$ after which nothing matters; $U_h(s_0,a_0,...,s_{N+k})=U_h(s_0,a_0,...,s_N)$ for any $k>0$)) or **infinite horizon** for decision making? With finite horizon, optimal policy may depend on time left (**nonstationary**, as opposed to **stationary**).\n\nHow to calculate utility of state sequences? We use **additive discounted rewards**: $U_h(s_0,a_0,...)=R(s_0,a_0,s_1)+\\gamma R(s_1,a_1,s_2)+\\gamma^2...$, where the **discount factor** $\\gamma \\in [0, 1]$ . Special case $\\gamma=1$ purely **additive rewards**. Discounted rewards might reflect empirical human behavior, time value of money, uncertainty, **stationarity** of preferences between state sequences (if two histories start the same, they should be preference ordered according to the rest of the histories, as if they were to start today), and conveniently promotes convergence (if $\\gamma<1$), utilities bounded in $[-R_{max}/(1-\\gamma), R_{max}/(1-\\gamma)]$ geo series. If environment contains terminal states and agent is guaranteed to get to one (**proper policy**), we will never need to compare infinite sequences; can use $\\gamma=1$ here. Infinite sequences can alternatively be compared by **average reward** per timestep.\n\nExpected utility obtained by executing $\\pi$ starting at $s$, reaching state $S_t$ at time $t$:\n$$U^\\pi(s)=E(\\sum_{t=0}^\\infty\\gamma^tR(S_t,\\pi(S_t),S_{t+1}))$$\nOptimal policy $\\pi^*_s=argmax_\\pi U^\\pi(s)$. If using discounted utilities with infinite horizons, optimal policy is independent of starting state (can write $\\pi^*$). True utility of a state is $U^{\\pi^*}(s)=:U(s)$. The **action-utility function** or **Q-function**: $Q(s,a)$ is expected utility of taking action $a$ in state $s$.\n\n**Bellman equations**:\n$$Q(s,a):=\\sum_{a'}P(s'\\mid s,a)[R(s,a,s')+\\gamma \\max_{a'}Q(s',a')]$$\n$$U(s)=\\max_a Q(s,a)$$\n$$\\pi^*(s)=argmax_a Q(s,a)$$\n\nLet $\\Phi(s)$ be *any* function of state $s$. Then, according to **shaping theorem**, the following transformation does not change optimal policy:\n$$R'(s,a,s')=R(s,a,s')+\\gamma \\Phi(s')-\\Phi(s)$$\nCan set $\\Phi(s)=U(s)$ so greedy policy $\\pi_G$ w.r.t. $R'$ is the optimal policy ($\\pi_G(s)=argmax_a\\sum_{s'}P(s'\\mid s,a)R'(s,a,s')$). Like providing treat to animal at each step in target sequence.\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\n**Value iteration**: $n$ equations, $n$ unknowns (non-linear because of \"max\" operator). Can calculate utilities with iterative approach until convergence. $U_i(s)$ utility at $i$th iteration. Apply update simultaneously to all states at each iteration:\n$$U_{i+1}(s)-\\max_{a\\in A(s)}Q_i(s,a)$$\nwill converge to optimal policy, unique solution for $U$. Can stop once $\\delta:=$ max change in $U(s)$, $\\delta\\leq\\epsilon(1-\\gamma)/\\gamma$ ($\\epsilon:=$ maximum error allowed in utility of any state). Value iteration propagates information through state space via local updates.\n\nLet Bellman update be operator $B$ applied simultaneously to utility of every state: $U_{i+1}:=BU_i$. **Utility vector** has components $U(s)$ for each $s$, length $||U||=\\max_s |U(s)|$ (**max norm**). We have $||BU_i - BU_i'||\\leq\\gamma||U_i-U_i'||$ (Bellman update is a contraction by a factor of $\\gamma$ on the space of utility vectors). Replace $U_i'$ with true utility $U$: error reduced by factor of at least $\\gamma$ at each iteration.\n$$N=ceil(\\log(2R_{max}/((1-\\gamma)\\epsilon))/\\log(1/\\gamma))$$\nIf $||U_{i+1}-U_i||<\\epsilon(1-\\gamma)/\\gamma$ then $||U_{i+1}-U||<\\epsilon$ (hence termination condition in value iteration algo).\n\n**Policy loss** $||U^{\\pi_i}-U||$ is the most the agent can lose by executing $\\pi_i$ instead of the optimal policy $\\pi^*$;\n$$|U_i-U||<\\epsilon \\implies ||U^{\\pi_i}-U||<2\\epsilon$$\nOften $\\pi_i$ becomes optimal before $U_i$ has converged.\n\n**Policy iteration** algorithm, starting from initial policy $\\pi_0$:\n- **Policy evaluation**: given a policy $\\pi_i$, calculate $U_i=U^{\\pi_i}$ (vector), using $U_i(s)=Q_i(s,\\pi_i(s))$\n- **Policy improvement**: calculate a new MEU policy $\\pi_{i+1}$ from $U_i$\nTerminate when policy improvement step yields no change to policy ($\\pi_i$ optimal).\n\nPolicy evaluation has $n$ linear equations (no \"max\"), can be solved in $O(n^3)$ time with linal methods. If $n$ large, do a small number of simple value iteration steps (no change to policy) to approximate utilities (**modified policy iteration**).\n\nCan also pick subsets of states and apply either kind of updating (policy improvement or simplified value iteration) to that subset (**asynchronous policy iteration**). Can use heuristic to only update states most likely to be reached by good policy.\n\n**Linear programming** (LP) can be used to solve MDPs; minimize $U(s)$ for each $s$, s.t. $U(s)\\geq Q(s,a)$ $\forall s,a$. DP usually more efficient.\n\nOnline approx algo for large MDPs: expectimax tree. The $\\epsilon$-horizon is the tree depth $H$ s.t. sum of rewards beyond any leaf is less than $\\epsilon$ (use heuristic to estimate beyond leaf nodes). Depth of $H=ceil(\\log_\\gamma(\\epsilon(1-\\gamma)/R_{max}))$ suffices (useful if $\\gamma$ not close to 1). If branching factor large, can generate $N$ samples and use the sample mean (will prob focus on the most likely outcomes). General approach: **real-time dynamic programming (RTDP)**. Can also apply Monte Carlo methods (UCT).\n\n\n## Module 8 Lecture Notes\n Your summary here \n\n## Module 8 Definitions\n* **Principle of maximum expected utility** - a rational agent should act to maximize expected utility; $action=\u0007rgmax_a EU(a)$\n* **Axioms of utility theory** - observability, transitivity, continuity, substitutability, monotonicity, decomposability\n* **Preference elicitation** - present choices and use observed preferences to determine underlying utility function\n* **Micromort** - value of a 1-in-million chance of death\n* **Certainty equivalent** - the value an agent will accept in lieu of a lottery\n* **Insurance premium** - difference between expected monetary value and certainty equivalent\n* **Unbiased expected utility estimate** - $E(\\hat{EU}(a)-EU(a))=0$\n* **Optimizer's curse** - expected utility of best choice tends to be an overestimate\n* **Normative theory** - describes how a rational agent should act\n* **Descriptive theory** - describes how actual agents (e.g. humans) act\n* **Objectivist/frequentist interpretation** - averages over repeated experiments\n* **Subjectivist / bayesian interpretation** - degrees of belief about unobserved variables\n* **** - \n* **An attribute of a history** - a transition $s_t to_{a_t}s_{t+1}$\n* **Contraction** - one-argument function, brings any 2-points closer (e.g. divby2); has only one fixed point, moves any point towards FP\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n\nAn **$n$-armed bandit** has $n$ levers, each yielding fixed/unknown probability distribution of winnings. Must choose levers successively.\n\nEach arm $M_i$ is an MRP. Overall Bandit problem is an MDP (Cartesian state space, actions $a_1,...,a_n$).\n\nOne-armed bandit $M$ yields sequence $R_0,R_1,...$, $M_\\lambda$ yields $\\lambda,\\lambda,...$ (cost of arm pull). Number of times you pull $M$ before switching to $M_\\lambda$ is **stopping time** $T$ (first pull of $M_1$ yields no extra info so stay forever). **Gittins index** of $M$ (value of $\\lambda$ for which agent is indifferent btwn playing $M$->$M_\\lambda$ game vs just taking $M_\\lambda$):\n$$\\lambda = \\max_{T>0} E[\\sum_{t=0}^{T-1}\\gamma^tR_t]/E[\\sum_{t=0}^{T-1}\\gamma^t]$$\nOptimal policy: pull the arm that has the highest Gittins index, then update the Gittins index of the arm you pulled.\nCompute Gittins index for arm $M$, construct MDP with additional action at each state: quit and receive infinite sequence of $\\lambda$-rewards; optimal policy value is $\\lambda/(1-\\gamma)$. Equivalently, construct **restart MDP** $M^s$; add new action to each state with same rewards/probs/transitions as initial state $s$ (replace infinite path of $\\lambda$ with restarting, since agent is indifferent b.d.o. Gittins index). Gittins index for arm $M$ in state $s$ is equal to $(1-\\gamma)$ times value of optimal policy for restart MDP $M^s$ (just solve MDP).\n\n**Bernoulli bandit**: each arm $M_i$ has count of 1s $s_i$, count of 0s $f_i$ (both initialized to 1), next outcome is 1 with probability $s_i/(s_i+f_i)$, 0 with probability $f_i/(s_i+f_i)$. Infinite states (can't solve with Gitten). Accurate appx: truncated MDP with states up to $s_i+f_i=100$ and $\\gamma=0.9$. Arms with higher payoff probability preferred, but exists an **exploration bonus** with rarely-tried arms.\n\nAppx algos (no Gittins comp):\n\n**Upper confidence bound** or UCB heuristic calculates **confidence interval** for the value of the arm from samples, choose arm with highest upper bound. Approximate index value for arm $M_i$:\n$$UCB(M_i) = \\hat{\\mu} + g(N)/\\sqrt{N_i}$$\n$N:=$ total num of samples drawn from all arms. Definition of $g$ determines **regret** relative to perfect policy. For undiscounted case, no algorithm can have regret that grows slower than $O(\\log N)$. USB policy can achieve this bound with $g(N)=\\sqrt{2\\log(1+N\\log^2(N))}$.\n\n**Bandit superprocess (BSP)**: each arm is a full MDP, e.g. multitasking, leads to greedier behavior in each MDP. Can solve as MDP with Cartesian state space (exponential, impractical). Better approach: consider **opportunity cost** in each MDP. A **dominating policy** ignores opportunity cost (optimally). Adding actions to states can create a relaxed MDP with a dominating policy; upper bound on value of arm. Lower bound: solve arm individually. If lower bound of one is above upper bound of others, solved. Otherwise, look-ahead search and recomputation of bounds -> optimal solution.\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n\n**Reinforcement learning (RL)**: agent interacts with world and periodically receives **rewards** or **reinforcements**. Same goal as MDPs, but not given details of MDP.\n\nProviding reward signal easier than providing labeled training examples (efficient, doesn't require domain knowledge e.g. chess).\n\n**Sparse rewards*: just reward for end goal. Better: inject expertise / provide intermediate rewards to help learn good strategies.\n\n**Model-based reinforcement learning**: agent uses transition model of environment to help interpret reward signals and make decisions. Model might be unknown/learned or provided (e.g. chess rules). Model-based RL systems try to learn utility function $U(s)$ for each state.\n\n**Model-free RL**: learns more direct representation of how to behave (no model):\n- **Action-Utility Learning** (**Q-learning**), agent learns $Q$ function\n- **Policy Search**: agent learns policy $\\pi(s)$ that maps states to actions (**reflex agent**)\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\n**Passive learning agent** executes set of **trials** in environment using policy $\\pi$.\n\n**Direct utility estimation**: the utility of a state is the expected reward-to-go; each trial provides *sample* of this quantity for each state visited. Calculate observed reward-to-go for each state and update estimated utility (running average). Infinite trials -> converge. Ignores Bellman equations / connections between states. Searches for $U$ in a hypothesis space much larger than necessary, converges slowly.\n\n**Adaptive dynamic programming (ADP)** uses constraints among utilities of states; learns transition model that connects them, then solves MDP using DP. For passive RL agent: plug learned transition models and rewards into linal solver (linear equations). Can also use **modified policy iteration** (see mod8) to update utility estimates after each change to learned model. Learning transition model easy (fully observable environment; keep track of transition probability averages). Intractable for large state spaces (cannot learn full transition model or solve so many equations).\n\n**Temporal-difference learning**: update $U^\\pi(s)$ after transition from $s$ to $'$ via $\\pi(s)$:\n$$U^\\pi(s)+=(alpha)[R(s,\\pi(s),s')+\\gamma U^\\pi(s')-U^\\pi(s)]$$\nwhere alpha is the **learning rate** (**temporal-difference** (TD) equations). The *average value* of $U^\\pi(s)$ will converge to the correct quantity in the limit, even if the value itself continues to fluctuate. Also if we decrease alpha as number of visits to $s$ increases, then $U^\\pi(s)$ will converge correctly. TD does not need a transition model to perform its updates. The environment provides the connection to neighboring states in the form of observed transitions. Crude but efficient first approximation to ADP. Each ADP adjustment is **pseudoexperience** (from TD pov) from simulating current transition model. Can extend TD to use transition model to generate pseudoexperiences (more closely approximate ADP, more comp. time).\n\nCan make ADP more efficient by appx algos for value/policy iteration. One idea: bound number of adjustments made after each observed transition. Alternatively: use heuristic to rank adjustments and only do most significant (**prioritized sweeping** heuristic, adjust states whose likely successors have just undergone large adjustment to utility estimates). Appx algos can converge nearly as fast, and do much less comp. Also, appx algo can use min adjust size that decreases as transition model becomes more accurate.\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\nActions provide *rewards* and *information*. Must tradeoff **exploitation** for short-term rewards and **exploration** to improve policy.\n\nAny optimal exploration scheme should be \"greedy in the limit of infinite exploration\" (**GLIE**). Must try each action in each state an unbounded number of times to avoid possibility that optimal action is missed. An ADP agent using such a scheme will eventually learn true transition model and can then exploit.\n\nSimplest GLIE scheme: choose random action at time step $t$ with prob $1/t$ and follow greedy policy otherwise. Converges slow. Better to give more weight to unexplored actions while disliking low-utility actions (like with MCTS). Alter utility equation to give higher estimate to less explored state-action pairs. Let $U^+(s)$ be optimistic utility estimate, $N(a,s)$ number of times $a$ tried at state $s$. Update equation:\n$$U^+(s) = \\max_s f(\\sum_{s'}P(s'\\mid s,a)[R(s,a,s')+\\gamma U^+(s)], N(s,a))$$\nwhere $f(u,n)$ is the **exploration function**: how greed (like high $u$) weighed against curiosity (like unexplored actions). Should increase in $u$ and decrease in $n$. Simple choice: $f(u,n)=u$ if $n\\geq N_e$ else $R^+$ (an optimistic estimate of best possible reward), where $N_e$ is fixed. Agent will try each state at least $N_e$ times.\n\nMany actions are **irreversible**, agent may enter **absorbing state** (no actions have any effect, no rewards, e.g. death). Better idea: choose policy that works reasonably well for whole range of possible / probable models. **Bayesian reinforcement learning** assumes prior prob $P(h)$ over hypothesis $h$ of true model; posterior $P(h\\mid e)$ obtained by Bayes' rule given observations. Then\n$$\\pi^* = argmax_\\pi\\sum_hP(h\\mid e)U_h^*$$\nAgent must consider effects of future observations. Problem becomes an **exploration poMDP**, belief states are distributions over models. Solving poMDP is usually intractable.\n\nAlternative: maintain set of possible models $H$ without considering probabilities, optimize for *worst case* over $H$:\n$$\\pi^* = argmax_\\pi \\min_h U_h^\\pi$$\nformulate as adversarial search problem. Problem: overly conservative; real life is full of risk-reward tradeoffs.\n\nHuman intervention can help algo learn safely (record initial series of actions, write constraints on behavior).\n\nActive **Q-learning**: learn action-utility $Q$ values instead of utility function $U$. Agent can then choose $argmax_a f(Q(s,a), N_{sa}[s,a])$. Q-learning TD update:\n$$Q(s,a)' = Q(s,a) + (alpha)[R(s,a,s') + \\gamma\\max_{a'}Q(s',a')-Q(s,a)]$$\ndoes not need transition model for learning or action selection. \n\n**SARSA** (state, action, reward, state, action) like Q-learning but SARSA updates with the Q-value of the action actually taken (no max). SARSA penalizes exploration to negative reward state.\n\nQ-learning is an **off-policy** learning algo; learns values for policy it isn't using, SARSA is an **on-policy** algo (learns Q-values for its own policy).\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\nConstructing compact appx of true utility or Q-function: **function approximation**. Might appx utility using weighted linear combo of features:\n$$\\hat{U}_{theta}(s) = theta_1f_1(s)+...+theta_nf_n(s)$$\njust learn parameters theta. Sometimes combine with lookahead search to produce more accurate decisions.\n\nFunction approximation allows for inductive generalizations about Q-function (can use experience from similar states).\n\nApproximating direct utility estimation: each state and sum of rewards received after constitute training example for a **supervised learning** algorithm. For RL makes sense to use an *online* learning algorithm that updates params after each trial. If $u_j(s)$ is observed total reward from $s$ onward in $j$th trial, error\n$$E_j(s)=(\\hat{U}_{theta}(s)-u_j(s))^2/2$$\nparameter update\n$$theta_i -= (alpha) (\\delta/\\delta theta_i)E_j(s)$$\n(the **Widrow-Hoff rule**, or the **delta rule** for online least-squares).\n\nFunction appx will make agent learn faster if hypothesis space is not too large and includes some functions that are a good fit to true utility function.\n\nApproximating TD and Q-learning:\n$$theta_i+=(alpha)[R(s,a,s')+\\gamma \\hat{U}_{theta}(s')-\\hat{U}_{theta}(s)](\\delta/\\delta theta_i)\\hat{U}_{theta}(s)$$\n$$theta_i+=(alpha)[R(s,a,s')+\\gamma \\max_{a'}\\hat{Q}_{theta}(s',a')-\\hat{Q}_{theta}(s,a)](\\delta/\\delta theta_i)\\hat{Q}_{theta}(s,a)$$\nfor passive TD learning, update rule converges to closest possible appx when function approximator is linear in the features. For nonlinear functions, no guarantees. Some other algos avoid these issues.\n\nWith a learned model, agent can do look-ahead search to improve decisions and carry out internal simulations to improve approximate representations of $U$ or $Q$ rather than requiring slow/expensive real-world experience. For a *partially observable environment*, learning more difficult because next percept is not a label for the state prediction problem anymore.\n\nDeep neural networks popular for function approximation in RL (deep RL). Parameter update via backpropagation.\n\n**Credit assignment** problem: figure out what elements of strategy to blame/praise for failure or success. Supply agent with additional **pseudorewards** for \"making progress\" (e.g. 17.1 $\\Phi$ transformation).\n\n**Heirarchical reinforcement learning (HRL)**: break long action of sequences into smaller pieces to make learning easier. HRL agent begins with **partial program** outlining heirarchical structure for agent's behavior. Adds primitives for unspecified choices to be filled in by learning. RL is special case of HRL. Higher-level program can decide on higher-level action, then each action has its own program, etc down to fundamentals. HRL has **joint state space** (each state $(s,m)$ has physical state and machine state). Heirarchical structure of behavior provides natural **additive decomposition** of overall utility function. Learning each term is easier than learning whole function.\n\n\n## AIMA Chapter 22.5  Policy Search\n\n**Policy search**: keep updating policy until performance improves, then stop. Interested in *paramaterized* representation of $\\pi$, fewer parameters than states. Could represent by collection of paramaterized Q-functions (one for each action), and then take\n$$\\pi(s) = argmax_a \\hat{Q}_{theta}(s,a)$$\nPolicy search will adjust the parameters theta to improve the policy. Not the same as Q-learning; find value of theta that results in good performance. Problem: policy is a discontinuous function of the parameters when actions are discrete. Makes gradient-based search difficult. Instead: use **stochastic policy** representation $\\pi_{theta}(s,a)$ which specifies the probability of selecting $a$ in state $s$, e.g. softmax function:\n$$\\pi_{theta}(s,a) = \\exp(b \\hat{Q}_{theta}(s,a)) /\\sum_{a'}\\exp(b \\hat{Q}_{theta}(s,a')) $$\nthe param $b>0$ (beta) modulates the softness of the softmax (large $b$ -> hardmax).\n\nImproving deterministic policy in determ environment: let $rho(theta)$ be the **policy value** (expected reward-to-go when $\\pi_{theta}$ executed). If we can formulate $rho(theta)$ and calculate the **policy gradient** vector $del_{theta}rho(theta)$ we can use standard optimization techniques. Otherwise, evaluate $\\pi_{theta}$ by executing it and observing accumulated reward. Follow the **empirical gradient** by hill climbing (evaluating change in policy value for small changes in each parameter).\n\nIn a nondeterm environment, more difficult; trying to compute $rho(theta)$ and $rho(theta+\\Delta theta)$ for some small $\\Delta theta$, but total reward for each trial may vary widely, so estimates of the policy value from a small number of trials will be unreliable. Comparing two estimates will be more unreliable. One solution: run lots of trials (but impractical, takes too much time, too many mistakes). Instead we can obtain an unbiased estimate of the gradient at $theta$, $del_{theta} rho(theta)$ from results of trials executed at $theta$. For episodic environment where each action $a$ receives reward $R(s_0, a, s_0)$ and environ restarts in $s_0$ (for simplicity):\n$$del_{theta} rho(theta) = del_{theta} \\sum_a R(s_0,a,s_0)\\pi_{theta}(s_0,a) = \\sum_a R(s_0,a,s_0) del_{theta}\\pi_{theta}(s_0,a)$$\nnow simple trick to approximate summation with samples from prob distribution defined by $\\pi_{theta}(s_0,a)$: ($N$ total trials, $a_j$ taken on $j$th trial, $u_j(s)$ total reward from state $s$ onward in $j$th trial):\n$$del_{theta} rho(theta) =approx N^{-1}\\sum_{j=1}^N (u_j(s) del_{theta} \\pi_{theta}(s_0,a_j)) / \\pi_{theta}(s_0,a_j)$$\n*Reinforce* algorithm, usually much more effective than hill climbing, but still slower than necessary.\n\nBetter approach: **correlated sampling**: compare policies with environment simulator in which teh random-number sequences can be repeated. Number of random sequences required to ensure that every policy is well estimated depends only on the complexity of the policy space, not complexity of underlying domain.\n\n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **Markov reward process (mrp)** - MDP with only 1 action $a_i$\n* **Selection problem** - (e.g. hiring): no index function exists (optimal policy switches preferences for two arms when third arm is added).\n* **Metalevel decision problem** - (what computations to make during a game-tree search before moving) is also a selection problem (not a bandit problem)\n* **Thompson sampling** - chooses an arm randomly according to the probability the arm is optimal, given samples so far ($O(\\log(n))$ regret)\n* **Bandit superprocess (bsp)** - each arm is a full MDP, e.g. multitasking, leads to greedier behavior in each MDP\n* **Passive reinforcement learning** - agent’s policy is fixed and the task is to learn the utilities of states (or of state–action pairs), and/or model\n* **Active reinforcement learning** - passive RL tasks + also figure out what to do (issue: exploration)\n* **Passive learning agent** - has fixed policy $\\pi(s)$ determining actions, trying to learn $U^\\pi(s)$ (similar to policy evaluation, but does not know model)\n* **Expected reward-to-go** - expected total reward from that state onward\n* **Greedy agent** - does no exploration - sometimes pays off and converges to optimal policy, but often doesn't due to not discovering better strategy\n* **Evaluation function** - a compact measure of desirability for potentially vast state spaces (approximate utility function)\n* **Catastrophic forgetting** - learned strategy is not 'subgame perfect', works in certain equilibrium but if deviated from these conditions, might act catastrophically suboptimally\n* **Experience replay** - (solution to catastrophic forgetting) make agent keep reliving youthful mistakes at regular intervals\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\nReal-world agents need to handle **uncertainty** (partial observability, nondeterminism, or adversaries). Agent can maintain **belief state**, repr of all possible world states that it might be in (and generate plan for every possible eventuality). Too complex, considers unlikely eventualities, must have a way to compare merits of plans that are not guaranteed.\n\nUSing logic to cope with complex and uncertain domain fails because of **laziness** (too much work to enumerate all antecedents or consequents needed to form exceptionless rule, too hard to use), **theoretical ignorance** (there is no complete theory for the domain), **practical ignorance** (even if all rules are known, might be uncertain about practical conditions in this case).\n\nAgent's knowledge can sometimes be at best a **degree of belief** in the relevant sentences (can use **probability theory**).\n\nProbability statements made with respect to knowledge state, not with respect to real world.\n\nDecision theroy = probability theory + utility theory.\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n\nLogical assertions say which possible worlds are strictly ruled out, probabilistic assertions talk about how probable the various worlds are. Prob theory: set of all possible worlds is called the **sample space** $\\Omega$ containing possible worlds $\\omega$. Possible worlds are *mutually exclusive* and *exhaustive*. \n\nA fully specified **probability model** associates a numerical prob $P(\\omega) \\in [0,1]$ with each possible world : $\\sum_\\omega P(\\omega) = 1$. \n\nIn logic a set of worlds is a **proposition**, in prob theory the prob of a proposition $\\phi$ is \n$$P(\\phi)=\\sum_{\\omega\\in\\phi}P(\\omega)$$\nRaw probabilities called **unconditional** or **prior probabilities**. Usually we have some information (**evidence**) that has already been revealed; then we look for the **conditional** or **posterior** probability. We have $P(a\\mid b) = P(a\\land b) / P(b)$ when $P(b)>0$. Equivalently, $P(a\\land b) = P(a\\mid b)P(b)$ (**product rule**).\n\nVariables called **random variables** map the domain of possible worlds $\\Omega$ to some **range** (set of values it can take on); can be finite, infinite, discrete, continuous. Can combine elementary propositions by using connectives of propositional logic, e.g. $P(cavity\\mid \\lnot toothache \\land teen)$. Vector of prob of each value of random variable: **P**(X)$=[0.6,0.1,...]$. Then **P** defines a **probability distribution** for X (assignment of prob to each value in range); finite discrete prob distribution called **categorical**.\n\nDefine continuous distribution by **probability density function**, some paramatrized function of $x$. PDF $P(x)$ is probability that $X$ falls within an arbitrarily small region beginning at $x$, divided by the width of the region. Can form distributions on multiple variables with commas (**joint probability distribution**).\n\nA possible world is an assignment of values to all the random variables under consideration.\n\nProbability model is completely determined by the joint distribution on all random variables (**full join probability distribution**).\n\n$P(\\lnot a) = 1-P(a)$, $P(a\\lor b) = P(a)+P(b)-P(a\\land b)$ (**inclusion exclusion principle**).\n\nIf an agent has some degree of belief in a proposition $a$, it should be able to state odds at which it is indifferent to a bet for or against $a$. If one's degrees of belief do not accurately reflect the world, they would lose money to an opposing agent whose beliefs more accurately reflect the state of the world. If agent expresses beliefs that violate axioms of probability, there is a combo of bets by agent 2 that will guarantee agent 1 to lose moeny each time. Refusing to bet like refusing to let time pass (we make decisions under uncertainty at each moment).\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n\n**Marginal probability** is the probability of one variable, extracted from a joint distribution through **marginalization** or **summing out** (sum probabilities for each possible value of other variables): $P(Y)=\\sum_z P(Y,Z=z)$.\n\n**Conditioning** rule: $P(Y)=\\sum_Z P(Y\\mid Z)P(Z)$.\n\nInference for query with single variable $X$. $E$ is list of evidence variables, $e$ is list of observed values for them, $Y$ is remaining unobserved variables. Query is\n$$P(X\\mid e) = aP(X,e) = a\\sum_y P(X,e,y)$$\nand $P(X,e,y)$ is ubset of probabilities from the full join distribution. However, answering probabilistic queries for discrete variables does not scale well; for domain with $n$ boolean variables, it requires an input table of size $O(2^n)$ and takes $O(2^n)$ time to process the table.\n\n\n## AIMA Chapter 12.4 Independence\n\n**Independence** (**marginal independence** and **absolute independence**): $P(a\\mid b) = P(a)$ or $P(b\\mid a)=P(b)$ or $P(a\\land b)=P(a)P(b)$ (all equivalent). Independence assertions usually based on knowledge of the domain. Can dramatically reduce amount of info necessary to specify the full joint distribution. If complete set of variables can be divided into independent subsets, then the full join distribution can be factored into separate joint distributions on those subsets.\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\n**Bayes' rule** (derived from product rule):\n$$P(b\\mid a) - P(a\\mid b)P(b)/P(a)$$\nMore general version conditionalized on some background evidence $e$:\n$$P(Y\\mid X,e)=P(X\\mid Y,e)P(Y\\mid e)/P(X\\mid e)$$\n$$P(cause\\mid effect) = P(effect\\mid cause) P(cause)/P(effect)$$\n$P(effect\\mid cause)$ quantifies relationship in **causal** direction, whereas $P(cause\\mid effect)$ describes the **diagnostic** direction.\n\nCan avoid assessing prior probability of the evidence by computing posterior for each value of query variable:\n$$P(Y\\mid X) = aP(X\\mid Y)P(Y)$$\nwhere $a$ is the normalization constant needed to make the entries sum to 1.\n\n**conditional independence** $P(X,Y\\mid Z) = P(X\\mid Z)P(Y\\mid Z)$, and $P(X\\mid Y,Z)=P(X\\mid Z)$ and $P(Y\\mid X,Z)=P(Y\\mid Z)$. Conditional independence assertions can allow probabilistic systems to scale up, and they are much more commonly available than independence assertions.\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\n**Naive Bayes model**: assume independence as a simplifying assumption (**Bayesian classifier**). Observed effects $E=e$, remaining effect variables $Y$ are unobserved:\n$$P(cause\\mid e) = a\\sum_y P(cause,e,y)$$\n$$P(cause\\mid e) = a P(cause) \\prod_j P(e_j\\mid cause)$$\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n\nNaive approach for language model: $P(next \\mid history) = P(history + next) / P(history)$ (compute frequency counts). Problem: requires too much data.\n\nChain rule:\n$$P(w_{1:n}) = \\prod_{k=1}^n P(w_k\\mid w_{1:k-1})$$\ndoesn't really help, still hard to estimate $P(w_k\\mid w_{1:k-1})$.\n\nIntuitino of n-gram model: approximate the history by just the last few words. Bigram model approximates $P(w_n\\mid w_{1:n-1})$ with $P(w_n\\mid w_{n-1})$. Assumption that the probability of a word depends only on previous word is called a **Markov** assumption. In general, an N-gram approximates $P(w_n\\mid w_{1:n-1})$ as $P(w_n\\mid w_{n-N+1:n-1})$.\n\nGiven bigram assumption, prob of complete word sequence is $\\prod_{k=1}^n P(w_k\\mid w_{k-1})$.\n\nMake an MLE estimate for the parameters of an n-gram model by getting counts from a corpus, and **normalizing** the counts between 0 and 1. E.g. for bigram: compute count $C(xy)$ of bigram, then\n$$P(w_n\\mid w_{n-1}) = C(w_{n-1}w_n)/C(w_{n-1})$$\nor for N-gram:\n$$P(w_n\\mid w_{n-N+1:n-1}) = C(w_{n-N+1:n-1}w_n)/C(w_{n-N+1:n-1})$$\nThis MLE maximizes the likelihood of the corpus given the model.\n\nPractical considerations: use pseudo-words at start of sentence and marker at end of sentence. Always represent/compute LM probabilities as **log probabilities**.\n\n\n## SLP Chapter 3.2 Evaluating Language Models\n\nTo evaluate LM, can embed in an application and measure how much app improves: **extrinsic evaluation**. But running NLP systems end-to-end is expensive. An **intrinsic evaluation** metric is one that measures the quality of a model independent of any application. We need a **test set** corpus. Whichever model assigns a higher probability to the test set (more accurately predicts the test set) is better. Don't train on test set, this can cause innacuracy in **perplexity**. Sometimes test set is used so frequently we implicity fit to it by choosing models / parameters. Then we can use initial test set **development test set** or **devset**, and use fresh test set for final eval.\n\nShould pick smallest test set to give enough statistical power to measure stat significant difference between the two potential models. In practice 80/10/10 training/devset/test. Can take large chunks or stripes out of corpus to make dif sets.\n\n**Perplexity** (PP) of a LM on test set $W=w_1,...,w_N$ is\n$$PP(W) = P(w_1,...,w_N)^{-(1/N)} = (\\prod_{i=1}^N P(w_i\\mid w_1,...,w_{i-1}))^{-(1/N)}$$\n(estimate $P(w_i\\mid w_1,...,w_{i-1}))$ as $P(w_i\\mid w_{i-n+1:i-1})$ for n-gram model). We want to minimize perplexity.\n\nShould include begin and end of sentence markers in prob comps, also include end-of-sentence marker (but not begin-marker) in total count of word tokens $N$.\n\nPerplexity is the **weighted average branching factor** of a language. Related to concept of entropy. The perplexity of two LMs is only comparable if they use identical vocabularies. Intrinsic improvement in perplexity does not guarantee extrinsic improvement in other task. But usually correlates with such improvements, can be used as a quick check on an algorithm. Should be confirmed with final extrnsic evaluation.\n\n\n## SLP Chapter 3.3 Sampling sentences from a language model\n\nTo visualize what kind of knowledge a language model embodies, can **sample** from it. \n\n\n## SLP Chapter 3.4 Generalization and Zeros\n\nAn n-gram model can bge highly influenced by idiosyncracies and phrases in its test set. Should use a test set that has a similar **genre** to whatever task we are trying to accomplish. Also get training data in the same **dialect** or **variety**.\n\nProblem of **sparsity**: any corpus is limited, so some perfectly acceptable English word sequences are bound to be missing. We'll have zero-probability n-grams that should have nonzero prob. These *zeros* (things that occur in training set but do in test set) causes us to underestimate the prob of all sorts of words that might occur, and also if any prob of any word in the test set is 0, the entire prob of test set is 0; perplexity undefined.\n\nIn a **closed vocabulary** system the test set can only contain words from lexicon of known words that can occur, so no unknown words. In other cases, must deal with words we haven't seen before, called **unknown words** or **out of vocabulary (OOV)** words. Percentage of OOV words in test set is **OOV rate**. An **open vocabulary system** is one in which we model unknown words by adding pseudo-word <UNK>. Can turn problem back into closed vocabulary by:\n1. **choosing a vocabulary** fixed in advance\n2. **converting** any OOV word in train set to unknown word token <UNK>\n3. **estimate** the prob for <UNK> from its counts just like any other regular word\n\nAlternatively, can replace by <UNK> all words that occur fewer than $n$ times in the training set (for some small number $n$) or choose vocab size $V$ in advance and populate vocab with most frequent words. Perplexity can be influenced by <UNK> model, so perplexity should only be compared for models with same vocabulary.\n\n\n## SLP Chapter 3.5 Smoothing\n\nHow to deal with unseen word sequences? Prevent 0 prob by shaving off a bit of prob from some more frequent events and giving it to events we've never seen (called **smoothing** or **discounting**).\n\n**Laplace smoothing**: add 1 to all n-gram counts before normalizing them into probabilities. Does not perform well enough for modern n-gram models, but gives a useful baseline. Define **adjusted count**\n$$c_i^* = (c_i+1)N/(N+V)$$\nSmoothing is like **discounting** (lowering) some non-zero counts to get prob mass that will be assigned to zero counts. Can describe smoothing algorithm in terms of relative **discount** $d_c$:\n$$d_c=c^*/c$$\n\nAlternative: **add-k smoothing**; add fractional count $k$ to each count:\n$$P^*_{Add-k}(w_n\\mid w_{n-1})=(C(w_{n-1}w_n)+k)/(C(w_{n-1}+kV)$$\nchoose $k$ using devset. Still doesn't work very well for LM.\n\nAlternative: **backoff** to using less context when sequence is unseen. Or use **interpolation**, always mixing prob estimates from each n-gram estimator, weighted combo. Can even compute weights $\\lambda$ by conditioning on the context. Learn weigghts from **held-out** corpus. Can use **EM** algorithm, iterative learning algo that converges on optimal $\\lambda$s.\n\nFor backoff model to give correct prob distribution, must **discount** the higher-order n-grams to save some prob mass for lower-order n-grams. Need explicit discount factor and function alpha to distribute prob mass to lower-order n-grams. Backoff with discounting is called **Katz backoff** (reply on discounted prob $P^*$ if we've seen n-gram before, otherwise backoff to Katz prob for shorter history). $P_{BO}(w_n\\mid w_{n-N+1:n-1}) = $P^*(w_n\\mid w_{n-N+1:n-1})$ if $C(w_{n-N+1:n})>0$ else $(alpha)(w_{n-N+1:n-1})P_{BO}(w_{n-N+2:n-1})$. Katz backoff often combined with smoothing method called **Good-Turing**; involves detailed computation for estimating Good-Turing smoothing and the $P^*$ and alpha values.\n\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n\nScalability: store string as 64-bit hash number in memory, probabilities quantized with 4-8 bits, n-grams stored in reverse tries, shrink model by pruning n-grams with counts below threshold, use entropy to prune less important n-grams, build approximate LMs using techniques like **Bloom filters**, use sorted arrays, efficiently combine probs and backoffs in single value, merge sorts to efficiently build prob tables in minimal number of passes through large corpus.\n\nWith very large models, simpler smoothing algorithm may be sufficient: **stupid backoff**; give up goal of true prob distribution; no discounting of higher-order probs. If higher-order n-gram has zero count, backoff to a lower order n-gram weighed by a fixed (context-independent) weight. Does not produce prob dist, so refer to it as $S$. $S(w_i\\mid w_{i-k+1:i-1}) = count(w_{i-k+1:i})/count(w_{i-k+1:i-1})$ if $count(w_{i-k+1:i})>0$ otherwise $\\lambda S(w_i\\mid w_{i-k+2:i-1})$. Backoff terminates in the unigram. $\\lambda=0.4$ works well.\n\n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Language models (lms)** - models that assign probabilities to sequences of words\n* **$n$-gram** - sequence of $n$ words\n* **Relative frequency** - observed frequency of a particular sentence divided by the observed frequency of a prefix\n* **Branching factor of a language** - number of possible next words that can follow any word\n* **Balanced corpora** - has roughly equal number of words from different genres\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n\nIndependence and conditional independence reduces num probabilities specified for full joint distribution.\n\n**Bayesian network** represents dependencies among variables. Can represent any full join prob distribution, often consisely. It is a DAG, each node annotated with quant probability info. Full spec:\n1. Each node represents a random variable (discrete or continuous)\n2. Directed edges connect nodes. Arrow from $X$ to $Y$ implies $X$ is a *parent* of $Y$.\n3. Each node $X_i$ has probability info $theta(X_i\\mid Parents(X_i))$ that quantifies effect of parents on the node using a finite number of **parameters**\n\nThe topology of the network specifies conditional independence relationships. Arrow represents that $X$ has a *direct influence* on $Y$; causes should be parents of effects. After topology, specify local probability info for each variable (conditional distribution given parents). \nLocal probability info takes the form of a **conditional probability table (CPT)** (for discrete-discrete). Each row: conditional probabilities for a **conditioning case** (possible combination of values for parent nodes; miniature possible world). A table for a boolean variable with $k$ boolean parents contains $2^k$ independently specifiable probabilities. Node with no parents has only 1 row with prior probabilities.\n\nJoint distribution defined as\n$$P(x_1,...,x_n)=\\prod_{i=1}^n theta(x_i\\mid parents(X_i))$$\nEach entry in joint distribution represented by product of appropriate elements of local conditional distributions. Parameters $theta(x_i\\mid parents(X_i)) = P(x_i\\mid parents(X_i))$ implied by the joint distribution. Hence,\n$$P(x_1,...,x_n)=\\prod_{i=1}^n P(x_i\\mid parents(X_i))$$\nwhen estimating values for local conditional distributions, need to be actual conditional probs for the variable given parents.\n\nHow to construct Bayes net. Rewrite entries in the joint distribution in terms of conditional probs using product rule: \n$$P(x_1,...,x_n)=\\prod_{i=1}^n P(x_i\\mid x_{i-1},...,x_1)$$\nEquivalent to assertion that $P(X_i\\mid X_{i-1},...,X_1)=P(X_i\\mid Parents(X_i))$ if $Parents(X_i)\\subseteq \\{X_{i-1},...,X_1\\}$. Can satisfy by numbering nodes according to topological sort of the DAG. Bayes net is correct representation of the domain if each node is conditionally independent of other predecessors in the node ordering, given its parents. Instructions:\n1. Determine set of variables required to model domain. Order them (net will be more compact if causes precede effects). Network structure, number of parameters, and even difficulty of filling in individual CPTs, depends on order of introduction. Example of difference between causal and diagnostic models. In causal model, can specify fewer numbers and each more easy.\n2. Choose minimal set of parents for $X_i$ from $X_1,...,X_{i-1}$ so that above-specified condition is satisfied. For each parent insert an edge from the parent to $X_i$. Write down conditional probability table $P(X_i\\mid Parents(X_i))$.\n\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n\nParents of node $X_i$ should contain all nodes in $X_1,...,X_{i-1}$ that *directly influence* $X_i$.\n\nBayes network can be specified by $2^k n$ numbers, while full joint distribution contains $2^n$ numbers.\n\nOften leave out links representing slight dependency (slight gain in accuracy not worth additional complexity in network).\n\nVariable is conditionally independent of non-descendants, given parents. Variable is conditionally independent of all other nodes in the network, given its parents, children, and children's parents (its **Markov blanket**).\n\nA set of nodes $X$ is conditionally independent of another set $Y$, given a third set $Z$, if $Z$ **d-separates** $X$ and $Y$ (otherwise, conditional independence not required):\n1. Consider just the **ancestral subgraph of $X$, $Y$, $Z$, and ancestors.\n2. Add links between any unliked pair of nodes that share a common child (now **moral graph**).\n3. Change directed links to undirected.\n4. If $Z$ blocks all path between $X$ and $Y$ in the resulting graph, then $Z$ d-separates $X$ and $Y$.\n\nWorst case: filling CPT requires $O(2^k)$ numbers and experience with all conditioning cases. Usually, relationships describable by **canonical distribution**, fits standard pattern (whole table specified just by pattern and parameters). Example **deterministic node** (value specified by parents, no uncertainty).\n\nUncertain relationships can be characterized by **noisy logical** relation: causal relationship between parent and child can be inhibited. List all possible causes (can add **leak node** that covers misc causes) and inhibition of each parent must be independent of inhibition of any other parents. Then only need $O(k)$ parameters instead of $O(2^k)$. \n\nCan handle continuous values with discretization (divide possible values into fixed set of intervals; tradeoff loss of accuracy and size of CPT). Alternatively, define continuous variable using standard pdf family. Another solution: **nonparametric representation**. New kinds of distributions: continuous given discrete (enumerate pdf for each discrete value), continuous given continuous (common choice: **linear-Gaussian**),\nand discrete given continuous (define function mapping continuous value to probability of each value, e.g. \"soft threshold\" integral of standard normal dist aka **probit model**, or **logistic function** $1/(1+e^{-x})$ aka **expit** or **inverse logit** model; also can take linear combo of continuous parents).\n\nNetwork containing only continuous variables with linear-Gaussian distributions has joint distribution (and posterior given any evidence) that is a multivariate Gaussian distribution. Inference only takes $O(n^3)$ time. If discrete variables are added only as parents, network defines **conditional Gaussian** (CG) distribution: given any assignments to the discrete variables, distribution over the continuous vars is multivariate Gaussian.\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n\nInference: compute posterior distribution for set of **query variables** given observed **event** $e$ (e.g. assignment of values to set of **evidence variables** $E$). Can easily extend single-variable queries to multivariable queries, so only consider single-variable ($X$) inference. Hidden (nonevidence, nonquery) variables: $Y$.\n\nInference by enumeration ($O(2^n)$ time, linear space complexity):\n$$P(X\\mid e)=(alpha)P(X,e)=(alpha)\\sum_yP(X,e,y)$$\nQuery can be answered with Bayes net with sums of products of conditional probabilities from net.\nEnumerate-All$(vars, e)$ computes probability of evidence in $e$ that is also in $vars$. Vars is list of variables we have left to look at, in topological order.\n\n**Variable elimination algorithm** uses DP. Time and space requirements dominated by size of largest factor constructed (determined by order of elimination variables and structure of net). Intractable to optimally order, but can use heuristics. Greedy: eliminate whichever var minimizes size of next factor to be constructed. Optimization: we can remove any leaf node from net if it is not a query variable or evidence variable. Continuing -> every variable that is not an ancestor of a query variable or evidence variable is irrelevant to the query. Variable elimination can remove these variables before evaluating the query.\n\nFor **singly connected networks** or **polytrees** (at most undirected path between any two nodes), the time and space complexity of variable elimination is linear in the size of the net (number of CPT entries) for any topological ordering of vars.\n\nFor **multiply connected networks**, variable elimination has worst case exponential time/space, even when number of parents per node is bounded.\n\nBecause it includes inference in propositional logic as a special case, inference in Bayes nets is NP-hard (reduce 3-SAT to it). Also #P-hard (strictly harder than NP-complete).\n\nTricks to make CSP solving more efficient (tree width, etc) can be applied to Bayes nets.\n\nWeighted model counting (WMC; computes sum of total weight of satisfying assignments, where weight is product of conditional probabilities for each variable assignment given its parents) solver can solve translated Bayes net.\n\nFor multiple variable queries, can use **clustering** (**join tree**) algorithms to get linear time in size of polytree Bayes net. For arbitrary topology, can join individual nodes such that the resulting network is a polytree. Two boolean nodes replaced with **meganode** that takes on 4 possible values. Special purpose inference algo (that handles meganodes that share vars) like constraint propagation.\n\n\n## Module 11 Lecture Notes\n\nBayes nets aka graphical models.\n\nNot every BN can represent every join distribution (topology enforces certain conditional independencies).\n\nTopology may happen to encode causal structure, really encodes conditional independence.\n\nA joint distribution over $N$ boolean variables is $2^N$ size, an N-node net with each node having up to $k$-parents is $O(N*2^{k+1})$ space.\n\n$X$ and $Y$ are conditionally independent given evidence variables $Z$ if $X$ and $Y$ are d-separated by $Z$; consider all (undirected) paths from $X$ to $Y$; no active paths = independence. A path is active if each triple is active. All it takes to block a path is a single inactive segment.\n\nThere might be more independence that can't be infered from structure of BN. Adding arcs increases set of distributions, but has costs. Full conditioning can encode any distribution.\n\n\n## Module 11 Definitions\n* **Conditioning case** - a possible combination of values for the parent nodes (a miniature possible world)\n* **Conditional probability table (cpt)** - CPT\n* **Locally structured (sparse) system** - each subcomponent interacts directly with only a bounded number of other components irrespective of total size (usually ~linear complexity)\n* **Context-specific independence (csi)** - variable is conditionally independent of some parents given *certain values* of others (can be specified with if-else syntax)\n* **Nonparametric representation** - define conditional distribution implicitly with collection of instances (examples of parent/child values)\n* **Hybrid bayesian network** - network with both discrete and continuous variables\n* **Linear-gaussian** - conditional distribution; child Gaussian distribution mean $\\mu$ relies linearly on the value of the parent and $\\sigma$ is fixed\n* **Hidden variables** - in Bayes net: neither input nor output, but keep network reasonably sparse with manageable number of parameters.\n* **Factor (variable elimination)** - prob table with unnormalized entries; matrix with one dimension for each variable ($Factor[val1][val2][...]\\propto P(var1=val1,var2=val2,...)$)\n* **Pointwise product of factors $f(...)(x)g(...)$** - new factor whose variables are union of variables in $f$ and $g$, elements given by product of corresponding elements\n* **Summing a variable out of a factor** - sum submatrices corresponding to each value of variable\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\nAny component of agent program can be improved with ML, depending on factors 1. which component to improve, 2. what prior knowledge the agent has, influencing the model it builds, 3. what data and feedback on that data is available.\n\nComponents of agent include \n1. direct mapping from conditions on current state to actions\n2. means to infer properties of world from percept sequence\n3. info about how world evolves and results of actions\n4. utility information indicating desirability of world states\n5. action value info indicating desirability of actions\n6. goals describing most desirable states\n7. problem generator, critic, and learning element enabling system to improve\n\nSeveral models for agent components: atomic, factored, and relational models based on logic or probability. Learning algos for all.\n\nML assumes little **prior knowledge** of agent; starts from scratch, learns from data.\n\nThree types of **feedback** can accompany inputs, determining type of learning:\n1. **supervised learning**: agent observes input-output (output=**label**) pairs and learns a function that maps input to output\n2. **unsupervised learning**: agent learns patterns in the input without any explicit feedback (e.g. **clustering**)\n3. **reinforcement learning**: agent learns from a series of reinforcements (rewards and punishments)\n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n\nGiven **training set** of $N$ example input-output pairs $\\{(x_i,y_i)\\}_{i=1}^N$ generated by $y_i=f(x_i)$, learn a function $h$ (**hypothesis**, drawn from **hypothesis space** $H$ of possible functions) that approximates $f$. With alternative vocab, we say $h$ is a **model** (or **function**) of the data, drawn from a **model class** (or **function class**) $H$. Output $y_i$ called **ground truth**.\n\nHow to choose hypothesis space? Might use prior knowledge, or perform **exploratory data analysis** (examining data with statistical tests and visualizations). Or can try multiple dif hypothesis spaces and evaluate which one works best.\n\nHow do we choose a good hypothesis from $H$? Could hope for **consistent hypothesis**: an $h$ such that each $x_i$ in the training set has $h(x_i)=y_i$. Unattainable with continuous-valued outputs; instead, look for **best-fit function** for which each $h(x_i)$ is close to $y_i$.\n\nTest of a hypothesis is how it handles inputs it hasn't yet seen. Evaluate test set. $h$ generalizes well if it accurately predicts the outputs of the test set.\n\nCan evaluate hypothesis space with the bias they impose (regardless of training dataset) and the variance they produce (from one training set to another).\n\nComplexity of model tough to define -- some NNs with billions of parameters can generalize well. Aim for \"appropriateness\", not \"simplicity\" in a model class\n\nSupervised learning can be done by choosing the hypothesis $h^*$ that is most probable given the data:\n$$h^* = argmax_{h\\in H} P(h\\mid data)$$\n$$h^* = argmax_{h\\in H} P(data\\mid h)P(h)$$\n(MLE except take into account prior probabilities; do we have some domain knowledge that makes some hypotheses better than others?)\n\nTradeoff between expressiveness of hypothesis space and the computational complexity of finding a good hypothesis within that space (**expressiveness-complexity tradeoff**). But an expressive language makes it possible for a simple hypothesis to fit the data, whereas restricting expressiveness of the language might mean that any consistent hypothesis must be complex.\n\nOverfitting becomes more likely as number of attributes grows, less likely as we increase the number of training examples. Larger hypothesis spaces have more capacity both to fit and to overfit. Some model classes are more prone to overfitting than others.\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n\nA **decision tree** maps vector of attribute values to output (\"decision\"). Reaches decision by performing sequence of tests (root to leaf). Each internal node tests an input attribute. I/O can be discrete or continuous.\n\nAny function in propositional logic can be expressed as a decision tree. Not all functions can be represented concisely.\n\nIntractable to find smallest consistent tree. With heuristics, can find one that is close to smallest. Learn-Decision-Tree algorithm: always test the most important attribute first (highest information gain), then recursively solve subproblems.\n\nEntropy (measure of uncertainty, units=bits) $H(V)$ of random variable $V$ with values $v_k$ having probability $P(v_k)$:\n$$H(V)=\\sum_k P(v_k)\\log_2(1/P(v_k)) = -\\sum_k P(v_k)\\log_2P(v_k)$$\nDefine $B(q)$ as entropy of boolean random variable that is true with probability $q$:\n$$B(q)=-(q\\log_2q+(1-q)\\log_2(1-q))$$\nIf a training set contains $p$ positive examples and $n$ negative examples, entropy of output on whole set is\n$$H(output) = B(p/(p+n))$$\nAn attribute $A$ with $d$ distinct values divides the training set $E$ into subsets $E_1,...,E_d$, where $E_k$ has $p_k$ positive examples and $n_k$ negative examples. Randomly chosen example from training set has the $k$th value for the attribute (i.e. is in $E_k$) with prob $(p_k+n_k)/(p+n)$. So expected entropy remaining after testing $A$ is:\n$$Remainder(A) = \\sum_{k=1}^dB(p_k/(p_k+n_k))(p_k+n_k)/(p+n)$$\n**information gain** from attribute test on $A$ is expected reduction in entropy:\n$$Gain(A)=B(p/(p+n))-Remainder(A)$$\n\n**Decision tree pruning** combats overfitting, increases noise tolerance. **Chi-squared pruning**: iteratively eliminate leaf nodes with information gain below threshold, set with **significance test**. **null hypothesis**: attribute is irrelevant (zero information gain for infinitely large sample space). Calculate prob that under null hypothesis, sample size $v=n+p$ would show observed deviation by comparing $p_k$ and $n_k$ with expected numbers under null: $\\hat{p}_k = p(p_k+n_k)/(p+n)$ and $\\hat{n}_k = n(p_k+n_k)/(p+n)$. Total deviation given by:\n$$del = \\sum_{k=1}^d(p_k-\\hat{p}_k)^2/\\hat{p}_k + (n_k-\\hat{n}_k)^2/\\hat{n}_k$$\nunder null hypothesis, $del$ distributed according to chi-squared distribution with $d-1$ DOF (use $\\chi^2$ to confirm or reject null hypothesis). \nDecision trees can be improved by handling: missing data (not all attribute values known), continuous and multivalued input attributes (**split point** inequality test on value of attribute; for discrete attributes with large domains, can use **information gain ratio** to avoid splitting into single-example subtrees, allow **equality test**), continuous-valued output attribute (need **regression tree**, each leaf is linear function of subset of numerical attributes, eventually stop splitting and apply linear regression on attributes).\n\n\n## AIMA Chapter 5.5 Stochastic Games\n\nGoal in ML: select a hypothesis that optimally fits future examples. Assume future examples will be like the past (**stationarity** assumption): examples are IID.\n\nFinding a good hypothesis involves **model selection** (choose good hypothesis space) and **optimization** (aka **training**) finds best hypothesis within space. Model selection can involve both domain knowledge and quant comparison using validation set.\n\nAs we add capacity to model class, eventually can exactly represent all training examples (model has **interpolated** the data). Model classes start to overfit as capacity approaches point of interpolation (stop remembering patterns, just memorize data).\n\nDNNs, kernel machines, random forests, and boosted ensembles have property that validation set error tends to decrease as capacity increases.\n\nAccuracy not full picture of kinds of errors. Want to minimize **loss function**, amount of utility lost by predicting $h(x)=\\hat{y}$ when correct answer is $f(x)=y$: $$L(x,y,\\hat{y}) = Utility(h(x)=y) - Utility(h(x)=\\hat{y})$$\noften simplified version specified without $x$. $L(y,y)=0$. Can create loss function with table for discrete outputs, but not real outputs. Some loss functions for real outputs:\n- Absolute-value loss: $L_1(y,\\hat{y})=|y-\\hat{y}|$\n- Squared-error loss: $L_2(y,\\hat{y})=(y-\\hat{y})^2$\n- 0/1 loss: $L_{0/1}(y,\\hat{y})=0$ if $y=\\hat{y}$ else 1\nExpected **generalization loss** for a hypothesis $h$ (where $\\epsilon$ represents set of all possible input-output examples):\n$$GenLoss_L(h)=\\sum_{(x,y)\\in \\epsilon}L(y,h(x))P(x,y)$$\nbest hypothesis $h^*$ minimizes expected generalization loss. To estimate generalization loss, use **empirical loss** on set of examples $E$ of size $N$:\n$$EmpLoss_{L,E}(h)=\\sum_{(x,y)\\in E}L(y,h(x))/N$$\nestimated best hypothesis $\\hat{h}^*$ minimizes empirical loss.\n\n$\\hat{h}^*$ may differ from true function $f$ due to unrealizability, variance (->0 as number of training examples increases if problem is realizable), noise, and computational complexity (if $H$ is a complicated function in a large hypothesis space, cannot search all possibilities; cannot guarantee finding best hypothesis).\n\nFight overfitting: can minimize weighted sum of empirical loss and complexity (\"total cost\"), with hyperparameter $\\lambda$ (if chosen well, nicely balances empirical loss of simple function with complicated function's tendency to overfit):\n$$Cost(h)=EmpLoss(h)+\\lambda Complexity(h)$$\nRegularization (penalizing complex hypotheses) involves choosing loss function and the complexity measure, aka **regularization function**. Choice of regularization function depends on hypothesis space.\n\nSimplify model by reducing dimensions that the models work with. **Feature selection**: discard irrelevant attributes.\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nHow many training examples needed? **Computational learning theory**: any seriously wrong hypothesis will by outed with high prob after small number of exmaples (will make incorrect prediction). Thus, any hypothesis consistent with sufficiently large set of training examples unlikely to be seriously wrong (must be **probably approximately correct (PAC)**). Learning algo that returns PAC hypotheses called a **PAC learning** algo. PAC learning motivated by stationarity assumption. Simplest deal with boolean functions ($L_{0/1}$ appropriate). **error rate** of hypothesis $h$ defined as expected generalization error for examples drawn from the stationary distribution, i.e. probability $h$ misclassifies a new example:\n$$error(h)=GenLoss_{L_{0/1}}(h)=\\sum_{x,y}L_{0/1}(y,h(x))P(x,y)$$\nHypothesis $h$ is **approximately correct** if $error(h)<\\epsilon$ for small constant $\\epsilon$ (inside $\\epsilon$-ball around true function $f$, surrounded by $H_{bad}$ section of hypothesis space). Can find $N$ such that all consisteny hypotheses approximately correct after training on $N$ examples. Prob seriously bad hypothesis $h_b\\in H_{bad}$ is consistent with first $N$ examples:\n$$P(h_b agrees with n examples)\\leq (1-\\epsilon)^N$$\nCan keep probability of $H_{bad}$ containing at least one consistent hypothesis below $\\delta$ if $N\\geq(\\ln(1/\\epsilon)+\\ln|H|)/\\epsilon$. With this $N$, learning algorithm will return an approximately correct hypothesis with probability at least $1-\\delta$. The number of required examples is called the **sample complexity**. To obtain real generalization to unseen examples, need to restrict hypothesis space $H$ without eliminating true function: use prior knowledge, insist algorithm return *simple* consistent hypothesis, or focus on learnable subsets of entire hypothesis space (assuming restricted space contains $h$ that is close enough to $f$).\n\n**Decision list** consists of series of tests (each conjunction of literals): if test succeeds when applied to example description, DL specifies value to return. If test fails, continue to next test. Can restrict size of each test to at most $k$ literals to force generalization ($k$-DL(n); uses $n$ boolean attributes; includes $k$-DT, set of decision trees of depth at most $k$). We have\n$$|k-DL(n)|=2^{O(n^k\\log_2(n^k))}$$\nnum examples needed for PAC-learning a $k-DL(n)$ function is polynomial in $n$:\n$$N\\geq(\\ln(1/\\delta)+O(n^k\\log_2(n_k)))/\\epsilon$$\nAlgo to find consistent decision list: greedily find test that agrees with subset of training set (choose smallest test $t$ that matches any uniformly classified subset), then construct remaining decision list with remaining examples (until no examples left).\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\n**Univariate linear function** (straight line) has form $h_w(x)=w_1x+w_0$ with weight vector $w=[w_0,w_1]$. Finding best $h_w$ to fit data is **linear regression**. $L_2$ loss:\n$$Loss(h_w)=\\sum_{j=1}^N(y_j-(w_1x_j+w_0))^2$$\nUnique solution:\n$$w_1=(N(\\sum x_jy_j) - (\\sum x_j)(\\sum y_j))/(N(\\sum x_j^2)-(\\sum x_j)^2)$$\n$$w_0=(\\sum y_j - w_1(\\sum x_j))/N$$\nPlotting loss function in weight space shows loss function is **convex**. This is true for *every* linear regression problem with an $L_2$ loss function (no nonglobal local minima).\n\nWe can search through a continuous weight space by incrementally modifying parameters (**hill climbing** for maximization, **gradient descent** for minimizing loss). Choose starting point in weight space, compute estimate of gradient and $(alpha)(\\delta/\\delta w_i)Loss(w)$ in steepest downhill direction, repeat until converge on point with (local) minimum loss. Parameter alpha is called **step size** or **learning rate**. Can be a fixed constant or decay over time as learning proceeds (guarantees convergence). A step that covers all the training examples is called an **epoch**. **batch gradient descent** (**deterministic gradient descent**) computes gradient over whole training set. **stochastic gradient descent** (SGD) (**online gradient descent**; useful when new data coming in one at a time) randomly selects small number of training examples at each step (traditionally just 1) and updates according to gradient on that set. **Minibatch** gradient descent chooses $m$ out of the $N$ examples. Can choose $m$ to take advantage of parallel vector operations, then $m$ is hyperparameter.\n\n**Multivariable linear regression**: each example $x_j$ is an $n$-element vector, hypothesis space is of functions of the form $h_W(x_j)=w_0+\\sum_i w_ix_{j,i}$. Invent dummy input attribute $x_{j,0}:=1$, then $h_w(x_j)=w\\cdot x_j=w^{T}x_j$. Want to minimize $L_2$ loss. Let $X$ be **data matrix** (one $n$-dimensional example per row):\n$$L(w)=||\\hat{y}-y||^2=||Xw-y||^2$$\n$$w^*=(X^TX)^{-1}X^Ty$$\nthis is called the **normal equation**, and $(X^TX)^{-1}$ is called the **pseudoinverse** of the data matrix.\n\nOverfitting common on multivariable linear functions. We minimize\n$$Cost(h)=EmpLoss(h)+\\lambda Complexity(h)$$\n(equivalent to minimizing $Loss(w)$ subject to $Complexity(w)\\leq c$ for constant $c$ related to $\\lambda$).\nFor linear functions, can consider family of regulization functions\n$$Complexity(h_w)=L_q(w)=\\sum_i|w_i|^q$$\n$L_1$ regularization produces **sparse model** (sets many weights to zero, like pruning). Number of examples needed to find good $h$ is logarithmic for $L_1$ and linear for $L_2$ regularization.\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\nText classification has many applications. Even language modeling can be seen as classification (each word is a class).\n\nIntroduce **multinomial naive Bayes classifier**. We represent a text document as a **bag-of-words** (unordered multiset, no position).\n\nChoose class $\\hat{x}$ for document $d$ such that \n$$\\hat{c}=argmax_{c\\in C}P(c\\mid d)=argmax_{c\\in C}P(d\\mid c)P(c)$$\ngenerative because it is an assumption about how the document is generated. Can represent document $d$ as set of featrures $f_1,...,f_n$:\n$$\\hat{c}=argmax_{c\\in C}P(f_1,...,f_n\\mid c)P(c)$$\napply **naive Bayes assumption**:\n$$\\hat{c}=argmax_{c\\in C}P(c\\mid d)=argmax_{c\\in C}P(c)\\prod_{f\\in F}P(f\\mid c)$$\nto apply to text with word positions:\n$$c_{NB}=argmax_{c\\in C}P(c)\\prod_{i\\in positions}P(w_i\\mid c)$$\ndone in log space to avoid underflow and increase speed\n$$c_{NB}=argmax_{c\\in C}\\log P(c)\\sum_{i\\in positions}\\log P(w_i\\mid c)$$\nin log space, this is linear combo of input features; NB is a linear classifier.\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n\nLearn $P(c)$ and $P(f_i\\mid c)$ with MLE ($N_c$ number of documents in training data with class $c$ and $N_{doc}$ total documents):\n$$\\hat{P}(x)=N_c/N_{doc}$$\nfor $P(f_i\\mid c)$, assume feature is just existence of word in document's bag of words:\n$$\\hat{P}(w_i\\mid c)=count(w_i,c)/(\\sum_{w\\in V}count(w,c))$$\nvocabulary $V$ consists of union of all word types in all classes\n\nif word does not occur, probability of feature is 0, so product is 0. Can use add-one (Laplace) smoothing, common in NB:\n$$\\hat{P}(w_i\\mid c)=(count(w_i,c)+1)/((\\sum_{w\\in V}count(w,c))+|V|)$$\n\nLaplace smoothing does poorly for $P(X\\mid Y)$ when $|X|$ is very large or $|Y|$ is very large. Another option: linear interpolation\n$$P_{LIN}(x\\mid y)=(alpha)\\hat{P}(x\\mid y)+(1-alpha)\\hat{P}(x)$$\n\nignore **unknown words** in test data that didn't show up in training. Also can ignore **stop words** (very frequent, e.g. the and a); sort vocabulary by frequency and define top words as stop words, or use predefined stop word list. Usually removing stop words doesn't improve performance.\n\n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n\nFor sentiment analysis, helps to clip word counts in each document at 1 (**binary multinomial naive Bayes** or **binary NB**). Also, helps to deal with negation; prepend NOT_ prefix to all words affected by logical negation term.\n\nCan also derive positive and negative word features from **sentiment lexicons**, lists of words that are pre-annotated with positive or negative sentiment. Add a feature that is counted whenever a word from that lexicon occurs. Adding these features works well only if training data is sparse and non-representative of test set. Using dense lexicon features instead of sparse individual-word features may generalize better.\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n\nSpam detection: rather than using all words as individual features, predefine likely sets of words or phrases as features, combined with features that are not purely linguistic.\n\nCharacter n-grams or byte n-gramscan be good features for other applications.\n\n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n\nIf features are just words and we use all words in text, NB model can be viewed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. Model assigns a probability to each sentence $P(s\\mid c)=\\prod_{i\\in positions}P(w_i\\mid c)$.\n\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n\nTo evaluate system for detecting things, can build a **confusion matrix** (visualizes how an algorithm performs with respect to human gold labels); dimensions are system output (positive, negative) and gold labels (positive, negative). \n\n**Accuracy**: what percentage of observations are correct: $(tp+tn)/(tp+fp+tn+fn)$. Doesn't work well when classes are unbalanced (e.g. with rare events, detection).\n\n**Precision** measures the percentage of items that the system detected that are in fact positive: $tp/(tp+fp)$\n\n**Recall** measures the percentage of items actually present in the input that were correctly identified by the system: $tp/(tp+fn)$.\n\nPrecision and recall emphasize true positives.\n\nCombination of precision and recall: F-measure\n$$F_b=(b^2+1)PR/(b^2P+R)$$\nwhere the $b$ (beta) parameter weighs importance of precision and recall, based on needs of application. $b>1$ favor recall, $b<1$ favor precision. $b=1$ balanced, called $F_1$. Weighs lower of two numbers more heavily.\n\nMulticlass: more cells in confusion matrix. In **macroaveraging**, we compute the performance for each class, and then average over classes. In **microaveraging**, we collect the decisions for all classes into a single confusion matrix, and compute precision and recall from that table. Microaverage dominated by more frequent class, macroaverage better when performance on all the classes is equally important.\n\n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n\nLet's call $M(A,x)$ the score system $A$ gets on test set $x$, and $\\delta(x)$ the performance difference between $A$ and $B$ on $x$:\n$$\\delta(x)=M(A,x)-M(B,x)$$\n$\\delta(x)$ is called the **effect size**. Would like to know if effect size is positive. Null hypothesis: effect size is non-positive. $p$-value is $P(\\delta(X)\\geq \\delta(x)\\mid H_0)$. Very small p-value means difference observed is very unlikely under $H_0$. If test passes p-value threshold, then the result \"$A$ is better than $B$\" is **statistically significant**.\n\nIn NLP, common to estimate $P(\\delta(X)\\geq \\delta(x)\\mid H_0)$ using non-parametric tests based on sampling: measure $\\delta(x')$ for lots of different test sets $x'$, which gives us a distribution.\n\nTwo common non-parametric tests in NLP: **approximate randomization** and the **bootstrap test**. **Paired** tests are those in which we compare two sets of observations that are aligned (each observation in one set can be paired with an observation in another).\n\n**bootstraping**: repeatedly drawing large numbers of smaller samples with replacement (called **bootstrap samples**) from an original larger sample\n\n**Paired bootstrap test**: create many virtual test sets from an observed test set by repeatedly sampling from it.\n$$p-value(x)=(\\sum_{i=1}^b int(\\delta(x^{(i)} \\geq 2\\delta(x)))/b$$\n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n\nRelease **model card** with info about how model was trained, motivation, intended use, users, and performance across different demographic or other groups and environments.\n\n\n## Module 12 Lecture Notes\n\nFor classification, classes separated by **decision boundary**, linear decision boundary is called a **linear separator** (compliant data are **linearly separable**). Write classification hypothesis $h_w(x) = 1$ if $w\\cdot x\\geq 0$ else 0. Equivalently, $h_w(x)=Threshold(w\\cdot x)$ where $Threshold(z)=int(z\\geq 0)$. Choose weights $w$ to minimize the loss. Weight update rule that converges to a solution if the data are linearly separable:\n$$w_i += (alpha)(y-h_w(x)) x_i$$\nthis is called the **perceptron learning rule**, typically applied one random example at a time. If alpha decays as $O(1/t)$ with iteration number $t$, perceptron rule will converge to a minimum-error solution (NP hard) even if data is not linearly separable.\n\nSoften threshold function, use logistic function:\n$$h_w(x)=Logistic(w\\cdot x)=1/(1+\\exp(-w\\cdot x))$$\nfor **logistic regression**, use gradient descent with $L_2$ loss function. For logistic function $g$,\n$h_w'(x) = h_w(x)(1-h_w(x))$$\nupdate rule\n$$w_i+=(alpha)(y-h_w(x))\\cdot h_w(x)(1-h_w(x))\\cdot x_i$$\nlogistic regression far better when data is noisy / not perfectly linearly separable.\n\nOften useful to use naive (or strong) baseline model to compare against new model.\n\n\n## Module 12 Definitions\n* **Transfer learning** - knowledge from one domain is transferred to a new domain, so learning can proceed faster with less data\n* **Induction** - going from a specific set of observations to a general rule (may be incorrect)\n* **Factored representation** - vector of attribute values\n* **Classification** - output is one of finite set of values\n* **Regression** - output is a number\n* **Bias** - tendency of a predictive hypothesis to deviate from expected value when averaged over different training sets (low complexity-> high bias)\n* **Hypothesis is underfitting** - when it fails to find a pattern in the data (too high bias)\n* **Variance** - amount of change in the hypothesis due to fluctuation in the training data\n* **Hypothesis is overfitting** - when it pays too much attention to particular dataset it's trained on (performs poorly on unseen data); too high variance\n* **Bias-variance tradeoff** - choice between more complex, low-bias hypotheses (fit training data well) and simpler, low-variance hypotheses that generalize better\n* **Boolean classification** - have outputs that are either true (**positive example**) or false (**negative example**)\n* **Boolean decision tree** - decision tree with discrete inputs and boolean output\n* **Learning curve** - used to evaluate the performance of a learning algorithm; accuracy vs training set size; usually increasing (also called **happy graphs**)\n* **Problems with decision trees** - suboptimal accuracy, expensive if deep, **unstable** (adding one new example can change entire tree); **Random forest model** improves\n* **Hyperparameters** - tunable parameters specifying the model class, as opposed to normal parameters (specify the model)\n* **K-fold cross validation** - split train data into $k$ pieces, do $k$ rounds of learning (a dif piece held out as validation set each round), consider average test score\n* **Leave-one-out cross-validation (loocv)** - k-fold cross-validation with $k=n$\n* **Learning problem is realizable** - if the hypothesis space $H$ actually contains the true function $f$\n* **Small-scale learning** - (traditional methods) dozens to thousands of training examples; generalization loss from approximation error and estimation error not enough examples\n* **Large-scale learning** - (more modern) millions of training example; generalization loss dominated by limits of computation\n* **Minimum description length (mdl) hypothesis** - complexity ~num bits of Turing machine program. empirical loss: num bits to encode data (correct=0, wrong depends on error size)\n* **(hyperparam tuning) hand tuning** - manual hyperparameter tuning\n* **(hyperparam tuning) grid search** - exhaustive search of all possible combos of hyperparam values\n* **(hyperparam tuning) random search** - randomly select from set of hyperparameter values, useful if too many combos for exhaustive search (or continuous)\n* **(hyperparam tuning) bayesian optimization** - trades off exploration and exploitation (staying near best values) UCS can be used to miminize regret\n* **(hyperparam tuning) population-based training (pbt)** - train group of models in parallel with dif hyperparams, choose next-gen based on previous results, random mutation (like genetic algo)\n* **Training curve** - measures classifier performance on a fixed training set as the learning process proceeds one update at a time\n* **Text categorization** - task of assigning a label or category to entire text or document\n* **Sentiment** - positive or nevative orientation that a writer expresses towards some subject\n* **Sentiment analysis** - extraction of sentiment from a text\n* **Probabilistic classifier** - additionally will tell us the probability of the observation being in the class\n* **Generative classifiers** - (e.g. naive Bayes) build a model of how a class could generate some input data\n* **Discriminative classifiers** - (e.g. logistic regression) learn what features from the input are most useful to discriminate between the different possible classes\n* **Linear classifier** - classifiers that use a linear combination of the inputs to make a classification decision\n* **Representational harms** - harms caused by a system that demeans a social group, e.g. by perpetuating negative stereotype\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\nNote sigmoid satisfies $1-\\sigma(x)=\\sigma(-x)$.\n\nLogistic regression classifier: $P(y=1)=\\sigma(w\\cdot x+b)$, $decision(x)$ = 1 if $P(y=1\\mid x) > 0.5$ else 0 (**decision boundary**).\n\nFeature engineering. For NB and logistic regression, can combine primitive features into more complex features or **feature interactions** (designed by hand). For many tasks, need large number of features, can be created automatically via **feature templates**. **Representation learning**, ways to learn features automatically in an unsupervised way.\n\nLogistic regression more accurate than NB when many correlated features (usually better for large datasets). NB can work very well on small datasets or short documents, easy to implement and fast to train.\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\nNeed loss function $L(\\hat{y},y)$ to specify how much $\\hat{y}$ differs from  true $y$. We want log function that prefers correct class labels of training examples to be *more likely*. **Conditional maximum likelihood estimation**: choose parameters $w,b$ that **maximize the log probability of the true y labels in the training data** given the observations $x$. Resulting function is the log likelihood loss, aka **cross entropy loss**:\n$$L_{CE}(\\hat{y},y)=-[y\\log\\sigma(w\\cdot x+b)+(1-y)\\log(1-\\sigma(w\\cdot x+b))]$$\n**cross entropy** between true probability dist $y$ and our estimated prob dist $\\hat{y}$: $y\\log(\\hat{y})+(1-y)\\log(1-\\hat{y})$.\n\nSolve with gradient descent. Loss function is convex. Gradient of cross entropy loss function for logistic regression:\n$$(\\delta/\\delta w_j)L_{CE}(\\hat{y},y)=[\\sigma(w\\cdot x + b)-y]x_j$$\n\n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n\n**Multinomial logistic regression** aka **softmax regression**, want to find $P(y=c\\mid x)$ for each $c\\in C$. Uses softmax to map vector to vector of probabilities:\n$$softmax(z_i)=\\exp(z_i)/\\sum_{j=1}^d\\exp(z_j)$$\nWe have\n$$p(y=c\\mid x)=\\exp(w_c\\cdot x+b_c)/(\\sum_{j=1}^K\\exp(w_j\\cdot x+b_j))$$\nLoss\n$$L_{CE}(\\hat{y},y)=-\\sum_{k=1}^K y_k\\log \\hat{P}(y=k\\mid x)$$\n$y$ is a **one-hot vector** (only one component has a 1, rest 0). So $L_{CE}(\\hat{y},y)=-\\log \\hat{y}_k$ where $k$ is the correct class;\n$$L_{CE}(\\hat{y},y) = -\\log (\\exp(w_k\\cdot x + b_k)/\\sum_{j=1}^K\\exp(w_j\\cdot x+b_j))$$\nThe gradient \n$$(\\delta/\\delta w_{k,i}) = -(int(y=k)-P(y=k\\mid x))x_i$$\n\nCan investigate which features are significant with statistical tests, or inspecting magnitude of weight.\n\n\n## SLP Chapter 7.1 Units\n\nBuilding block of neural net is **unit**, taking set of real valued numbers as input, performs some computation, then produces an output. Neural unit:\n$$z=w\\cdot x + b$$\nthen apply non-linear function $f$ to $z$ to get **activation** value.\n\nCommon activation functions: sigmoid, tanh, ReLU.\n\nSigmoid and tanh have problem that values of $y$ can become **saturated** (extremely close to 1, derivative close to 0), causing **vanishing gradient**.\n\nPerceptron acts as linear **decision boundary**, and XOR is not **linearly separable**.\n\nNeural net with nonlinear activation function can model XOR (linear activation function translates to 1-layer net).\n\n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n\n**Feedforward network** (**multi-layer perceptron** or MLP) is a multilayer network in which units are connected with no cycles, outputs passed strictly forward.\n\nHas input layer, hideden layers, output layer. Standard architecture: layers fully connected. Parameters for hidden layer: weight matrix $W$ and bias vector $b$ for whole layer. $W_{ji}$ represents weight of connection from $i$th input unit $x_i$ to the $j$th hidden unit $h_j$. Hidden layer:\n$$h = \\sigma(Wx+b)$$\n\nOne hidden layer case:\nInput layer is layer 0, $n_0$ is number of inputs, so $x\\in R^{n_0}$, hidden layer has $h\\in R^{n_1}$. Weight matrix $W\\in R^{n_1*n_0}$. Hidden layer computes **representation** of input. Output layer has weight matrix $U$, produces intermediate output $z=Uh$, then normalize with softmax function:\" $y = softmax(z)$.\n\nGeneral notation: superscripts in square brackets mean layer number, starting at 0 for input layer. $n_j$ means number of units at layer $j$. $a^{[i]} means output from layer $i$, and $z^{[i]}$ represents combination of weights and biases at that layer.\n\nTo represent bias, add dummy node $a^{[i]}_0:=1$ so $h=\\sigma(Wx)$.\n\nIn NLP: represent text inputs to NNs as word embeddings $[e_{w_1};,,,;e_{e_w_n}]$ -- **pretraining**.\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n\nNeural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction. On the other hand, they are more complex, slower to train, and less interpretable. For many smaller tasks n-gram model can still be the right tool.\n\nNeural LM takes context as input and outputs word probabilities. Model stores patterns learned from training data, representing words as embeddings.\n\nFeedforward NLM has moving window that can see $N$ words into the past. Multiply each one-hot vector (word) by embedding weight matrix $E$ (column for each word, $d$ rows); multiplying by one-hot vector selects column vector (word embedding) for that word. Then concatenate embedding vectors to produce $e$, the embedding layer. Follow by hidden layer and an output layer whose softmax produces prob distribution.\n\n\n## SLP Chapter 7.6 Training Neural Nets\n\nLoss function for NN: use cross entropy loss (same as logistic regression). Train with gradient descent. Compute gradient w.r.t. parameters with **error backpropagation** or **backwards differentiation**.\n$$L_{CE}(\\hat{y},y) = -\\log \\hat{y}_i$$\nwhere $i$ is the correct class.\n\nComputation graph like NN. **Backwards pass** / backwards differentiation: just use chain rule.\n\nUseful derivatives:\n$$(\\delta/\\delta z)\\sigma(z) = \\sigma(z)(1-\\sigma(z))$$\n$$(\\delta/\\delta z)tanh(z) = 1-tanh^2(z)$$\n$$(\\delta/\\delta z)ReLU(z) = 0 for z<0, 1 for z\\geq 0$$\n\nOptimization in NNs is a non-convex optimization problem, more complex than for logistic regression. Should initialize weights with small random numbers. Helpful to normalize input values to have 0 mean and unit variance. Regularization technique: **dropout**, randomly dropping some units and their connections from the NN during training. Tuning hyperparameters is also important (learning rate, mini-batch size, model architecture, number of hidden nodes per layer, the choice of activation functions, how to regularize, etc). Modern NNs are optimmized for parallelization on vector-based GPUs.\n\n\n## SLP Chapter 7.7 Training the neural language model\n\nOK to **freeze** embedding layer $E$ (use some pretraining algorithm, then hold $E$ constant while modifying params of NN).\n\nCan also train NN and $E$ together. Make sure weights from document window one hot vectors to embedding layer share *same* $E$ (not dif for each one hot vector). $E$ can then be used for other tasks.\n\n\n## Module 13 Lecture Notes\n\nCan improve perceptrons by adjusting weight update. MIRA: choose update size that fixes current mistake but minimizes change to $w$:\n$$\\min_w (\\sum_y ||w_y-w_y'||^2)/2$$\n$$w_y^*\\cdot f(x) \\geq w_y\\cdot f(x)+1$$\n\nAnother solution: cap step size with constant $C$ (optimization that assumes non-separable data); usually converges faster than perceptron. Usually better, especially on noisy data.\n\n**Support Vector Machines (SVMs)** maximize margin; only support vectors matter, other training examples are ignorable.\n\nTerm document matrix: rows are words, columns are documents, cells are counts.\n\nTerm-term matrix (term-context matrix): value in cell indicates how often word appears in a context window surrounding other word.\n\nAlternative to counts: TF-IDF (term frequency in a document) / (number of documents term occured in)\n\nTrain classifier to predict neighboring words, and learn dense embeddings for words in the training corpus. Skip gram: positive examples from context window, negative examples (noise words) from outside window. Pick noise words according to \n$$P_a(w)=count(w)^a/\\sum_w count(w)^a$$\n$a=3/4$ works well (rare noise words slightly higher prob)\n\nWord embeddings can solve analogies, vector arithmetic, visualizations.\n\n\n## Module 13 Definitions\n* **Sigmoid function** - \\sigma(z)=1/(1+\\exp(-z))\n* **Tanh function** - variant of sigmoid that ranges [-1,1]: $y=(e^z-e^{-z})/(e^z+e^{-z})$\n* **Relu function** - simplest / most common activation function: $y = \\max(z,0)$\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\n\nCan improve perceptrons by adjusting weight update. MIRA: choose update size that fixes current mistake but minimizes change to $w$:\n$$\\min_w (\\sum_y ||w_y-w_y'||^2)/2$$\n$$w_y^*\\cdot f(x) \\geq w_y\\cdot f(x)+1$$\n\nAnother solution: cap step size with constant $C$ (optimization that assumes non-separable data); usually converges faster than perceptron. Usually better, especially on noisy data.\n\n**Support Vector Machines (SVMs)** maximize margin; only support vectors matter, other training examples are ignorable.\n\nTerm document matrix: rows are words, columns are documents, cells are counts.\n\nTerm-term matrix (term-context matrix): value in cell indicates how often word appears in a context window surrounding other word.\n\nAlternative to counts: TF-IDF (term frequency in a document) / (number of documents term occured in)\n\nTrain classifier to predict neighboring words, and learn dense embeddings for words in the training corpus. Skip gram: positive examples from context window, negative examples (noise words) from outside window. Pick noise words according to \n$$P_a(w)=count(w)^a/\\sum_w count(w)^a$$\n$a=3/4$ works well (rare noise words slightly higher prob)\n\nWord embeddings can solve analogies, vector arithmetic, visualizations.\n\n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}