{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc33e143-2e52-45b3-bbc9-886d0803253a"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "72b4d993-ded7-4607-c915-2296f410ffe9"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"Lili Stoesser\"\n",
        "permission = True\n",
        "anonymous = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "equation_1 = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "equation_2 = r\"\"\" $EU(a)) = \\sum_s P(\\textnormal{RESULT}(a)=s')U(s')$ \"\"\"\n",
        "equation_3 = r\"\"\" $action = \\textnormal{argmax}_{a}EU(a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" An makes a decision: has avaliable some actions a, assigns a probability\n",
        "P(s) to each possible state. The transition model is given by P(s'|s,a) the probability that \n",
        "action a in state s reaches state s' \"\"\" + equation_1 + \"\"\" An agent's preferences \n",
        "are captured by a utility function. Expected utility: \"\"\" + equation_2 + \"\"\" MEU: \"\"\" + equation_3 + \"\"\" The MEU\n",
        "formalizes the geneearal notion that an intelligent agent should do the right thing but it \n",
        "does not operationalize that advice. An agent may not know how good a state is until it\n",
        "knows where it can go from that state. So if an agent acts so as to maximize a utility function\n",
        "that correctly reflects the performance measure, then the agent will achieve the highest \n",
        "possible perfomace score. \"\"\"\n",
        "\n",
        "chapter16_2 =  \"\"\" An agent's preferences (handwritten). Lottery (handwritten). All formulas\n",
        "are also handwritten. The axioms of utility theory are really axioms about prefereneces. But \n",
        "we can derrive the existence of a utility function, and the expected utility of a lottery.\n",
        "The outcome of a nondeterministic action is a lottery, an agent can only act rationally by \n",
        "choosing an action that maximizes expected utility. In a deterministic environment an agent\n",
        "needs only a preference ranking on states- the numbers dont matter\"\"\"\n",
        "\n",
        "chapter16_3 =  \"\"\" Utility functions map from lotteries to real numbers. \n",
        "\n",
        "16.3.1 Utility assessment and utility scales\n",
        "To built decision-theoretic system that helps a human make decisions. We must find \n",
        "the human's utility function, preference elicitation. We can assist the utility of \n",
        "any prize S by asking teh agentt to choose between S and a standard lottery. \n",
        "\n",
        "16.3.2 The utility of money\n",
        "If an agent prefers more money the agent has a monotonic preference for \n",
        "money. The utility of money was almost exactly proportional to the logarithm of the amount. \n",
        "\n",
        "16.3.3 Expected utility and post-decision disappointment:\n",
        "The rational way to choose the best action a*, is to maximize the expected utility: a* = argmax_aEU(a)\n",
        "Probability density function\"\"\" + r\"\"\" $P(x) = \\frac{d}{dx}((F(x)^{k})=kf(x)(F(x))^{k-1}$ \"\"\" + \"\"\" With\n",
        "more choices, extremely optimistic estimates are more likely to arise. We can avoid hte optimizer's curse\n",
        "with a Bayesian approach that uses and explicit probability model P(EU^|EU). \n",
        "\n",
        "16.3.4 Human judgment and irrationality\n",
        "Decision theory is a normative theory. Humans are \"predicatably irrational. An explanation\n",
        "for the apparently irrational preference is the certainty effect. People feel more comfortable\n",
        "making relative utility judgements rather than absolute ones. The very fact that different\n",
        "wordings of the same decision problem elicit different choices suggests that the decision problem itself is not getting through.\"\"\"\n",
        "\n",
        "equation_4 = r\"\"\" $U_{h}([s_{0},a_{0},s_{1},a_{1},...]) = R(s_{0},a_{0},s_{1})+\\gamma R(s_{1},a_{1},s_{2})+\\gamma^{2} R(s_{1},a_{1},s_{2})$ \"\"\"\n",
        "\n",
        "equation_5 = r\"\"\" $U^{\\pi}(s) = E[\\sum_{t=0}^{\\infty} (\\gamma^{t}R(S_{t},\\pi(S_{t}),S_{t+1})]$ \"\"\"\n",
        "\n",
        "equation_6 = r\"\"\" $\\pi_{s}^{*}=(\\textnormal{argmax}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ \"\"\"\n",
        "\n",
        "equation_7 = r\"\"\" $U(s)= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ \"\"\"\n",
        "\n",
        "equation_8 = r\"\"\" $U(s)= \\textnormal{max}_{a}Q(s,a)$ \"\"\"\n",
        "equation_9 = r\"\"\" $\\pi ^{*}= \\textnormal{argmax}_{a}Q(s,a)$ \"\"\"\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\" If the environment were deterministic, a solution would be easy.\n",
        "But the environment is stochastic so it's actions are unreliable. The transition model\n",
        "describes the outcome of each action in each state. P(s'|s,a) is the probability of reaching\n",
        "state s' if action a is done in state s. If a decision problem is sequential the utility\n",
        "function will depend on an environment history. For every transition from s to s' via a the\n",
        "the agent recieves a reward R(s,a,s'), they may be + or - but are bounded by +/- Rmax.\n",
        "The utility of an environment history is the sum of the rewards recieved. Methods for solving\n",
        "MDPs usually involve dynamic programming. No matter what the outcome of the action, the\n",
        "resulting state will be in the policy, and the agent will know what to do next. The quality\n",
        "of a policy is measured by the expected utility of the possiblke environment histories genterated\n",
        "by it. Optimal policy is denoted by pi*.\n",
        "\n",
        "17.1.1 Utilities over time\n",
        "First check if there is a finite horizon or an infinite horizon for decision making. \n",
        "With a finite horizon, an optimal action in a given state may depend on the time left.\n",
        "With no fixed time limit there is no reason to behave differently in the same state at differetn times\n",
        "With additive discounted rewards the utility of a history is:\"\"\" + equation_4 +\"\"\"three \n",
        "solutions to equations that lead to infinite histories: 1. with discounted\n",
        "rewards the utility of an infinite sequence is finite 2. if the environment has terminal\n",
        "states and we are guaranteed to get on eeventually then we will never need to compare infinite\n",
        "sequences 3. infinite sequences can be compared in terms of the average reward obtained \n",
        "per time step.\n",
        "\n",
        "17.1.2 Optimal policies and the utilities of states\n",
        "Compare policies by comparing their expected utilities them. \"\"\" + equation_5 +\"\"\"Optimal \n",
        "policy is\"\"\" + equation_6 +\"\"\"Consequence of using discounted utilities with infinite horizons is\n",
        "the optimal policy being independent of the starting state. The utility of a state is the expected reward for the\n",
        "next transition plus the discounted utility of the next state Bellman equation:\"\"\" + equation_7 + \"\"\"Q \n",
        "function:\"\"\" + equation_8 + \"\"\" We can extract the optimal policy from the q\n",
        "function\"\"\" + equation_9 + \"\"\" ``` \n",
        "function Q-value(mdp, s,a,U) returns a utility value\n",
        "  return sum_s' P(s'|s,a){[R(s,a,s')+gammaU[s']] ```\n",
        "  \n",
        "17.1.4 Representing MDPs\n",
        "Dynamic decision networks, or DDNs. They typically have an exponential complexity advantage\n",
        "over atomic representations and can model quite substantial real-world problems.\"\"\"\n",
        "equation_10 = r\"\"\" $U(s)_{i+1}= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U_{i}(s')])$ \"\"\"\n",
        "\n",
        "chapter17_2 =  \"\"\" Bellman update: \"\"\" + equation_10 +\"\"\"is to be applied simultaneously to \n",
        "all states at each iteration\n",
        "``` \n",
        " function Value-Iteration(mdp, e) returns a utility function\n",
        "    inputs mdp, with states S, actions A(s) transistion model P(s'|s,a), rewards R(s,a,a') discount gamma\n",
        "      e, the maximum error allowed in the utility of any state\n",
        "    local variables: U,U', vectors of utilities for stares in S, initially zero\n",
        "        delta, the maximum relative change in the utility of any state\n",
        "    repeat\n",
        "      U<- U'; delta <-0\n",
        "      for each state s in S do\n",
        "        U'[s]<-max_aeA(s) Q-Value(mdp,s,a,U)\n",
        "        if|U'[s]-U[s]| > delta then delta <-|U'[s] - U[s]|\n",
        "      until delta<= e(i-gamma)/gamma\n",
        "      return U\n",
        "``` \n",
        "Value iteration eventually converges to a unique set of solutions to the bellman equation. Contraction\n",
        "has one fixed point, when applied the function must get closer to the fixed point.\n",
        "\n",
        "Policy iteration stops when the policy improvemtn step yields no change in utilities\n",
        "``` \n",
        " function Policy-Iteration(mdp) returns a policy\n",
        "  inputs: mdp wiht stares, actions, transition model\n",
        "  local variables: U, a vector of utilites for states in S (initially 0)\n",
        "      pi, policy vector indexed by state, initially random\n",
        "\n",
        "  repeat\n",
        "    U<-Policy-Evaluation(pi, U, mdp)\n",
        "    unchanged <- true\n",
        "    for each state s in S do\n",
        "      a*<- argmax Q-value(mdp,s,a,U)\n",
        "      if Q-Value(mdp,s,a*,U)> Q-Value(mdp,s,pi[s],U) then\n",
        "        pi[s] <- a*; unchanged? <- false\n",
        "    until unchanged?\n",
        "    retunr pi\n",
        "``` \n",
        "Asynchronous policy iterstion is guaranteed to converege to an optimal policy. Linear Programming\n",
        "seeks to minimize U(s) for all s subject to the inequalities.\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\" In our search algorithms so far, the\n",
        "transition model was deterministic and\n",
        "described the outcome of each action in\n",
        "each state.\n",
        "The transition function is sometimes\n",
        "written as T(s, a, s’), or explicitly as a\n",
        "probability:\n",
        "\n",
        "Reward function\n",
        "We will specify a utility or reward function for the agent.\n",
        "The “rewards“ can be positive or negative but are bounded by some maximum value.\n",
        "Because the decision process is sequential, we must specify the utility function on a\n",
        "sequence of states and actions.\n",
        "Instead of only giving a reward at the goal states, the agent can receive a reward at\n",
        "each time step, based on its transition from s to s’ via action a.\n",
        "This is defined by a reward function\n",
        "R( s, a, s’ )\n",
        "For example, we could give the Millennium Falcon a small negative reward of -0.04 for every\n",
        "transition except for entering the terminal states (+1 for entering the planet’s orbit or -1 for\n",
        "smashing into an asteroid).\n",
        "The rewards are additive, so if the Millennium Falcon takes 4 steps before entering\n",
        "the planet's orbit, it gets -0.04 + -0.04 + -0.04 + -0.04 + 1 = 0.84 for that solution\n",
        "\n",
        "Markov Decision Process\n",
        "A Markov decision process or MDP is\n",
        "• a sequential decision problem\n",
        "• for a fully observable environment\n",
        "• with a stochastic transition model\n",
        "• that has additive rewards\n",
        "MDPs are non-deterministic search\n",
        "problems. One way of solving them is via\n",
        "expectimax search.\n",
        "\n",
        "Markov Decision Process\n",
        "To find a solution to an MDP, you need to define the following things:\n",
        "• A set of states s ∈ S\n",
        "• A set of actions a ∈ A\n",
        "• A transition function T(s, a, s’)\n",
        "• Probability that executing action a in s will lead to s’ P(s’ | s, a)\n",
        "• The probability is called the model\n",
        "• A reward function R(s, a, s’)\n",
        "• Sometimes just R(s) or R(s’)\n",
        "• An initial state s0\n",
        "• Optionally, one or more terminal states\n",
        "\n",
        "Solution == Policy\n",
        "In search problems a solution was a\n",
        "sequence of action that corresponded to\n",
        "the shortest path.\n",
        "Because of the non-determinism in MDPs\n",
        "we cannot simply give a sequence of\n",
        "actions.\n",
        "Instead, the solution to an MDP is a policy.\n",
        "A policy maps from a state onto the action\n",
        "to take if the agent is in that state.\n",
        "𝝿(s) = a\n",
        "\n",
        "Optimal Quantities\n",
        "▪ The value (utility) of a state s:\n",
        "V*(s) = expected utility starting in s and\n",
        "acting optimally\n",
        "▪ The value (utility) of a q-state (s,a): Q*(s,a) = expected utility starting out\n",
        "having taken action a from state s and\n",
        "(thereafter) acting optimally\n",
        "▪ The optimal policy:\n",
        "Pi*(s) = optimal action from state s\n",
        "\n",
        "Policy Iteration\n",
        "Alternative approach for optimal values:\n",
        "• Step 1: Policy evaluation: calculate utilities for some fixed policy (not\n",
        "optimal utilities!) until convergence\n",
        "• Step 2: Policy improvement: update policy using one-step lookahead with resulting converged (but not optimal!) utilities as future\n",
        "values\n",
        "• Repeat steps until policy converges\n",
        "This is policy iteration\n",
        "• It’s still optimal!\n",
        "• Can converge (much) faster under some conditions\n",
        "\n",
        "Comparison\n",
        "Both value iteration and policy iteration compute the same thing (all optimal\n",
        "values)\n",
        "In value iteration:\n",
        "• Every iteration updates both the values and (implicitly) the policy\n",
        "• We don’t track the policy, but taking the max over actions implicitly recomputes\n",
        "it\n",
        "In policy iteration:\n",
        "• We do several passes that update utilities with fixed policy (each pass is fast\n",
        "because we consider only one action, not all of them)\n",
        "• After the policy is evaluated, a new policy is chosen (slow like a value iteration\n",
        "pass)\n",
        "• The new policy will be better (or we’re done)\n",
        "Both are dynamic programs for solving MDPs\n",
        "\n",
        "Summary: MDP Algorithms\n",
        "So you want to….\n",
        "• Compute optimal values: use value iteration or policy iteration\n",
        "• Compute values for a particular policy: use policy evaluation\n",
        "• Turn your values into a policy: use policy extraction (one-step\n",
        "lookahead)\n",
        "These all look the same!\n",
        "• They basically are – they are all variations of Bellman updates\n",
        "• They all use one-step lookahead expectimax fragments\n",
        "• They differ only in whether we plug in a fixed policy or max over\n",
        "actions\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"assigns a single number to represend the desrability of a state\",\n",
        "\"expected utility\" : \"the average utility value of the outcomes, weighted by the probability that the outcome occurs\",\n",
        "\"maximum expected utility (MEU)\": \"a rational agent should choose the action that maximizes the agent's expected utility\",\n",
        "\"lottery\": \"the set of outcomes for each action\",\n",
        "\"orderability\": \"given 2 lotteries, a rational agent must either prefer\",\n",
        "\"transitivity\": \"given any 3 lotteries if an agent preferes A to B and B to C then the agent must prefer A to C\",\n",
        "\"continuity\": \"probability p the agent is indifferent between B and L that yields A with p and C with p-1\",\n",
        "\"substitutibility\": \"if agent is indifferent between A and B then also indifferent between more complex lotteries with b sub a\",\n",
        "\"monotonicity\": \"if an agent prefers A to B then the agent must prefer the lottery that has a higher prob for A\",\n",
        "\"decomposability\": \"compound lotteries can be reduced to simpler ones using the laws of probability\",\n",
        "\"existence of utility function\": \"if agent's preferences obey axioms of utility then there is func U that obeys\",\n",
        "\"expected utility of a lottery\": \"the utility of a lottery is the sum of the probabilities of each outcome times their utilities\",\n",
        "\"value function\": \"preference ranking on states\",\n",
        "\"ordinal utility function\": \"value function\",\n",
        "\"preference elicitation\": \"present choices to the human and using the observed preferences to find the utility function\",\n",
        "\"Normalized utilities us a scale with worst = 0 and best = 1\": \"\",\n",
        "\"value of a statistical life\": \"determines the costs and benefits of regulations and interventions\",\n",
        "\"micromort\": \"one in a million chance of death\",\n",
        "\"QALY\": \"quality adjusted life year\",\n",
        "\"risk-averse\": \"prefer a sure thing with a payoff that is less than the EMV of a gamble\",\n",
        "\"risk-seeking\": \"the opposite of risk averse\",\n",
        "\"certainty equivalent\": \"the value an agent will accept in lieu of a lottery\",\n",
        "\"insurance premium\": \"the difference between the EMV of a lottery and its certanty equivalent\",\n",
        "\"risk-neutral\": \"an agent that has a linear curve\",\n",
        "\"unbiased\": \"the expected value of the error is zero\",\n",
        "\"order statistic\": \"the distrobution of any particular ranked element of a sample\",\n",
        "\"optimizer's curse\": \"the tendency for the estimated expected utility of the best choice to be too high\",\n",
        "\"normative theory\": \"describes how a rational agent should act\",\n",
        "\"descriptive theory\": \"describes how actual agents really do act\",\n",
        "\"certainty effect\": \"people are strongly attracted to gains that are certain\",\n",
        "\"ambiguity aversion\": \"most people elect the known proabability rather than the unknown unknowns\",\n",
        "\"framing effect\": \"the exact wording of a decision problem can have a big impact on the agent's choices\",\n",
        "\"anchoring effect\": \"proposing something outrageous so that settling with somthing slightly crazy isnt crazy\",\n",
        "\"fully observable\": \"if the agent's sensors give it access to the complete state of the environment at every point in time\",\n",
        "\"Markovian\": \"the probability of reaching s' from s depends only on s not the history\",\n",
        "\"environment history\": \"a sequence of states and action\",\n",
        "\"Markov decision process MDP\": \"sequential decision problem for a fully observable, stochastic environment with Markovian transition model and additive rewards\",\n",
        "\"dynamic programming\": \"simplifying a problem by recursively breaking it into smaller pieces\",\n",
        "\"policy\": \"what an agent should do for any state that the agent might reach\",\n",
        "\"optimal policy\": \"yields the highest expected utility\",\n",
        "\"finite horizon\": \"there is a fixed time N after which nothing matters-the game is over\",\n",
        "\"nonstationary\": \"a policy that depends on the time\",\n",
        "\"stationary\": \"optimal policy depends only on the current state\",\n",
        "\"discount factor (gamma)\": \"[0,1] describes the agent's preferences for current rewards over future rewards\",\n",
        "\"additive reward\": \"when gamma is 1\",\n",
        "\"attribute\": \"each transition s_t -> s_t+1 is an attribute of the the history\",\n",
        "\"propper policy\": \"a policy that is guaranteed to reach a terminal state\",\n",
        "\"Q-function (action-utility function)\": \"expected utility of taking a given action in a given state\",\n",
        "\"sparse\": \"when most entries in MDP tables are zerso\",\n",
        "\"dynamic decision network\": \"factored representations, \",\n",
        "\"factored representations\": \"\",\n",
        "\"value iteration\": \"n possible states then n bellman equations\",\n",
        "\"contraction\": \"function of one argumentthat when applied to 2 inputs produces 2 closer together outputs\",\n",
        "\"max norm\": \"measures the distance between utility vectors, measures the length of a vector by abs value of its biggest component.\",\n",
        "\"policy loss\": \"||U^pi_i-U||\",\n",
        "\"policy iteration\": \"alternates  policy evaluation and policy improvent starting at an initial policy pi0\",\n",
        "\"policy evaluation\": \"given pi_i calculate Ui=U^pi_i the utility of each state if pi_i were to be executed\",\n",
        "\"policy improvement\": \"calculate a new MEU policy pi_i+1, using one step look ahead based on Ui\",\n",
        "\"modified policy iteration\": \"uses simplified bellman equation\",\n",
        "\"asynchronus policy iteration\": \"pick any subset ofof states and apply either kind of updating (policy improvement or value iteration)\",\n",
        "\"Linear programming\": \"general approach for formulating constrained optimization problems\",\n",
        "\"real time dynamic programming\": \"like minimax, alternates max and chance nodes\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "b93d6872-6fad-4fd0-8205-12485054dadc"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n An makes a decision: has avaliable some actions a, assigns a probability\nP(s) to each possible state. The transition model is given by P(s'|s,a) the probability that \naction a in state s reaches state s'  $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$  An agent's preferences \nare captured by a utility function. Expected utility:  $EU(a)) = \\sum_s P(\\textnormal{RESULT}(a)=s')U(s')$  MEU:  $action = \\textnormal{argmax}_{a}EU(a)$  The MEU\nformalizes the geneearal notion that an intelligent agent should do the right thing but it \ndoes not operationalize that advice. An agent may not know how good a state is until it\nknows where it can go from that state. So if an agent acts so as to maximize a utility function\nthat correctly reflects the performance measure, then the agent will achieve the highest \npossible perfomace score. \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n An agent's preferences (handwritten). Lottery (handwritten). All formulas\nare also handwritten. The axioms of utility theory are really axioms about prefereneces. But \nwe can derrive the existence of a utility function, and the expected utility of a lottery.\nThe outcome of a nondeterministic action is a lottery, an agent can only act rationally by \nchoosing an action that maximizes expected utility. In a deterministic environment an agent\nneeds only a preference ranking on states- the numbers dont matter\n\n## AIMA Chapter 16.3 Utility Functions\n Utility functions map from lotteries to real numbers. \n\n16.3.1 Utility assessment and utility scales\nTo built decision-theoretic system that helps a human make decisions. We must find \nthe human's utility function, preference elicitation. We can assist the utility of \nany prize S by asking teh agentt to choose between S and a standard lottery. \n\n16.3.2 The utility of money\nIf an agent prefers more money the agent has a monotonic preference for \nmoney. The utility of money was almost exactly proportional to the logarithm of the amount. \n\n16.3.3 Expected utility and post-decision disappointment:\nThe rational way to choose the best action a*, is to maximize the expected utility: a* = argmax_aEU(a)\nProbability density function $P(x) = \\frac{d}{dx}((F(x)^{k})=kf(x)(F(x))^{k-1}$  With\nmore choices, extremely optimistic estimates are more likely to arise. We can avoid hte optimizer's curse\nwith a Bayesian approach that uses and explicit probability model P(EU^|EU). \n\n16.3.4 Human judgment and irrationality\nDecision theory is a normative theory. Humans are \"predicatably irrational. An explanation\nfor the apparently irrational preference is the certainty effect. People feel more comfortable\nmaking relative utility judgements rather than absolute ones. The very fact that different\nwordings of the same decision problem elicit different choices suggests that the decision problem itself is not getting through.\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n If the environment were deterministic, a solution would be easy.\nBut the environment is stochastic so it's actions are unreliable. The transition model\ndescribes the outcome of each action in each state. P(s'|s,a) is the probability of reaching\nstate s' if action a is done in state s. If a decision problem is sequential the utility\nfunction will depend on an environment history. For every transition from s to s' via a the\nthe agent recieves a reward R(s,a,s'), they may be + or - but are bounded by +/- Rmax.\nThe utility of an environment history is the sum of the rewards recieved. Methods for solving\nMDPs usually involve dynamic programming. No matter what the outcome of the action, the\nresulting state will be in the policy, and the agent will know what to do next. The quality\nof a policy is measured by the expected utility of the possiblke environment histories genterated\nby it. Optimal policy is denoted by pi*.\n\n17.1.1 Utilities over time\nFirst check if there is a finite horizon or an infinite horizon for decision making. \nWith a finite horizon, an optimal action in a given state may depend on the time left.\nWith no fixed time limit there is no reason to behave differently in the same state at differetn times\nWith additive discounted rewards the utility of a history is: $U_{h}([s_{0},a_{0},s_{1},a_{1},...]) = R(s_{0},a_{0},s_{1})+\\gamma R(s_{1},a_{1},s_{2})+\\gamma^{2} R(s_{1},a_{1},s_{2})$ three \nsolutions to equations that lead to infinite histories: 1. with discounted\nrewards the utility of an infinite sequence is finite 2. if the environment has terminal\nstates and we are guaranteed to get on eeventually then we will never need to compare infinite\nsequences 3. infinite sequences can be compared in terms of the average reward obtained \nper time step.\n\n17.1.2 Optimal policies and the utilities of states\nCompare policies by comparing their expected utilities them.  $U^{\\pi}(s) = E[\\sum_{t=0}^{\\infty} (\\gamma^{t}R(S_{t},\\pi(S_{t}),S_{t+1})]$ Optimal \npolicy is $\\pi_{s}^{*}=(\\textnormal{argmax}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ Consequence of using discounted utilities with infinite horizons is\nthe optimal policy being independent of the starting state. The utility of a state is the expected reward for the\nnext transition plus the discounted utility of the next state Bellman equation: $U(s)= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ Q \nfunction: $U(s)= \\textnormal{max}_{a}Q(s,a)$  We can extract the optimal policy from the q\nfunction $\\pi ^{*}= \\textnormal{argmax}_{a}Q(s,a)$  ``` \nfunction Q-value(mdp, s,a,U) returns a utility value\n  return sum_s' P(s'|s,a){[R(s,a,s')+gammaU[s']] ```\n  \n17.1.4 Representing MDPs\nDynamic decision networks, or DDNs. They typically have an exponential complexity advantage\nover atomic representations and can model quite substantial real-world problems.\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Bellman update:  $U(s)_{i+1}= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U_{i}(s')])$ is to be applied simultaneously to \nall states at each iteration\n``` \n function Value-Iteration(mdp, e) returns a utility function\n    inputs mdp, with states S, actions A(s) transistion model P(s'|s,a), rewards R(s,a,a') discount gamma\n      e, the maximum error allowed in the utility of any state\n    local variables: U,U', vectors of utilities for stares in S, initially zero\n        delta, the maximum relative change in the utility of any state\n    repeat\n      U<- U'; delta <-0\n      for each state s in S do\n        U'[s]<-max_aeA(s) Q-Value(mdp,s,a,U)\n        if|U'[s]-U[s]| > delta then delta <-|U'[s] - U[s]|\n      until delta<= e(i-gamma)/gamma\n      return U\n``` \nValue iteration eventually converges to a unique set of solutions to the bellman equation. Contraction\nhas one fixed point, when applied the function must get closer to the fixed point.\n\nPolicy iteration stops when the policy improvemtn step yields no change in utilities\n``` \n function Policy-Iteration(mdp) returns a policy\n  inputs: mdp wiht stares, actions, transition model\n  local variables: U, a vector of utilites for states in S (initially 0)\n      pi, policy vector indexed by state, initially random\n\n  repeat\n    U<-Policy-Evaluation(pi, U, mdp)\n    unchanged <- true\n    for each state s in S do\n      a*<- argmax Q-value(mdp,s,a,U)\n      if Q-Value(mdp,s,a*,U)> Q-Value(mdp,s,pi[s],U) then\n        pi[s] <- a*; unchanged? <- false\n    until unchanged?\n    retunr pi\n``` \nAsynchronous policy iterstion is guaranteed to converege to an optimal policy. Linear Programming\nseeks to minimize U(s) for all s subject to the inequalities.\n\n\n## Module 8 Lecture Notes\n In our search algorithms so far, the\ntransition model was deterministic and\ndescribed the outcome of each action in\neach state.\nThe transition function is sometimes\nwritten as T(s, a, s’), or explicitly as a\nprobability:\n\nReward function\nWe will specify a utility or reward function for the agent.\nThe “rewards“ can be positive or negative but are bounded by some maximum value.\nBecause the decision process is sequential, we must specify the utility function on a\nsequence of states and actions.\nInstead of only giving a reward at the goal states, the agent can receive a reward at\neach time step, based on its transition from s to s’ via action a.\nThis is defined by a reward function\nR( s, a, s’ )\nFor example, we could give the Millennium Falcon a small negative reward of -0.04 for every\ntransition except for entering the terminal states (+1 for entering the planet’s orbit or -1 for\nsmashing into an asteroid).\nThe rewards are additive, so if the Millennium Falcon takes 4 steps before entering\nthe planet's orbit, it gets -0.04 + -0.04 + -0.04 + -0.04 + 1 = 0.84 for that solution\n\nMarkov Decision Process\nA Markov decision process or MDP is\n• a sequential decision problem\n• for a fully observable environment\n• with a stochastic transition model\n• that has additive rewards\nMDPs are non-deterministic search\nproblems. One way of solving them is via\nexpectimax search.\n\nMarkov Decision Process\nTo find a solution to an MDP, you need to define the following things:\n• A set of states s ∈ S\n• A set of actions a ∈ A\n• A transition function T(s, a, s’)\n• Probability that executing action a in s will lead to s’ P(s’ | s, a)\n• The probability is called the model\n• A reward function R(s, a, s’)\n• Sometimes just R(s) or R(s’)\n• An initial state s0\n• Optionally, one or more terminal states\n\nSolution == Policy\nIn search problems a solution was a\nsequence of action that corresponded to\nthe shortest path.\nBecause of the non-determinism in MDPs\nwe cannot simply give a sequence of\nactions.\nInstead, the solution to an MDP is a policy.\nA policy maps from a state onto the action\nto take if the agent is in that state.\n𝝿(s) = a\n\nOptimal Quantities\n▪ The value (utility) of a state s:\nV*(s) = expected utility starting in s and\nacting optimally\n▪ The value (utility) of a q-state (s,a): Q*(s,a) = expected utility starting out\nhaving taken action a from state s and\n(thereafter) acting optimally\n▪ The optimal policy:\nPi*(s) = optimal action from state s\n\nPolicy Iteration\nAlternative approach for optimal values:\n• Step 1: Policy evaluation: calculate utilities for some fixed policy (not\noptimal utilities!) until convergence\n• Step 2: Policy improvement: update policy using one-step lookahead with resulting converged (but not optimal!) utilities as future\nvalues\n• Repeat steps until policy converges\nThis is policy iteration\n• It’s still optimal!\n• Can converge (much) faster under some conditions\n\nComparison\nBoth value iteration and policy iteration compute the same thing (all optimal\nvalues)\nIn value iteration:\n• Every iteration updates both the values and (implicitly) the policy\n• We don’t track the policy, but taking the max over actions implicitly recomputes\nit\nIn policy iteration:\n• We do several passes that update utilities with fixed policy (each pass is fast\nbecause we consider only one action, not all of them)\n• After the policy is evaluated, a new policy is chosen (slow like a value iteration\npass)\n• The new policy will be better (or we’re done)\nBoth are dynamic programs for solving MDPs\n\nSummary: MDP Algorithms\nSo you want to….\n• Compute optimal values: use value iteration or policy iteration\n• Compute values for a particular policy: use policy evaluation\n• Turn your values into a policy: use policy extraction (one-step\nlookahead)\nThese all look the same!\n• They basically are – they are all variations of Bellman updates\n• They all use one-step lookahead expectimax fragments\n• They differ only in whether we plug in a fixed policy or max over\nactions\n \n\n## Module 8 Definitions\n* **Utility function** - assigns a single number to represend the desrability of a state\n* **Expected utility** - the average utility value of the outcomes, weighted by the probability that the outcome occurs\n* **Maximum expected utility (meu)** - a rational agent should choose the action that maximizes the agent's expected utility\n* **Lottery** - the set of outcomes for each action\n* **Orderability** - given 2 lotteries, a rational agent must either prefer\n* **Transitivity** - given any 3 lotteries if an agent preferes A to B and B to C then the agent must prefer A to C\n* **Continuity** - probability p the agent is indifferent between B and L that yields A with p and C with p-1\n* **Substitutibility** - if agent is indifferent between A and B then also indifferent between more complex lotteries with b sub a\n* **Monotonicity** - if an agent prefers A to B then the agent must prefer the lottery that has a higher prob for A\n* **Decomposability** - compound lotteries can be reduced to simpler ones using the laws of probability\n* **Existence of utility function** - if agent's preferences obey axioms of utility then there is func U that obeys\n* **Expected utility of a lottery** - the utility of a lottery is the sum of the probabilities of each outcome times their utilities\n* **Value function** - preference ranking on states\n* **Ordinal utility function** - value function\n* **Preference elicitation** - present choices to the human and using the observed preferences to find the utility function\n* **Normalized utilities us a scale with worst = 0 and best = 1** - \n* **Value of a statistical life** - determines the costs and benefits of regulations and interventions\n* **Micromort** - one in a million chance of death\n* **Qaly** - quality adjusted life year\n* **Risk-averse** - prefer a sure thing with a payoff that is less than the EMV of a gamble\n* **Risk-seeking** - the opposite of risk averse\n* **Certainty equivalent** - the value an agent will accept in lieu of a lottery\n* **Insurance premium** - the difference between the EMV of a lottery and its certanty equivalent\n* **Risk-neutral** - an agent that has a linear curve\n* **Unbiased** - the expected value of the error is zero\n* **Order statistic** - the distrobution of any particular ranked element of a sample\n* **Optimizer's curse** - the tendency for the estimated expected utility of the best choice to be too high\n* **Normative theory** - describes how a rational agent should act\n* **Descriptive theory** - describes how actual agents really do act\n* **Certainty effect** - people are strongly attracted to gains that are certain\n* **Ambiguity aversion** - most people elect the known proabability rather than the unknown unknowns\n* **Framing effect** - the exact wording of a decision problem can have a big impact on the agent's choices\n* **Anchoring effect** - proposing something outrageous so that settling with somthing slightly crazy isnt crazy\n* **Fully observable** - if the agent's sensors give it access to the complete state of the environment at every point in time\n* **Markovian** - the probability of reaching s' from s depends only on s not the history\n* **Environment history** - a sequence of states and action\n* **Markov decision process mdp** - sequential decision problem for a fully observable, stochastic environment with Markovian transition model and additive rewards\n* **Dynamic programming** - simplifying a problem by recursively breaking it into smaller pieces\n* **Policy** - what an agent should do for any state that the agent might reach\n* **Optimal policy** - yields the highest expected utility\n* **Finite horizon** - there is a fixed time N after which nothing matters-the game is over\n* **Nonstationary** - a policy that depends on the time\n* **Stationary** - optimal policy depends only on the current state\n* **Discount factor (gamma)** - [0,1] describes the agent's preferences for current rewards over future rewards\n* **Additive reward** - when gamma is 1\n* **Attribute** - each transition s_t -> s_t+1 is an attribute of the the history\n* **Propper policy** - a policy that is guaranteed to reach a terminal state\n* **Q-function (action-utility function)** - expected utility of taking a given action in a given state\n* **Sparse** - when most entries in MDP tables are zerso\n* **Dynamic decision network** - factored representations, \n* **Factored representations** - \n* **Value iteration** - n possible states then n bellman equations\n* **Contraction** - function of one argumentthat when applied to 2 inputs produces 2 closer together outputs\n* **Max norm** - measures the distance between utility vectors, measures the length of a vector by abs value of its biggest component.\n* **Policy loss** - ||U^pi_i-U||\n* **Policy iteration** - alternates  policy evaluation and policy improvent starting at an initial policy pi0\n* **Policy evaluation** - given pi_i calculate Ui=U^pi_i the utility of each state if pi_i were to be executed\n* **Policy improvement** - calculate a new MEU policy pi_i+1, using one step look ahead based on Ui\n* **Modified policy iteration** - uses simplified bellman equation\n* **Asynchronus policy iteration** - pick any subset ofof states and apply either kind of updating (policy improvement or value iteration)\n* **Linear programming** - general approach for formulating constrained optimization problems\n* **Real time dynamic programming** - like minimax, alternates max and chance nodes\n* **** - \n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\" There is a tradeoff between exploitation of the current best option\n",
        "and exploration of previously unknown states to gain information. There are several\n",
        "definitions of bandit problems. The most general: each arm M is a Markov reward process. \n",
        "With states Si, transition model Pi, and reward Ri. The arm defines a distrobution over\n",
        "sequences of rewards Ri0, Ri1,... A policy for any bandit problem: pull the arm that\n",
        "has the highest gittens index then update the gittens indicies.\n",
        "\n",
        "17.3.2 The Bernoulli bandit\n",
        "The state of arm Mi is defined by si and fi, the counts of successes and failures,\n",
        "the transition probability predicts the next outcome to be 1 with probability (si)/(si+fi)\n",
        "and 0 with probability (fi)/(si+fi). There is an exploration bonus associated with arms\n",
        "that have only been tried a few times\n",
        "\n",
        "17.3.3 Approximately optimal bandit policies\n",
        "First class of methods use the upper confidence bound.\n",
        "Thompson sampling chooses an arm randomly according to the probability that the arm is\n",
        "in fact optimal\n",
        "\n",
        "17.3.4 Non-indexable variants\n",
        "Selection problem - making a decision in the midst of conditions. \n",
        "the ordinary term for BSPs is multitaksing. We can take advantage of the loose nature\n",
        "of the interaction between the arms. This can be modeled by opportunity cost.\n",
        "\"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\" Reinforcement learning is an alternative to needing huge databases of \n",
        "moves and their actions. Reinforcement learning differes from just solving the MDP because\n",
        "the agent is not given the MDP to solve, the agent is in the MDP. If we can provide the\n",
        "correct reward signal to the agent reinforcement learning provides a very general way to build AI\n",
        "systems. Especially in simulated environmentes when there is no shortage of opportunities\n",
        "to gain experience. Some key basic reinfrocement learning approaches are:\n",
        "\n",
        "Model-based reinforcement learning: agent uses a transition model of the environment to \n",
        "help interpret the reward signals and decide how to act. If the model is unknow the agent\n",
        "learns it by observing the effects of its actions\n",
        "\n",
        "Model-free reinforcement learning: The agent never learns or knows the transition model.\n",
        "It learns a more direct representation of how to behave. In one of 2 varieties: 1) action \n",
        "utility learning like Q learning. Where the agent learns a Q-function then chooses what to\n",
        "do in s by finding the action with the highest q. 2) policy search: the agent learns a policy pi(s)\n",
        "that maps directly from states to actions. this is a reflex agent\"\"\"\n",
        "equation_11 = r\"\"\" $U^{\\pi}(s)\\leftarrow ^{\\pi} \\alpha[R(s',s,a)+\\gamma U^{\\pi}(s')-U^{\\pi}(s)]$ \"\"\"\n",
        "chapter22_2 =  \"\"\" Passive learning is similar to policy evaluation, except passive \n",
        "learning agent does not know the reansistion model or the reward function. The expected \n",
        "utility associated with each state s:\"\"\" + equation_5 +\"\"\"\n",
        "\n",
        "22.2.1 Direct utility estimation \n",
        "Reduces reinfrocement learnign to a standard supervised learning problem in which\n",
        "each example is a (state, reward-to-go) pair. The utility of a state is determined by the\n",
        "reward and expected utility of the successor states-> the utility obeys the bellman equation.\n",
        "By ignoring the connections between states, direct utility estimation misses opportunities\n",
        "for learning.\n",
        "\n",
        "22.2.2 Adaptive dynamic programming\n",
        "Takes advantage of the constraints among the utilities of states by learning the transition\n",
        "model that connects them, then solving the corresponding Markov decision process using\n",
        "dynamic programming. For a passive learning agent we would plug the learned transition model and the\n",
        "observed rewards into the bellman equation.\n",
        "```\n",
        "function Passibe-ADP-Learner(precept) returns an action\n",
        "  inputs: percept, a percept indicating the current state s' and reward signal r\n",
        "  persistent: pi, a fixed policy\n",
        "              mdp, an MDP with model P, rewards R, actions A, discount gamma\n",
        "              U, a table of utilities for states, initially empty\n",
        "              Ns'|sa a table of outcome count vectors indexed by state and action, initaially zero\n",
        "              s, a the previous state and action, initailly null\n",
        "  if s' new then U[s']<-0\n",
        "  if s not null then \n",
        "    increment Ns'|s,a[s,a][s']\n",
        "    R[s,a,s']<-r\n",
        "    add a to A[s]\n",
        "    P( |s,a)<-NORMALIZE(N_s'|s,a [s,a])\n",
        "    U<- POLICY EVALUATION(pi, U, mdp)\n",
        "    s,a <- s', pi[s']\n",
        "    return a\n",
        "```\n",
        "22.2.3 Temporal difference learning\n",
        "You can also use the observed transistions to adjust the utilites of the oberved\n",
        "transitions to adjust the utiliteis of the oberved states so that they agree with\n",
        "the constraint equations. When the transition occurs apply this update\n",
        "\"\"\" + equation_11 +\"\"\" where alpha is the learning rate. This is know as the temporal-difference\n",
        "equation.\n",
        "```\n",
        "function Passive-TD-Learner(percept) returns an action\n",
        "  inputs: percept, a percept indicating the current state s' and reward signal r\n",
        "  persistent: pi, a fixed policy\n",
        "              s, the previous state, initially null\n",
        "              U, a table of utilities for states, initially empty\n",
        "              Ns, a table of frequencies for states, initially zero\n",
        "  if s' is new then U'[s']<-0\n",
        "  if s is not null then \n",
        "    increment Ns[s]\n",
        "    temporal difference equation\n",
        "  s<-s'\n",
        "  return pi[s']\n",
        "```\n",
        "\"\"\"\n",
        "equation12 = r\"\"\" $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\sum_{h}P(h|e)U_{h}^{\\pi})$ \"\"\"\n",
        "equation13 = r\"\"\" $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\textnormal{min}_{h}U_{h}^{\\pi})$ \"\"\"\n",
        "chapter22_3 =  \"\"\" An active learning agent gets to decid what actions to take. Adaptive\n",
        "dynamic programming (ADP) agent: first needs to learn a complete transition model with \n",
        "outcome probabilities for all actions, next take into account the fact that hte agent \n",
        "has a choice of actions. The utilities it needs to learn are those defined by the optimal\n",
        "policy (obey the bellman equation). With U the agent can extract an optimal action by \n",
        "one-step look-ahead to maximize the expected utility. If it uses policy iteration it could simply\n",
        "execute the action the optimal policy recommends.\n",
        "\n",
        "22.3.1 Exploration\n",
        "Sometimes greed pays off but it often does not. A learned model is not the same as\n",
        "the true environment, what is learned could be suboptimal in the real environment. Greedy\n",
        "agent overlooks the fact that actions provide more than rewards they provide information\n",
        "in the form of percepts in the resulting state. An ADP using a GLIE scheme will eventually learn \n",
        "the true transition model. A good approach is to give some weight to actions the agent has not\n",
        "tried very often while avoiding those that are believed to be of low utility.\n",
        "\n",
        "22.3.2 Safe exploration\n",
        "Many actions are irreversible, in the sense defined for online search agents: no set of \n",
        "actions can restore it to where it was before. Worst case the agent enters an absorbing state.\n",
        "A better idea to avoid catastrophe is to choose a plicy that works reasobably for a range of \n",
        "models that have a chance of being true, even if it is suboptimal for the maximum-likelihood model.\n",
        "There are 3 mathematical approaches for this: 1) Bayesian reinforcement learning: posterior probablility P(h|e) is \n",
        "obtatined by bayes' rule given the observations to date.\"\"\" + equation12 + \"\"\"2) robust control theory allows for a\n",
        "set of possible models H wihtout assigning probabilities to them adn defines an optimal\n",
        "robust policy as on that gives the best outcome in the worst case over H:\"\"\" + equation13 + \"\"\" This\n",
        "is often the set of models that exceed some likelihood threshold on P(h|e). The problem is that\n",
        "this results in overly conservative behavior.\n",
        "\n",
        "22.3.3 Temporal-difference Q-learning\n",
        "This avoids the need for a model by learning an action-utility function instead of a utility\n",
        "function. Q(s,a) denotes the expected total discounted reward if the agent takes action a in s\n",
        "and acts optimally after.\n",
        "\n",
        "``` \n",
        "function Q-learning-agent(percept) returns an action\n",
        "  inputs: percepts, a percept indicating teh current state s' and reward signal r\n",
        "  persistent: Q, a table of action values indexed by state adn action, initially zero\n",
        "              Nsa, a table of frequencies for state-action pairs, initially zero\n",
        "              s, a ther previous stat and action, initially null\n",
        "  if s is not null then\n",
        "    increment Nsa[s,a]\n",
        "    Q[s,a]<- Q[s,a] + alpha(Nsa[s,a])(r + gamma*max_a' Q[s',a']-Q[s,a]\n",
        "  s,a <- s', argmax_a' f(Q[s',a'], N_sa[s',a'])\n",
        "  return a\n",
        "``` \n",
        "\"\"\"\n",
        "chapter22_4 =  \"\"\" Evaluation functions are approximate utility functions.\n",
        "Adding look-ahead search means that effective behavior can be generated from a much simpler\n",
        "utility function approximator that is learnable from far fewer experiences. Allows for inductive\n",
        "generalization: the agent can generalize from states it has visited to states it has not. Backgamon!\n",
        "\n",
        "22.4.1 Approximating direct utility estimation\n",
        "The state and the sum of rewards received constitute a training example for a supervised learning\n",
        "algorithm. Reinforcement learning makes more sense to use an online learning algorithm that\n",
        "updates after each trial. The rate of change of the error wrt each parameter (theta_i)\n",
        "is partial dEj/dthetai. Changing the parameters theta_i in response to an \n",
        "observed transition between two states changes the values of U^theta for every other state!\n",
        "Linear function approximation:the function must be linear in the features. But these can\n",
        "be arbitrary nonlinear functions of the state variables.\n",
        "\n",
        "22.4.2 Approximaating temporal-difference learning\n",
        "Adjust the parameters to try to reduce the temporal difference between successive states\n",
        "the update rule can be shown to converge to the closest possible approximation to the\n",
        "true function when the function approximator is linear in the features. With active\n",
        "learning and nonlinear functions such as neural networks, tricky.\n",
        "Learning a model for an observable environment is a supervised learning problem,\n",
        "because the next percept gives the outcome state. Partially observable environments have \n",
        "a much harder learning problem. The next percept is no longer known.\n",
        "\n",
        "22.4.3 Deep reinforcement learning\n",
        "It is always possible to represent U or Q as linear combinations of features, if they aren't\n",
        "in an efficiently computible form the linear function approximator\n",
        "may be insufficient. The parameters are the layers of the network. Obstacles:\n",
        "it is often difficult to get good performance and the trained system may behave\n",
        "very unpredictably if the environment differs even a little from the training data.\n",
        " \n",
        "22.4.4 Reward shaping\n",
        "Real world environments may have sparse rewards: many actions are required\n",
        "to achieve any nonzero reward. R'(s,a,s') = R(s,a,s') + gamma*phi(s') - phi(s)\n",
        "phi can be constructed to reflect any desirable aspects of the state.\n",
        "\n",
        "22.4.5 Hierarchical reinforcement learning \n",
        "Deal with long action sequences by breaking them into small pieces, then again until\n",
        "they are short enough to make learning easy. Begins with a partial program. The theoretical\n",
        "foundations of HRL are based on the concept of the joint spate space. This is all essentially\n",
        "like solving an MDP with: states->choice states sigma of the joint space, the \n",
        "actions at sigma are the choices avaliable in sigma from the partial program, the \n",
        "reward functionp(sigma,c,sigma') is the expected sum of rewards for all physical transition, the\n",
        "transition model t(sigma,c,sigma') -> if c invokes a physical\n",
        "action a, then t borrows from the physical model P(s'|s,a)\"\"\"\n",
        "chapter22_5 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" That wasn’t planning, it was learning!\n",
        "Specifically, reinforcement learning\n",
        "There was an MDP, but you couldn’t solve it with just computation\n",
        "You needed to actually act to figure it out\n",
        "Important ideas in reinforcement learning that came up\n",
        "Exploration: you have to try unknown actions to get information\n",
        "Exploitation: eventually, you have to use what you know\n",
        "Regret: even if you learn intelligently, you make mistakes\n",
        "Sampling: because of chance, you have to try things repeatedly\n",
        "Difficulty: learning can be much harder than solving a known MDP\n",
        "\n",
        "Reinforcement Learning\n",
        "Still assume a Markov decision process (MDP):\n",
        "A set of states s ∈ S\n",
        "A set of actions (per state) A\n",
        "A model T(s,a,s’)\n",
        "A reward function R(s,a,s’)\n",
        "Still looking for a policy π(s)\n",
        "New twist: don’t know T or R\n",
        "I.e. we don’t know which states are good or what the actions do\n",
        "Must actually try actions and states out to learn\n",
        "\n",
        "Model-Based Learning\n",
        "Model-Based Idea:\n",
        "Learn an approximate model based on experiences\n",
        "Solve for values as if the learned model were correct\n",
        "Step 1: Learn empirical MDP model\n",
        "Count outcomes s’ for each s, a\n",
        "Normalize to give an estimate of T\n",
        "Discover each R when we experience (s, a, s’)\n",
        "Step 2: Solve the learned MDP\n",
        "For example, use value iteration, as before\n",
        "\n",
        "Passive Reinforcement Learning\n",
        "Simplified task: policy evaluation\n",
        "Input: a fixed policy π(s)\n",
        "You don’t know the transitions T(s,a,s’)\n",
        "You don’t know the rewards R(s,a,s’)\n",
        "Goal: learn the state values\n",
        "In this case:\n",
        "Learner is “along for the ride”\n",
        "No choice about what actions to take\n",
        "Just execute the policy and learn from experience\n",
        "This is NOT offline planning! You actually take actions in the world.\n",
        "\n",
        "How to Explore?\n",
        "Several schemes for forcing exploration\n",
        "Simplest: random actions (ε-greedy)\n",
        "Every time step, flip a coin\n",
        "With (small) probability ε, act randomly\n",
        "With (large) probability 1-ε, act on current policy\n",
        "Problems with random actions?\n",
        "You do eventually explore the space, but keep\n",
        "thrashing around once learning is done\n",
        "One solution: lower ε over time\n",
        "Another solution: exploration functions\n",
        "\n",
        "Even if you learn the optimal policy,\n",
        "you still make mistakes along the way!\n",
        "Regret is a measure of your total\n",
        "mistake cost: the difference between\n",
        "your (expected) rewards, including\n",
        "youthful suboptimality, and optimal\n",
        "(expected) rewards\n",
        "Minimizing regret goes beyond\n",
        "learning to be optimal – it requires\n",
        "optimally learning to be optimal\n",
        "Example: random exploration and\n",
        "exploration functions both end up\n",
        "optimal, but random exploration has\n",
        "higher regret\n",
        "\"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"one armed bandit\": \"slot machine\",\n",
        "\"exploitation\": \"taking advantage of the current best option\",\n",
        "\"exploration\": \"looking at previously unknown states\",\n",
        "\"markov reward process\": \"mdp with only one possible action ai\",\n",
        "\"stopping time\": \"if you pull a arm T times the stopping time is T\",\n",
        "\"bernoulli bandit\": \"each arm Mi produces a reward of 0 or 1 with fixed but unknown probability mu_i\",\n",
        "\"upper confidence bound\": \"use the samples from each arm to establish a confidence interfal for the value of the arm\",\n",
        "\"confidence interval\": \"range within which the value can be estimate to lie with high confidence\",\n",
        "\"bandit superprocess\": \"each arm is a full MDP in its own\",\n",
        "\"opportunity cost\": \"how much utility is given up per time step by not devoting that time toanother arm\",\n",
        "\"reinforcement learning\": \"an agent interacts with the world and periodically recieves rewards or reinforcements\",\n",
        "\"rewards\": \"reflect how well the agent is doing\",\n",
        "\"reinforcements\": \"psychology term for rewards\",\n",
        "\"passive reinforcement learning\": \"the agent's policy is fixed and the task is to learn the utilities of the states\",\n",
        "\"active reinforcement learning\": \"the agent must also figure out what to do: exploration\",\n",
        "\"exploration\": \"an agent must explore lots of its terrain to learn how to behave in it\",\n",
        "\"apprenticeship learning\": \"training a learning agent using demonstrations rather than reward signals\",\n",
        "\"direct utility estimation\": \"the utility of a state is defined as the expected total rewardfrom that state on\",\n",
        "\"reward-to-go\": \"expected total reward from that state onward\",\n",
        "\"temporal difference methods\": \"adjust the utility estimate towards the ideal equilibrium\",\n",
        "\"pseudoexperience\": \"generated by simulating the current transition model\",\n",
        "\"prioritized sweeping\": \"heuristic prefers to make adjustments to states whose likely successors have just undergone a large adjustment in their own utility estimates.\",\n",
        "\"active learning agent\": \"decides what actions to take\",\n",
        "\"greedy agent\": \"takes the action that it currently believes to be optimal at each step\",\n",
        "\"GLIE (greedy in the limit of infinite exploration)\": \"must try each action in each state an unbounded number of times to avoid missing an optimal action\",\n",
        "\"exploration function\": \"f(u,n) bellman equation, number of times a state has been tried\",\n",
        "\"absorbing stare\": \"no actions have any effect and no rewards are recieved\",\n",
        "\"SARSA (state action reward state action)\": \"updates with the Q-value of the action a'\",\n",
        "\"off-poliicy\": \"learns q values that answer what would this be worht in this state, assuming i stop using the current policy?\",\n",
        "\"on-policy\": \"what would this action be worth in the state, assuming i stick with my policy\",\n",
        "\"bayesian reinforcement learning\": \"assumes prior probability P(h) over hypothesis h about what the model is\",\n",
        "\"function approximation\": \"process of constructing a compact approximation of the true utility function or Q-function\",\n",
        "\"widrow-hoff rule\": \"delta rule equation 22.10\",\n",
        "\"catastrophic forgetting\": \"a nonzero weight on a inear feature can cause a car to slide bc it forgets danger\",\n",
        "\"experience replay\": \"ensures the car relives its youthful crashing behavior so it doesnt forget\",\n",
        "\"deep reinforcement learning system\": \"reinforcement learning system that uses a deep network as a function approximator\",\n",
        "\"reward shaing\": \"supplying teh agent with additional rewards for making progress\",\n",
        "\"pseudorewards\": \"extra rewards\",\n",
        "\"partial program\": \"outlines a hierarchial structure for the agent's behavior\",\n",
        "\"joint state space\": \"each state (s,m) is composed of a physical state and a machine state\",\n",
        "\"choice state sigma=(s,m)\": \"program counter for m is at a choice point in the agent program\",\n",
        "\"additive decomposition\": \"overall utility decomposes into several terms, depeding on a few variables\"\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "8f7ace09-59ce-4315-95eb-141bc9d98f44"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n There is a tradeoff between exploitation of the current best option\nand exploration of previously unknown states to gain information. There are several\ndefinitions of bandit problems. The most general: each arm M is a Markov reward process. \nWith states Si, transition model Pi, and reward Ri. The arm defines a distrobution over\nsequences of rewards Ri0, Ri1,... A policy for any bandit problem: pull the arm that\nhas the highest gittens index then update the gittens indicies.\n\n17.3.2 The Bernoulli bandit\nThe state of arm Mi is defined by si and fi, the counts of successes and failures,\nthe transition probability predicts the next outcome to be 1 with probability (si)/(si+fi)\nand 0 with probability (fi)/(si+fi). There is an exploration bonus associated with arms\nthat have only been tried a few times\n\n17.3.3 Approximately optimal bandit policies\nFirst class of methods use the upper confidence bound.\nThompson sampling chooses an arm randomly according to the probability that the arm is\nin fact optimal\n\n17.3.4 Non-indexable variants\nSelection problem - making a decision in the midst of conditions. \nthe ordinary term for BSPs is multitaksing. We can take advantage of the loose nature\nof the interaction between the arms. This can be modeled by opportunity cost.\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n Reinforcement learning is an alternative to needing huge databases of \nmoves and their actions. Reinforcement learning differes from just solving the MDP because\nthe agent is not given the MDP to solve, the agent is in the MDP. If we can provide the\ncorrect reward signal to the agent reinforcement learning provides a very general way to build AI\nsystems. Especially in simulated environmentes when there is no shortage of opportunities\nto gain experience. Some key basic reinfrocement learning approaches are:\n\nModel-based reinforcement learning: agent uses a transition model of the environment to \nhelp interpret the reward signals and decide how to act. If the model is unknow the agent\nlearns it by observing the effects of its actions\n\nModel-free reinforcement learning: The agent never learns or knows the transition model.\nIt learns a more direct representation of how to behave. In one of 2 varieties: 1) action \nutility learning like Q learning. Where the agent learns a Q-function then chooses what to\ndo in s by finding the action with the highest q. 2) policy search: the agent learns a policy pi(s)\nthat maps directly from states to actions. this is a reflex agent\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Passive learning is similar to policy evaluation, except passive \nlearning agent does not know the reansistion model or the reward function. The expected \nutility associated with each state s: $U^{\\pi}(s) = E[\\sum_{t=0}^{\\infty} (\\gamma^{t}R(S_{t},\\pi(S_{t}),S_{t+1})]$ \n\n22.2.1 Direct utility estimation \nReduces reinfrocement learnign to a standard supervised learning problem in which\neach example is a (state, reward-to-go) pair. The utility of a state is determined by the\nreward and expected utility of the successor states-> the utility obeys the bellman equation.\nBy ignoring the connections between states, direct utility estimation misses opportunities\nfor learning.\n\n22.2.2 Adaptive dynamic programming\nTakes advantage of the constraints among the utilities of states by learning the transition\nmodel that connects them, then solving the corresponding Markov decision process using\ndynamic programming. For a passive learning agent we would plug the learned transition model and the\nobserved rewards into the bellman equation.\n```\nfunction Passibe-ADP-Learner(precept) returns an action\n  inputs: percept, a percept indicating the current state s' and reward signal r\n  persistent: pi, a fixed policy\n              mdp, an MDP with model P, rewards R, actions A, discount gamma\n              U, a table of utilities for states, initially empty\n              Ns'|sa a table of outcome count vectors indexed by state and action, initaially zero\n              s, a the previous state and action, initailly null\n  if s' new then U[s']<-0\n  if s not null then \n    increment Ns'|s,a[s,a][s']\n    R[s,a,s']<-r\n    add a to A[s]\n    P( |s,a)<-NORMALIZE(N_s'|s,a [s,a])\n    U<- POLICY EVALUATION(pi, U, mdp)\n    s,a <- s', pi[s']\n    return a\n```\n22.2.3 Temporal difference learning\nYou can also use the observed transistions to adjust the utilites of the oberved\ntransitions to adjust the utiliteis of the oberved states so that they agree with\nthe constraint equations. When the transition occurs apply this update\n $U^{\\pi}(s)\\leftarrow ^{\\pi} \\alpha[R(s',s,a)+\\gamma U^{\\pi}(s')-U^{\\pi}(s)]$  where alpha is the learning rate. This is know as the temporal-difference\nequation.\n```\nfunction Passive-TD-Learner(percept) returns an action\n  inputs: percept, a percept indicating the current state s' and reward signal r\n  persistent: pi, a fixed policy\n              s, the previous state, initially null\n              U, a table of utilities for states, initially empty\n              Ns, a table of frequencies for states, initially zero\n  if s' is new then U'[s']<-0\n  if s is not null then \n    increment Ns[s]\n    temporal difference equation\n  s<-s'\n  return pi[s']\n```\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n An active learning agent gets to decid what actions to take. Adaptive\ndynamic programming (ADP) agent: first needs to learn a complete transition model with \noutcome probabilities for all actions, next take into account the fact that hte agent \nhas a choice of actions. The utilities it needs to learn are those defined by the optimal\npolicy (obey the bellman equation). With U the agent can extract an optimal action by \none-step look-ahead to maximize the expected utility. If it uses policy iteration it could simply\nexecute the action the optimal policy recommends.\n\n22.3.1 Exploration\nSometimes greed pays off but it often does not. A learned model is not the same as\nthe true environment, what is learned could be suboptimal in the real environment. Greedy\nagent overlooks the fact that actions provide more than rewards they provide information\nin the form of percepts in the resulting state. An ADP using a GLIE scheme will eventually learn \nthe true transition model. A good approach is to give some weight to actions the agent has not\ntried very often while avoiding those that are believed to be of low utility.\n\n22.3.2 Safe exploration\nMany actions are irreversible, in the sense defined for online search agents: no set of \nactions can restore it to where it was before. Worst case the agent enters an absorbing state.\nA better idea to avoid catastrophe is to choose a plicy that works reasobably for a range of \nmodels that have a chance of being true, even if it is suboptimal for the maximum-likelihood model.\nThere are 3 mathematical approaches for this: 1) Bayesian reinforcement learning: posterior probablility P(h|e) is \nobtatined by bayes' rule given the observations to date. $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\sum_{h}P(h|e)U_{h}^{\\pi})$ 2) robust control theory allows for a\nset of possible models H wihtout assigning probabilities to them adn defines an optimal\nrobust policy as on that gives the best outcome in the worst case over H: $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\textnormal{min}_{h}U_{h}^{\\pi})$  This\nis often the set of models that exceed some likelihood threshold on P(h|e). The problem is that\nthis results in overly conservative behavior.\n\n22.3.3 Temporal-difference Q-learning\nThis avoids the need for a model by learning an action-utility function instead of a utility\nfunction. Q(s,a) denotes the expected total discounted reward if the agent takes action a in s\nand acts optimally after.\n\n``` \nfunction Q-learning-agent(percept) returns an action\n  inputs: percepts, a percept indicating teh current state s' and reward signal r\n  persistent: Q, a table of action values indexed by state adn action, initially zero\n              Nsa, a table of frequencies for state-action pairs, initially zero\n              s, a ther previous stat and action, initially null\n  if s is not null then\n    increment Nsa[s,a]\n    Q[s,a]<- Q[s,a] + alpha(Nsa[s,a])(r + gamma*max_a' Q[s',a']-Q[s,a]\n  s,a <- s', argmax_a' f(Q[s',a'], N_sa[s',a'])\n  return a\n``` \n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Evaluation functions are approximate utility functions.\nAdding look-ahead search means that effective behavior can be generated from a much simpler\nutility function approximator that is learnable from far fewer experiences. Allows for inductive\ngeneralization: the agent can generalize from states it has visited to states it has not. Backgamon!\n\n22.4.1 Approximating direct utility estimation\nThe state and the sum of rewards received constitute a training example for a supervised learning\nalgorithm. Reinforcement learning makes more sense to use an online learning algorithm that\nupdates after each trial. The rate of change of the error wrt each parameter (theta_i)\nis partial dEj/dthetai. Changing the parameters theta_i in response to an \nobserved transition between two states changes the values of U^theta for every other state!\nLinear function approximation:the function must be linear in the features. But these can\nbe arbitrary nonlinear functions of the state variables.\n\n22.4.2 Approximaating temporal-difference learning\nAdjust the parameters to try to reduce the temporal difference between successive states\nthe update rule can be shown to converge to the closest possible approximation to the\ntrue function when the function approximator is linear in the features. With active\nlearning and nonlinear functions such as neural networks, tricky.\nLearning a model for an observable environment is a supervised learning problem,\nbecause the next percept gives the outcome state. Partially observable environments have \na much harder learning problem. The next percept is no longer known.\n\n22.4.3 Deep reinforcement learning\nIt is always possible to represent U or Q as linear combinations of features, if they aren't\nin an efficiently computible form the linear function approximator\nmay be insufficient. The parameters are the layers of the network. Obstacles:\nit is often difficult to get good performance and the trained system may behave\nvery unpredictably if the environment differs even a little from the training data.\n \n22.4.4 Reward shaping\nReal world environments may have sparse rewards: many actions are required\nto achieve any nonzero reward. R'(s,a,s') = R(s,a,s') + gamma*phi(s') - phi(s)\nphi can be constructed to reflect any desirable aspects of the state.\n\n22.4.5 Hierarchical reinforcement learning \nDeal with long action sequences by breaking them into small pieces, then again until\nthey are short enough to make learning easy. Begins with a partial program. The theoretical\nfoundations of HRL are based on the concept of the joint spate space. This is all essentially\nlike solving an MDP with: states->choice states sigma of the joint space, the \nactions at sigma are the choices avaliable in sigma from the partial program, the \nreward functionp(sigma,c,sigma') is the expected sum of rewards for all physical transition, the\ntransition model t(sigma,c,sigma') -> if c invokes a physical\naction a, then t borrows from the physical model P(s'|s,a)\n\n## Module 9 Lecture Notes\n That wasn’t planning, it was learning!\nSpecifically, reinforcement learning\nThere was an MDP, but you couldn’t solve it with just computation\nYou needed to actually act to figure it out\nImportant ideas in reinforcement learning that came up\nExploration: you have to try unknown actions to get information\nExploitation: eventually, you have to use what you know\nRegret: even if you learn intelligently, you make mistakes\nSampling: because of chance, you have to try things repeatedly\nDifficulty: learning can be much harder than solving a known MDP\n\nReinforcement Learning\nStill assume a Markov decision process (MDP):\nA set of states s ∈ S\nA set of actions (per state) A\nA model T(s,a,s’)\nA reward function R(s,a,s’)\nStill looking for a policy π(s)\nNew twist: don’t know T or R\nI.e. we don’t know which states are good or what the actions do\nMust actually try actions and states out to learn\n\nModel-Based Learning\nModel-Based Idea:\nLearn an approximate model based on experiences\nSolve for values as if the learned model were correct\nStep 1: Learn empirical MDP model\nCount outcomes s’ for each s, a\nNormalize to give an estimate of T\nDiscover each R when we experience (s, a, s’)\nStep 2: Solve the learned MDP\nFor example, use value iteration, as before\n\nPassive Reinforcement Learning\nSimplified task: policy evaluation\nInput: a fixed policy π(s)\nYou don’t know the transitions T(s,a,s’)\nYou don’t know the rewards R(s,a,s’)\nGoal: learn the state values\nIn this case:\nLearner is “along for the ride”\nNo choice about what actions to take\nJust execute the policy and learn from experience\nThis is NOT offline planning! You actually take actions in the world.\n\nHow to Explore?\nSeveral schemes for forcing exploration\nSimplest: random actions (ε-greedy)\nEvery time step, flip a coin\nWith (small) probability ε, act randomly\nWith (large) probability 1-ε, act on current policy\nProblems with random actions?\nYou do eventually explore the space, but keep\nthrashing around once learning is done\nOne solution: lower ε over time\nAnother solution: exploration functions\n\nEven if you learn the optimal policy,\nyou still make mistakes along the way!\nRegret is a measure of your total\nmistake cost: the difference between\nyour (expected) rewards, including\nyouthful suboptimality, and optimal\n(expected) rewards\nMinimizing regret goes beyond\nlearning to be optimal – it requires\noptimally learning to be optimal\nExample: random exploration and\nexploration functions both end up\noptimal, but random exploration has\nhigher regret\n\n\n## Module 9 Definitions\n* **One armed bandit** - slot machine\n* **Exploitation** - taking advantage of the current best option\n* **Exploration** - an agent must explore lots of its terrain to learn how to behave in it\n* **Markov reward process** - mdp with only one possible action ai\n* **Stopping time** - if you pull a arm T times the stopping time is T\n* **Bernoulli bandit** - each arm Mi produces a reward of 0 or 1 with fixed but unknown probability mu_i\n* **Upper confidence bound** - use the samples from each arm to establish a confidence interfal for the value of the arm\n* **Confidence interval** - range within which the value can be estimate to lie with high confidence\n* **Bandit superprocess** - each arm is a full MDP in its own\n* **Opportunity cost** - how much utility is given up per time step by not devoting that time toanother arm\n* **Reinforcement learning** - an agent interacts with the world and periodically recieves rewards or reinforcements\n* **Rewards** - reflect how well the agent is doing\n* **Reinforcements** - psychology term for rewards\n* **Passive reinforcement learning** - the agent's policy is fixed and the task is to learn the utilities of the states\n* **Active reinforcement learning** - the agent must also figure out what to do: exploration\n* **Apprenticeship learning** - training a learning agent using demonstrations rather than reward signals\n* **Direct utility estimation** - the utility of a state is defined as the expected total rewardfrom that state on\n* **Reward-to-go** - expected total reward from that state onward\n* **Temporal difference methods** - adjust the utility estimate towards the ideal equilibrium\n* **Pseudoexperience** - generated by simulating the current transition model\n* **Prioritized sweeping** - heuristic prefers to make adjustments to states whose likely successors have just undergone a large adjustment in their own utility estimates.\n* **Active learning agent** - decides what actions to take\n* **Greedy agent** - takes the action that it currently believes to be optimal at each step\n* **Glie (greedy in the limit of infinite exploration)** - must try each action in each state an unbounded number of times to avoid missing an optimal action\n* **Exploration function** - f(u,n) bellman equation, number of times a state has been tried\n* **Absorbing stare** - no actions have any effect and no rewards are recieved\n* **Sarsa (state action reward state action)** - updates with the Q-value of the action a'\n* **Off-poliicy** - learns q values that answer what would this be worht in this state, assuming i stop using the current policy?\n* **On-policy** - what would this action be worth in the state, assuming i stick with my policy\n* **Bayesian reinforcement learning** - assumes prior probability P(h) over hypothesis h about what the model is\n* **Function approximation** - process of constructing a compact approximation of the true utility function or Q-function\n* **Widrow-hoff rule** - delta rule equation 22.10\n* **Catastrophic forgetting** - a nonzero weight on a inear feature can cause a car to slide bc it forgets danger\n* **Experience replay** - ensures the car relives its youthful crashing behavior so it doesnt forget\n* **Deep reinforcement learning system** - reinforcement learning system that uses a deep network as a function approximator\n* **Reward shaing** - supplying teh agent with additional rewards for making progress\n* **Pseudorewards** - extra rewards\n* **Partial program** - outlines a hierarchial structure for the agent's behavior\n* **Joint state space** - each state (s,m) is composed of a physical state and a machine state\n* **Choice state sigma=(s,m)** - program counter for m is at a choice point in the agent program\n* **Additive decomposition** - overall utility decomposes into several terms, depeding on a few variables\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\" Agents in the real world need to handle uncertainty. We have seeen\n",
        "agents handle this with a belief state. This works on simple problems but has drawbacks:\n",
        "the agent must consider every possible explanation for its sensor observation, a correct\n",
        "contingent plan must consdider arbitarily unlikely contigencies, sometimes there is no plan\n",
        "guaranted to achieve the goal- but the agent must act. The right thing to do—the rational\n",
        "decision—therefore depends on both the relative importance of various goals and the\n",
        "likelihood that, and degree to which, they will be achieved.\n",
        "\n",
        "12.1.1 Summarizing uncertainty\n",
        "Trying to use logic to cope wiht domain like medical diagnosis fails for 3 main reasons:\n",
        "1) laziness 2) theoretical ignorance 3) practical ignorance. Agent's knowledge can at best\n",
        "provide only a degree of belief. This can be dealt with probability theory. The ontological \n",
        "commitments of logic and probability theory are the same. The theory of probability provides\n",
        "a way of summarizing the uncertainty that comes from our laziness and ignorance.\n",
        "\n",
        "12.1.2 Uncertainty and rational decisions\n",
        "To make choices and agent must have preferences among the different possible outcomes of \n",
        "the various plasn. Use utility theory to represent preferences and reason. A utility\n",
        "function can account for any set of preferences—quirky or typical, noble or perverse. \n",
        "Decision theory = probability theory + utility theory. An agent is rational if and only\n",
        "if it chooses the action that yields the highest expected utility, averaged over all\n",
        "the possible outcomes of the action.\n",
        "\n",
        "``` \n",
        "function DT-agent(percept) returns and action\n",
        "  persistent: belief_state, probabilistic beliefs about the current state of the world\n",
        "              axtion, the agent's action\n",
        "  update belief_state based on action and percept\n",
        "  calculate outcome probabilities for actions\n",
        "    given action descriptions and current belief_state\n",
        "  select action with highest expected utility\n",
        "    given probabilites of outcomes and utility information\n",
        "  return action\n",
        "``` \n",
        " \"\"\"\n",
        "chapter12_2 = \"\"\" For our agent to represent and use probabilistic information,\n",
        "we need a formal language. But the language of probability theory has traditionally\n",
        "been informal.\n",
        "\n",
        "12.2.1 What probabilites are about \n",
        "probabilistic assertions talk about how probable the various worlds are. A fully specified\n",
        "probability model associates a numerical probability P(w) wiht each possibe word.\n",
        "every possible world has a probability between 0 and 1 and that the total probability\n",
        "of the set of possible worlds is 1. In logic a set of words corresponds ot a proposition\n",
        "in formal language. Most of the time we have some evidence that has been uncovered.\n",
        "\n",
        "12.2.2 The language of propostitions in probability \n",
        "propositions describing sets of possible worlds are usually written in a notation\n",
        "that combines elements of propositional logic and constraint satisfaction notation.\n",
        "It is a factored represention in which a possibly word is represented by a set of variable/value \n",
        "pairs. It can also be shown as a structure representation. Variables are called random variables\n",
        "For continuous variables, it is not possible to write out the entire distribution as a vector,\n",
        "because there are infinitely many values. Instead, we can define the probability that a\n",
        "random variable takes on some value x as a parameterized function of x, usually called\n",
        "a probability density function. A possible world is defined to be an assignment of values\n",
        "to all of the random variables under consideration.\n",
        "\n",
        "12.2.3 Probability axioms and their reasonableness\n",
        "\n",
        "\"\"\"\n",
        "chapter12_3 = \"\"\"Simple method for probabilittic inference. We use the full joint\n",
        "distribution as the “knowledge base” from which answers to all questions may be derived. \n",
        "One common practice is to extract the distrobution over some subset of variable(s).\n",
        "Marginalization adn conditioning are useful rules for all kinds of derivations involving\n",
        "probability expressions \"\"\"\n",
        "chapter12_4 = \"\"\" Independence assertions are usually based on knowledge of the domain.\n",
        "If the complete set of variables can be divided into independent subsets, then the full\n",
        "joint distribution can be factored into separate joint distributions on those subsets.\n",
        "When they are available, then, independence assertions can help in reducing the size of\n",
        "the domain representation and the complexity of the inference problem. Unfortunately,\n",
        "clean separation of entire sets of variables by independence is quite rare.  \"\"\"\n",
        "chapter12_5 = \"\"\" Bayes’ rule is useful in practice because there are many cases where we\n",
        "do have good probability estimates for these three numbers and need to compute the fourth.\n",
        "Often, we perceive as evidence the effect of some unknown cause and we would like to\n",
        "determine that cause. a is cause b is effect. Causal direction P(effect|cause).\n",
        "Diagnostic direction P(cause|effect). diagnostic knowledge is often more fragile than\n",
        "causal knowledge. If there is a sudden epidemic of meningitis, the unconditional\n",
        "probability of meningitis, , will go up\n",
        "\n",
        "12.5.2 Using Bayes’ rule: Combining evidence\n",
        "To make progress, we need to find some additional assertions about the domain that will\n",
        "enable us to simplify the expressions. The notion of independence in provides a clue,\n",
        "but needs refining. These variables are independent, however, given the presence or\n",
        "the absence of a cavity.expresses the conditional independence of toothache and catch given cavity. \n",
        "We can plug it into  to obtain the probability of a cavity: (handwritten).\n",
        "In general, for n symptoms that are all conditionally independent given cavity,\n",
        "the size of the representation grows as O(n) instead of (O2^n). That means that conditional\n",
        "independence assertions can allow probabilistic systems to scale up; moreover, they are\n",
        "much more commonly available than absolute independence assertions. Conceptually, \n",
        "cavity separates toothache and catch because it is a direct cause of both of them. \n",
        "The decomposition of large probabilistic domains into weakly connected subsets through \n",
        "conditional independence is one of the most important developments in the recent history of AI.\"\"\"\n",
        "chapter12_6 = \"\"\" Naieve bayes model, probability distrobution (written). \n",
        "In practice, naive Bayes systems often work very well, even when the conditional independence\n",
        "assumption is not strictly true. The run time of this calculation is linear in the number\n",
        "of observed effects and does not depend on the number of unobserved effects.\n",
        "\n",
        "12.6.1 Text classification with naive Bayes \n",
        "Here the “cause” is the  variable, and the “effect” variables are the presence or\n",
        "absence of certain key words, HasWordi. The naive Bayes model consists of the prior probabilities \n",
        "P(category) and the conditional probabilities P(HasWordi|Category). For each category c,\n",
        "P(category = c) is estimated as the fraction of all previously seen documents that are \n",
        "of category .\"\"\"\n",
        "chapter12_7 = \"\"\" ure logical inference can conclude nothing about which square is most\n",
        "likely to be safe, so a logical agent might have to choose randomly. We will see that a\n",
        "probabilistic agent can do much better than the logical agent. The first term is the conditional \n",
        "probability distribution of a breeze configuration, given a pit configuration; its values are \n",
        "1 if all the breezy squares are adjacent to the pits and 0 otherwise. The second term is the prior \n",
        "probability of a pit configuration. Each square contains a pit with probability 0.2, \n",
        "independently of the other squares. What this section has shown is that even seemingly complicated \n",
        "problems can be formulated precisely in probability theory and solved with simple algorithms. \n",
        "To get efficient solutions, independence and conditional independence relationships can be used \n",
        "to simplify the summations required. These relationships often correspond to our natural \n",
        "understanding of how the problem should be decomposed.\"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" P(w|h), the probability of a word w given some history h. One way\n",
        "is with relatice frequency counts. Take a large corpus, count the number of times we see\n",
        "h and count the number of times this is followed by w. even the web isn’t big enough to \n",
        "give us good estimates in most cases. This is because language is creative; new sentences\n",
        "are created all the time, and we won’t always be able to count entire sentences.\n",
        "The intuition of the n-gram model is that instead of computing the probability of\n",
        "a word given its entire history, we can approximate the history by just the last few\n",
        "words.The bigram model, for example, approximates the probability of a word given\n",
        "all the previous words P(wn|w1:n−1) by using only the conditional probability of the\n",
        "preceding word P(wn|wn−1). we are thus making the following approximation: P(wn|w1:n−1) ≈ P(wn|wn−1)\n",
        "The assumption that the probability of a word depends only on the previous word is\n",
        "Markov called a Markov assumption.An intuitive way to\n",
        "estimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood\n",
        "estimation the MLE estimate for the parameters of an n-gram model by getting counts from a\n",
        "corpus, and normalizing the counts so that they lie between 0 and 1.\n",
        "For the general case of MLE n-gram parameter estimation:\n",
        "P(wn|wn−N+1:n−1) = C(wn−N+1:n−1 wn)/C(wn−N+1:n−1) estimates the n-gram probability by dividing the\n",
        "observed frequency of a particular sequence by the observed frequency of a prefix.\n",
        "This ratio is called a relative frequency. Some of the bigram probabilities above encode some facts that we think of as strictly\n",
        "syntactic in nature, like the fact that what comes after eat is usually a noun or an\n",
        "adjective, or that what comes after to is usually a verb. Others might be a fact about\n",
        "the personal assistant task, like the high probability of sentences beginning with\n",
        "the words I. And some might even be cultural rather than linguistic.\n",
        "We always represent and compute language model probabilities in log format\n",
        "as log probabilities. Since probabilities are (by definition) less than or equal to log\n",
        "probabilities 1, the more probabilities we multiply together, the smaller the product becomes\"\"\"\n",
        "SLP_chapter3_2 = \"\"\" The best way to evaluate the performance of a language model is to embed it in\n",
        "an application and measure how much the application improves (extrinsic evaluation).\n",
        "running big NLP systems end-to-end is often very expensive. Instead, it would be nice to have a metric \n",
        "that can be used to quickly evaluate potential improvements in a language model. An \n",
        "intrinsic evaluation metric is one that measures the quality of a model independent of any application.\n",
        "we need a test set. the probabilities of an n-gram model come from the corpus it is trained on, the training set or training corpus. We can then measure\n",
        "the quality of an n-gram model by its performance on some unseen data called the\n",
        "test set test set or test corpus. whichever model assigns a higher probability to the\n",
        "test set—meaning it more accurately predicts the test set—is a better model.\n",
        "Training on the test set introduces a bias that makes the probabilities all look too high, and causes huge\n",
        "inaccuracies in perplexity. The perplexity of a language model on a test set is the inverse probability of the test set, normalized\n",
        "by the number of words.PP(W) = P(w1w2 ...wN)^(-1/N). There is another way to think about\n",
        "perplexity: as the weighted average branching factor of a language. An (intrinsic) \n",
        "improvement in perplexity does not guarantee an (extrinsic) improvement in the performance \n",
        "of a language processing task like speech recognition or machine translation. Nonetheless, \n",
        "because perplexity often correlates with such improvements, it is commonly used as a \n",
        "quick check on an algorithm. But a model’s improvement in perplexity should always be \n",
        "confirmed by an end-to-end evaluation of a real task before concluding the evaluation of the model.\"\"\"\n",
        "SLP_chapter3_3 = \"\"\" One important way to visualize what kind of knowledge a language model embodies\n",
        "is to sample from it. Sampling from a distribution means to choose random points\n",
        "according to their likelihood. Thus sampling from a language model—which represents \n",
        "a distribution over sentences—means to generate some sentences, choosing\n",
        "each sentence according to its likelihood as defined by the model. We can use the \n",
        "same technique to generate bigrams by first generating a random bigram that starts with \n",
        "s but in < > (according to its bigram probability). Let’s say the second word of that bigram is w. \n",
        "We next choose a random bigram starting with w (again, drawn according to its bigram probability), and so on. \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" The n-gram model, like many statistical models, is dependent on the training corpus.\n",
        "One implication of this is that the probabilities often encode specific facts about a\n",
        "given training corpus. Another implication is that n-grams do a better and better job\n",
        "of modeling the training corpus as we increase the value of N. One step\n",
        "is to be sure to use a training corpus that has a similar genre to whatever task we are\n",
        "trying to accomplish. It is equally important to get training data in the appropriate dialect or variety,\n",
        "especially when processing social media posts or spoken transcripts. Our models may still be\n",
        "subject to the problem of sparsity. For any n-gram that occurred a sufficient number\n",
        "of times, we might have a good estimate of its probability. But because any corpus is\n",
        "limited, some perfectly acceptable English word sequences are bound to be missing\n",
        "from it. That is, we’ll have many cases of putative “zero probability n-grams” that\n",
        "should really have some non-zero probability. The percentage of OOV\n",
        "words that appear in the test set is called the OOV rate. An open vocabulary system open\n",
        "vocabulary is one in which we model these potential unknown words in the test set by adding a\n",
        "pseudo-word called <UNK>. The first one is to turn the problem back into a closed vocabulary one\n",
        "by choosing a fixed vocabulary in advance: 1. Choose a vocabulary (word list) that is fixed in advance.\n",
        "2. Convert in the training set any word that is not in this set (any OOV word) to the unknown \n",
        "word token <UNK> in a text normalization step. 3. Estimate the probabilities for <UNK> from its \n",
        "counts just like any other regular word in the training set. The second alternative, in \n",
        "situations where we don’t have a prior vocabulary in advance, is to create such a vocabulary \n",
        "implicitly, replacing words in the training data by <UNK> based on their frequency.\"\"\"\n",
        "SLP_chapter3_5 = \"\"\" To keep a language model from assigning zero probability to these unseen \n",
        "events, we’ll have to shave off a bit of probability mass from some more frequent events \n",
        "and give it to the events we’ve never seen. This modification is called smoothing or discounting.\n",
        "Laplace smoothing:  All the counts that used to be zero will now\n",
        "have a count of 1, the counts of 1 will be 2, and so on. This does not perform well but is \n",
        "a good introduction, and its useful for other tasks like text classification.\n",
        "Instead of changing both the numerator and denominator, it is convenient to\n",
        "describe how a smoothing algorithm affects the numerator, by defining an adjusted\n",
        "count c∗ . This adjusted count is easier to compare directly with the MLE counts and\n",
        "can be turned into a probability like an MLE count by normalizing by N. \n",
        "A related way to view smoothing is as discounting (lowering) some non-zero\n",
        "counts in order to get the probability mass that will be assigned to the zero counts.\n",
        "We might describe a smoothing algorithm in terms of a relative discount dc=c*/c\n",
        "add-k smoothing. P∗_Add-k(wn|wn−1) = C(wn−1wn) +k / C(wn−1) +kV\n",
        "Add-k smoothing requires that we have a method for choosing k. Although add-k is useful for some\n",
        "tasks (including text classification), it turns out that it still doesn’t work well for\n",
        "language modeling. \n",
        "\n",
        "Backoff and Interpolation:\n",
        "Sometimes using less context is a good thing, helping to generalize more for contexts \n",
        "that the model hasn’t learned much about. In backoff, we use the trigram if the evidence is\n",
        "sufficient, otherwise we use the bigram, otherwise the unigram. In other words, we\n",
        "only “back off” to a lower-order n-gram if we have zero evidence for a higher-order\n",
        "interpolation n-gram. By contrast, in interpolation, we always mix the probability estimates from\n",
        "all the n-gram estimators, weighing and combining the trigram, bigram, and unigram\n",
        "counts.\"\"\"\n",
        "SLP_chapter3_7 = \"\"\"Efficiency considerations are important when building language models that use\n",
        "such large sets of n-grams. Rather than store each word as a string, it is generally\n",
        "represented in memory as a 64-bit hash number, with the words themselves stored\n",
        "on disk. Probabilities are generally quantized using only 4-8 bits (instead of 8-byte\n",
        "floats), and n-grams are stored in reverse tries. An n-gram language model can also be shrunk by pruning. \n",
        "Stupid backoff gives up the idea of trying to make the language\n",
        "model a true probability distribution. There is no discounting of the higher-order\n",
        "probabilities. If a higher-order n-gram has a zero count, we simply backoff to a\n",
        "lower order n-gram, weighed by a fixed (context-independent) weight\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" Uncertainty\n",
        "General situation:\n",
        "• Observed variables (evidence): Agent knows certain things about the state of the world (e.g.,\n",
        "sensor readings or symptoms)\n",
        "• Unobserved variables (states): Agent needs to reason about other aspects (e.g. where an object\n",
        "is or what disease is present)\n",
        "• Model: Agent knows something about how the known variables relate to the unknown variables\n",
        "Probabilistic reasoning gives us a framework\n",
        "for managing our beliefs and knowledge\n",
        "What Probabilities Are About\n",
        "Like logical assertions, probabilities are about possible worlds. Instead of strictly\n",
        "ruling out possibilities (where a logical assertion is false), probabilities quantify how\n",
        "likely a particular possible world is.\n",
        "In probability theory, the possible worlds are called the sample space, and they\n",
        "mutually exclusive and exhaustive.\n",
        "A fully specified probability model associates a probability P(w) with each possible\n",
        "world w. \n",
        "\n",
        "A random variable is some aspect of the world\n",
        "about which we (may) have uncertainty\n",
        "• R = Is it raining?\n",
        "• U = Is the professor carrying an umbrella?\n",
        "We denote random variables with capital letters\n",
        "\n",
        "Probabilistic Inference\n",
        "Probabilistic inference: compute a desired\n",
        "probability from other known probabilities (e.g.\n",
        "conditional from joint)\n",
        "We generally compute conditional probabilities\n",
        "• P(on time | no reported accidents) = 0.90\n",
        "• These represent the agent’s beliefs given the\n",
        "evidence\n",
        "Probabilities change with new evidence:\n",
        "• P(on time | no accidents, 5 a.m.) = 0.95\n",
        "• P(on time | no accidents, 5 a.m., raining) = 0.80\n",
        "• Observing new evidence causes beliefs to be updated\n",
        "\n",
        "Probabilistic Language Models\n",
        "One goal: assign a probability to a sentence\n",
        "• Autocomplete for texting\n",
        "• Machine Translation\n",
        "• Spelling Correction\n",
        "• Speech Recognition\n",
        "Other Natural Language Generation tasks: summarization, question-answering, dialog\n",
        "systems\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"belief state\": \"epresentation of the set of all possible world states that it might be in\",\n",
        "\"qualification problem\": \"when we can't properly qualify a problem and therefore can't infer the proper plan\",\n",
        "\"rational decision\": \"the right thing to do \",\n",
        "\"laziness\": \"too much work to list the complete set of antecedents or consequents needed to ensure an exceptionless rule\",\n",
        "\"theoretical ignorance\": \"Medical science has no complete theory for the domain.\",\n",
        "\"practical ignorance\": \"if we know all rules we still might be uncertain if all necessary tests havent been run\",\n",
        "\"ontological commitments\": \"the world is composed of facts that do or do not hold in any particular case\",\n",
        "\"epistemological commitments\": \"logical agent believes each sentence to be true or false or has no opinion\",\n",
        "\"utility theory\": \"very state (or state sequence) has a degree of usefulness, or utility, to an agent\",\n",
        "\"decision theory\": \"Preferences, as expressed by utilities, are combined with probabilities\",\n",
        "\"sample space\": \"the set of all possible words\",\n",
        "\"unconditional or prior probabilities\": \"degrees of belief in propositions in the absence of any other information\",\n",
        "\"conditional or posterior probability\": \"probability of doing something given that we have already done something\",\n",
        "\"random variable\": \"is a function that maps from the domain of possible worlds omega to some range\",\n",
        "\"bernoulli distrobution\": \"the set {0,1}\",\n",
        "\"joint probability distrobution\": \"n x m table of probabilities based on the probability of P(n, m)\",\n",
        "\"full joint probability distrobution\": \"probability model is completely determined by the joint distribution for all of the random variables\",\n",
        "\"probabilistic inference\": \"computation of posterior probabilities for query propositions given observed evidence\",\n",
        "\"marginalization or summing out\": \"sum up the probabilities for each possible value of the other variables, thereby taking them out of the equation\",\n",
        "\"independence\": \"separating out probabilites when one cannot influence the others\",\n",
        "\"text classification\": \"decide which of a predefined set of classes or categories it belongs to\"\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "8c48e1fd-9a17-4316-8190-df015b64c378"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Agents in the real world need to handle uncertainty. We have seeen\nagents handle this with a belief state. This works on simple problems but has drawbacks:\nthe agent must consider every possible explanation for its sensor observation, a correct\ncontingent plan must consdider arbitarily unlikely contigencies, sometimes there is no plan\nguaranted to achieve the goal- but the agent must act. The right thing to do—the rational\ndecision—therefore depends on both the relative importance of various goals and the\nlikelihood that, and degree to which, they will be achieved.\n\n12.1.1 Summarizing uncertainty\nTrying to use logic to cope wiht domain like medical diagnosis fails for 3 main reasons:\n1) laziness 2) theoretical ignorance 3) practical ignorance. Agent's knowledge can at best\nprovide only a degree of belief. This can be dealt with probability theory. The ontological \ncommitments of logic and probability theory are the same. The theory of probability provides\na way of summarizing the uncertainty that comes from our laziness and ignorance.\n\n12.1.2 Uncertainty and rational decisions\nTo make choices and agent must have preferences among the different possible outcomes of \nthe various plasn. Use utility theory to represent preferences and reason. A utility\nfunction can account for any set of preferences—quirky or typical, noble or perverse. \nDecision theory = probability theory + utility theory. An agent is rational if and only\nif it chooses the action that yields the highest expected utility, averaged over all\nthe possible outcomes of the action.\n\n``` \nfunction DT-agent(percept) returns and action\n  persistent: belief_state, probabilistic beliefs about the current state of the world\n              axtion, the agent's action\n  update belief_state based on action and percept\n  calculate outcome probabilities for actions\n    given action descriptions and current belief_state\n  select action with highest expected utility\n    given probabilites of outcomes and utility information\n  return action\n``` \n \n\n## AIMA Chapter 12.2 Basic Probability Notation\n For our agent to represent and use probabilistic information,\nwe need a formal language. But the language of probability theory has traditionally\nbeen informal.\n\n12.2.1 What probabilites are about \nprobabilistic assertions talk about how probable the various worlds are. A fully specified\nprobability model associates a numerical probability P(w) wiht each possibe word.\nevery possible world has a probability between 0 and 1 and that the total probability\nof the set of possible worlds is 1. In logic a set of words corresponds ot a proposition\nin formal language. Most of the time we have some evidence that has been uncovered.\n\n12.2.2 The language of propostitions in probability \npropositions describing sets of possible worlds are usually written in a notation\nthat combines elements of propositional logic and constraint satisfaction notation.\nIt is a factored represention in which a possibly word is represented by a set of variable/value \npairs. It can also be shown as a structure representation. Variables are called random variables\nFor continuous variables, it is not possible to write out the entire distribution as a vector,\nbecause there are infinitely many values. Instead, we can define the probability that a\nrandom variable takes on some value x as a parameterized function of x, usually called\na probability density function. A possible world is defined to be an assignment of values\nto all of the random variables under consideration.\n\n12.2.3 Probability axioms and their reasonableness\n\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\nSimple method for probabilittic inference. We use the full joint\ndistribution as the “knowledge base” from which answers to all questions may be derived. \nOne common practice is to extract the distrobution over some subset of variable(s).\nMarginalization adn conditioning are useful rules for all kinds of derivations involving\nprobability expressions \n\n## AIMA Chapter 12.4 Independence\n Independence assertions are usually based on knowledge of the domain.\nIf the complete set of variables can be divided into independent subsets, then the full\njoint distribution can be factored into separate joint distributions on those subsets.\nWhen they are available, then, independence assertions can help in reducing the size of\nthe domain representation and the complexity of the inference problem. Unfortunately,\nclean separation of entire sets of variables by independence is quite rare.  \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Bayes’ rule is useful in practice because there are many cases where we\ndo have good probability estimates for these three numbers and need to compute the fourth.\nOften, we perceive as evidence the effect of some unknown cause and we would like to\ndetermine that cause. a is cause b is effect. Causal direction P(effect|cause).\nDiagnostic direction P(cause|effect). diagnostic knowledge is often more fragile than\ncausal knowledge. If there is a sudden epidemic of meningitis, the unconditional\nprobability of meningitis, , will go up\n\n12.5.2 Using Bayes’ rule: Combining evidence\nTo make progress, we need to find some additional assertions about the domain that will\nenable us to simplify the expressions. The notion of independence in provides a clue,\nbut needs refining. These variables are independent, however, given the presence or\nthe absence of a cavity.expresses the conditional independence of toothache and catch given cavity. \nWe can plug it into  to obtain the probability of a cavity: (handwritten).\nIn general, for n symptoms that are all conditionally independent given cavity,\nthe size of the representation grows as O(n) instead of (O2^n). That means that conditional\nindependence assertions can allow probabilistic systems to scale up; moreover, they are\nmuch more commonly available than absolute independence assertions. Conceptually, \ncavity separates toothache and catch because it is a direct cause of both of them. \nThe decomposition of large probabilistic domains into weakly connected subsets through \nconditional independence is one of the most important developments in the recent history of AI.\n\n## AIMA Chapter 12.6 Naive Bayes Models\n Naieve bayes model, probability distrobution (written). \nIn practice, naive Bayes systems often work very well, even when the conditional independence\nassumption is not strictly true. The run time of this calculation is linear in the number\nof observed effects and does not depend on the number of unobserved effects.\n\n12.6.1 Text classification with naive Bayes \nHere the “cause” is the  variable, and the “effect” variables are the presence or\nabsence of certain key words, HasWordi. The naive Bayes model consists of the prior probabilities \nP(category) and the conditional probabilities P(HasWordi|Category). For each category c,\nP(category = c) is estimated as the fraction of all previously seen documents that are \nof category .\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n ure logical inference can conclude nothing about which square is most\nlikely to be safe, so a logical agent might have to choose randomly. We will see that a\nprobabilistic agent can do much better than the logical agent. The first term is the conditional \nprobability distribution of a breeze configuration, given a pit configuration; its values are \n1 if all the breezy squares are adjacent to the pits and 0 otherwise. The second term is the prior \nprobability of a pit configuration. Each square contains a pit with probability 0.2, \nindependently of the other squares. What this section has shown is that even seemingly complicated \nproblems can be formulated precisely in probability theory and solved with simple algorithms. \nTo get efficient solutions, independence and conditional independence relationships can be used \nto simplify the summations required. These relationships often correspond to our natural \nunderstanding of how the problem should be decomposed.\n\n## SLP Chapter 3.1 N-Grams\n P(w|h), the probability of a word w given some history h. One way\nis with relatice frequency counts. Take a large corpus, count the number of times we see\nh and count the number of times this is followed by w. even the web isn’t big enough to \ngive us good estimates in most cases. This is because language is creative; new sentences\nare created all the time, and we won’t always be able to count entire sentences.\nThe intuition of the n-gram model is that instead of computing the probability of\na word given its entire history, we can approximate the history by just the last few\nwords.The bigram model, for example, approximates the probability of a word given\nall the previous words P(wn|w1:n−1) by using only the conditional probability of the\npreceding word P(wn|wn−1). we are thus making the following approximation: P(wn|w1:n−1) ≈ P(wn|wn−1)\nThe assumption that the probability of a word depends only on the previous word is\nMarkov called a Markov assumption.An intuitive way to\nestimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood\nestimation the MLE estimate for the parameters of an n-gram model by getting counts from a\ncorpus, and normalizing the counts so that they lie between 0 and 1.\nFor the general case of MLE n-gram parameter estimation:\nP(wn|wn−N+1:n−1) = C(wn−N+1:n−1 wn)/C(wn−N+1:n−1) estimates the n-gram probability by dividing the\nobserved frequency of a particular sequence by the observed frequency of a prefix.\nThis ratio is called a relative frequency. Some of the bigram probabilities above encode some facts that we think of as strictly\nsyntactic in nature, like the fact that what comes after eat is usually a noun or an\nadjective, or that what comes after to is usually a verb. Others might be a fact about\nthe personal assistant task, like the high probability of sentences beginning with\nthe words I. And some might even be cultural rather than linguistic.\nWe always represent and compute language model probabilities in log format\nas log probabilities. Since probabilities are (by definition) less than or equal to log\nprobabilities 1, the more probabilities we multiply together, the smaller the product becomes\n\n## SLP Chapter 3.2 Evaluating Language Models\n The best way to evaluate the performance of a language model is to embed it in\nan application and measure how much the application improves (extrinsic evaluation).\nrunning big NLP systems end-to-end is often very expensive. Instead, it would be nice to have a metric \nthat can be used to quickly evaluate potential improvements in a language model. An \nintrinsic evaluation metric is one that measures the quality of a model independent of any application.\nwe need a test set. the probabilities of an n-gram model come from the corpus it is trained on, the training set or training corpus. We can then measure\nthe quality of an n-gram model by its performance on some unseen data called the\ntest set test set or test corpus. whichever model assigns a higher probability to the\ntest set—meaning it more accurately predicts the test set—is a better model.\nTraining on the test set introduces a bias that makes the probabilities all look too high, and causes huge\ninaccuracies in perplexity. The perplexity of a language model on a test set is the inverse probability of the test set, normalized\nby the number of words.PP(W) = P(w1w2 ...wN)^(-1/N). There is another way to think about\nperplexity: as the weighted average branching factor of a language. An (intrinsic) \nimprovement in perplexity does not guarantee an (extrinsic) improvement in the performance \nof a language processing task like speech recognition or machine translation. Nonetheless, \nbecause perplexity often correlates with such improvements, it is commonly used as a \nquick check on an algorithm. But a model’s improvement in perplexity should always be \nconfirmed by an end-to-end evaluation of a real task before concluding the evaluation of the model.\n\n## SLP Chapter 3.3 Sampling sentences from a language model\n One important way to visualize what kind of knowledge a language model embodies\nis to sample from it. Sampling from a distribution means to choose random points\naccording to their likelihood. Thus sampling from a language model—which represents \na distribution over sentences—means to generate some sentences, choosing\neach sentence according to its likelihood as defined by the model. We can use the \nsame technique to generate bigrams by first generating a random bigram that starts with \ns but in < > (according to its bigram probability). Let’s say the second word of that bigram is w. \nWe next choose a random bigram starting with w (again, drawn according to its bigram probability), and so on. \n\n## SLP Chapter 3.4 Generalization and Zeros\n The n-gram model, like many statistical models, is dependent on the training corpus.\nOne implication of this is that the probabilities often encode specific facts about a\ngiven training corpus. Another implication is that n-grams do a better and better job\nof modeling the training corpus as we increase the value of N. One step\nis to be sure to use a training corpus that has a similar genre to whatever task we are\ntrying to accomplish. It is equally important to get training data in the appropriate dialect or variety,\nespecially when processing social media posts or spoken transcripts. Our models may still be\nsubject to the problem of sparsity. For any n-gram that occurred a sufficient number\nof times, we might have a good estimate of its probability. But because any corpus is\nlimited, some perfectly acceptable English word sequences are bound to be missing\nfrom it. That is, we’ll have many cases of putative “zero probability n-grams” that\nshould really have some non-zero probability. The percentage of OOV\nwords that appear in the test set is called the OOV rate. An open vocabulary system open\nvocabulary is one in which we model these potential unknown words in the test set by adding a\npseudo-word called <UNK>. The first one is to turn the problem back into a closed vocabulary one\nby choosing a fixed vocabulary in advance: 1. Choose a vocabulary (word list) that is fixed in advance.\n2. Convert in the training set any word that is not in this set (any OOV word) to the unknown \nword token <UNK> in a text normalization step. 3. Estimate the probabilities for <UNK> from its \ncounts just like any other regular word in the training set. The second alternative, in \nsituations where we don’t have a prior vocabulary in advance, is to create such a vocabulary \nimplicitly, replacing words in the training data by <UNK> based on their frequency.\n\n## SLP Chapter 3.5 Smoothing\n To keep a language model from assigning zero probability to these unseen \nevents, we’ll have to shave off a bit of probability mass from some more frequent events \nand give it to the events we’ve never seen. This modification is called smoothing or discounting.\nLaplace smoothing:  All the counts that used to be zero will now\nhave a count of 1, the counts of 1 will be 2, and so on. This does not perform well but is \na good introduction, and its useful for other tasks like text classification.\nInstead of changing both the numerator and denominator, it is convenient to\ndescribe how a smoothing algorithm affects the numerator, by defining an adjusted\ncount c∗ . This adjusted count is easier to compare directly with the MLE counts and\ncan be turned into a probability like an MLE count by normalizing by N. \nA related way to view smoothing is as discounting (lowering) some non-zero\ncounts in order to get the probability mass that will be assigned to the zero counts.\nWe might describe a smoothing algorithm in terms of a relative discount dc=c*/c\nadd-k smoothing. P∗_Add-k(wn|wn−1) = C(wn−1wn) +k / C(wn−1) +kV\nAdd-k smoothing requires that we have a method for choosing k. Although add-k is useful for some\ntasks (including text classification), it turns out that it still doesn’t work well for\nlanguage modeling. \n\nBackoff and Interpolation:\nSometimes using less context is a good thing, helping to generalize more for contexts \nthat the model hasn’t learned much about. In backoff, we use the trigram if the evidence is\nsufficient, otherwise we use the bigram, otherwise the unigram. In other words, we\nonly “back off” to a lower-order n-gram if we have zero evidence for a higher-order\ninterpolation n-gram. By contrast, in interpolation, we always mix the probability estimates from\nall the n-gram estimators, weighing and combining the trigram, bigram, and unigram\ncounts.\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\nEfficiency considerations are important when building language models that use\nsuch large sets of n-grams. Rather than store each word as a string, it is generally\nrepresented in memory as a 64-bit hash number, with the words themselves stored\non disk. Probabilities are generally quantized using only 4-8 bits (instead of 8-byte\nfloats), and n-grams are stored in reverse tries. An n-gram language model can also be shrunk by pruning. \nStupid backoff gives up the idea of trying to make the language\nmodel a true probability distribution. There is no discounting of the higher-order\nprobabilities. If a higher-order n-gram has a zero count, we simply backoff to a\nlower order n-gram, weighed by a fixed (context-independent) weight\n\n## Module 10 Lecture Notes\n Uncertainty\nGeneral situation:\n• Observed variables (evidence): Agent knows certain things about the state of the world (e.g.,\nsensor readings or symptoms)\n• Unobserved variables (states): Agent needs to reason about other aspects (e.g. where an object\nis or what disease is present)\n• Model: Agent knows something about how the known variables relate to the unknown variables\nProbabilistic reasoning gives us a framework\nfor managing our beliefs and knowledge\nWhat Probabilities Are About\nLike logical assertions, probabilities are about possible worlds. Instead of strictly\nruling out possibilities (where a logical assertion is false), probabilities quantify how\nlikely a particular possible world is.\nIn probability theory, the possible worlds are called the sample space, and they\nmutually exclusive and exhaustive.\nA fully specified probability model associates a probability P(w) with each possible\nworld w. \n\nA random variable is some aspect of the world\nabout which we (may) have uncertainty\n• R = Is it raining?\n• U = Is the professor carrying an umbrella?\nWe denote random variables with capital letters\n\nProbabilistic Inference\nProbabilistic inference: compute a desired\nprobability from other known probabilities (e.g.\nconditional from joint)\nWe generally compute conditional probabilities\n• P(on time | no reported accidents) = 0.90\n• These represent the agent’s beliefs given the\nevidence\nProbabilities change with new evidence:\n• P(on time | no accidents, 5 a.m.) = 0.95\n• P(on time | no accidents, 5 a.m., raining) = 0.80\n• Observing new evidence causes beliefs to be updated\n\nProbabilistic Language Models\nOne goal: assign a probability to a sentence\n• Autocomplete for texting\n• Machine Translation\n• Spelling Correction\n• Speech Recognition\nOther Natural Language Generation tasks: summarization, question-answering, dialog\nsystems\n\n\n## Module 10 Definitions\n* **Belief state** - epresentation of the set of all possible world states that it might be in\n* **Qualification problem** - when we can't properly qualify a problem and therefore can't infer the proper plan\n* **Rational decision** - the right thing to do \n* **Laziness** - too much work to list the complete set of antecedents or consequents needed to ensure an exceptionless rule\n* **Theoretical ignorance** - Medical science has no complete theory for the domain.\n* **Practical ignorance** - if we know all rules we still might be uncertain if all necessary tests havent been run\n* **Ontological commitments** - the world is composed of facts that do or do not hold in any particular case\n* **Epistemological commitments** - logical agent believes each sentence to be true or false or has no opinion\n* **Utility theory** - very state (or state sequence) has a degree of usefulness, or utility, to an agent\n* **Decision theory** - Preferences, as expressed by utilities, are combined with probabilities\n* **Sample space** - the set of all possible words\n* **Unconditional or prior probabilities** - degrees of belief in propositions in the absence of any other information\n* **Conditional or posterior probability** - probability of doing something given that we have already done something\n* **Random variable** - is a function that maps from the domain of possible worlds omega to some range\n* **Bernoulli distrobution** - the set {0,1}\n* **Joint probability distrobution** - n x m table of probabilities based on the probability of P(n, m)\n* **Full joint probability distrobution** - probability model is completely determined by the joint distribution for all of the random variables\n* **Probabilistic inference** - computation of posterior probabilities for query propositions given observed evidence\n* **Marginalization or summing out** - sum up the probabilities for each possible value of the other variables, thereby taking them out of the equation\n* **Independence** - separating out probabilites when one cannot influence the others\n* **Text classification** - decide which of a predefined set of classes or categories it belongs to\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" Bayesian networks can represent essentially any full joint probability distribution \n",
        "and in many cases can do so very concisely. 1) Each node corresponds to a random variable, \n",
        "which may be discrete or continuous. 2) Directed links or arrows connect pairs of nodes. \n",
        "If there is an arrow from node X to node Y, X is said to be a parent of Y. 3) Each node\n",
        "Xi has associated probability information theta(Xi|parents(Xi)) that quantifies the\n",
        "effect of the parents on the node using a finite number of parameters. Once the topology\n",
        "of the Bayes net is laid out, we need only specify the local probability information for\n",
        "each variable, in the form of a conditional distribution given its parents. The full joint\n",
        "distribution for all the variables is defined by the topology and the local probability information.\n",
        "The local probability information attached to each node takes the form of aconditional\n",
        "probability table (CPT). (CPTs can be used only for discrete variables). Each row in a CPT \n",
        "contains the conditional probability of each node value for a conditioning case.\"\"\"\n",
        "chapter13_2 = \"\"\" The semantics of Bayes nets defines each entry in the joint distribution\n",
        "as follows: equation . the Bayesian network is a correct representation of the domain\n",
        "only if each node is conditionally independent of its other predecessors in the node\n",
        "ordering, given its parents. 1) Nodes: determine the set of variables that are required\n",
        "to model the domain, now order them. 2) Links: for i = 1 to n do: choose a minimal set of parents\n",
        "for Xi from X1 to Xi-1, for each paretn insert a link from the parent to Xi, CPts: write\n",
        "down the cpt P(Xi|parents(Xi)). a Bayes net can often be far more compact than the full \n",
        "joint distribution. This property is what makes it feasible to handle domains with many variables.\n",
        "If we stick to a causal model, we end up having to specify fewer numbers, and the numbers will\n",
        "often be easier to come up with.\n",
        "\n",
        "13.2.1 Conditional independence relations in Bayesian networks \n",
        "Each variable is conditionally independent of its non-descendants, given its parents.\n",
        "instead of defining the full joint distribution as the product of conditional distributions,\n",
        "the network defines a set of conditional independence properties. The full joint\n",
        "distribution can be derived from those properties. Is a set of nodes X is conditionally \n",
        "independent of another set Y, given a third set Z. This can be determined efficiently by \n",
        "examining the Bayes net to see whether Z d-separates X and Y. The process works as follows:\n",
        "1) Consider just the ancestral subgraph consisting of X, Y, Z, and their ancestors. 2) Add \n",
        "links between any unlinked pair of nodes that share a common child; now we have the \n",
        "so-called moral graph. 3)Replace all directed links by undirected links. 4) If Z blocks \n",
        "all paths between X and Y in the resulting graph, then Z d-separates X and Y. In \n",
        "that case, X is conditionally independent of Y, given Z.\n",
        "\n",
        "13.2.2 Efficient Representation of Conditional Distributions \n",
        "lling in the CPT for a node requires up to O(2^k) numbers and perhaps a great deal\n",
        "of experience with all the possible conditioning cases. Usually, such relationships are\n",
        "describable by a canonical distribution that fits some standard pattern. In such cases, the\n",
        "complete table can be specified just by naming the pattern and perhaps supplying a few parameters.\n",
        "Uncertain relationships can often be characterized by so-called noisy logical relationships. \n",
        "\n",
        "13.2.3 Bayesian nets with continuous variables\n",
        "discretization. define a continuous variable using one of the standard families of probability\n",
        "density functions. Nonparametric representation.\n",
        "\"\"\"\n",
        "chapter13_3 = \"\"\" The basic task for any probabilistic inference system is to compute the\n",
        "posterior probability distribution for a set of query variables, given some observed\n",
        "event—usually, some assignment of values to a set of evidence variables. \n",
        "\n",
        "13.3.1 Inference by enumeration\n",
        "a query can be answered using a Bayes net by computing sums of products of conditional\n",
        "probabilities from the network. Equation (handwritten) with complexity O(2^n) after nesting\n",
        "summations\n",
        "\n",
        "``` \n",
        "Function enumeration-ask(X,e,bn) returns a distrobution over X\n",
        "  inputs: X, the query\n",
        "          e, observed values for variables E\n",
        "          bn, a bayes net with variables vars\n",
        "  Q(X) <- a distrobution over x, initially empty\n",
        "  for each value xi of X do\n",
        "    Q(xi) <- enumerate-all(vars, exi)\n",
        "      where exi is ectended with X = xi\n",
        "  return normalize(Q(X))\n",
        "\n",
        "function ennumerate-ALL(vars, e) returns a real number\n",
        "  if empty?(vars) then return 1\n",
        "  V<-First(vars)\n",
        "  if V is evidence variable with value v in e\n",
        "    then return P(v|parents(V))x enumerate-all(REST(vars),e)\n",
        "    else return sum P(v|parents(V))x enumerate-all(REST(vars),ev)\n",
        "      where ev is extended with V = v\n",
        "``` \n",
        "\n",
        "13.3.2 The variable elimination algorithm\n",
        "The final question is, of course, how to do inference in the network to make predictions. \n",
        "We turn now to this question. For each inference method that we describe, we will evaluate \n",
        "the method on the insurance net to measure the time and space requirements of the method.\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" Bayes’ Nets: Big Picture\n",
        "- Two problems with using full joint distribution tables\n",
        "as our probabilistic models:\n",
        "- Unless there are only a few variables, the joint is WAY too\n",
        "big to represent explicitly\n",
        "- Hard to learn (estimate) anything empirically about more\n",
        "than a few variables at a time\n",
        "- Bayes’ nets: a technique for describing complex joint\n",
        "distributions (models) using simple, local\n",
        "distributions (conditional probabilities)\n",
        "- More properly called graphical models\n",
        "- We describe how variables locally interact\n",
        "- Local interactions chain together to give global, indirect\n",
        "interactions\n",
        "\n",
        "Graphical Model Notation\n",
        "- Nodes: variables (with domains)\n",
        "- Can be assigned (observed) or unassigned\n",
        "(unobserved)\n",
        "- Arcs: interactions\n",
        "- Indicate “direct influence” between variables\n",
        "- Formally: encode conditional independence\n",
        "(more later)\n",
        "- For now: imagine that arrows mean\n",
        "direct causation (in general, they don’t!)\n",
        "\n",
        "Bayes’ Net Semantics\n",
        "- A set of nodes, one per variable X\n",
        "- A directed, acyclic graph\n",
        "- A conditional distribution for each node\n",
        "- A collection of distributions over X, one for each\n",
        "combination of parents’ values\n",
        "- CPT: conditional probability table\n",
        "- Description of a noisy “causal” process\n",
        "- Bayes’ nets implicitly encode joint distributions\n",
        "- As a product of local conditional distributions\n",
        "- To see what probability a BN gives to a full assignment, multiply all the\n",
        "relevant conditionals together:\n",
        "\n",
        "Causality?\n",
        "- When Bayes’ nets reflect the true causal patterns:\n",
        "- Often simpler (nodes have fewer parents)\n",
        "- Often easier to think about\n",
        "- Often easier to elicit from experts\n",
        "- BNs need not actually be causal\n",
        "- Sometimes no causal net exists over the domain\n",
        "(especially if variables are missing)\n",
        "- E.g. consider the variables Traffic and Drips\n",
        "- End up with arrows that reflect correlation, not causation\n",
        "- What do the arrows really mean?\n",
        "- Topology may happen to encode causal structure\n",
        "- Topology really encodes conditional independence\n",
        "\n",
        "Active / Inactive Paths\n",
        "- Question: Are X and Y conditionally independent given\n",
        "evidence variables {Z}?\n",
        "- Yes, if X and Y “d-separated” by Z\n",
        "- Consider all (undirected) paths from X to Y\n",
        "- No active paths = independence!\n",
        "- A path is active if each triple is active:\n",
        "- Causal chain A ® B ® C where B is unobserved (either direction)\n",
        "- Common cause A ¬ B ® C where B is unobserved\n",
        "- Common effect (aka v-structure)\n",
        "A ® B ¬ C where B or one of its descendents is observed\n",
        "- All it takes to block a path is a single inactive segment\n",
        "\n",
        "Bayes Nets Representation Summary\n",
        "- Bayes nets compactly encode joint distributions\n",
        "- Guaranteed independencies of distributions can be\n",
        "deduced from BN graph structure\n",
        "- D-separation gives precise conditional independence\n",
        "guarantees from graph alone\n",
        "- A Bayes’ Net’s joint distribution may have further\n",
        "(conditional) independence that is not detectable until\n",
        "you inspect its specific distribution \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"bayesian network\": \"directed graph in which each node is annotated with quantitative probability information.\",\n",
        "\"conditioning case\": \"a possible combination of values for the parent nodes—a miniature possible world\",\n",
        "\"syntax of a bayes net\": \"onsists of a directed acyclic graph with some local probability information attached to each node\",\n",
        "\"semantics\": \"defines how the syntax corresponds to a joint distribution over the variables of the network.\",\n",
        "\"topological order\": \"any order consistent with the directed graph structure.\",\n",
        "\"locally structured\": \"each subcomponent interacts directly with only a bounded number of other components\",\n",
        "\"sparse\": \"locally structured\",\n",
        "\"deterministic node\": \" has its value specified exactly by the values of its parents, with no uncertainty.\",\n",
        "\"context specific independence\": \"a variable is conditionally independent of some of its parents given certain values of others.\",\n",
        "\"noisy OR model\": \"allows for uncertainty about the ability of each parent to cause the child to be true\",\n",
        "\"discretization\": \"dividing up the possible values into a fixed set of intervals.\",\n",
        "\"nonparametric representation\": \"define the conditional distribution implicitly with a collection of instances, each containing specific values of the parent and child variables\",\n",
        "\"hybrid bayseian network\": \"A network with both discrete and continuous variables\",\n",
        "\"linear-gaussian condition distrobution\": \"child has a Gaussian distribution whose mean mu varies linearly with the value of the parent and whose standard deviation sigma is fixed\",\n",
        "\"probit model \": \"probability unit\",\n",
        "\"expit or inverse logit model\": \"uses the logistic function 1/(1+e^-x) to produce a soft threshold—it maps any  to a value between 0 and 1\",\n",
        "\"variable elimination algorithm\": \"evaluating expressions such as Equation in right-to-left order. Intermediate results are stored, and summations that depend on the variable are done.\"  \n",
        "}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "3711654d-fbe8-4f54-f7d2-f89f0c40249c"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Bayesian networks can represent essentially any full joint probability distribution \nand in many cases can do so very concisely. 1) Each node corresponds to a random variable, \nwhich may be discrete or continuous. 2) Directed links or arrows connect pairs of nodes. \nIf there is an arrow from node X to node Y, X is said to be a parent of Y. 3) Each node\nXi has associated probability information theta(Xi|parents(Xi)) that quantifies the\neffect of the parents on the node using a finite number of parameters. Once the topology\nof the Bayes net is laid out, we need only specify the local probability information for\neach variable, in the form of a conditional distribution given its parents. The full joint\ndistribution for all the variables is defined by the topology and the local probability information.\nThe local probability information attached to each node takes the form of aconditional\nprobability table (CPT). (CPTs can be used only for discrete variables). Each row in a CPT \ncontains the conditional probability of each node value for a conditioning case.\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n The semantics of Bayes nets defines each entry in the joint distribution\nas follows: equation . the Bayesian network is a correct representation of the domain\nonly if each node is conditionally independent of its other predecessors in the node\nordering, given its parents. 1) Nodes: determine the set of variables that are required\nto model the domain, now order them. 2) Links: for i = 1 to n do: choose a minimal set of parents\nfor Xi from X1 to Xi-1, for each paretn insert a link from the parent to Xi, CPts: write\ndown the cpt P(Xi|parents(Xi)). a Bayes net can often be far more compact than the full \njoint distribution. This property is what makes it feasible to handle domains with many variables.\nIf we stick to a causal model, we end up having to specify fewer numbers, and the numbers will\noften be easier to come up with.\n\n13.2.1 Conditional independence relations in Bayesian networks \nEach variable is conditionally independent of its non-descendants, given its parents.\ninstead of defining the full joint distribution as the product of conditional distributions,\nthe network defines a set of conditional independence properties. The full joint\ndistribution can be derived from those properties. Is a set of nodes X is conditionally \nindependent of another set Y, given a third set Z. This can be determined efficiently by \nexamining the Bayes net to see whether Z d-separates X and Y. The process works as follows:\n1) Consider just the ancestral subgraph consisting of X, Y, Z, and their ancestors. 2) Add \nlinks between any unlinked pair of nodes that share a common child; now we have the \nso-called moral graph. 3)Replace all directed links by undirected links. 4) If Z blocks \nall paths between X and Y in the resulting graph, then Z d-separates X and Y. In \nthat case, X is conditionally independent of Y, given Z.\n\n13.2.2 Efficient Representation of Conditional Distributions \nlling in the CPT for a node requires up to O(2^k) numbers and perhaps a great deal\nof experience with all the possible conditioning cases. Usually, such relationships are\ndescribable by a canonical distribution that fits some standard pattern. In such cases, the\ncomplete table can be specified just by naming the pattern and perhaps supplying a few parameters.\nUncertain relationships can often be characterized by so-called noisy logical relationships. \n\n13.2.3 Bayesian nets with continuous variables\ndiscretization. define a continuous variable using one of the standard families of probability\ndensity functions. Nonparametric representation.\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n The basic task for any probabilistic inference system is to compute the\nposterior probability distribution for a set of query variables, given some observed\nevent—usually, some assignment of values to a set of evidence variables. \n\n13.3.1 Inference by enumeration\na query can be answered using a Bayes net by computing sums of products of conditional\nprobabilities from the network. Equation (handwritten) with complexity O(2^n) after nesting\nsummations\n\n``` \nFunction enumeration-ask(X,e,bn) returns a distrobution over X\n  inputs: X, the query\n          e, observed values for variables E\n          bn, a bayes net with variables vars\n  Q(X) <- a distrobution over x, initially empty\n  for each value xi of X do\n    Q(xi) <- enumerate-all(vars, exi)\n      where exi is ectended with X = xi\n  return normalize(Q(X))\n\nfunction ennumerate-ALL(vars, e) returns a real number\n  if empty?(vars) then return 1\n  V<-First(vars)\n  if V is evidence variable with value v in e\n    then return P(v|parents(V))x enumerate-all(REST(vars),e)\n    else return sum P(v|parents(V))x enumerate-all(REST(vars),ev)\n      where ev is extended with V = v\n``` \n\n13.3.2 The variable elimination algorithm\nThe final question is, of course, how to do inference in the network to make predictions. \nWe turn now to this question. For each inference method that we describe, we will evaluate \nthe method on the insurance net to measure the time and space requirements of the method.\n\n\n## Module 11 Lecture Notes\n Bayes’ Nets: Big Picture\n- Two problems with using full joint distribution tables\nas our probabilistic models:\n- Unless there are only a few variables, the joint is WAY too\nbig to represent explicitly\n- Hard to learn (estimate) anything empirically about more\nthan a few variables at a time\n- Bayes’ nets: a technique for describing complex joint\ndistributions (models) using simple, local\ndistributions (conditional probabilities)\n- More properly called graphical models\n- We describe how variables locally interact\n- Local interactions chain together to give global, indirect\ninteractions\n\nGraphical Model Notation\n- Nodes: variables (with domains)\n- Can be assigned (observed) or unassigned\n(unobserved)\n- Arcs: interactions\n- Indicate “direct influence” between variables\n- Formally: encode conditional independence\n(more later)\n- For now: imagine that arrows mean\ndirect causation (in general, they don’t!)\n\nBayes’ Net Semantics\n- A set of nodes, one per variable X\n- A directed, acyclic graph\n- A conditional distribution for each node\n- A collection of distributions over X, one for each\ncombination of parents’ values\n- CPT: conditional probability table\n- Description of a noisy “causal” process\n- Bayes’ nets implicitly encode joint distributions\n- As a product of local conditional distributions\n- To see what probability a BN gives to a full assignment, multiply all the\nrelevant conditionals together:\n\nCausality?\n- When Bayes’ nets reflect the true causal patterns:\n- Often simpler (nodes have fewer parents)\n- Often easier to think about\n- Often easier to elicit from experts\n- BNs need not actually be causal\n- Sometimes no causal net exists over the domain\n(especially if variables are missing)\n- E.g. consider the variables Traffic and Drips\n- End up with arrows that reflect correlation, not causation\n- What do the arrows really mean?\n- Topology may happen to encode causal structure\n- Topology really encodes conditional independence\n\nActive / Inactive Paths\n- Question: Are X and Y conditionally independent given\nevidence variables {Z}?\n- Yes, if X and Y “d-separated” by Z\n- Consider all (undirected) paths from X to Y\n- No active paths = independence!\n- A path is active if each triple is active:\n- Causal chain A ® B ® C where B is unobserved (either direction)\n- Common cause A ¬ B ® C where B is unobserved\n- Common effect (aka v-structure)\nA ® B ¬ C where B or one of its descendents is observed\n- All it takes to block a path is a single inactive segment\n\nBayes Nets Representation Summary\n- Bayes nets compactly encode joint distributions\n- Guaranteed independencies of distributions can be\ndeduced from BN graph structure\n- D-separation gives precise conditional independence\nguarantees from graph alone\n- A Bayes’ Net’s joint distribution may have further\n(conditional) independence that is not detectable until\nyou inspect its specific distribution \n\n## Module 11 Definitions\n* **Bayesian network** - directed graph in which each node is annotated with quantitative probability information.\n* **Conditioning case** - a possible combination of values for the parent nodes—a miniature possible world\n* **Syntax of a bayes net** - onsists of a directed acyclic graph with some local probability information attached to each node\n* **Semantics** - defines how the syntax corresponds to a joint distribution over the variables of the network.\n* **Topological order** - any order consistent with the directed graph structure.\n* **Locally structured** - each subcomponent interacts directly with only a bounded number of other components\n* **Sparse** - locally structured\n* **Deterministic node** -  has its value specified exactly by the values of its parents, with no uncertainty.\n* **Context specific independence** - a variable is conditionally independent of some of its parents given certain values of others.\n* **Noisy or model** - allows for uncertainty about the ability of each parent to cause the child to be true\n* **Discretization** - dividing up the possible values into a fixed set of intervals.\n* **Nonparametric representation** - define the conditional distribution implicitly with a collection of instances, each containing specific values of the parent and child variables\n* **Hybrid bayseian network** - A network with both discrete and continuous variables\n* **Linear-gaussian condition distrobution** - child has a Gaussian distribution whose mean mu varies linearly with the value of the parent and whose standard deviation sigma is fixed\n* **Probit model ** - probability unit\n* **Expit or inverse logit model** - uses the logistic function 1/(1+e^-x) to produce a soft threshold—it maps any  to a value between 0 and 1\n* **Variable elimination algorithm** - evaluating expressions such as Equation in right-to-left order. Intermediate results are stored, and summations that depend on the variable are done.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" Any component of an agent program can be improved by machine learning.\n",
        "The improvements, and the techniques used to make them, depend on these factors: 1) Which\n",
        "component is to be improved. 2) What prior knowledge the agent has, which influences the\n",
        "model it builds. 3) What data and feedback on that data is available. The components of these agents include:\n",
        "1) A direct mapping from conditions on the current state to actions. 2) A means to \n",
        "infer relevant properties of the world from the percept sequence. 3) Information about \n",
        "the way the world evolves and about the results of possible actions the agent can take.\n",
        "4) Utility information indicating the desirability of world states. 5) Action-value \n",
        "information indicating the desirability of actions. 6) Goals that describe the most \n",
        "desirable states. 7) A problem generator, critic, and learning element that enable\n",
        "the system to improve. we consider transfer learning, in which knowledge from one domain\n",
        "is transferred to a new domain, so that learning can proceed faster with less data.\n",
        "We do assume, however, that the designer of the system chooses a model framework that\n",
        "can lead to effective learning. This chapter concentrates on problems where the input\n",
        "is a factored representation—a vector of attribute values. It is also possible for the\n",
        "input to be any kind of data structure, including atomic and relational. We look at\n",
        "classification and regression problems. Three types of feedback that accompany the inputs:\n",
        "1) supervised learning 2) unsupervised learning 3) reinforcement learning.\n",
        "\"\"\"\n",
        "chapter19_2 = \"\"\" the task of supervised learning is this: Given a training set of N\n",
        "example input–output pairs (x1,y1), ...(xN,yN), where each pair was generated by an unknown\n",
        "function y=f(x), discover a function h that approximates the true function f. The function \n",
        "h is called a hypothesis about the world. It is drawn from a hypothesis space H of possible\n",
        "functions. or we can say a function drawn from a function class. We call the output \n",
        "yi the ground truth—the true answer we are asking our model to predict. With continuous-valued\n",
        "outputs we can’t expect an exact match to the ground truth; instead we look for a best-fit\n",
        "function for which each h(xi) is close to yi. The true measure of a hypothesis is not how it\n",
        "does on the training set, but rather how well it handles inputs it has not yet seen.\n",
        "Bias often results from restrictions imposed by the hypothesis space. Often there is\n",
        "a bias–variance tradeoff: a choice between more complex, low-bias hypotheses that fit\n",
        "the training data well and simpler, low-variance hypotheses that may generalize better. \n",
        "The problem is that there is a tradeoff between the expressiveness of a hypothesis space\n",
        "and the computational complexity of finding a good hypothesis within that space. \"\"\"\n",
        "chapter19_3 = \"\"\" Each internal node in the tree corresponds to a test of the value \n",
        "of one of the input attributes, the branches from the node are labeled with the possible\n",
        "values of the attribute, and the leaf nodes specify what value is to be returned by\n",
        "the function.\n",
        "\n",
        "19.3.1 Expressiveness of decision trees \n",
        "A Boolean decision tree is equivalent to a logical statement of the form: Output <-> (path1 V path2 V...)\n",
        "Any function in propostional logic and be expressed as a decision tree. There is no representation\n",
        "that is efficient for all kinds of functions.\n",
        "\n",
        "19.3.2 Learning decision trees from examples \n",
        "There are four cases to consider for these recursive subproblems: 1) If the remaining examples \n",
        "are all positive (or all negative), then we are done: we can answer Yes or No.\n",
        "2) If there are some positive and some negative examples, then choose the best attribute\n",
        "to split them. 3) If there are no examples left, it means that no example has been observed\n",
        "for this combination of attribute values, and we return the most common output value from \n",
        "the set. 4) If there are no attributes left, but both positive and negative examples, \n",
        "it means that these examples have exactly the same description, but different classifications. \n",
        "``` \n",
        "function learn-decision-tree(examples, attributes, parent_examples) returns a tree\n",
        "  if examples is empty then return plurality-value(parent_examples)\n",
        "  else if all examples have the same classification then return the classification\n",
        "  else if attributes is empty then reutn plurality-value(examples)\n",
        "  else\n",
        "    A<- argmax_aEattributes Importance(a, examples)\n",
        "    tree <- a new decision tree wiht root test A\n",
        "    for each value v of A do \n",
        "      exs <-{e: e E ecamples and eA = v}\n",
        "      subtree <- learn-decision-tree(exs, attributes - A, examples)\n",
        "      add a branch to tree wiht label (A=v)and subtree subtree\n",
        "    return tree\n",
        "``` \n",
        "We can evaluate the performance of a learning algorithm with a learning curve.\n",
        "\n",
        "19.3.3 Choosing attribute tests \n",
        "The decision tree learning algorithm chooses the attribute with the highest Importance.\n",
        "This is defined in terms of entropy.\n",
        "\n",
        "19.3.4 Generalization and overfitting \n",
        "Overfitting becomes more likely as the number of attributes grows, and less likely as \n",
        "we increase the number of training examples. Larger hypothesis spaces have more capacity\n",
        "both to fit and to overfit\n",
        "\n",
        "19.3.5 Broadening the applicability of decision trees \n",
        "Decision trees can be made more widely useful by handling the following complications:\n",
        "Missing data, continuous na dmultivalued input attributes, continuous-valued output \n",
        "attribute\n",
        "\"\"\"\n",
        "chapter19_4 = \"\"\" Our goal in machine learning is to select a hypothesis that will\n",
        "optimally fit future examples. To make that precise we need to define “future example”\n",
        "and “optimal fit.” the optimal fit is the hypothesis that minimizes the error rate.\n",
        "split the examples you have into two sets: a training set to create the hypothesis,\n",
        "and a test set to evaluate it. That means you need three data sets: 1) A training set \n",
        "to train candidate models. 2) A validation set, also known as a development set or dev\n",
        "set, to evaluate the candidate models and choose the best one. 3) A test set to do a \n",
        "final unbiased evaluation of the best model. K-fold cross-validation: plit the data into k\n",
        "equal subsets. We then perform k rounds of learning; on each round 1/k of the data \n",
        "are held out as a validation set and the remaining examples are used as the training set.\n",
        "task of finding a good hypothesis as two subtasks: model selection and optimization.\n",
        "\n",
        "19.4.1 Model selection \n",
        "``` \n",
        "function model-selection(learner, examples, k) returns a (hypothesis, error rate) pair\n",
        "  err <- an array, indexed by size, storing validation-set error rates\n",
        "  training_set, test_set <- a partition of examples into two sets\n",
        "  for size = 1 to inf do\n",
        "    err[size] <- cross-validation(learner, size, training_set, k)\n",
        "    if err is starting to increase significantly then\n",
        "      best_size <- the value of size with the minnimum err[size]\n",
        "      h <- Learner(best_size, training_set)\n",
        "      return h, error-rate(h, test_set)\n",
        "\n",
        "function cross-validation(Learner,size,examples,k) returns error rate\n",
        "  N<- the number of examples\n",
        "  errs <- 0\n",
        "  for i = 1 to k do \n",
        "    validation_set <- examples[(i-1)xN/k:ixN/k]\n",
        "    training_set <- examples - validation_set\n",
        "    h <- Learner(size, training_set)\n",
        "    errs <- errs + ERROR-RATE(h, validation_set)\n",
        "  return errs/k\n",
        "``` \n",
        "Model classes typically start to overfit as the capacity approaches the point of interpolation\n",
        "\n",
        "19.4.2 From error rates to loss \n",
        "in machine learning it is traditional to express this as a negative: to minimize a loss \n",
        "function rather than maximize a utility function. Best hypothesis is the one with \n",
        "the minimum expected generalization loss. Since P(x,Y) is not known in most cases, the\n",
        "learning agent can only estimate generalization loss with empirical loss. h^* may differ\n",
        "from the true function because of unrealizability, variance, noise, and computational complexity.\n",
        "\n",
        "19.4.4 Hyperparameter tuning\n",
        "Simplest is hand tuning: Repeat until you have satisfactory performance.\n",
        "If there are only a few hyperparameters a grid search is appropriate. try all combinations \n",
        "of values and see which performs best on the validation data. If there are too many combinations \n",
        "of possible values, then random search. Random sampling is also a good way to handle continuous values.\n",
        "Bayesian optimization. Population based training: PBT, as well as by random mutation, \n",
        "as in genetic algorithms\n",
        "\"\"\"\n",
        "chapter19_5 = \"\"\" computational learning theory, which lies at the intersection of AI, \n",
        "statistics, and theoretical computer science. The underlying principle is that any hypothesis \n",
        "that is seriously wrong will almost certainly be “found out” with high probability after a \n",
        "small number of examples, because it will make an incorrect prediction. Must be probably \n",
        "approximately correct.\n",
        "\n",
        "19.5.1 PAC learning example: Learning decision lists \n",
        "If a test succeeds when applied to an example description, the decision list specifies \n",
        "the value to be returned. If the test fails, processing continues with the next test in \n",
        "the list. Decision lists resemble decision trees, but their overall structure is simpler: \n",
        "they branch only in one direction. \n",
        "\n",
        "``` \n",
        "function Decision-list-learning(examples) returns a decision list or failure\n",
        "  if examples is empty then return the trivial decision list nO\n",
        "  t <- a test that matches a nonempty subset examples_t are all positive or negative\n",
        "      such that all the members of examples_t are all positive or all negative\n",
        "  if there is no such t then return a failure\n",
        "  if the examples in examples_t are positive then o<- YES else o<- NO\n",
        "  return a decision list with initiial test t and outcome o adn remaining tests given by \n",
        "    Decision-List_learning(examples-examples_t)\n",
        "``` \n",
        "\"\"\"\n",
        "chapter19_6 = \"\"\" 19.6.1 Univariate linear regression \n",
        "h_w(x) = w1x +w0 use w bc coefficients are weights. The loss function is convex \n",
        "this is true for every linear regression problem with an loss function, and implies \n",
        "that there are no local minima. \n",
        "\n",
        "19.6.2 Gradient descent \n",
        "choose any starting point in weight space—here, a point in the (\n",
        " \n",
        ") plane—and then compute an estimate of the gradient and move a small amount in the \n",
        "steepest downhill direction, repeating until we converge on a point in weight space with \n",
        "(local) minimum loss. \n",
        "``` \n",
        "w <- any point in the parameter space\n",
        "while not converged do\n",
        "  for each wi in w do\n",
        "    w<- wi - alpha partial/partial wi Loss(w)\n",
        "``` \n",
        "SGD is widely applied to models other than linear regression, in particular neural networks. \n",
        "Even when the loss surface is not convex, the approach has proven effective in finding \n",
        "good local minima that are close to the global minimum.\n",
        "\n",
        "19.6.3 Multivariable linear regression \n",
        "it is common to use regularization on multivariable linear functions to avoid overfitting. \n",
        "Recall that with regularization we minimize the total cost of a hypothesis, counting \n",
        "both the empirical loss and the complexity of the hypothesis:\n",
        "\n",
        "19.6.4 Linear classifiers with a hard threshold \n",
        " we can think of h as the result of passing the linear function w*x through a threshold function:\n",
        " \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\" multinomial naive Bayes classifier, so called because it is a Bayesian \n",
        "classifier that makes a simplifying (naive) assumption about how the features interact.\n",
        "We represent a text document bag-of-words as if it were a bag-of-words, that is, an \n",
        "unordered set of words with their position ignored, keeping only their frequency in the document\n",
        "cˆ = argmax_c∈C P(c|d) break down any conditional probability P(x|y) into\n",
        "three other probabilities: P(x|y) = P(y|x)P(x)/P(y) then cˆ = argmax_c∈C P(c|d) = argmax_c∈C P(d|c)P(c)/P(d) \n",
        "To return to classification: we compute the most probable class ˆc given some\n",
        "document d by choosing the class which has the highest product of two probabilities:\n",
        "the prior probability of the class P(c) and the likelihood of the document P(d|c):\n",
        "Classifiers that use a linear combination of the inputs\n",
        "to make a classification decision —like naive Bayes and also logistic regression—\n",
        "are called linear classifiers.\n",
        "\"\"\"\n",
        "slp_chapter4_2 = \"\"\"How can we learn the probabilities P(c) and P(fi|c)?  Pˆ(c) = Nc/Ndoc\n",
        "Pˆ(wi|c) = count(wi, c)/ summation w∈V count(w, c). \n",
        "What do we do about words that occur in our test data but are not in our vocabulary at all because they did not occur in any training document in any class? The\n",
        "unknown word solution for such unknown words is to ignore them—remove them from the test\n",
        "document and not include any probability for them at all. \n",
        "Finally, some systems choose to completely ignore another class of words: stop\n",
        "stop words words, very frequent words like the and a. This can be done by sorting the vocabulary by frequency in the training set, and defining the top 10–100 vocabulary entries\n",
        "as stop words, or alternatively by using one of the many predefined stop word lists\n",
        "available online.\n",
        "\n",
        "``` \n",
        "function TRAIN NAIVE BAYES(D, C) returns log P(c) and log P(w|c)\n",
        "for each class c ∈ C # Calculate P(c) terms\n",
        "  Ndoc = number of documents in D\n",
        "  Nc = number of documents from D in class c\n",
        "  logprior[c]← log Nc/Ndoc\n",
        "  V←vocabulary of D\n",
        "  bigdoc[c]←append(d) for d ∈ D with class c\n",
        "  for each word w in V # Calculate P(w|c) terms\n",
        "    count(w,c)←# of occurrences of w in bigdoc[c]\n",
        "    loglikelihood[w,c]← log (count(w, c) + 1)/summ_w' in V(count (w0, c) + 1)\n",
        "return logprior, loglikelihood, V\n",
        "\n",
        "function TEST NAIVE BAYES(testdoc, logprior, loglikelihood, C, V) returns best c\n",
        "for each class c ∈ C\n",
        "  sum[c]← logprior[c]\n",
        "  for each position i in testdoc\n",
        "    word←testdoc[i]\n",
        "    if word ∈ V\n",
        "      sum[c]←sum[c]+ loglikelihood[word,c]\n",
        "  return argmaxc sum[c]\n",
        "``` \n",
        "\n",
        " \"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\" for sentiment classification and a number of other text classification tasks,\n",
        "whether a word occurs or not seems to matter more than its frequency. Thus it\n",
        "often improves performance to clip the word counts in each document at 1. Binary\n",
        "multinomial naive Bayes or binary NB. The variant uses the same Eq. 4.10 except\n",
        "that for each document we remove all duplicate words before concatenating them\n",
        "into the single big document. A second important addition commonly made when doing text classification for\n",
        "sentiment is to deal with negation. Consider the difference between I really like this\n",
        "movie (positive) and I didn’t like this movie (negative). The negation expressed by\n",
        "didn’t completely alters the inferences we draw from the predicate like. Similarly,\n",
        "negation can modify a negative word to produce a positive review (don’t dismiss this\n",
        "film, doesn’t let us get bored). Finally, in some situations we might have insufficient labeled training data to\n",
        "train accurate naive Bayes classifiers using all words in the training set to estimate\n",
        "positive and negative sentiment. In such cases we can instead derive the positive \n",
        "and negative word features from sentiment lexicons, lists of words that are pre- sentiment\n",
        "lexicons annotated with positive or negative sentiment\n",
        "\"\"\"\n",
        "slp_chapter4_5 = \"\"\" A common solution here, rather than using all the words as individual features,\n",
        "is to predefine likely sets of words or phrases as features, combined with features\n",
        "that are not purely linguistic. For example the open-source SpamAssassin tool1\n",
        "predefines features like the phrase “one hundred percent guaranteed”, or the feature\n",
        "mentions millions of dollars, which is a regular expression that matches suspiciously\n",
        "large sums of money. But it also includes features like HTML has a low ratio of text\n",
        "to image area, that aren’t purely linguistic and might require some sophisticated\n",
        "computation, or totally non-linguistic features about, say, the path that the email\n",
        "took to arrive. For other tasks, like language id—determining what language a given piece\n",
        "of text is written in—the most effective naive Bayes features are not words at all,\n",
        "but character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,\n",
        "‘thei’), or, even simpler byte n-grams, where instead of using the multibyte Unicode\n",
        "character representations called codepoints, we just pretend everything is a string of\n",
        "raw bytes. Because spaces count as a byte, byte n-grams can model statistics about\n",
        "the beginning or ending of words \"\"\"\n",
        "slp_chapter4_6 = \"\"\" Bayes classifiers can use any sort of feature: dictionaries, URLs, email addresses, network features, phrases, and so on. But\n",
        "if, as in the previous section, we use only individual word features, and we use all\n",
        "of the words in the text (not a subset), then naive Bayes has an important similarity to language modeling. Specifically, a naive Bayes model can be viewed as a\n",
        "set of class-specific unigram language models, in which the model for each class\n",
        "instantiates a unigram language model.\n",
        "Since the likelihood features from the naive Bayes model assign a probability to\n",
        "each word P(word|c), the model also assigns a probability to each sentence:\n",
        "P(s|c) = PI _ i∈positions P(wi|c)\"\"\"\n",
        "slp_chapter4_7 = \"\"\"In both cases, we need a metric for knowing how well our spam detector (or\n",
        "pie-tweet-detector) is doing. To evaluate any system for detecting things, we start\n",
        "by building a confusion matrix. A confusion matrix confusion\n",
        "is a table for visualizing how an algorithm performs with respect to the human gold\n",
        "labels, using two dimensions (system output and gold labels), and each cell labeling\n",
        "a set of possible outcomes. precision and recall. Precision measures the percentage of the items that\n",
        "the system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\n",
        "are positive according to the human gold labels). Precision is defined as\n",
        "Precision = true positives/true positives + false positives\n",
        "Recall measures the percentage of items actually present in the input that were\n",
        "correctly identified by the system. Recall is defined as\n",
        "Recall = true positives/true positives + false negatives. There are many ways to define a single metric that incorporates aspects of both\n",
        "F-measure precision and recall. The simplest of these combinations is the F-measure, defined as:\n",
        "Fβ = (β2 +1)PR/β^2P+R\n",
        "The β parameter differentially weights the importance of recall and precision,\n",
        "based perhaps on the needs of an application. Values of β > 1 favor recall, while\n",
        "values of β < 1 favor precision. When β = 1, precision and recall are equally balanced.\n",
        "In macroaveraging, we compute the performance microaveraging for each class, and then \n",
        "average over classes. In microaveraging, we collect the decisions for all classes into \n",
        "a single confusion matrix, and then compute precision and recall from that table.\n",
        " \"\"\"\n",
        "slp_chapter4_8 = \"\"\" we use the training set to train the model, then use\n",
        "the development test set (also called a devset) to perhaps tune some parameters,and \n",
        "in general decide what the best model is. In cross-validation, we choose a number k, and partition our data into k disjoint\n",
        "subsets called folds. Now we choose one of those k folds as a test set, train our\n",
        "classifier on the remaining k − 1 folds, and then compute the error rate on the test\n",
        "set. Then we repeat with another fold as the test set, again training on the other k−1\n",
        "folds. We do this sampling process k times and average the test set error rate from\n",
        "these k runs to get an average error rate.\n",
        " \"\"\"\n",
        "slp_chapter4_9 = \"\"\" We would like to know if δ(x) > 0, meaning that our logistic regression classifier\n",
        "has a higher F1 than our naive Bayes classifier on X. δ(x) is called the effect size;\n",
        "a bigger δ means that A seems to be way better than B; a small δ means A seems to\n",
        "be only a little better. The hypothesis H0, called the null hypothesis, supposes that δ(x) is actually negative or zero, meaning that A is not better than B. We would like to know if we can\n",
        "confidently rule out this hypothesis, and instead support H1, that A is better. We do this by creating a random variable X ranging over all test sets. Now we\n",
        "ask how likely is it, if the null hypothesis H0 was correct, that among these test sets\n",
        "we would encounter the value of δ(x) that we found. We formalize this likelihood\n",
        "as the p-value: the probability, assuming the null hypothesis H0 is true, of seeing\n",
        "the δ(x) that we saw or one even greater P(δ(X) ≥ δ(x)|H0 is true).We say that a result \n",
        "is statistically significant if the δ we saw has a probability that is below the \n",
        "threshold and we therefore reject this null hypothesis. There are two common non-parametric \n",
        "tests used in NLP: approximate randomization and the bootstrap test, showing the paired \n",
        "version of the test, which again is most common in NLP. Paired tests are those in which we compare two sets of observations that are aligned:\n",
        "each observation in one set can be paired with an observation in another. This happens naturally when we are comparing the performance of two systems on the same\n",
        "test set; we can pair the performance of system A on an individual observation xi\n",
        "with the performance of system B on the same xi.\n",
        "\n",
        "``` \n",
        "function BOOTSTRAP(test set x, num of samples b) returns p-value(x)\n",
        "Calculate δ(x) # how much better does algorithm A do than B on x\n",
        "s = 0\n",
        "for i = 1 to b do\n",
        "for j = 1 to n do # Draw a bootstrap sample x^(i) of size n\n",
        "    Select a member of x at random and add it to x^(i)\n",
        "  Calculate δ(x^(i)) # how much better does algorithm A do than B on x^(i)\n",
        "  s←s + 1 if δ(x^(i)) ≥ 2δ(x)\n",
        "p-value(x) ≈ s/b\n",
        "# on what % of the b samples did algorithm A beat expectations?\n",
        "return p-value(x) # if very few did, our observed δ is probably not accidental\n",
        "``` \n",
        "\"\"\"\n",
        "slp_chapter4_10 = \"\"\" One class of harms is representational harms, \n",
        "harms caused by a system that demeans a social group, for example by perpetuating negative \n",
        "stereotypes about them. In other tasks classifiers may lead to both representational harms and other\n",
        "harms, such as censorship. For example the important text classification task of\n",
        "toxicity detection is the task of detecting hate speech, abuse, harassment, or other toxicity\n",
        "detection\n",
        "kinds of toxic language. While the goal of such classifiers is to help reduce societal harm,\n",
        "toxicity classifiers can themselves cause harms.   \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\" Model-Based Classification\n",
        "Model-based approach\n",
        "• Build a model (e.g. Bayes’ net) where both the label and\n",
        "features are random variables\n",
        "• Instantiate any observed features\n",
        "• Query for the distribution of the\n",
        "label conditioned on the features\n",
        "\n",
        "What do we need in order to use Naïve Bayes?\n",
        "• Inference method (we just saw this part)\n",
        "• Start with a bunch of probabilities: P(Y) and the P(Fi\n",
        "|Y) tables\n",
        "• Use standard inference to compute P(Y|F1…Fn)\n",
        "• Nothing new here\n",
        "• Estimates of local conditional probability tables\n",
        "• P(Y), the prior over labels\n",
        "• P(Fi\n",
        "|Y) for each feature (evidence variable)\n",
        "• These probabilities are collectively called the parameters of the\n",
        "model and denoted by q\n",
        "• Up until now, we assumed these appeared by magic, but…\n",
        "• …they typically come from training data counts: we’ll look at this\n",
        "soon\n",
        "\n",
        "Bag-of-words Naïve Bayes:\n",
        "• Features: Wi is the word at positon i\n",
        "• As before: predict label conditioned on feature variables (spam vs. ham)\n",
        "• As before: assume features are conditionally independent given label\n",
        "• New: each Wi is identically distributed\n",
        "Generative model:\n",
        "“Tied” distributions and bag-of-words\n",
        "• Usually, each variable gets its own conditional probability distribution P(F|Y)\n",
        "• In a bag-of-words model\n",
        "• Each position is identically distributed\n",
        "• All positions share the same conditional probs P(W|Y)\n",
        "• Why make this assumption?\n",
        "• Called “bag-of-words” because model is insensitive to word order or reordering\n",
        "\n",
        "Relative frequency parameters will overfit the training data!\n",
        "• Just because we never saw a 3 with pixel (15,15) on during training doesn’t mean we won’t\n",
        "see it at test time\n",
        "• Unlikely that every occurrence of “minute” is 100% spam\n",
        "• Unlikely that every occurrence of “seriously” is 100% ham\n",
        "• What about all the words that don’t occur in the training set at all?\n",
        "• In general, we can’t go around giving unseen events zero probability\n",
        "As an extreme case, imagine using the entire email as the only feature\n",
        "• Would get the training data perfect (if deterministic labeling)\n",
        "• Wouldn’t generalize at all\n",
        "• Just making the bag-of-words assumption gives us some generalization, but isn’t enough\n",
        "To generalize better: we need to smooth or regularize the estimates\n",
        "\n",
        "What to Do About Errors?\n",
        "Need more features– words aren’t enough!\n",
        "• Have you emailed the sender before?\n",
        "• Have 1K other people just gotten the same email?\n",
        "• Is the sending information consistent?\n",
        "• Is the email in ALL CAPS?\n",
        "• Do inline URLs point where they say they point?\n",
        "• Does the email address you by (your) name?\n",
        "Can add these information sources as new\n",
        "variables in the NB model\n",
        "Next class we’ll talk about classifiers which let\n",
        "you easily add arbitrary features more easily\n",
        "\n",
        "First step: get a baseline\n",
        "• Baselines are very simple “straw man” procedures\n",
        "• Help determine how hard the task is\n",
        "• Help know what a “good” accuracy is\n",
        "Weak baseline: most frequent label classifier\n",
        "• Gives all test instances whatever label was most common in the training set\n",
        "• E.g. for spam filtering, might label everything as ham\n",
        "• Accuracy might be very high if the problem is skewed\n",
        "• E.g. calling everything “ham” gets 66%, so a classifier that gets 70% isn’t very good…\n",
        "For real research, usually use previous work as a (strong) baseline\n",
        "\n",
        "The confidence of a probabilistic classifier:\n",
        "• Posterior over the top label\n",
        "• Represents how sure the classifier is of\n",
        "the classification\n",
        "• Any probabilistic model will have\n",
        "confidences\n",
        "• No guarantee confidence is correct\n",
        "Calibration\n",
        "• Weak calibration: higher confidences\n",
        "mean higher accuracy\n",
        "• Strong calibration: confidence predicts\n",
        "accuracy rate\n",
        "• What’s the value of calibration?\n",
        "\n",
        "Summary\n",
        "Bayes rule lets us do diagnostic queries with causal probabilities\n",
        "The naïve Bayes assumption takes all features to be independent given\n",
        "the class label\n",
        "We can build classifiers out of a naïve Bayes model using training data\n",
        "Smoothing estimates is important in real systems\n",
        "Classifier confidences are useful, when you can get them\n",
        "\n",
        "Problems with the Perceptron\n",
        "Noise: if the data isn’t separable,\n",
        "weights might thrash\n",
        "• Averaging weight vectors over\n",
        "time can help (averaged\n",
        "perceptron)\n",
        "Mediocre generalization: finds a\n",
        "“barely” separating solution\n",
        "Overtraining: test / held-out\n",
        "accuracy usually rises, then falls\n",
        "• Overtraining is a kind of\n",
        "overfitting\n",
        "\n",
        "Fixing the Perceptron\n",
        "Idea: adjust the weight update to mitigate these effects\n",
        "MIRA*: choose an update size that fixes the current\n",
        "mistake…\n",
        "… but, minimizes the change to w\n",
        "\n",
        "In practice, it’s also bad to make updates that are too large\n",
        "• Example may be labeled incorrectly\n",
        "• You may not have enough features\n",
        "• Solution: cap the maximum possible value of t with some\n",
        "constant C\n",
        "• Corresponds to an optimization that assumes nonseparable data\n",
        "• Usually converges faster than perceptron\n",
        "• Usually better, especially on noisy data\n",
        "\n",
        "Classification: Comparison\n",
        "Naïve Bayes\n",
        "• Builds a model training data\n",
        "• Gives prediction probabilities\n",
        "• Strong assumptions about feature independence\n",
        "• One pass through data (counting)\n",
        "Perceptrons / MIRA:\n",
        "• Makes less assumptions about data\n",
        "• Mistake-driven learning\n",
        "• Multiple passes through data (prediction)\n",
        "• Often more accurate\n",
        "\n",
        "Perceptions can be used for all kinds of classification problems.\n",
        "Think of the inputs as features representing something we want to classify.\n",
        "The feature values for inputs are fixed, but we can choose different weight vectors.\n",
        "Depending on the weight vector that we pick, we will get a different classifier. \n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"induction\": \"Going from a specific set of observations to a general rule\",  \n",
        "\"deductive conclusions\": \"guaranteed to be correct if the premises are correct\",\n",
        "\"factored representation\": \"vector of attribute values\",\n",
        "\"classification\": \"when the output of the learning problem is a finite set of values\",\n",
        "\"regression\": \"when the output of the learning problem is a number\",\n",
        "\"supervised learning\": \"the agent observes input-output pairs and learns a function that maps from input to output\",\n",
        "\"unsupervised learning\": \"the agent learns patterns in the input without any explicit feedback\",\n",
        "\"clustering\": \"detecting potentially useful clusters of input examples\",\n",
        "\"reinforcement learning\": \"the agent learns from a series of reinforcements: rewards and punishments\",\n",
        "\"exploratory data analysis\": \"examining the data with statistical tests and visualizations—histograms, scatter plots, box plots—get some insight into what hypothesis space might be appropriate\",\n",
        "\"consistent hypothesis\": \"h such that each xi in the training set has h(xi) = yi\",\n",
        "\"test set\": \"second sample of (xi, yi) pairs\",\n",
        "\"generalizes well\": \"if it accurately predicts the outpusts of the test set\",\n",
        "\"bias\": \"the tendency of a predictive hypothesis to deviate from the expected value when averaged over different training sets\",\n",
        "\"underfitting\": \"when it fails to find a pattern in the data.\",\n",
        "\"variance\": \"the amount of change in the hypothesis due to fluctuation in the training data\",\n",
        "\"decistion tree\": \"representation of a function that maps a vector of attribute values to a single output value—a “decision.”\",\n",
        "\"boolean classification\": \"consider only inputs consisting of discrete values and outputs that are either true or false \",\n",
        "\"happy graphs\": \"learning curves\",\n",
        "\"entropy\": \" measure of the uncertainty of a random variable\",\n",
        "\"information gain\": \"from the attribute test on A is the expected reduction in entropy:\",\n",
        "\"decision tree pruning\": \"combats overfitting by eliminating nodes that are not clearly relevant.\",\n",
        "\"statistical significance test\": \"how large a gain should we require to split on an attribute?\",\n",
        "\"early stopping\": \"have the decision tree algorithm stop generating nodes when there is no good attribute to split on\",\n",
        "\"missing data\": \"in many domains not all the attribute values will be known for every example\",\n",
        "\"continuous and multivalued input attributes\": \"may be that every example has a different attribute value. The information gain measure would give its highest score to such an attribute\",\n",
        "\"split point test\": \"an inequality test on the value of an attribute\",\n",
        "\"regression tree\": \"has at each leaf a linear function of some subset of numerical attributes, rather than a single output value\",\n",
        "\"decision trees are unstable\": \"adding just one new example can change the test at the root\",\n",
        "\"stationary assumption\": \"\",\n",
        "\"independent and identically distributed or i.i.d\": \"each example has the same prior probabilityt distrobution adn is independent of the previous examples\",\n",
        "\"error rate\": \"the proportion of times that h(x) != y for an (x,y) example\",\n",
        "\"hyper parameters\": \"try different thresholds for x^2 pruning of decision trees, or different degrees for polynomials\",\n",
        "\"leave-one-out cross-validation or LOOCV\": \"k-fold cross-validation when k=n\",\n",
        "\"model selection\": \"choose a good hypothesis space\",\n",
        "\"optimization (training)\": \"finds the best hypothesis for the space\",\n",
        "\"interpolated\": \"a model that exactly fits all the training data\",\n",
        "\"realizable\": \"hypothesis space H actually contains the true function f\",\n",
        "\"varriance\": \"a learning algorithm will in general return different hypotheses for different sets of examples\",\n",
        "\"computationally intractable\": \"a search can explore part of the space and return a reasonably good hypothesis, but can’t always guarantee the best one\",\n",
        "\"small scale learning\": \"the number of training examples ranged from dozens to the low thousands\",\n",
        "\"large scale learning\": \"millions of examples\",\n",
        "\"regularization\": \"explicitly penalizing complex hypotheses is called\",\n",
        "\"regularization function\": \"the complexity function\",\n",
        "\"frature selection\": \"discard attributes that appear to be irrelevant\",\n",
        "\"minimum description length or MDL hypothesis\": \"minimizes the total number of bits required\",\n",
        "\"hand tuning\": \"guess based on past experience, train model,measure performance on validation data, analyze results, and use intuition to suggest new parameter values\",\n",
        "\"random search\": \"samples uniformly from the set of all possible hyperparameter settings, repeating for as long as you are willing to spend the time and computational resources\",\n",
        "\"bayesian optimization\": \"treats the task of choosing good hyperparameter values as a machine learning problem in itself\",\n",
        "\"PBT\": \"trains in 2 waves, 2nd generation can choose based on the successful values from the previous generation.\",\n",
        "\"computational learning theory\": \"which lies at the intersection of AI, statistics, and theoretical computer science.\",\n",
        "\"pac learning algorithm\": \"Any learning algorithm that returns hypotheses that are probably approximately correct\",\n",
        "\"approximately correct\": \"error(h) < epsilon which is a small constatne\",\n",
        "\"sample complexity\": \"number of required examples as a function of epsilon and delta\",\n",
        "\"decision lists\": \"onsists of a series of tests, each of which is a conjunction of literals\",\n",
        "\"linear regression\": \"task of finding the hw that best fits the data\",\n",
        "\"weight space\": \"the space defined by all possible settings of the weights\",\n",
        "\"epoch\": \"a step that covers all the training examples\",\n",
        "\"stochastic gradient descent or SGD\": \"randomly selects a small number of training examples at each step, and updates according to\",\n",
        "\"decistion boundary\": \"is a line (or a surface, in higher dimensions) that separates the two classes.\",\n",
        "\"linear separator\": \"linear decision boundary\",\n",
        "\"training curve\": \"measures the classifier performance on a fixed training set as the learning process proceeds one update at a time on that training set.\"\n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "539a7774-7ed8-4337-8537-fd7c85fe3d86"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "(\"AIMA Chapter 19.1 Forms of Learning\", chapter19_1),\n",
        "(\"AIMA Chapter 19.2 Supervised Learning\", chapter19_2),\n",
        "(\"AIMA Chapter 19.3 Learning Decision Trees\", chapter19_3),\n",
        "(\"AIMA Chapter 19.4 Model Selection and Optimization\", chapter19_4),\n",
        "(\"AIMA Chapter 19.5 The Theory of Learning\", chapter19_5),\n",
        "(\"AIMA Chapter 19.6 Linear Regression and Classification\", chapter19_6),\n",
        "(\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "(\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "(\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "(\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "(\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "(\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "(\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "(\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "(\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "(\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "(\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\",\n",
        "module12_sections,\n",
        "module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 19.1 Forms of Learning\n Any component of an agent program can be improved by machine learning.\nThe improvements, and the techniques used to make them, depend on these factors: 1) Which\ncomponent is to be improved. 2) What prior knowledge the agent has, which influences the\nmodel it builds. 3) What data and feedback on that data is available. The components of these agents include:\n1) A direct mapping from conditions on the current state to actions. 2) A means to \ninfer relevant properties of the world from the percept sequence. 3) Information about \nthe way the world evolves and about the results of possible actions the agent can take.\n4) Utility information indicating the desirability of world states. 5) Action-value \ninformation indicating the desirability of actions. 6) Goals that describe the most \ndesirable states. 7) A problem generator, critic, and learning element that enable\nthe system to improve. we consider transfer learning, in which knowledge from one domain\nis transferred to a new domain, so that learning can proceed faster with less data.\nWe do assume, however, that the designer of the system chooses a model framework that\ncan lead to effective learning. This chapter concentrates on problems where the input\nis a factored representation—a vector of attribute values. It is also possible for the\ninput to be any kind of data structure, including atomic and relational. We look at\nclassification and regression problems. Three types of feedback that accompany the inputs:\n1) supervised learning 2) unsupervised learning 3) reinforcement learning.\n\n\n## AIMA Chapter 19.2 Supervised Learning\n the task of supervised learning is this: Given a training set of N\nexample input–output pairs (x1,y1), ...(xN,yN), where each pair was generated by an unknown\nfunction y=f(x), discover a function h that approximates the true function f. The function \nh is called a hypothesis about the world. It is drawn from a hypothesis space H of possible\nfunctions. or we can say a function drawn from a function class. We call the output \nyi the ground truth—the true answer we are asking our model to predict. With continuous-valued\noutputs we can’t expect an exact match to the ground truth; instead we look for a best-fit\nfunction for which each h(xi) is close to yi. The true measure of a hypothesis is not how it\ndoes on the training set, but rather how well it handles inputs it has not yet seen.\nBias often results from restrictions imposed by the hypothesis space. Often there is\na bias–variance tradeoff: a choice between more complex, low-bias hypotheses that fit\nthe training data well and simpler, low-variance hypotheses that may generalize better. \nThe problem is that there is a tradeoff between the expressiveness of a hypothesis space\nand the computational complexity of finding a good hypothesis within that space. \n\n## AIMA Chapter 19.3 Learning Decision Trees\n Each internal node in the tree corresponds to a test of the value \nof one of the input attributes, the branches from the node are labeled with the possible\nvalues of the attribute, and the leaf nodes specify what value is to be returned by\nthe function.\n\n19.3.1 Expressiveness of decision trees \nA Boolean decision tree is equivalent to a logical statement of the form: Output <-> (path1 V path2 V...)\nAny function in propostional logic and be expressed as a decision tree. There is no representation\nthat is efficient for all kinds of functions.\n\n19.3.2 Learning decision trees from examples \nThere are four cases to consider for these recursive subproblems: 1) If the remaining examples \nare all positive (or all negative), then we are done: we can answer Yes or No.\n2) If there are some positive and some negative examples, then choose the best attribute\nto split them. 3) If there are no examples left, it means that no example has been observed\nfor this combination of attribute values, and we return the most common output value from \nthe set. 4) If there are no attributes left, but both positive and negative examples, \nit means that these examples have exactly the same description, but different classifications. \n``` \nfunction learn-decision-tree(examples, attributes, parent_examples) returns a tree\n  if examples is empty then return plurality-value(parent_examples)\n  else if all examples have the same classification then return the classification\n  else if attributes is empty then reutn plurality-value(examples)\n  else\n    A<- argmax_aEattributes Importance(a, examples)\n    tree <- a new decision tree wiht root test A\n    for each value v of A do \n      exs <-{e: e E ecamples and eA = v}\n      subtree <- learn-decision-tree(exs, attributes - A, examples)\n      add a branch to tree wiht label (A=v)and subtree subtree\n    return tree\n``` \nWe can evaluate the performance of a learning algorithm with a learning curve.\n\n19.3.3 Choosing attribute tests \nThe decision tree learning algorithm chooses the attribute with the highest Importance.\nThis is defined in terms of entropy.\n\n19.3.4 Generalization and overfitting \nOverfitting becomes more likely as the number of attributes grows, and less likely as \nwe increase the number of training examples. Larger hypothesis spaces have more capacity\nboth to fit and to overfit\n\n19.3.5 Broadening the applicability of decision trees \nDecision trees can be made more widely useful by handling the following complications:\nMissing data, continuous na dmultivalued input attributes, continuous-valued output \nattribute\n\n\n## AIMA Chapter 19.4 Model Selection and Optimization\n Our goal in machine learning is to select a hypothesis that will\noptimally fit future examples. To make that precise we need to define “future example”\nand “optimal fit.” the optimal fit is the hypothesis that minimizes the error rate.\nsplit the examples you have into two sets: a training set to create the hypothesis,\nand a test set to evaluate it. That means you need three data sets: 1) A training set \nto train candidate models. 2) A validation set, also known as a development set or dev\nset, to evaluate the candidate models and choose the best one. 3) A test set to do a \nfinal unbiased evaluation of the best model. K-fold cross-validation: plit the data into k\nequal subsets. We then perform k rounds of learning; on each round 1/k of the data \nare held out as a validation set and the remaining examples are used as the training set.\ntask of finding a good hypothesis as two subtasks: model selection and optimization.\n\n19.4.1 Model selection \n``` \nfunction model-selection(learner, examples, k) returns a (hypothesis, error rate) pair\n  err <- an array, indexed by size, storing validation-set error rates\n  training_set, test_set <- a partition of examples into two sets\n  for size = 1 to inf do\n    err[size] <- cross-validation(learner, size, training_set, k)\n    if err is starting to increase significantly then\n      best_size <- the value of size with the minnimum err[size]\n      h <- Learner(best_size, training_set)\n      return h, error-rate(h, test_set)\n\nfunction cross-validation(Learner,size,examples,k) returns error rate\n  N<- the number of examples\n  errs <- 0\n  for i = 1 to k do \n    validation_set <- examples[(i-1)xN/k:ixN/k]\n    training_set <- examples - validation_set\n    h <- Learner(size, training_set)\n    errs <- errs + ERROR-RATE(h, validation_set)\n  return errs/k\n``` \nModel classes typically start to overfit as the capacity approaches the point of interpolation\n\n19.4.2 From error rates to loss \nin machine learning it is traditional to express this as a negative: to minimize a loss \nfunction rather than maximize a utility function. Best hypothesis is the one with \nthe minimum expected generalization loss. Since P(x,Y) is not known in most cases, the\nlearning agent can only estimate generalization loss with empirical loss. h^* may differ\nfrom the true function because of unrealizability, variance, noise, and computational complexity.\n\n19.4.4 Hyperparameter tuning\nSimplest is hand tuning: Repeat until you have satisfactory performance.\nIf there are only a few hyperparameters a grid search is appropriate. try all combinations \nof values and see which performs best on the validation data. If there are too many combinations \nof possible values, then random search. Random sampling is also a good way to handle continuous values.\nBayesian optimization. Population based training: PBT, as well as by random mutation, \nas in genetic algorithms\n\n\n## AIMA Chapter 19.5 The Theory of Learning\n computational learning theory, which lies at the intersection of AI, \nstatistics, and theoretical computer science. The underlying principle is that any hypothesis \nthat is seriously wrong will almost certainly be “found out” with high probability after a \nsmall number of examples, because it will make an incorrect prediction. Must be probably \napproximately correct.\n\n19.5.1 PAC learning example: Learning decision lists \nIf a test succeeds when applied to an example description, the decision list specifies \nthe value to be returned. If the test fails, processing continues with the next test in \nthe list. Decision lists resemble decision trees, but their overall structure is simpler: \nthey branch only in one direction. \n\n``` \nfunction Decision-list-learning(examples) returns a decision list or failure\n  if examples is empty then return the trivial decision list nO\n  t <- a test that matches a nonempty subset examples_t are all positive or negative\n      such that all the members of examples_t are all positive or all negative\n  if there is no such t then return a failure\n  if the examples in examples_t are positive then o<- YES else o<- NO\n  return a decision list with initiial test t and outcome o adn remaining tests given by \n    Decision-List_learning(examples-examples_t)\n``` \n\n\n## AIMA Chapter 19.6 Linear Regression and Classification\n 19.6.1 Univariate linear regression \nh_w(x) = w1x +w0 use w bc coefficients are weights. The loss function is convex \nthis is true for every linear regression problem with an loss function, and implies \nthat there are no local minima. \n\n19.6.2 Gradient descent \nchoose any starting point in weight space—here, a point in the (\n \n) plane—and then compute an estimate of the gradient and move a small amount in the \nsteepest downhill direction, repeating until we converge on a point in weight space with \n(local) minimum loss. \n``` \nw <- any point in the parameter space\nwhile not converged do\n  for each wi in w do\n    w<- wi - alpha partial/partial wi Loss(w)\n``` \nSGD is widely applied to models other than linear regression, in particular neural networks. \nEven when the loss surface is not convex, the approach has proven effective in finding \ngood local minima that are close to the global minimum.\n\n19.6.3 Multivariable linear regression \nit is common to use regularization on multivariable linear functions to avoid overfitting. \nRecall that with regularization we minimize the total cost of a hypothesis, counting \nboth the empirical loss and the complexity of the hypothesis:\n\n19.6.4 Linear classifiers with a hard threshold \n we can think of h as the result of passing the linear function w*x through a threshold function:\n \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n multinomial naive Bayes classifier, so called because it is a Bayesian \nclassifier that makes a simplifying (naive) assumption about how the features interact.\nWe represent a text document bag-of-words as if it were a bag-of-words, that is, an \nunordered set of words with their position ignored, keeping only their frequency in the document\ncˆ = argmax_c∈C P(c|d) break down any conditional probability P(x|y) into\nthree other probabilities: P(x|y) = P(y|x)P(x)/P(y) then cˆ = argmax_c∈C P(c|d) = argmax_c∈C P(d|c)P(c)/P(d) \nTo return to classification: we compute the most probable class ˆc given some\ndocument d by choosing the class which has the highest product of two probabilities:\nthe prior probability of the class P(c) and the likelihood of the document P(d|c):\nClassifiers that use a linear combination of the inputs\nto make a classification decision —like naive Bayes and also logistic regression—\nare called linear classifiers.\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\nHow can we learn the probabilities P(c) and P(fi|c)?  Pˆ(c) = Nc/Ndoc\nPˆ(wi|c) = count(wi, c)/ summation w∈V count(w, c). \nWhat do we do about words that occur in our test data but are not in our vocabulary at all because they did not occur in any training document in any class? The\nunknown word solution for such unknown words is to ignore them—remove them from the test\ndocument and not include any probability for them at all. \nFinally, some systems choose to completely ignore another class of words: stop\nstop words words, very frequent words like the and a. This can be done by sorting the vocabulary by frequency in the training set, and defining the top 10–100 vocabulary entries\nas stop words, or alternatively by using one of the many predefined stop word lists\navailable online.\n\n``` \nfunction TRAIN NAIVE BAYES(D, C) returns log P(c) and log P(w|c)\nfor each class c ∈ C # Calculate P(c) terms\n  Ndoc = number of documents in D\n  Nc = number of documents from D in class c\n  logprior[c]← log Nc/Ndoc\n  V←vocabulary of D\n  bigdoc[c]←append(d) for d ∈ D with class c\n  for each word w in V # Calculate P(w|c) terms\n    count(w,c)←# of occurrences of w in bigdoc[c]\n    loglikelihood[w,c]← log (count(w, c) + 1)/summ_w' in V(count (w0, c) + 1)\nreturn logprior, loglikelihood, V\n\nfunction TEST NAIVE BAYES(testdoc, logprior, loglikelihood, C, V) returns best c\nfor each class c ∈ C\n  sum[c]← logprior[c]\n  for each position i in testdoc\n    word←testdoc[i]\n    if word ∈ V\n      sum[c]←sum[c]+ loglikelihood[word,c]\n  return argmaxc sum[c]\n``` \n\n \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n for sentiment classification and a number of other text classification tasks,\nwhether a word occurs or not seems to matter more than its frequency. Thus it\noften improves performance to clip the word counts in each document at 1. Binary\nmultinomial naive Bayes or binary NB. The variant uses the same Eq. 4.10 except\nthat for each document we remove all duplicate words before concatenating them\ninto the single big document. A second important addition commonly made when doing text classification for\nsentiment is to deal with negation. Consider the difference between I really like this\nmovie (positive) and I didn’t like this movie (negative). The negation expressed by\ndidn’t completely alters the inferences we draw from the predicate like. Similarly,\nnegation can modify a negative word to produce a positive review (don’t dismiss this\nfilm, doesn’t let us get bored). Finally, in some situations we might have insufficient labeled training data to\ntrain accurate naive Bayes classifiers using all words in the training set to estimate\npositive and negative sentiment. In such cases we can instead derive the positive \nand negative word features from sentiment lexicons, lists of words that are pre- sentiment\nlexicons annotated with positive or negative sentiment\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n A common solution here, rather than using all the words as individual features,\nis to predefine likely sets of words or phrases as features, combined with features\nthat are not purely linguistic. For example the open-source SpamAssassin tool1\npredefines features like the phrase “one hundred percent guaranteed”, or the feature\nmentions millions of dollars, which is a regular expression that matches suspiciously\nlarge sums of money. But it also includes features like HTML has a low ratio of text\nto image area, that aren’t purely linguistic and might require some sophisticated\ncomputation, or totally non-linguistic features about, say, the path that the email\ntook to arrive. For other tasks, like language id—determining what language a given piece\nof text is written in—the most effective naive Bayes features are not words at all,\nbut character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,\n‘thei’), or, even simpler byte n-grams, where instead of using the multibyte Unicode\ncharacter representations called codepoints, we just pretend everything is a string of\nraw bytes. Because spaces count as a byte, byte n-grams can model statistics about\nthe beginning or ending of words \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Bayes classifiers can use any sort of feature: dictionaries, URLs, email addresses, network features, phrases, and so on. But\nif, as in the previous section, we use only individual word features, and we use all\nof the words in the text (not a subset), then naive Bayes has an important similarity to language modeling. Specifically, a naive Bayes model can be viewed as a\nset of class-specific unigram language models, in which the model for each class\ninstantiates a unigram language model.\nSince the likelihood features from the naive Bayes model assign a probability to\neach word P(word|c), the model also assigns a probability to each sentence:\nP(s|c) = PI _ i∈positions P(wi|c)\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\nIn both cases, we need a metric for knowing how well our spam detector (or\npie-tweet-detector) is doing. To evaluate any system for detecting things, we start\nby building a confusion matrix. A confusion matrix confusion\nis a table for visualizing how an algorithm performs with respect to the human gold\nlabels, using two dimensions (system output and gold labels), and each cell labeling\na set of possible outcomes. precision and recall. Precision measures the percentage of the items that\nthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\nare positive according to the human gold labels). Precision is defined as\nPrecision = true positives/true positives + false positives\nRecall measures the percentage of items actually present in the input that were\ncorrectly identified by the system. Recall is defined as\nRecall = true positives/true positives + false negatives. There are many ways to define a single metric that incorporates aspects of both\nF-measure precision and recall. The simplest of these combinations is the F-measure, defined as:\nFβ = (β2 +1)PR/β^2P+R\nThe β parameter differentially weights the importance of recall and precision,\nbased perhaps on the needs of an application. Values of β > 1 favor recall, while\nvalues of β < 1 favor precision. When β = 1, precision and recall are equally balanced.\nIn macroaveraging, we compute the performance microaveraging for each class, and then \naverage over classes. In microaveraging, we collect the decisions for all classes into \na single confusion matrix, and then compute precision and recall from that table.\n \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n we use the training set to train the model, then use\nthe development test set (also called a devset) to perhaps tune some parameters,and \nin general decide what the best model is. In cross-validation, we choose a number k, and partition our data into k disjoint\nsubsets called folds. Now we choose one of those k folds as a test set, train our\nclassifier on the remaining k − 1 folds, and then compute the error rate on the test\nset. Then we repeat with another fold as the test set, again training on the other k−1\nfolds. We do this sampling process k times and average the test set error rate from\nthese k runs to get an average error rate.\n \n\n## SLP Chapter 4.9 Statistical Significance Testing\n We would like to know if δ(x) > 0, meaning that our logistic regression classifier\nhas a higher F1 than our naive Bayes classifier on X. δ(x) is called the effect size;\na bigger δ means that A seems to be way better than B; a small δ means A seems to\nbe only a little better. The hypothesis H0, called the null hypothesis, supposes that δ(x) is actually negative or zero, meaning that A is not better than B. We would like to know if we can\nconfidently rule out this hypothesis, and instead support H1, that A is better. We do this by creating a random variable X ranging over all test sets. Now we\nask how likely is it, if the null hypothesis H0 was correct, that among these test sets\nwe would encounter the value of δ(x) that we found. We formalize this likelihood\nas the p-value: the probability, assuming the null hypothesis H0 is true, of seeing\nthe δ(x) that we saw or one even greater P(δ(X) ≥ δ(x)|H0 is true).We say that a result \nis statistically significant if the δ we saw has a probability that is below the \nthreshold and we therefore reject this null hypothesis. There are two common non-parametric \ntests used in NLP: approximate randomization and the bootstrap test, showing the paired \nversion of the test, which again is most common in NLP. Paired tests are those in which we compare two sets of observations that are aligned:\neach observation in one set can be paired with an observation in another. This happens naturally when we are comparing the performance of two systems on the same\ntest set; we can pair the performance of system A on an individual observation xi\nwith the performance of system B on the same xi.\n\n``` \nfunction BOOTSTRAP(test set x, num of samples b) returns p-value(x)\nCalculate δ(x) # how much better does algorithm A do than B on x\ns = 0\nfor i = 1 to b do\nfor j = 1 to n do # Draw a bootstrap sample x^(i) of size n\n    Select a member of x at random and add it to x^(i)\n  Calculate δ(x^(i)) # how much better does algorithm A do than B on x^(i)\n  s←s + 1 if δ(x^(i)) ≥ 2δ(x)\np-value(x) ≈ s/b\n# on what % of the b samples did algorithm A beat expectations?\nreturn p-value(x) # if very few did, our observed δ is probably not accidental\n``` \n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n One class of harms is representational harms, \nharms caused by a system that demeans a social group, for example by perpetuating negative \nstereotypes about them. In other tasks classifiers may lead to both representational harms and other\nharms, such as censorship. For example the important text classification task of\ntoxicity detection is the task of detecting hate speech, abuse, harassment, or other toxicity\ndetection\nkinds of toxic language. While the goal of such classifiers is to help reduce societal harm,\ntoxicity classifiers can themselves cause harms.   \n\n## Module 12 Lecture Notes\n Model-Based Classification\nModel-based approach\n• Build a model (e.g. Bayes’ net) where both the label and\nfeatures are random variables\n• Instantiate any observed features\n• Query for the distribution of the\nlabel conditioned on the features\n\nWhat do we need in order to use Naïve Bayes?\n• Inference method (we just saw this part)\n• Start with a bunch of probabilities: P(Y) and the P(Fi\n|Y) tables\n• Use standard inference to compute P(Y|F1…Fn)\n• Nothing new here\n• Estimates of local conditional probability tables\n• P(Y), the prior over labels\n• P(Fi\n|Y) for each feature (evidence variable)\n• These probabilities are collectively called the parameters of the\nmodel and denoted by q\n• Up until now, we assumed these appeared by magic, but…\n• …they typically come from training data counts: we’ll look at this\nsoon\n\nBag-of-words Naïve Bayes:\n• Features: Wi is the word at positon i\n• As before: predict label conditioned on feature variables (spam vs. ham)\n• As before: assume features are conditionally independent given label\n• New: each Wi is identically distributed\nGenerative model:\n“Tied” distributions and bag-of-words\n• Usually, each variable gets its own conditional probability distribution P(F|Y)\n• In a bag-of-words model\n• Each position is identically distributed\n• All positions share the same conditional probs P(W|Y)\n• Why make this assumption?\n• Called “bag-of-words” because model is insensitive to word order or reordering\n\nRelative frequency parameters will overfit the training data!\n• Just because we never saw a 3 with pixel (15,15) on during training doesn’t mean we won’t\nsee it at test time\n• Unlikely that every occurrence of “minute” is 100% spam\n• Unlikely that every occurrence of “seriously” is 100% ham\n• What about all the words that don’t occur in the training set at all?\n• In general, we can’t go around giving unseen events zero probability\nAs an extreme case, imagine using the entire email as the only feature\n• Would get the training data perfect (if deterministic labeling)\n• Wouldn’t generalize at all\n• Just making the bag-of-words assumption gives us some generalization, but isn’t enough\nTo generalize better: we need to smooth or regularize the estimates\n\nWhat to Do About Errors?\nNeed more features– words aren’t enough!\n• Have you emailed the sender before?\n• Have 1K other people just gotten the same email?\n• Is the sending information consistent?\n• Is the email in ALL CAPS?\n• Do inline URLs point where they say they point?\n• Does the email address you by (your) name?\nCan add these information sources as new\nvariables in the NB model\nNext class we’ll talk about classifiers which let\nyou easily add arbitrary features more easily\n\nFirst step: get a baseline\n• Baselines are very simple “straw man” procedures\n• Help determine how hard the task is\n• Help know what a “good” accuracy is\nWeak baseline: most frequent label classifier\n• Gives all test instances whatever label was most common in the training set\n• E.g. for spam filtering, might label everything as ham\n• Accuracy might be very high if the problem is skewed\n• E.g. calling everything “ham” gets 66%, so a classifier that gets 70% isn’t very good…\nFor real research, usually use previous work as a (strong) baseline\n\nThe confidence of a probabilistic classifier:\n• Posterior over the top label\n• Represents how sure the classifier is of\nthe classification\n• Any probabilistic model will have\nconfidences\n• No guarantee confidence is correct\nCalibration\n• Weak calibration: higher confidences\nmean higher accuracy\n• Strong calibration: confidence predicts\naccuracy rate\n• What’s the value of calibration?\n\nSummary\nBayes rule lets us do diagnostic queries with causal probabilities\nThe naïve Bayes assumption takes all features to be independent given\nthe class label\nWe can build classifiers out of a naïve Bayes model using training data\nSmoothing estimates is important in real systems\nClassifier confidences are useful, when you can get them\n\nProblems with the Perceptron\nNoise: if the data isn’t separable,\nweights might thrash\n• Averaging weight vectors over\ntime can help (averaged\nperceptron)\nMediocre generalization: finds a\n“barely” separating solution\nOvertraining: test / held-out\naccuracy usually rises, then falls\n• Overtraining is a kind of\noverfitting\n\nFixing the Perceptron\nIdea: adjust the weight update to mitigate these effects\nMIRA*: choose an update size that fixes the current\nmistake…\n… but, minimizes the change to w\n\nIn practice, it’s also bad to make updates that are too large\n• Example may be labeled incorrectly\n• You may not have enough features\n• Solution: cap the maximum possible value of t with some\nconstant C\n• Corresponds to an optimization that assumes nonseparable data\n• Usually converges faster than perceptron\n• Usually better, especially on noisy data\n\nClassification: Comparison\nNaïve Bayes\n• Builds a model training data\n• Gives prediction probabilities\n• Strong assumptions about feature independence\n• One pass through data (counting)\nPerceptrons / MIRA:\n• Makes less assumptions about data\n• Mistake-driven learning\n• Multiple passes through data (prediction)\n• Often more accurate\n\nPerceptions can be used for all kinds of classification problems.\nThink of the inputs as features representing something we want to classify.\nThe feature values for inputs are fixed, but we can choose different weight vectors.\nDepending on the weight vector that we pick, we will get a different classifier. \n \n\n## Module 12 Definitions\n* **Induction** - Going from a specific set of observations to a general rule\n* **Deductive conclusions** - guaranteed to be correct if the premises are correct\n* **Factored representation** - vector of attribute values\n* **Classification** - when the output of the learning problem is a finite set of values\n* **Regression** - when the output of the learning problem is a number\n* **Supervised learning** - the agent observes input-output pairs and learns a function that maps from input to output\n* **Unsupervised learning** - the agent learns patterns in the input without any explicit feedback\n* **Clustering** - detecting potentially useful clusters of input examples\n* **Reinforcement learning** - the agent learns from a series of reinforcements: rewards and punishments\n* **Exploratory data analysis** - examining the data with statistical tests and visualizations—histograms, scatter plots, box plots—get some insight into what hypothesis space might be appropriate\n* **Consistent hypothesis** - h such that each xi in the training set has h(xi) = yi\n* **Test set** - second sample of (xi, yi) pairs\n* **Generalizes well** - if it accurately predicts the outpusts of the test set\n* **Bias** - the tendency of a predictive hypothesis to deviate from the expected value when averaged over different training sets\n* **Underfitting** - when it fails to find a pattern in the data.\n* **Variance** - the amount of change in the hypothesis due to fluctuation in the training data\n* **Decistion tree** - representation of a function that maps a vector of attribute values to a single output value—a “decision.”\n* **Boolean classification** - consider only inputs consisting of discrete values and outputs that are either true or false \n* **Happy graphs** - learning curves\n* **Entropy** -  measure of the uncertainty of a random variable\n* **Information gain** - from the attribute test on A is the expected reduction in entropy:\n* **Decision tree pruning** - combats overfitting by eliminating nodes that are not clearly relevant.\n* **Statistical significance test** - how large a gain should we require to split on an attribute?\n* **Early stopping** - have the decision tree algorithm stop generating nodes when there is no good attribute to split on\n* **Missing data** - in many domains not all the attribute values will be known for every example\n* **Continuous and multivalued input attributes** - may be that every example has a different attribute value. The information gain measure would give its highest score to such an attribute\n* **Split point test** - an inequality test on the value of an attribute\n* **Regression tree** - has at each leaf a linear function of some subset of numerical attributes, rather than a single output value\n* **Decision trees are unstable** - adding just one new example can change the test at the root\n* **Stationary assumption** - \n* **Independent and identically distributed or i.i.d** - each example has the same prior probabilityt distrobution adn is independent of the previous examples\n* **Error rate** - the proportion of times that h(x) != y for an (x,y) example\n* **Hyper parameters** - try different thresholds for x^2 pruning of decision trees, or different degrees for polynomials\n* **Leave-one-out cross-validation or loocv** - k-fold cross-validation when k=n\n* **Model selection** - choose a good hypothesis space\n* **Optimization (training)** - finds the best hypothesis for the space\n* **Interpolated** - a model that exactly fits all the training data\n* **Realizable** - hypothesis space H actually contains the true function f\n* **Varriance** - a learning algorithm will in general return different hypotheses for different sets of examples\n* **Computationally intractable** - a search can explore part of the space and return a reasonably good hypothesis, but can’t always guarantee the best one\n* **Small scale learning** - the number of training examples ranged from dozens to the low thousands\n* **Large scale learning** - millions of examples\n* **Regularization** - explicitly penalizing complex hypotheses is called\n* **Regularization function** - the complexity function\n* **Frature selection** - discard attributes that appear to be irrelevant\n* **Minimum description length or mdl hypothesis** - minimizes the total number of bits required\n* **Hand tuning** - guess based on past experience, train model,measure performance on validation data, analyze results, and use intuition to suggest new parameter values\n* **Random search** - samples uniformly from the set of all possible hyperparameter settings, repeating for as long as you are willing to spend the time and computational resources\n* **Bayesian optimization** - treats the task of choosing good hyperparameter values as a machine learning problem in itself\n* **Pbt** - trains in 2 waves, 2nd generation can choose based on the successful values from the previous generation.\n* **Computational learning theory** - which lies at the intersection of AI, statistics, and theoretical computer science.\n* **Pac learning algorithm** - Any learning algorithm that returns hypotheses that are probably approximately correct\n* **Approximately correct** - error(h) < epsilon which is a small constatne\n* **Sample complexity** - number of required examples as a function of epsilon and delta\n* **Decision lists** - onsists of a series of tests, each of which is a conjunction of literals\n* **Linear regression** - task of finding the hw that best fits the data\n* **Weight space** - the space defined by all possible settings of the weights\n* **Epoch** - a step that covers all the training examples\n* **Stochastic gradient descent or sgd** - randomly selects a small number of training examples at each step, and updates according to\n* **Decistion boundary** - is a line (or a surface, in higher dimensions) that separates the two classes.\n* **Linear separator** - linear decision boundary\n* **Training curve** - measures the classifier performance on a fixed training set as the learning process proceeds one update at a time on that training set.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\" Consider a single input observation x, which we will represent by a vector of features [x1, x2,..., xn] (we’ll show sample features in the next subsection). The classifier\n",
        "output y can be 1 (meaning the observation is a member of the class) or 0 (the observation is not a member of the class). We want to know the probability P(y = 1|x)\n",
        "that this observation is a member of the class. So perhaps the decision is “positive \n",
        "sentiment” versus “negative sentiment”, the features represent counts of words in a\n",
        "document, P(y = 1|x) is the probability that the document has positive sentiment,\n",
        "and P(y = 0|x) is the probability that the document has negative sentiment.\n",
        "Logistic regression solves this task by learning, from a training set, a vector of\n",
        "weights and a bias term. Each weight wi is a real number, and is associated with one\n",
        "of the input features xi. The weight wi represents how important that input feature\n",
        "is to the classification decision, and can be positive or negative. The bias term, also called the intercept, is\n",
        "intercept another real number that’s added to the weighted inputs. To create a probability, we’ll pass z through the sigmoid function, σ(z). The\n",
        "sigmoid function (named because it looks like an s) is also called the logistic function, \n",
        "and gives logistic regression its name. The sigmoid has the following equation, σ(z) = 1/1+e^−z=1/1+exp(−z)\n",
        "The sigmoid function has the property 1−σ(x) = σ(−x)  so we could also have expressed P(y = 0) as σ(−(w· x+b)).\n",
        "Now we have an algorithm that given an instance x computes the probability\n",
        "P(y = 1|x). How do we make a decision? For a test instance x, we say yes if the\n",
        "probability P(y = 1|x) is more than .5, and no otherwise. We call .5 the decision\n",
        "boundary: decision(x) = \u001a1 if P(y = 1|x) > 0.5\n",
        "                        0 otherwise\n",
        "For many tasks (especially when feature values can reference specific words)\n",
        "we’ll need large numbers of features. Often these are created automatically via feature templates,\n",
        "In order to avoid the extensive human effort of feature design, recent research in\n",
        "NLP has focused on representation learning: ways to learn features automatically\n",
        "in an unsupervised way from the input. Logistic regression has a number of advantages over naive\n",
        "Bayes. Naive Bayes has overly strong conditional independence assumptions. Consider two features which are strongly correlated; in fact, imagine that we just add the\n",
        "same feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate, multiplying them both in, overestimating the evidence. B\n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\" the distance between the system output and the gold output, and we call this distance\n",
        "loss the loss function or the cost function. The second thing we need is an optimization algorithm for iteratively updating\n",
        "the weights so as to minimize this loss function. The standard algorithm for this is\n",
        "gradient descent; we’ll introduce the stochastic gradient descent algorithm in the\n",
        "following section. \"\"\"\n",
        "slp_chapter5_3 = \"\"\" We need a loss function that expresses, for an observation x, how close the classifier\n",
        "output ( ˆy = σ(w· x+b)) is to the correct output (y, which is 0 or 1). We’ll call this:\n",
        "L(yˆ, y) = How much ˆy differs from the true y This is called conditional maximum likelihood\n",
        "estimation: we choose the parameters w,b that maximize the log probability of\n",
        "the true y labels in the training data given the observations x. The resulting loss\n",
        "function is the negative log likelihood loss, generally called the cross-entropy loss. \"\"\"\n",
        "slp_chapter5_4 = \"\"\" So the goal is to find the set of weights which minimizes the loss function, averaged\n",
        "over all examples:θˆ = argmin_θ 1/m SUM_i=1 L_CE(f(x(i);θ), y(i)). For logistic regression, \n",
        "this loss function is conveniently convex. A convex function has just one minimum; \n",
        "there are no local minima to get stuck in, so gradient descent starting from any point \n",
        "is guaranteed to find the minimum. The gradient descent algorithm answers this question \n",
        "by finding the gradient of the loss function at the current point and moving in the \n",
        "opposite direction. The magnitude of the amount to move in gradient descent is the value of the\n",
        "slope d/dw L(f(x;w), y) weighted by a learning rate η. A higher (faster) learning\n",
        "rate means that we should move w more on each step.The gradient is just such a vector; it expresses the directional components of the\n",
        "sharpest slope along each of those N dimensions. If we’re just imagining two weight\n",
        "dimensions (say for one weight w and one bias b), the gradient might be a vector with\n",
        "two orthogonal components, each of which tells us how much the ground slopes in\n",
        "the w dimension and in the b dimension. Logistic regression: gradient with respect to a single weight wj represents a\n",
        "very intuitive value: the difference between the true y and our estimated \n",
        "ˆy = σ(w·x+b) for that observation, multiplied by the corresponding input value xj.\n",
        "\n",
        "``` \n",
        "function STOCHASTIC GRADIENT DESCENT(L(), f(), x, y) returns θ\n",
        "# where: L is the loss function\n",
        "# f is a function parameterized by θ\n",
        "# x is the set of training inputs x^(1), x^(2),..., x^(m)\n",
        "# y is the set of training outputs (labels) y^(1), y^(2),..., y^(m)\n",
        "θ ←0\n",
        "repeat til done # see caption\n",
        "  For each training tuple (x(i), y(i)) (in random order)\n",
        "    1. Optional (for reporting): # How are we doing on this tuple?\n",
        "    Compute ˆy(i) = f(x(i);θ) # What is our estimated output ˆy?\n",
        "    Compute the loss L(yˆ(i), y(i)) # How far off is ˆy(i)from the true output y(i)?\n",
        "    2. g←∇θ L(f(x(i);θ), y(i)) # How should we move θ to maximize loss?\n",
        "    3. θ ←θ − η g # Go the other way instead\n",
        "return θ\n",
        "``` \n",
        "\n",
        "\"\"\"\n",
        "slp_chapter5_5 = \"\"\" To avoid overfitting, a new regularization term R(θ) is added to the objective\n",
        "function θˆ = argmax_θ SUM^m_i=1 logP(y^(i)|x^(i))−αR(θ) \n",
        "The L2 regularized objective function becomes:\n",
        "θˆ = argmax_θ [SUm_i=1 logP(y^(i)|x^(i))]−α SUM_j=1 θ^\n",
        "L1 regularization is a linear function of the weight values, named after the L1 norm L1\n",
        "regularization||W||1, the sum of the absolute values of the weights, or Manhattan distance (the\n",
        "Manhattan distance is the distance you’d have to walk between two points in a city\n",
        "with a street grid like New York):θˆ = argmax_θ [SUM_1=i logP(y^(i)|x^(i))]−αSUM_j=1|θj|\n",
        "L1 regularization is called lasso lasso regression and L2 regularization is called ridge regression,\n",
        "ridge and both are commonly used in language processing\n",
        "\"\"\"\n",
        "slp_chapter5_6 = \"\"\" The multinomial logistic classifier uses a generalization of the sigmoid, called\n",
        "softmax the softmax function, to compute the probability p(y = c|x). The softmax function\n",
        "takes a vector z = [z1,z2,...,zk] of k arbitrary values and maps them to a probability\n",
        "distribution, with each value in the range (0,1), and all the values summing to 1.\n",
        "Like the sigmoid, it is an exponential function.\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\" Classifier components\n",
        "Machine learning classifiers require a training corpus of M observations\n",
        "input/output pairs (x(i)\n",
        ",y(i)\n",
        ").\n",
        "1. A feature representation of the input. For each input observation\n",
        "x(i)\n",
        ", this will be a vector of features [x1,x2,...,xn].\n",
        "2. A classification function that computes the estimated class �! via\n",
        "p(y|x).\n",
        "3. An objective function for learning, usually involving minimizing\n",
        "error on training examples.\n",
        "4. An algorithm for optimizing the objective function.\n",
        "\n",
        "Sentiment classifier\n",
        "For sentiment classification, consider an input observation x,\n",
        "represented by a vector of features [x1,x2,...,xn]. The classifier output y\n",
        "can be 1 (positive sentiment) or 0 (negative sentiment). We want to\n",
        "estimate P(y = 1|x).\n",
        "Logistic regression solves this task by learning, from a training set, a\n",
        "vector of weights and a bias term.\n",
        "z = sum_i wixi + b\n",
        "We can also write this as a dot product:\n",
        "z = w*x+b\n",
        "\n",
        "Now we have an algorithm that given an instance x computes the\n",
        "probability P(y = 1|x). How do we make a decision?\n",
        "y^ = {1 if P(y=1|x)>0.5   0 otherwise\n",
        "For a test instance x, we say yes if the probability P(y = 1|x) is more than\n",
        ".5, and no otherwise. We call .5 the decision boundary\n",
        "\n",
        "Learning in logistic regression\n",
        "How do we get the weights of the model? We learn the parameters\n",
        "(weights + bias) via learning. This requires 2 components:\n",
        "1. An objective function or loss function that tells us distance between\n",
        "the system output and the gold output. We will use cross-entropy\n",
        "loss.\n",
        "2. An algorithm for optimizing the objective function. We will use\n",
        "stochastic gradient descent to minimize the loss function. \n",
        "\n",
        "We need to determine for some observation x how close the classifier\n",
        "output (y^!= σ (w · x + b)) is to the correct output (y, which is 0 or 1).\n",
        "L(y^,y) = how much y^! differs from the true y\n",
        "One example is mean squared error\n",
        "L_MSE(y^,y) = 1/2(y^-y)^2 \n",
        "The resulting loss function is the negative log likelihood loss, more\n",
        "commonly called the cross entropy loss.\n",
        "The result is cross-entropy loss:\n",
        "L_CE(y^,y)= −log(py|x) = −[y log y^! + (1 − y )log(1 − y^)]\n",
        "Finally, plug in the definition for y^<= σ (w · x) + b\n",
        "L_CE(y^,y) = −[y log σ(w·x+b) + (1 − y) log(1 − σ(w·x+b))]\n",
        "We want the loss to be smaller if the model’s estimate is close to correct,\n",
        "and we want the loss to be bigger if it is confused.\n",
        "\n",
        "Gradient descent is a method that finds a minimum of a function by figuring out in\n",
        "which direction (in the space of the parameters θ) the function’s slope is rising the most\n",
        "steeply, and moving in the opposite direction. \n",
        "\n",
        "Stochastic gradient descent chooses a single random example at a time and updates\n",
        "its weights on that example. As a result the updates can fluctuate.\n",
        "An alternate is batch training, which computes the gradient over the entire dataset.\n",
        "This gives a much better estimate of which direction to move the weights but takes a\n",
        "long time to compute.\n",
        "A commonly used compromise is mini-batch training, where we train on a small\n",
        "batch. The batch size can be 512 or 1024, often selected based on computational\n",
        "resources, so that all examples in the mini-batch can be processed in parallel. The loss\n",
        "is then accumulated.\n",
        "\n",
        "Overfitting is a problem with many machine learning models. Overfitting results in\n",
        "poor generalization and poor performance on unseen test set.\n",
        "In logistic regression, if a feature only occurs in one class then it will get a high weight.\n",
        "Sometimes we are just modelling noisy factors that just accidentally correlate with the\n",
        "class.\n",
        "Regularization is a way to penalize large weights. A regularization term is added to\n",
        "the loss function.\n",
        "Lasso regression uses L1 regularization\n",
        "Ridge regression uses L2 regularization \n",
        "\n",
        "Instead of directly outputting z = w·x+b, which is a linear function of x, neuron units\n",
        "apply a non-linear function f to z.\n",
        "The output of this function is called the activation value for the unit, represented by\n",
        "the variable a. The output of a neural network is called y, so if the activation of a node\n",
        "is the final output of a network then\n",
        "y=a= f(z)\n",
        "There are 3 commonly used non-linear functions used for f:\n",
        "The sigmoid function\n",
        "The tanh function\n",
        "The rectified linear unit ReLU\n",
        "\n",
        "The simplest kind of NN is the Feed-Forward Neural Network\n",
        "Multilayer network, all units are usually fully-connected, and no cycles.\n",
        "The outputs from each layer are passed to units in the next higher layer, and no\n",
        "outputs are passed back to lower layers. \n",
        "\n",
        "The advantage of using a single matrix W for the weights of the entire layer is the\n",
        "hidden layer computation can be done efficiently with simple matrix operations.\n",
        "The computation has three steps:\n",
        "1. multiplying the weight matrix by the input vector x,\n",
        "2. adding the bias vector b, and\n",
        "3. applying the activation function g (such as Sigmoid)\n",
        "The output of the hidden layer, the vector h, is thus the following, using the sigmoid\n",
        "function σ:\n",
        "h = σ(Wx+b) \n",
        "\n",
        "Like logistic regression, we want to learn the best parameters for the neural net to\n",
        "make its predictions �/ as close to possible as the gold standard labels in our training\n",
        "data y.\n",
        "What do we need?\n",
        "A loss function – cross-entropy loss\n",
        "An optimization algorithm – gradient descent\n",
        "A way of computing the gradient of the loss function – error propagation \n",
        "Although backpropagation was invented for neural nets, it is related to general\n",
        "procedure called backward differentiation, which depends on the notion of\n",
        "computation graphs.\n",
        "A computation graph represents the process of computing a mathematical expression.\n",
        "The computation is broken down into separate operations. Each operation is a node in\n",
        "a graph.\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",   \n",
        "}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "7eab952f-2b51-4b25-8e10-c40fd7eeeb7d"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Consider a single input observation x, which we will represent by a vector of features [x1, x2,..., xn] (we’ll show sample features in the next subsection). The classifier\noutput y can be 1 (meaning the observation is a member of the class) or 0 (the observation is not a member of the class). We want to know the probability P(y = 1|x)\nthat this observation is a member of the class. So perhaps the decision is “positive \nsentiment” versus “negative sentiment”, the features represent counts of words in a\ndocument, P(y = 1|x) is the probability that the document has positive sentiment,\nand P(y = 0|x) is the probability that the document has negative sentiment.\nLogistic regression solves this task by learning, from a training set, a vector of\nweights and a bias term. Each weight wi is a real number, and is associated with one\nof the input features xi. The weight wi represents how important that input feature\nis to the classification decision, and can be positive or negative. The bias term, also called the intercept, is\nintercept another real number that’s added to the weighted inputs. To create a probability, we’ll pass z through the sigmoid function, σ(z). The\nsigmoid function (named because it looks like an s) is also called the logistic function, \nand gives logistic regression its name. The sigmoid has the following equation, σ(z) = 1/1+e^−z=1/1+exp(−z)\nThe sigmoid function has the property 1−σ(x) = σ(−x)  so we could also have expressed P(y = 0) as σ(−(w· x+b)).\nNow we have an algorithm that given an instance x computes the probability\nP(y = 1|x). How do we make a decision? For a test instance x, we say yes if the\nprobability P(y = 1|x) is more than .5, and no otherwise. We call .5 the decision\nboundary: decision(x) = \u001a1 if P(y = 1|x) > 0.5\n                        0 otherwise\nFor many tasks (especially when feature values can reference specific words)\nwe’ll need large numbers of features. Often these are created automatically via feature templates,\nIn order to avoid the extensive human effort of feature design, recent research in\nNLP has focused on representation learning: ways to learn features automatically\nin an unsupervised way from the input. Logistic regression has a number of advantages over naive\nBayes. Naive Bayes has overly strong conditional independence assumptions. Consider two features which are strongly correlated; in fact, imagine that we just add the\nsame feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate, multiplying them both in, overestimating the evidence. B\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n the distance between the system output and the gold output, and we call this distance\nloss the loss function or the cost function. The second thing we need is an optimization algorithm for iteratively updating\nthe weights so as to minimize this loss function. The standard algorithm for this is\ngradient descent; we’ll introduce the stochastic gradient descent algorithm in the\nfollowing section. \n\n## SLP Chapter 5.3 The cross-entropy loss function\n We need a loss function that expresses, for an observation x, how close the classifier\noutput ( ˆy = σ(w· x+b)) is to the correct output (y, which is 0 or 1). We’ll call this:\nL(yˆ, y) = How much ˆy differs from the true y This is called conditional maximum likelihood\nestimation: we choose the parameters w,b that maximize the log probability of\nthe true y labels in the training data given the observations x. The resulting loss\nfunction is the negative log likelihood loss, generally called the cross-entropy loss. \n\n## SLP Chapter 5.4 Gradient Descent\n So the goal is to find the set of weights which minimizes the loss function, averaged\nover all examples:θˆ = argmin_θ 1/m SUM_i=1 L_CE(f(x(i);θ), y(i)). For logistic regression, \nthis loss function is conveniently convex. A convex function has just one minimum; \nthere are no local minima to get stuck in, so gradient descent starting from any point \nis guaranteed to find the minimum. The gradient descent algorithm answers this question \nby finding the gradient of the loss function at the current point and moving in the \nopposite direction. The magnitude of the amount to move in gradient descent is the value of the\nslope d/dw L(f(x;w), y) weighted by a learning rate η. A higher (faster) learning\nrate means that we should move w more on each step.The gradient is just such a vector; it expresses the directional components of the\nsharpest slope along each of those N dimensions. If we’re just imagining two weight\ndimensions (say for one weight w and one bias b), the gradient might be a vector with\ntwo orthogonal components, each of which tells us how much the ground slopes in\nthe w dimension and in the b dimension. Logistic regression: gradient with respect to a single weight wj represents a\nvery intuitive value: the difference between the true y and our estimated \nˆy = σ(w·x+b) for that observation, multiplied by the corresponding input value xj.\n\n``` \nfunction STOCHASTIC GRADIENT DESCENT(L(), f(), x, y) returns θ\n# where: L is the loss function\n# f is a function parameterized by θ\n# x is the set of training inputs x^(1), x^(2),..., x^(m)\n# y is the set of training outputs (labels) y^(1), y^(2),..., y^(m)\nθ ←0\nrepeat til done # see caption\n  For each training tuple (x(i), y(i)) (in random order)\n    1. Optional (for reporting): # How are we doing on this tuple?\n    Compute ˆy(i) = f(x(i);θ) # What is our estimated output ˆy?\n    Compute the loss L(yˆ(i), y(i)) # How far off is ˆy(i)from the true output y(i)?\n    2. g←∇θ L(f(x(i);θ), y(i)) # How should we move θ to maximize loss?\n    3. θ ←θ − η g # Go the other way instead\nreturn θ\n``` \n\n\n\n## SLP Chapter 5.5 Regularization\n To avoid overfitting, a new regularization term R(θ) is added to the objective\nfunction θˆ = argmax_θ SUM^m_i=1 logP(y^(i)|x^(i))−αR(θ) \nThe L2 regularized objective function becomes:\nθˆ = argmax_θ [SUm_i=1 logP(y^(i)|x^(i))]−α SUM_j=1 θ^\nL1 regularization is a linear function of the weight values, named after the L1 norm L1\nregularization||W||1, the sum of the absolute values of the weights, or Manhattan distance (the\nManhattan distance is the distance you’d have to walk between two points in a city\nwith a street grid like New York):θˆ = argmax_θ [SUM_1=i logP(y^(i)|x^(i))]−αSUM_j=1|θj|\nL1 regularization is called lasso lasso regression and L2 regularization is called ridge regression,\nridge and both are commonly used in language processing\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n The multinomial logistic classifier uses a generalization of the sigmoid, called\nsoftmax the softmax function, to compute the probability p(y = c|x). The softmax function\ntakes a vector z = [z1,z2,...,zk] of k arbitrary values and maps them to a probability\ndistribution, with each value in the range (0,1), and all the values summing to 1.\nLike the sigmoid, it is an exponential function.\n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n Classifier components\nMachine learning classifiers require a training corpus of M observations\ninput/output pairs (x(i)\n,y(i)\n).\n1. A feature representation of the input. For each input observation\nx(i)\n, this will be a vector of features [x1,x2,...,xn].\n2. A classification function that computes the estimated class �! via\np(y|x).\n3. An objective function for learning, usually involving minimizing\nerror on training examples.\n4. An algorithm for optimizing the objective function.\n\nSentiment classifier\nFor sentiment classification, consider an input observation x,\nrepresented by a vector of features [x1,x2,...,xn]. The classifier output y\ncan be 1 (positive sentiment) or 0 (negative sentiment). We want to\nestimate P(y = 1|x).\nLogistic regression solves this task by learning, from a training set, a\nvector of weights and a bias term.\nz = sum_i wixi + b\nWe can also write this as a dot product:\nz = w*x+b\n\nNow we have an algorithm that given an instance x computes the\nprobability P(y = 1|x). How do we make a decision?\ny^ = {1 if P(y=1|x)>0.5   0 otherwise\nFor a test instance x, we say yes if the probability P(y = 1|x) is more than\n.5, and no otherwise. We call .5 the decision boundary\n\nLearning in logistic regression\nHow do we get the weights of the model? We learn the parameters\n(weights + bias) via learning. This requires 2 components:\n1. An objective function or loss function that tells us distance between\nthe system output and the gold output. We will use cross-entropy\nloss.\n2. An algorithm for optimizing the objective function. We will use\nstochastic gradient descent to minimize the loss function. \n\nWe need to determine for some observation x how close the classifier\noutput (y^!= σ (w · x + b)) is to the correct output (y, which is 0 or 1).\nL(y^,y) = how much y^! differs from the true y\nOne example is mean squared error\nL_MSE(y^,y) = 1/2(y^-y)^2 \nThe resulting loss function is the negative log likelihood loss, more\ncommonly called the cross entropy loss.\nThe result is cross-entropy loss:\nL_CE(y^,y)= −log(py|x) = −[y log y^! + (1 − y )log(1 − y^)]\nFinally, plug in the definition for y^<= σ (w · x) + b\nL_CE(y^,y) = −[y log σ(w·x+b) + (1 − y) log(1 − σ(w·x+b))]\nWe want the loss to be smaller if the model’s estimate is close to correct,\nand we want the loss to be bigger if it is confused.\n\nGradient descent is a method that finds a minimum of a function by figuring out in\nwhich direction (in the space of the parameters θ) the function’s slope is rising the most\nsteeply, and moving in the opposite direction. \n\nStochastic gradient descent chooses a single random example at a time and updates\nits weights on that example. As a result the updates can fluctuate.\nAn alternate is batch training, which computes the gradient over the entire dataset.\nThis gives a much better estimate of which direction to move the weights but takes a\nlong time to compute.\nA commonly used compromise is mini-batch training, where we train on a small\nbatch. The batch size can be 512 or 1024, often selected based on computational\nresources, so that all examples in the mini-batch can be processed in parallel. The loss\nis then accumulated.\n\nOverfitting is a problem with many machine learning models. Overfitting results in\npoor generalization and poor performance on unseen test set.\nIn logistic regression, if a feature only occurs in one class then it will get a high weight.\nSometimes we are just modelling noisy factors that just accidentally correlate with the\nclass.\nRegularization is a way to penalize large weights. A regularization term is added to\nthe loss function.\nLasso regression uses L1 regularization\nRidge regression uses L2 regularization \n\nInstead of directly outputting z = w·x+b, which is a linear function of x, neuron units\napply a non-linear function f to z.\nThe output of this function is called the activation value for the unit, represented by\nthe variable a. The output of a neural network is called y, so if the activation of a node\nis the final output of a network then\ny=a= f(z)\nThere are 3 commonly used non-linear functions used for f:\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\n\nThe simplest kind of NN is the Feed-Forward Neural Network\nMultilayer network, all units are usually fully-connected, and no cycles.\nThe outputs from each layer are passed to units in the next higher layer, and no\noutputs are passed back to lower layers. \n\nThe advantage of using a single matrix W for the weights of the entire layer is the\nhidden layer computation can be done efficiently with simple matrix operations.\nThe computation has three steps:\n1. multiplying the weight matrix by the input vector x,\n2. adding the bias vector b, and\n3. applying the activation function g (such as Sigmoid)\nThe output of the hidden layer, the vector h, is thus the following, using the sigmoid\nfunction σ:\nh = σ(Wx+b) \n\nLike logistic regression, we want to learn the best parameters for the neural net to\nmake its predictions �/ as close to possible as the gold standard labels in our training\ndata y.\nWhat do we need?\nA loss function – cross-entropy loss\nAn optimization algorithm – gradient descent\nA way of computing the gradient of the loss function – error propagation \nAlthough backpropagation was invented for neural nets, it is related to general\nprocedure called backward differentiation, which depends on the notion of\ncomputation graphs.\nA computation graph represents the process of computing a mathematical expression.\nThe computation is broken down into separate operations. Each operation is a node in\na graph.\n\n\n## Module 13 Definitions\n* **** - \n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_10 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_11 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\"How should we represent the meaning of a word?\n",
        "In N-gram LMs we represented words as a string of letters or as an index in a\n",
        "vocabulary list.\n",
        "Ideally, we want a meaning representation to encode:\n",
        "1. Synonyms – words that have similar meanings\n",
        "2. Antonyms – words that have opposite meanings\n",
        "3. Connotations – words that are positive or negative\n",
        "4. Semantic Roles – buy, sell, and pay are different parts of the same underlying\n",
        "purchasing event\n",
        "5. Support for entailment\n",
        "Most words don’t have many synonyms, but they do have a lot of similar words. Cat is\n",
        "not a synonym of dog, but cats and dogs are certainly similar words. Context windows\n",
        "Occur in a window of +/- 2 words, in the same sentence, in the same document\n",
        "Instead of window of words use more complex contexts:\n",
        "dependency patters. Subj-of-verb, adj-mod, obj-of-verb\n",
        "Languages have long distance dependencies.\n",
        "Alternates to counts\n",
        "Raw word frequency is not a great measure of association between words. It’s very\n",
        "skewed “the” and “of” are very frequent, but maybe not the most discriminative\n",
        "We’d rather have a measure that asks whether a context word is particularly\n",
        "informative about the target word.\n",
        "Instead of raw counts, it’s common to transform vectors using TF-IDF or PPMI\n",
        "Sparse v. Dense Vectors\n",
        "Co-occurrence matrix (weighted by TF-IDF or mutual information)\n",
        "• Long (length |V| = 50,000+)\n",
        "• Sparse (most elements are zeros)\n",
        "Alternative: learn vectors that are\n",
        "• Short (length 200-1000)\n",
        "• Dense (most elements are non-zero)\n",
        "Language modeling Goal: Learn a function that returns the joint probability\n",
        "Primary difficulty:\n",
        "1. There are too many parameters to accurately estimate. This is\n",
        "sometimes called the “curse of dimensionality”\n",
        "2. In n-gram-based models we fail to generalize to related words / word\n",
        "sequences that we have observed.\n",
        "\n",
        "In LMs we user chain rule to get the conditional probability of the next\n",
        "word in the sequence given all of the previous words:\n",
        "\n",
        "What happens when we have a new (unseen) combination of n words?\n",
        "1. Back-off\n",
        "2. Smoothing / interpolation\n",
        "We are basically just stitching together short sequences of observed\n",
        "words.\n",
        "\n",
        "Neural Language models have several advantages over n-gram LMs:\n",
        "1. They don’t need smoothing\n",
        "2. They can handle much longer histories.\n",
        "3. They can generalize over contexts of similar words.\n",
        "4. Neural LMs tend to have much higher predictive accuracy than n-gram LMs.\n",
        "Disadvantage: slower to train than traditional n-gram LMs\n",
        "\n",
        "Advantages of NN LMs\n",
        "Better results. They achieve better preplexity scores than SOTA n-gram LMs.\n",
        "Larger N. NN LMs can scale to much larger orders of n. This is achievable because\n",
        "parameters are associated only with individual words, and not with n-grams.\n",
        "They generalize across contexts. For example, by observing that the words blue,\n",
        "green, red, black, etc. appear in similar contexts, the model will be able to assign a\n",
        "reasonable score to the green car even though it never observed in training, because it\n",
        "did observe blue car and red car.\n",
        "A by-product of training are word embeddings\n",
        "\n",
        "A Neural Probabilistic LM\n",
        "Given:\n",
        "A training set w1 … wt where wt ∈V\n",
        "Learn:\n",
        "f(w1 … wt) = P(wt|w1 … wt-1)\n",
        "Subject to giving a high probability to an unseen text/dev set (e.g. minimizing\n",
        "the perplexity)\n",
        "Constraint:\n",
        "Create a proper probability distribution (e.g. sums to 1) so that we can take\n",
        "the product of conditional probabilities to get the joint probability of a\n",
        "sentence\n",
        "\n",
        "Summary: How to learn word2vec (skip-gram) embeddings\n",
        "Start with V random 300-dimensional vectors as initial\n",
        "embeddings\n",
        "Use logistic regression\n",
        "• Take a corpus and take pairs of words that co-occur as positive\n",
        "examples\n",
        "• Take pairs of words that don't co-occur as negative examples\n",
        "• Train the classifier to distinguish these by slowly adjusting all the\n",
        "embeddings to improve the classifier performance\n",
        "• Throw away the classifier code and keep the embeddings.\n",
        "\n",
        "In a word-analogy task we are given two pairs of words that share a\n",
        "relation (e.g. “man:woman”, “king:queen”).\n",
        "The identity of the fourth word (“queen”) is hidden, and we need to infer\n",
        "it based on the other three by answering\n",
        "“man is to woman as king is to — ?”\n",
        "More generally, we will say a:a∗ as b:b∗.\n",
        "Can we solve these with word vectors?\n",
        "\n",
        "Implicit Association test (Greenwald et al 1998): How associated are\n",
        "• concepts (flowers, insects) & attributes (pleasantness, unpleasantness)?\n",
        "• Studied by measuring timing latencies for categorization.\n",
        "Psychological findings on US participants:\n",
        "• African-American names are associated with unpleasant words (more\n",
        "than European-American names)\n",
        "• Male names associated more with math, female names with arts\n",
        "• Old people's names with unpleasant words, young people with pleasant\n",
        "words.\n",
        "\n",
        "Conclusion\n",
        "Embeddings = vector models of meaning\n",
        "• More fine-grained than just a string or index\n",
        "• Especially good at modeling similarity/analogy\n",
        "• Just download them and use cosines!!\n",
        "• Can use sparse models (tf-idf) or dense models (word2vec, GLoVE)\n",
        "• Useful in practice but know they encode cultural stereotypes\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",\n",
        "\"\": \"\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "cellView": "form",
        "outputId": "7c996ef6-bba3-454b-d9d1-db86d4931b9d"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 14 Lecture Notes\", module14_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 14 Lecture Notes\nHow should we represent the meaning of a word?\nIn N-gram LMs we represented words as a string of letters or as an index in a\nvocabulary list.\nIdeally, we want a meaning representation to encode:\n1. Synonyms – words that have similar meanings\n2. Antonyms – words that have opposite meanings\n3. Connotations – words that are positive or negative\n4. Semantic Roles – buy, sell, and pay are different parts of the same underlying\npurchasing event\n5. Support for entailment\nMost words don’t have many synonyms, but they do have a lot of similar words. Cat is\nnot a synonym of dog, but cats and dogs are certainly similar words. Context windows\nOccur in a window of +/- 2 words, in the same sentence, in the same document\nInstead of window of words use more complex contexts:\ndependency patters. Subj-of-verb, adj-mod, obj-of-verb\nLanguages have long distance dependencies.\nAlternates to counts\nRaw word frequency is not a great measure of association between words. It’s very\nskewed “the” and “of” are very frequent, but maybe not the most discriminative\nWe’d rather have a measure that asks whether a context word is particularly\ninformative about the target word.\nInstead of raw counts, it’s common to transform vectors using TF-IDF or PPMI\nSparse v. Dense Vectors\nCo-occurrence matrix (weighted by TF-IDF or mutual information)\n• Long (length |V| = 50,000+)\n• Sparse (most elements are zeros)\nAlternative: learn vectors that are\n• Short (length 200-1000)\n• Dense (most elements are non-zero)\nLanguage modeling Goal: Learn a function that returns the joint probability\nPrimary difficulty:\n1. There are too many parameters to accurately estimate. This is\nsometimes called the “curse of dimensionality”\n2. In n-gram-based models we fail to generalize to related words / word\nsequences that we have observed.\n\nIn LMs we user chain rule to get the conditional probability of the next\nword in the sequence given all of the previous words:\n\nWhat happens when we have a new (unseen) combination of n words?\n1. Back-off\n2. Smoothing / interpolation\nWe are basically just stitching together short sequences of observed\nwords.\n\nNeural Language models have several advantages over n-gram LMs:\n1. They don’t need smoothing\n2. They can handle much longer histories.\n3. They can generalize over contexts of similar words.\n4. Neural LMs tend to have much higher predictive accuracy than n-gram LMs.\nDisadvantage: slower to train than traditional n-gram LMs\n\nAdvantages of NN LMs\nBetter results. They achieve better preplexity scores than SOTA n-gram LMs.\nLarger N. NN LMs can scale to much larger orders of n. This is achievable because\nparameters are associated only with individual words, and not with n-grams.\nThey generalize across contexts. For example, by observing that the words blue,\ngreen, red, black, etc. appear in similar contexts, the model will be able to assign a\nreasonable score to the green car even though it never observed in training, because it\ndid observe blue car and red car.\nA by-product of training are word embeddings\n\nA Neural Probabilistic LM\nGiven:\nA training set w1 … wt where wt ∈V\nLearn:\nf(w1 … wt) = P(wt|w1 … wt-1)\nSubject to giving a high probability to an unseen text/dev set (e.g. minimizing\nthe perplexity)\nConstraint:\nCreate a proper probability distribution (e.g. sums to 1) so that we can take\nthe product of conditional probabilities to get the joint probability of a\nsentence\n\nSummary: How to learn word2vec (skip-gram) embeddings\nStart with V random 300-dimensional vectors as initial\nembeddings\nUse logistic regression\n• Take a corpus and take pairs of words that co-occur as positive\nexamples\n• Take pairs of words that don't co-occur as negative examples\n• Train the classifier to distinguish these by slowly adjusting all the\nembeddings to improve the classifier performance\n• Throw away the classifier code and keep the embeddings.\n\nIn a word-analogy task we are given two pairs of words that share a\nrelation (e.g. “man:woman”, “king:queen”).\nThe identity of the fourth word (“queen”) is hidden, and we need to infer\nit based on the other three by answering\n“man is to woman as king is to — ?”\nMore generally, we will say a:a∗ as b:b∗.\nCan we solve these with word vectors?\n\nImplicit Association test (Greenwald et al 1998): How associated are\n• concepts (flowers, insects) & attributes (pleasantness, unpleasantness)?\n• Studied by measuring timing latencies for categorization.\nPsychological findings on US participants:\n• African-American names are associated with unpleasant words (more\nthan European-American names)\n• Male names associated more with math, female names with arts\n• Old people's names with unpleasant words, young people with pleasant\nwords.\n\nConclusion\nEmbeddings = vector models of meaning\n• More fine-grained than just a string or index\n• Especially good at modeling similarity/analogy\n• Just download them and use cosines!!\n• Can use sparse models (tf-idf) or dense models (word2vec, GLoVE)\n• Useful in practice but know they encode cultural stereotypes\n\n\n## Module 14 Definitions\n* **** - \n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T3cm1WV542_a",
        "outputId": "a870e3d9-c295-4f9f-f582-fef4a36d25fd"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,\n",
        "                             module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n An makes a decision: has avaliable some actions a, assigns a probability\nP(s) to each possible state. The transition model is given by P(s'|s,a) the probability that \naction a in state s reaches state s'  $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$  An agent's preferences \nare captured by a utility function. Expected utility:  $EU(a)) = \\sum_s P(\\textnormal{RESULT}(a)=s')U(s')$  MEU:  $action = \\textnormal{argmax}_{a}EU(a)$  The MEU\nformalizes the geneearal notion that an intelligent agent should do the right thing but it \ndoes not operationalize that advice. An agent may not know how good a state is until it\nknows where it can go from that state. So if an agent acts so as to maximize a utility function\nthat correctly reflects the performance measure, then the agent will achieve the highest \npossible perfomace score. \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n An agent's preferences (handwritten). Lottery (handwritten). All formulas\nare also handwritten. The axioms of utility theory are really axioms about prefereneces. But \nwe can derrive the existence of a utility function, and the expected utility of a lottery.\nThe outcome of a nondeterministic action is a lottery, an agent can only act rationally by \nchoosing an action that maximizes expected utility. In a deterministic environment an agent\nneeds only a preference ranking on states- the numbers dont matter\n\n## AIMA Chapter 16.3 Utility Functions\n Utility functions map from lotteries to real numbers. \n\n16.3.1 Utility assessment and utility scales\nTo built decision-theoretic system that helps a human make decisions. We must find \nthe human's utility function, preference elicitation. We can assist the utility of \nany prize S by asking teh agentt to choose between S and a standard lottery. \n\n16.3.2 The utility of money\nIf an agent prefers more money the agent has a monotonic preference for \nmoney. The utility of money was almost exactly proportional to the logarithm of the amount. \n\n16.3.3 Expected utility and post-decision disappointment:\nThe rational way to choose the best action a*, is to maximize the expected utility: a* = argmax_aEU(a)\nProbability density function $P(x) = \\frac{d}{dx}((F(x)^{k})=kf(x)(F(x))^{k-1}$  With\nmore choices, extremely optimistic estimates are more likely to arise. We can avoid hte optimizer's curse\nwith a Bayesian approach that uses and explicit probability model P(EU^|EU). \n\n16.3.4 Human judgment and irrationality\nDecision theory is a normative theory. Humans are \"predicatably irrational. An explanation\nfor the apparently irrational preference is the certainty effect. People feel more comfortable\nmaking relative utility judgements rather than absolute ones. The very fact that different\nwordings of the same decision problem elicit different choices suggests that the decision problem itself is not getting through.\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n If the environment were deterministic, a solution would be easy.\nBut the environment is stochastic so it's actions are unreliable. The transition model\ndescribes the outcome of each action in each state. P(s'|s,a) is the probability of reaching\nstate s' if action a is done in state s. If a decision problem is sequential the utility\nfunction will depend on an environment history. For every transition from s to s' via a the\nthe agent recieves a reward R(s,a,s'), they may be + or - but are bounded by +/- Rmax.\nThe utility of an environment history is the sum of the rewards recieved. Methods for solving\nMDPs usually involve dynamic programming. No matter what the outcome of the action, the\nresulting state will be in the policy, and the agent will know what to do next. The quality\nof a policy is measured by the expected utility of the possiblke environment histories genterated\nby it. Optimal policy is denoted by pi*.\n\n17.1.1 Utilities over time\nFirst check if there is a finite horizon or an infinite horizon for decision making. \nWith a finite horizon, an optimal action in a given state may depend on the time left.\nWith no fixed time limit there is no reason to behave differently in the same state at differetn times\nWith additive discounted rewards the utility of a history is: $U_{h}([s_{0},a_{0},s_{1},a_{1},...]) = R(s_{0},a_{0},s_{1})+\\gamma R(s_{1},a_{1},s_{2})+\\gamma^{2} R(s_{1},a_{1},s_{2})$ three \nsolutions to equations that lead to infinite histories: 1. with discounted\nrewards the utility of an infinite sequence is finite 2. if the environment has terminal\nstates and we are guaranteed to get on eeventually then we will never need to compare infinite\nsequences 3. infinite sequences can be compared in terms of the average reward obtained \nper time step.\n\n17.1.2 Optimal policies and the utilities of states\nCompare policies by comparing their expected utilities them.  $U^{\\pi}(s) = E[\\sum_{t=0}^{\\infty} (\\gamma^{t}R(S_{t},\\pi(S_{t}),S_{t+1})]$ Optimal \npolicy is $\\pi_{s}^{*}=(\\textnormal{argmax}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ Consequence of using discounted utilities with infinite horizons is\nthe optimal policy being independent of the starting state. The utility of a state is the expected reward for the\nnext transition plus the discounted utility of the next state Bellman equation: $U(s)= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ Q \nfunction: $U(s)= \\textnormal{max}_{a}Q(s,a)$  We can extract the optimal policy from the q\nfunction $\\pi ^{*}= \\textnormal{argmax}_{a}Q(s,a)$  ``` \nfunction Q-value(mdp, s,a,U) returns a utility value\n  return sum_s' P(s'|s,a){[R(s,a,s')+gammaU[s']] ```\n  \n17.1.4 Representing MDPs\nDynamic decision networks, or DDNs. They typically have an exponential complexity advantage\nover atomic representations and can model quite substantial real-world problems.\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Bellman update:  $U(s)_{i+1}= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U_{i}(s')])$ is to be applied simultaneously to \nall states at each iteration\n``` \n function Value-Iteration(mdp, e) returns a utility function\n    inputs mdp, with states S, actions A(s) transistion model P(s'|s,a), rewards R(s,a,a') discount gamma\n      e, the maximum error allowed in the utility of any state\n    local variables: U,U', vectors of utilities for stares in S, initially zero\n        delta, the maximum relative change in the utility of any state\n    repeat\n      U<- U'; delta <-0\n      for each state s in S do\n        U'[s]<-max_aeA(s) Q-Value(mdp,s,a,U)\n        if|U'[s]-U[s]| > delta then delta <-|U'[s] - U[s]|\n      until delta<= e(i-gamma)/gamma\n      return U\n``` \nValue iteration eventually converges to a unique set of solutions to the bellman equation. Contraction\nhas one fixed point, when applied the function must get closer to the fixed point.\n\nPolicy iteration stops when the policy improvemtn step yields no change in utilities\n``` \n function Policy-Iteration(mdp) returns a policy\n  inputs: mdp wiht stares, actions, transition model\n  local variables: U, a vector of utilites for states in S (initially 0)\n      pi, policy vector indexed by state, initially random\n\n  repeat\n    U<-Policy-Evaluation(pi, U, mdp)\n    unchanged <- true\n    for each state s in S do\n      a*<- argmax Q-value(mdp,s,a,U)\n      if Q-Value(mdp,s,a*,U)> Q-Value(mdp,s,pi[s],U) then\n        pi[s] <- a*; unchanged? <- false\n    until unchanged?\n    retunr pi\n``` \nAsynchronous policy iterstion is guaranteed to converege to an optimal policy. Linear Programming\nseeks to minimize U(s) for all s subject to the inequalities.\n\n\n## Module 8 Lecture Notes\n In our search algorithms so far, the\ntransition model was deterministic and\ndescribed the outcome of each action in\neach state.\nThe transition function is sometimes\nwritten as T(s, a, s’), or explicitly as a\nprobability:\n\nReward function\nWe will specify a utility or reward function for the agent.\nThe “rewards“ can be positive or negative but are bounded by some maximum value.\nBecause the decision process is sequential, we must specify the utility function on a\nsequence of states and actions.\nInstead of only giving a reward at the goal states, the agent can receive a reward at\neach time step, based on its transition from s to s’ via action a.\nThis is defined by a reward function\nR( s, a, s’ )\nFor example, we could give the Millennium Falcon a small negative reward of -0.04 for every\ntransition except for entering the terminal states (+1 for entering the planet’s orbit or -1 for\nsmashing into an asteroid).\nThe rewards are additive, so if the Millennium Falcon takes 4 steps before entering\nthe planet's orbit, it gets -0.04 + -0.04 + -0.04 + -0.04 + 1 = 0.84 for that solution\n\nMarkov Decision Process\nA Markov decision process or MDP is\n• a sequential decision problem\n• for a fully observable environment\n• with a stochastic transition model\n• that has additive rewards\nMDPs are non-deterministic search\nproblems. One way of solving them is via\nexpectimax search.\n\nMarkov Decision Process\nTo find a solution to an MDP, you need to define the following things:\n• A set of states s ∈ S\n• A set of actions a ∈ A\n• A transition function T(s, a, s’)\n• Probability that executing action a in s will lead to s’ P(s’ | s, a)\n• The probability is called the model\n• A reward function R(s, a, s’)\n• Sometimes just R(s) or R(s’)\n• An initial state s0\n• Optionally, one or more terminal states\n\nSolution == Policy\nIn search problems a solution was a\nsequence of action that corresponded to\nthe shortest path.\nBecause of the non-determinism in MDPs\nwe cannot simply give a sequence of\nactions.\nInstead, the solution to an MDP is a policy.\nA policy maps from a state onto the action\nto take if the agent is in that state.\n𝝿(s) = a\n\nOptimal Quantities\n▪ The value (utility) of a state s:\nV*(s) = expected utility starting in s and\nacting optimally\n▪ The value (utility) of a q-state (s,a): Q*(s,a) = expected utility starting out\nhaving taken action a from state s and\n(thereafter) acting optimally\n▪ The optimal policy:\nPi*(s) = optimal action from state s\n\nPolicy Iteration\nAlternative approach for optimal values:\n• Step 1: Policy evaluation: calculate utilities for some fixed policy (not\noptimal utilities!) until convergence\n• Step 2: Policy improvement: update policy using one-step lookahead with resulting converged (but not optimal!) utilities as future\nvalues\n• Repeat steps until policy converges\nThis is policy iteration\n• It’s still optimal!\n• Can converge (much) faster under some conditions\n\nComparison\nBoth value iteration and policy iteration compute the same thing (all optimal\nvalues)\nIn value iteration:\n• Every iteration updates both the values and (implicitly) the policy\n• We don’t track the policy, but taking the max over actions implicitly recomputes\nit\nIn policy iteration:\n• We do several passes that update utilities with fixed policy (each pass is fast\nbecause we consider only one action, not all of them)\n• After the policy is evaluated, a new policy is chosen (slow like a value iteration\npass)\n• The new policy will be better (or we’re done)\nBoth are dynamic programs for solving MDPs\n\nSummary: MDP Algorithms\nSo you want to….\n• Compute optimal values: use value iteration or policy iteration\n• Compute values for a particular policy: use policy evaluation\n• Turn your values into a policy: use policy extraction (one-step\nlookahead)\nThese all look the same!\n• They basically are – they are all variations of Bellman updates\n• They all use one-step lookahead expectimax fragments\n• They differ only in whether we plug in a fixed policy or max over\nactions\n \n\n## Module 8 Definitions\n* **Utility function** - assigns a single number to represend the desrability of a state\n* **Expected utility** - the average utility value of the outcomes, weighted by the probability that the outcome occurs\n* **Maximum expected utility (meu)** - a rational agent should choose the action that maximizes the agent's expected utility\n* **Lottery** - the set of outcomes for each action\n* **Orderability** - given 2 lotteries, a rational agent must either prefer\n* **Transitivity** - given any 3 lotteries if an agent preferes A to B and B to C then the agent must prefer A to C\n* **Continuity** - probability p the agent is indifferent between B and L that yields A with p and C with p-1\n* **Substitutibility** - if agent is indifferent between A and B then also indifferent between more complex lotteries with b sub a\n* **Monotonicity** - if an agent prefers A to B then the agent must prefer the lottery that has a higher prob for A\n* **Decomposability** - compound lotteries can be reduced to simpler ones using the laws of probability\n* **Existence of utility function** - if agent's preferences obey axioms of utility then there is func U that obeys\n* **Expected utility of a lottery** - the utility of a lottery is the sum of the probabilities of each outcome times their utilities\n* **Value function** - preference ranking on states\n* **Ordinal utility function** - value function\n* **Preference elicitation** - present choices to the human and using the observed preferences to find the utility function\n* **Normalized utilities us a scale with worst = 0 and best = 1** - \n* **Value of a statistical life** - determines the costs and benefits of regulations and interventions\n* **Micromort** - one in a million chance of death\n* **Qaly** - quality adjusted life year\n* **Risk-averse** - prefer a sure thing with a payoff that is less than the EMV of a gamble\n* **Risk-seeking** - the opposite of risk averse\n* **Certainty equivalent** - the value an agent will accept in lieu of a lottery\n* **Insurance premium** - the difference between the EMV of a lottery and its certanty equivalent\n* **Risk-neutral** - an agent that has a linear curve\n* **Unbiased** - the expected value of the error is zero\n* **Order statistic** - the distrobution of any particular ranked element of a sample\n* **Optimizer's curse** - the tendency for the estimated expected utility of the best choice to be too high\n* **Normative theory** - describes how a rational agent should act\n* **Descriptive theory** - describes how actual agents really do act\n* **Certainty effect** - people are strongly attracted to gains that are certain\n* **Ambiguity aversion** - most people elect the known proabability rather than the unknown unknowns\n* **Framing effect** - the exact wording of a decision problem can have a big impact on the agent's choices\n* **Anchoring effect** - proposing something outrageous so that settling with somthing slightly crazy isnt crazy\n* **Fully observable** - if the agent's sensors give it access to the complete state of the environment at every point in time\n* **Markovian** - the probability of reaching s' from s depends only on s not the history\n* **Environment history** - a sequence of states and action\n* **Markov decision process mdp** - sequential decision problem for a fully observable, stochastic environment with Markovian transition model and additive rewards\n* **Dynamic programming** - simplifying a problem by recursively breaking it into smaller pieces\n* **Policy** - what an agent should do for any state that the agent might reach\n* **Optimal policy** - yields the highest expected utility\n* **Finite horizon** - there is a fixed time N after which nothing matters-the game is over\n* **Nonstationary** - a policy that depends on the time\n* **Stationary** - optimal policy depends only on the current state\n* **Discount factor (gamma)** - [0,1] describes the agent's preferences for current rewards over future rewards\n* **Additive reward** - when gamma is 1\n* **Attribute** - each transition s_t -> s_t+1 is an attribute of the the history\n* **Propper policy** - a policy that is guaranteed to reach a terminal state\n* **Q-function (action-utility function)** - expected utility of taking a given action in a given state\n* **Sparse** - when most entries in MDP tables are zerso\n* **Dynamic decision network** - factored representations, \n* **Factored representations** - \n* **Value iteration** - n possible states then n bellman equations\n* **Contraction** - function of one argumentthat when applied to 2 inputs produces 2 closer together outputs\n* **Max norm** - measures the distance between utility vectors, measures the length of a vector by abs value of its biggest component.\n* **Policy loss** - ||U^pi_i-U||\n* **Policy iteration** - alternates  policy evaluation and policy improvent starting at an initial policy pi0\n* **Policy evaluation** - given pi_i calculate Ui=U^pi_i the utility of each state if pi_i were to be executed\n* **Policy improvement** - calculate a new MEU policy pi_i+1, using one step look ahead based on Ui\n* **Modified policy iteration** - uses simplified bellman equation\n* **Asynchronus policy iteration** - pick any subset ofof states and apply either kind of updating (policy improvement or value iteration)\n* **Linear programming** - general approach for formulating constrained optimization problems\n* **Real time dynamic programming** - like minimax, alternates max and chance nodes\n* **** - \n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n There is a tradeoff between exploitation of the current best option\nand exploration of previously unknown states to gain information. There are several\ndefinitions of bandit problems. The most general: each arm M is a Markov reward process. \nWith states Si, transition model Pi, and reward Ri. The arm defines a distrobution over\nsequences of rewards Ri0, Ri1,... A policy for any bandit problem: pull the arm that\nhas the highest gittens index then update the gittens indicies.\n\n17.3.2 The Bernoulli bandit\nThe state of arm Mi is defined by si and fi, the counts of successes and failures,\nthe transition probability predicts the next outcome to be 1 with probability (si)/(si+fi)\nand 0 with probability (fi)/(si+fi). There is an exploration bonus associated with arms\nthat have only been tried a few times\n\n17.3.3 Approximately optimal bandit policies\nFirst class of methods use the upper confidence bound.\nThompson sampling chooses an arm randomly according to the probability that the arm is\nin fact optimal\n\n17.3.4 Non-indexable variants\nSelection problem - making a decision in the midst of conditions. \nthe ordinary term for BSPs is multitaksing. We can take advantage of the loose nature\nof the interaction between the arms. This can be modeled by opportunity cost.\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n Reinforcement learning is an alternative to needing huge databases of \nmoves and their actions. Reinforcement learning differes from just solving the MDP because\nthe agent is not given the MDP to solve, the agent is in the MDP. If we can provide the\ncorrect reward signal to the agent reinforcement learning provides a very general way to build AI\nsystems. Especially in simulated environmentes when there is no shortage of opportunities\nto gain experience. Some key basic reinfrocement learning approaches are:\n\nModel-based reinforcement learning: agent uses a transition model of the environment to \nhelp interpret the reward signals and decide how to act. If the model is unknow the agent\nlearns it by observing the effects of its actions\n\nModel-free reinforcement learning: The agent never learns or knows the transition model.\nIt learns a more direct representation of how to behave. In one of 2 varieties: 1) action \nutility learning like Q learning. Where the agent learns a Q-function then chooses what to\ndo in s by finding the action with the highest q. 2) policy search: the agent learns a policy pi(s)\nthat maps directly from states to actions. this is a reflex agent\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Passive learning is similar to policy evaluation, except passive \nlearning agent does not know the reansistion model or the reward function. The expected \nutility associated with each state s: $U^{\\pi}(s) = E[\\sum_{t=0}^{\\infty} (\\gamma^{t}R(S_{t},\\pi(S_{t}),S_{t+1})]$ \n\n22.2.1 Direct utility estimation \nReduces reinfrocement learnign to a standard supervised learning problem in which\neach example is a (state, reward-to-go) pair. The utility of a state is determined by the\nreward and expected utility of the successor states-> the utility obeys the bellman equation.\nBy ignoring the connections between states, direct utility estimation misses opportunities\nfor learning.\n\n22.2.2 Adaptive dynamic programming\nTakes advantage of the constraints among the utilities of states by learning the transition\nmodel that connects them, then solving the corresponding Markov decision process using\ndynamic programming. For a passive learning agent we would plug the learned transition model and the\nobserved rewards into the bellman equation.\n```\nfunction Passibe-ADP-Learner(precept) returns an action\n  inputs: percept, a percept indicating the current state s' and reward signal r\n  persistent: pi, a fixed policy\n              mdp, an MDP with model P, rewards R, actions A, discount gamma\n              U, a table of utilities for states, initially empty\n              Ns'|sa a table of outcome count vectors indexed by state and action, initaially zero\n              s, a the previous state and action, initailly null\n  if s' new then U[s']<-0\n  if s not null then \n    increment Ns'|s,a[s,a][s']\n    R[s,a,s']<-r\n    add a to A[s]\n    P( |s,a)<-NORMALIZE(N_s'|s,a [s,a])\n    U<- POLICY EVALUATION(pi, U, mdp)\n    s,a <- s', pi[s']\n    return a\n```\n22.2.3 Temporal difference learning\nYou can also use the observed transistions to adjust the utilites of the oberved\ntransitions to adjust the utiliteis of the oberved states so that they agree with\nthe constraint equations. When the transition occurs apply this update\n $U^{\\pi}(s)\\leftarrow ^{\\pi} \\alpha[R(s',s,a)+\\gamma U^{\\pi}(s')-U^{\\pi}(s)]$  where alpha is the learning rate. This is know as the temporal-difference\nequation.\n```\nfunction Passive-TD-Learner(percept) returns an action\n  inputs: percept, a percept indicating the current state s' and reward signal r\n  persistent: pi, a fixed policy\n              s, the previous state, initially null\n              U, a table of utilities for states, initially empty\n              Ns, a table of frequencies for states, initially zero\n  if s' is new then U'[s']<-0\n  if s is not null then \n    increment Ns[s]\n    temporal difference equation\n  s<-s'\n  return pi[s']\n```\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n An active learning agent gets to decid what actions to take. Adaptive\ndynamic programming (ADP) agent: first needs to learn a complete transition model with \noutcome probabilities for all actions, next take into account the fact that hte agent \nhas a choice of actions. The utilities it needs to learn are those defined by the optimal\npolicy (obey the bellman equation). With U the agent can extract an optimal action by \none-step look-ahead to maximize the expected utility. If it uses policy iteration it could simply\nexecute the action the optimal policy recommends.\n\n22.3.1 Exploration\nSometimes greed pays off but it often does not. A learned model is not the same as\nthe true environment, what is learned could be suboptimal in the real environment. Greedy\nagent overlooks the fact that actions provide more than rewards they provide information\nin the form of percepts in the resulting state. An ADP using a GLIE scheme will eventually learn \nthe true transition model. A good approach is to give some weight to actions the agent has not\ntried very often while avoiding those that are believed to be of low utility.\n\n22.3.2 Safe exploration\nMany actions are irreversible, in the sense defined for online search agents: no set of \nactions can restore it to where it was before. Worst case the agent enters an absorbing state.\nA better idea to avoid catastrophe is to choose a plicy that works reasobably for a range of \nmodels that have a chance of being true, even if it is suboptimal for the maximum-likelihood model.\nThere are 3 mathematical approaches for this: 1) Bayesian reinforcement learning: posterior probablility P(h|e) is \nobtatined by bayes' rule given the observations to date. $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\sum_{h}P(h|e)U_{h}^{\\pi})$ 2) robust control theory allows for a\nset of possible models H wihtout assigning probabilities to them adn defines an optimal\nrobust policy as on that gives the best outcome in the worst case over H: $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\textnormal{min}_{h}U_{h}^{\\pi})$  This\nis often the set of models that exceed some likelihood threshold on P(h|e). The problem is that\nthis results in overly conservative behavior.\n\n22.3.3 Temporal-difference Q-learning\nThis avoids the need for a model by learning an action-utility function instead of a utility\nfunction. Q(s,a) denotes the expected total discounted reward if the agent takes action a in s\nand acts optimally after.\n\n``` \nfunction Q-learning-agent(percept) returns an action\n  inputs: percepts, a percept indicating teh current state s' and reward signal r\n  persistent: Q, a table of action values indexed by state adn action, initially zero\n              Nsa, a table of frequencies for state-action pairs, initially zero\n              s, a ther previous stat and action, initially null\n  if s is not null then\n    increment Nsa[s,a]\n    Q[s,a]<- Q[s,a] + alpha(Nsa[s,a])(r + gamma*max_a' Q[s',a']-Q[s,a]\n  s,a <- s', argmax_a' f(Q[s',a'], N_sa[s',a'])\n  return a\n``` \n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Evaluation functions are approximate utility functions.\nAdding look-ahead search means that effective behavior can be generated from a much simpler\nutility function approximator that is learnable from far fewer experiences. Allows for inductive\ngeneralization: the agent can generalize from states it has visited to states it has not. Backgamon!\n\n22.4.1 Approximating direct utility estimation\nThe state and the sum of rewards received constitute a training example for a supervised learning\nalgorithm. Reinforcement learning makes more sense to use an online learning algorithm that\nupdates after each trial. The rate of change of the error wrt each parameter (theta_i)\nis partial dEj/dthetai. Changing the parameters theta_i in response to an \nobserved transition between two states changes the values of U^theta for every other state!\nLinear function approximation:the function must be linear in the features. But these can\nbe arbitrary nonlinear functions of the state variables.\n\n22.4.2 Approximaating temporal-difference learning\nAdjust the parameters to try to reduce the temporal difference between successive states\nthe update rule can be shown to converge to the closest possible approximation to the\ntrue function when the function approximator is linear in the features. With active\nlearning and nonlinear functions such as neural networks, tricky.\nLearning a model for an observable environment is a supervised learning problem,\nbecause the next percept gives the outcome state. Partially observable environments have \na much harder learning problem. The next percept is no longer known.\n\n22.4.3 Deep reinforcement learning\nIt is always possible to represent U or Q as linear combinations of features, if they aren't\nin an efficiently computible form the linear function approximator\nmay be insufficient. The parameters are the layers of the network. Obstacles:\nit is often difficult to get good performance and the trained system may behave\nvery unpredictably if the environment differs even a little from the training data.\n \n22.4.4 Reward shaping\nReal world environments may have sparse rewards: many actions are required\nto achieve any nonzero reward. R'(s,a,s') = R(s,a,s') + gamma*phi(s') - phi(s)\nphi can be constructed to reflect any desirable aspects of the state.\n\n22.4.5 Hierarchical reinforcement learning \nDeal with long action sequences by breaking them into small pieces, then again until\nthey are short enough to make learning easy. Begins with a partial program. The theoretical\nfoundations of HRL are based on the concept of the joint spate space. This is all essentially\nlike solving an MDP with: states->choice states sigma of the joint space, the \nactions at sigma are the choices avaliable in sigma from the partial program, the \nreward functionp(sigma,c,sigma') is the expected sum of rewards for all physical transition, the\ntransition model t(sigma,c,sigma') -> if c invokes a physical\naction a, then t borrows from the physical model P(s'|s,a)\n\n## Module 9 Lecture Notes\n That wasn’t planning, it was learning!\nSpecifically, reinforcement learning\nThere was an MDP, but you couldn’t solve it with just computation\nYou needed to actually act to figure it out\nImportant ideas in reinforcement learning that came up\nExploration: you have to try unknown actions to get information\nExploitation: eventually, you have to use what you know\nRegret: even if you learn intelligently, you make mistakes\nSampling: because of chance, you have to try things repeatedly\nDifficulty: learning can be much harder than solving a known MDP\n\nReinforcement Learning\nStill assume a Markov decision process (MDP):\nA set of states s ∈ S\nA set of actions (per state) A\nA model T(s,a,s’)\nA reward function R(s,a,s’)\nStill looking for a policy π(s)\nNew twist: don’t know T or R\nI.e. we don’t know which states are good or what the actions do\nMust actually try actions and states out to learn\n\nModel-Based Learning\nModel-Based Idea:\nLearn an approximate model based on experiences\nSolve for values as if the learned model were correct\nStep 1: Learn empirical MDP model\nCount outcomes s’ for each s, a\nNormalize to give an estimate of T\nDiscover each R when we experience (s, a, s’)\nStep 2: Solve the learned MDP\nFor example, use value iteration, as before\n\nPassive Reinforcement Learning\nSimplified task: policy evaluation\nInput: a fixed policy π(s)\nYou don’t know the transitions T(s,a,s’)\nYou don’t know the rewards R(s,a,s’)\nGoal: learn the state values\nIn this case:\nLearner is “along for the ride”\nNo choice about what actions to take\nJust execute the policy and learn from experience\nThis is NOT offline planning! You actually take actions in the world.\n\nHow to Explore?\nSeveral schemes for forcing exploration\nSimplest: random actions (ε-greedy)\nEvery time step, flip a coin\nWith (small) probability ε, act randomly\nWith (large) probability 1-ε, act on current policy\nProblems with random actions?\nYou do eventually explore the space, but keep\nthrashing around once learning is done\nOne solution: lower ε over time\nAnother solution: exploration functions\n\nEven if you learn the optimal policy,\nyou still make mistakes along the way!\nRegret is a measure of your total\nmistake cost: the difference between\nyour (expected) rewards, including\nyouthful suboptimality, and optimal\n(expected) rewards\nMinimizing regret goes beyond\nlearning to be optimal – it requires\noptimally learning to be optimal\nExample: random exploration and\nexploration functions both end up\noptimal, but random exploration has\nhigher regret\n\n\n## Module 9 Definitions\n* **One armed bandit** - slot machine\n* **Exploitation** - taking advantage of the current best option\n* **Exploration** - an agent must explore lots of its terrain to learn how to behave in it\n* **Markov reward process** - mdp with only one possible action ai\n* **Stopping time** - if you pull a arm T times the stopping time is T\n* **Bernoulli bandit** - each arm Mi produces a reward of 0 or 1 with fixed but unknown probability mu_i\n* **Upper confidence bound** - use the samples from each arm to establish a confidence interfal for the value of the arm\n* **Confidence interval** - range within which the value can be estimate to lie with high confidence\n* **Bandit superprocess** - each arm is a full MDP in its own\n* **Opportunity cost** - how much utility is given up per time step by not devoting that time toanother arm\n* **Reinforcement learning** - an agent interacts with the world and periodically recieves rewards or reinforcements\n* **Rewards** - reflect how well the agent is doing\n* **Reinforcements** - psychology term for rewards\n* **Passive reinforcement learning** - the agent's policy is fixed and the task is to learn the utilities of the states\n* **Active reinforcement learning** - the agent must also figure out what to do: exploration\n* **Apprenticeship learning** - training a learning agent using demonstrations rather than reward signals\n* **Direct utility estimation** - the utility of a state is defined as the expected total rewardfrom that state on\n* **Reward-to-go** - expected total reward from that state onward\n* **Temporal difference methods** - adjust the utility estimate towards the ideal equilibrium\n* **Pseudoexperience** - generated by simulating the current transition model\n* **Prioritized sweeping** - heuristic prefers to make adjustments to states whose likely successors have just undergone a large adjustment in their own utility estimates.\n* **Active learning agent** - decides what actions to take\n* **Greedy agent** - takes the action that it currently believes to be optimal at each step\n* **Glie (greedy in the limit of infinite exploration)** - must try each action in each state an unbounded number of times to avoid missing an optimal action\n* **Exploration function** - f(u,n) bellman equation, number of times a state has been tried\n* **Absorbing stare** - no actions have any effect and no rewards are recieved\n* **Sarsa (state action reward state action)** - updates with the Q-value of the action a'\n* **Off-poliicy** - learns q values that answer what would this be worht in this state, assuming i stop using the current policy?\n* **On-policy** - what would this action be worth in the state, assuming i stick with my policy\n* **Bayesian reinforcement learning** - assumes prior probability P(h) over hypothesis h about what the model is\n* **Function approximation** - process of constructing a compact approximation of the true utility function or Q-function\n* **Widrow-hoff rule** - delta rule equation 22.10\n* **Catastrophic forgetting** - a nonzero weight on a inear feature can cause a car to slide bc it forgets danger\n* **Experience replay** - ensures the car relives its youthful crashing behavior so it doesnt forget\n* **Deep reinforcement learning system** - reinforcement learning system that uses a deep network as a function approximator\n* **Reward shaing** - supplying teh agent with additional rewards for making progress\n* **Pseudorewards** - extra rewards\n* **Partial program** - outlines a hierarchial structure for the agent's behavior\n* **Joint state space** - each state (s,m) is composed of a physical state and a machine state\n* **Choice state sigma=(s,m)** - program counter for m is at a choice point in the agent program\n* **Additive decomposition** - overall utility decomposes into several terms, depeding on a few variables\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Agents in the real world need to handle uncertainty. We have seeen\nagents handle this with a belief state. This works on simple problems but has drawbacks:\nthe agent must consider every possible explanation for its sensor observation, a correct\ncontingent plan must consdider arbitarily unlikely contigencies, sometimes there is no plan\nguaranted to achieve the goal- but the agent must act. The right thing to do—the rational\ndecision—therefore depends on both the relative importance of various goals and the\nlikelihood that, and degree to which, they will be achieved.\n\n12.1.1 Summarizing uncertainty\nTrying to use logic to cope wiht domain like medical diagnosis fails for 3 main reasons:\n1) laziness 2) theoretical ignorance 3) practical ignorance. Agent's knowledge can at best\nprovide only a degree of belief. This can be dealt with probability theory. The ontological \ncommitments of logic and probability theory are the same. The theory of probability provides\na way of summarizing the uncertainty that comes from our laziness and ignorance.\n\n12.1.2 Uncertainty and rational decisions\nTo make choices and agent must have preferences among the different possible outcomes of \nthe various plasn. Use utility theory to represent preferences and reason. A utility\nfunction can account for any set of preferences—quirky or typical, noble or perverse. \nDecision theory = probability theory + utility theory. An agent is rational if and only\nif it chooses the action that yields the highest expected utility, averaged over all\nthe possible outcomes of the action.\n\n``` \nfunction DT-agent(percept) returns and action\n  persistent: belief_state, probabilistic beliefs about the current state of the world\n              axtion, the agent's action\n  update belief_state based on action and percept\n  calculate outcome probabilities for actions\n    given action descriptions and current belief_state\n  select action with highest expected utility\n    given probabilites of outcomes and utility information\n  return action\n``` \n \n\n## AIMA Chapter 12.2 Basic Probability Notation\n For our agent to represent and use probabilistic information,\nwe need a formal language. But the language of probability theory has traditionally\nbeen informal.\n\n12.2.1 What probabilites are about \nprobabilistic assertions talk about how probable the various worlds are. A fully specified\nprobability model associates a numerical probability P(w) wiht each possibe word.\nevery possible world has a probability between 0 and 1 and that the total probability\nof the set of possible worlds is 1. In logic a set of words corresponds ot a proposition\nin formal language. Most of the time we have some evidence that has been uncovered.\n\n12.2.2 The language of propostitions in probability \npropositions describing sets of possible worlds are usually written in a notation\nthat combines elements of propositional logic and constraint satisfaction notation.\nIt is a factored represention in which a possibly word is represented by a set of variable/value \npairs. It can also be shown as a structure representation. Variables are called random variables\nFor continuous variables, it is not possible to write out the entire distribution as a vector,\nbecause there are infinitely many values. Instead, we can define the probability that a\nrandom variable takes on some value x as a parameterized function of x, usually called\na probability density function. A possible world is defined to be an assignment of values\nto all of the random variables under consideration.\n\n12.2.3 Probability axioms and their reasonableness\n\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\nSimple method for probabilittic inference. We use the full joint\ndistribution as the “knowledge base” from which answers to all questions may be derived. \nOne common practice is to extract the distrobution over some subset of variable(s).\nMarginalization adn conditioning are useful rules for all kinds of derivations involving\nprobability expressions \n\n## AIMA Chapter 12.4 Independence\n Independence assertions are usually based on knowledge of the domain.\nIf the complete set of variables can be divided into independent subsets, then the full\njoint distribution can be factored into separate joint distributions on those subsets.\nWhen they are available, then, independence assertions can help in reducing the size of\nthe domain representation and the complexity of the inference problem. Unfortunately,\nclean separation of entire sets of variables by independence is quite rare.  \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Bayes’ rule is useful in practice because there are many cases where we\ndo have good probability estimates for these three numbers and need to compute the fourth.\nOften, we perceive as evidence the effect of some unknown cause and we would like to\ndetermine that cause. a is cause b is effect. Causal direction P(effect|cause).\nDiagnostic direction P(cause|effect). diagnostic knowledge is often more fragile than\ncausal knowledge. If there is a sudden epidemic of meningitis, the unconditional\nprobability of meningitis, , will go up\n\n12.5.2 Using Bayes’ rule: Combining evidence\nTo make progress, we need to find some additional assertions about the domain that will\nenable us to simplify the expressions. The notion of independence in provides a clue,\nbut needs refining. These variables are independent, however, given the presence or\nthe absence of a cavity.expresses the conditional independence of toothache and catch given cavity. \nWe can plug it into  to obtain the probability of a cavity: (handwritten).\nIn general, for n symptoms that are all conditionally independent given cavity,\nthe size of the representation grows as O(n) instead of (O2^n). That means that conditional\nindependence assertions can allow probabilistic systems to scale up; moreover, they are\nmuch more commonly available than absolute independence assertions. Conceptually, \ncavity separates toothache and catch because it is a direct cause of both of them. \nThe decomposition of large probabilistic domains into weakly connected subsets through \nconditional independence is one of the most important developments in the recent history of AI.\n\n## AIMA Chapter 12.6 Naive Bayes Models\n Naieve bayes model, probability distrobution (written). \nIn practice, naive Bayes systems often work very well, even when the conditional independence\nassumption is not strictly true. The run time of this calculation is linear in the number\nof observed effects and does not depend on the number of unobserved effects.\n\n12.6.1 Text classification with naive Bayes \nHere the “cause” is the  variable, and the “effect” variables are the presence or\nabsence of certain key words, HasWordi. The naive Bayes model consists of the prior probabilities \nP(category) and the conditional probabilities P(HasWordi|Category). For each category c,\nP(category = c) is estimated as the fraction of all previously seen documents that are \nof category .\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n ure logical inference can conclude nothing about which square is most\nlikely to be safe, so a logical agent might have to choose randomly. We will see that a\nprobabilistic agent can do much better than the logical agent. The first term is the conditional \nprobability distribution of a breeze configuration, given a pit configuration; its values are \n1 if all the breezy squares are adjacent to the pits and 0 otherwise. The second term is the prior \nprobability of a pit configuration. Each square contains a pit with probability 0.2, \nindependently of the other squares. What this section has shown is that even seemingly complicated \nproblems can be formulated precisely in probability theory and solved with simple algorithms. \nTo get efficient solutions, independence and conditional independence relationships can be used \nto simplify the summations required. These relationships often correspond to our natural \nunderstanding of how the problem should be decomposed.\n\n## SLP Chapter 3.1 N-Grams\n P(w|h), the probability of a word w given some history h. One way\nis with relatice frequency counts. Take a large corpus, count the number of times we see\nh and count the number of times this is followed by w. even the web isn’t big enough to \ngive us good estimates in most cases. This is because language is creative; new sentences\nare created all the time, and we won’t always be able to count entire sentences.\nThe intuition of the n-gram model is that instead of computing the probability of\na word given its entire history, we can approximate the history by just the last few\nwords.The bigram model, for example, approximates the probability of a word given\nall the previous words P(wn|w1:n−1) by using only the conditional probability of the\npreceding word P(wn|wn−1). we are thus making the following approximation: P(wn|w1:n−1) ≈ P(wn|wn−1)\nThe assumption that the probability of a word depends only on the previous word is\nMarkov called a Markov assumption.An intuitive way to\nestimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood\nestimation the MLE estimate for the parameters of an n-gram model by getting counts from a\ncorpus, and normalizing the counts so that they lie between 0 and 1.\nFor the general case of MLE n-gram parameter estimation:\nP(wn|wn−N+1:n−1) = C(wn−N+1:n−1 wn)/C(wn−N+1:n−1) estimates the n-gram probability by dividing the\nobserved frequency of a particular sequence by the observed frequency of a prefix.\nThis ratio is called a relative frequency. Some of the bigram probabilities above encode some facts that we think of as strictly\nsyntactic in nature, like the fact that what comes after eat is usually a noun or an\nadjective, or that what comes after to is usually a verb. Others might be a fact about\nthe personal assistant task, like the high probability of sentences beginning with\nthe words I. And some might even be cultural rather than linguistic.\nWe always represent and compute language model probabilities in log format\nas log probabilities. Since probabilities are (by definition) less than or equal to log\nprobabilities 1, the more probabilities we multiply together, the smaller the product becomes\n\n## SLP Chapter 3.2 Evaluating Language Models\n The best way to evaluate the performance of a language model is to embed it in\nan application and measure how much the application improves (extrinsic evaluation).\nrunning big NLP systems end-to-end is often very expensive. Instead, it would be nice to have a metric \nthat can be used to quickly evaluate potential improvements in a language model. An \nintrinsic evaluation metric is one that measures the quality of a model independent of any application.\nwe need a test set. the probabilities of an n-gram model come from the corpus it is trained on, the training set or training corpus. We can then measure\nthe quality of an n-gram model by its performance on some unseen data called the\ntest set test set or test corpus. whichever model assigns a higher probability to the\ntest set—meaning it more accurately predicts the test set—is a better model.\nTraining on the test set introduces a bias that makes the probabilities all look too high, and causes huge\ninaccuracies in perplexity. The perplexity of a language model on a test set is the inverse probability of the test set, normalized\nby the number of words.PP(W) = P(w1w2 ...wN)^(-1/N). There is another way to think about\nperplexity: as the weighted average branching factor of a language. An (intrinsic) \nimprovement in perplexity does not guarantee an (extrinsic) improvement in the performance \nof a language processing task like speech recognition or machine translation. Nonetheless, \nbecause perplexity often correlates with such improvements, it is commonly used as a \nquick check on an algorithm. But a model’s improvement in perplexity should always be \nconfirmed by an end-to-end evaluation of a real task before concluding the evaluation of the model.\n\n## SLP Chapter 3.3 Sampling sentences from a language model\n One important way to visualize what kind of knowledge a language model embodies\nis to sample from it. Sampling from a distribution means to choose random points\naccording to their likelihood. Thus sampling from a language model—which represents \na distribution over sentences—means to generate some sentences, choosing\neach sentence according to its likelihood as defined by the model. We can use the \nsame technique to generate bigrams by first generating a random bigram that starts with \ns but in < > (according to its bigram probability). Let’s say the second word of that bigram is w. \nWe next choose a random bigram starting with w (again, drawn according to its bigram probability), and so on. \n\n## SLP Chapter 3.4 Generalization and Zeros\n The n-gram model, like many statistical models, is dependent on the training corpus.\nOne implication of this is that the probabilities often encode specific facts about a\ngiven training corpus. Another implication is that n-grams do a better and better job\nof modeling the training corpus as we increase the value of N. One step\nis to be sure to use a training corpus that has a similar genre to whatever task we are\ntrying to accomplish. It is equally important to get training data in the appropriate dialect or variety,\nespecially when processing social media posts or spoken transcripts. Our models may still be\nsubject to the problem of sparsity. For any n-gram that occurred a sufficient number\nof times, we might have a good estimate of its probability. But because any corpus is\nlimited, some perfectly acceptable English word sequences are bound to be missing\nfrom it. That is, we’ll have many cases of putative “zero probability n-grams” that\nshould really have some non-zero probability. The percentage of OOV\nwords that appear in the test set is called the OOV rate. An open vocabulary system open\nvocabulary is one in which we model these potential unknown words in the test set by adding a\npseudo-word called <UNK>. The first one is to turn the problem back into a closed vocabulary one\nby choosing a fixed vocabulary in advance: 1. Choose a vocabulary (word list) that is fixed in advance.\n2. Convert in the training set any word that is not in this set (any OOV word) to the unknown \nword token <UNK> in a text normalization step. 3. Estimate the probabilities for <UNK> from its \ncounts just like any other regular word in the training set. The second alternative, in \nsituations where we don’t have a prior vocabulary in advance, is to create such a vocabulary \nimplicitly, replacing words in the training data by <UNK> based on their frequency.\n\n## SLP Chapter 3.5 Smoothing\n To keep a language model from assigning zero probability to these unseen \nevents, we’ll have to shave off a bit of probability mass from some more frequent events \nand give it to the events we’ve never seen. This modification is called smoothing or discounting.\nLaplace smoothing:  All the counts that used to be zero will now\nhave a count of 1, the counts of 1 will be 2, and so on. This does not perform well but is \na good introduction, and its useful for other tasks like text classification.\nInstead of changing both the numerator and denominator, it is convenient to\ndescribe how a smoothing algorithm affects the numerator, by defining an adjusted\ncount c∗ . This adjusted count is easier to compare directly with the MLE counts and\ncan be turned into a probability like an MLE count by normalizing by N. \nA related way to view smoothing is as discounting (lowering) some non-zero\ncounts in order to get the probability mass that will be assigned to the zero counts.\nWe might describe a smoothing algorithm in terms of a relative discount dc=c*/c\nadd-k smoothing. P∗_Add-k(wn|wn−1) = C(wn−1wn) +k / C(wn−1) +kV\nAdd-k smoothing requires that we have a method for choosing k. Although add-k is useful for some\ntasks (including text classification), it turns out that it still doesn’t work well for\nlanguage modeling. \n\nBackoff and Interpolation:\nSometimes using less context is a good thing, helping to generalize more for contexts \nthat the model hasn’t learned much about. In backoff, we use the trigram if the evidence is\nsufficient, otherwise we use the bigram, otherwise the unigram. In other words, we\nonly “back off” to a lower-order n-gram if we have zero evidence for a higher-order\ninterpolation n-gram. By contrast, in interpolation, we always mix the probability estimates from\nall the n-gram estimators, weighing and combining the trigram, bigram, and unigram\ncounts.\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\nEfficiency considerations are important when building language models that use\nsuch large sets of n-grams. Rather than store each word as a string, it is generally\nrepresented in memory as a 64-bit hash number, with the words themselves stored\non disk. Probabilities are generally quantized using only 4-8 bits (instead of 8-byte\nfloats), and n-grams are stored in reverse tries. An n-gram language model can also be shrunk by pruning. \nStupid backoff gives up the idea of trying to make the language\nmodel a true probability distribution. There is no discounting of the higher-order\nprobabilities. If a higher-order n-gram has a zero count, we simply backoff to a\nlower order n-gram, weighed by a fixed (context-independent) weight\n\n## Module 10 Lecture Notes\n Uncertainty\nGeneral situation:\n• Observed variables (evidence): Agent knows certain things about the state of the world (e.g.,\nsensor readings or symptoms)\n• Unobserved variables (states): Agent needs to reason about other aspects (e.g. where an object\nis or what disease is present)\n• Model: Agent knows something about how the known variables relate to the unknown variables\nProbabilistic reasoning gives us a framework\nfor managing our beliefs and knowledge\nWhat Probabilities Are About\nLike logical assertions, probabilities are about possible worlds. Instead of strictly\nruling out possibilities (where a logical assertion is false), probabilities quantify how\nlikely a particular possible world is.\nIn probability theory, the possible worlds are called the sample space, and they\nmutually exclusive and exhaustive.\nA fully specified probability model associates a probability P(w) with each possible\nworld w. \n\nA random variable is some aspect of the world\nabout which we (may) have uncertainty\n• R = Is it raining?\n• U = Is the professor carrying an umbrella?\nWe denote random variables with capital letters\n\nProbabilistic Inference\nProbabilistic inference: compute a desired\nprobability from other known probabilities (e.g.\nconditional from joint)\nWe generally compute conditional probabilities\n• P(on time | no reported accidents) = 0.90\n• These represent the agent’s beliefs given the\nevidence\nProbabilities change with new evidence:\n• P(on time | no accidents, 5 a.m.) = 0.95\n• P(on time | no accidents, 5 a.m., raining) = 0.80\n• Observing new evidence causes beliefs to be updated\n\nProbabilistic Language Models\nOne goal: assign a probability to a sentence\n• Autocomplete for texting\n• Machine Translation\n• Spelling Correction\n• Speech Recognition\nOther Natural Language Generation tasks: summarization, question-answering, dialog\nsystems\n\n\n## Module 10 Definitions\n* **Belief state** - epresentation of the set of all possible world states that it might be in\n* **Qualification problem** - when we can't properly qualify a problem and therefore can't infer the proper plan\n* **Rational decision** - the right thing to do \n* **Laziness** - too much work to list the complete set of antecedents or consequents needed to ensure an exceptionless rule\n* **Theoretical ignorance** - Medical science has no complete theory for the domain.\n* **Practical ignorance** - if we know all rules we still might be uncertain if all necessary tests havent been run\n* **Ontological commitments** - the world is composed of facts that do or do not hold in any particular case\n* **Epistemological commitments** - logical agent believes each sentence to be true or false or has no opinion\n* **Utility theory** - very state (or state sequence) has a degree of usefulness, or utility, to an agent\n* **Decision theory** - Preferences, as expressed by utilities, are combined with probabilities\n* **Sample space** - the set of all possible words\n* **Unconditional or prior probabilities** - degrees of belief in propositions in the absence of any other information\n* **Conditional or posterior probability** - probability of doing something given that we have already done something\n* **Random variable** - is a function that maps from the domain of possible worlds omega to some range\n* **Bernoulli distrobution** - the set {0,1}\n* **Joint probability distrobution** - n x m table of probabilities based on the probability of P(n, m)\n* **Full joint probability distrobution** - probability model is completely determined by the joint distribution for all of the random variables\n* **Probabilistic inference** - computation of posterior probabilities for query propositions given observed evidence\n* **Marginalization or summing out** - sum up the probabilities for each possible value of the other variables, thereby taking them out of the equation\n* **Independence** - separating out probabilites when one cannot influence the others\n* **Text classification** - decide which of a predefined set of classes or categories it belongs to\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Bayesian networks can represent essentially any full joint probability distribution \nand in many cases can do so very concisely. 1) Each node corresponds to a random variable, \nwhich may be discrete or continuous. 2) Directed links or arrows connect pairs of nodes. \nIf there is an arrow from node X to node Y, X is said to be a parent of Y. 3) Each node\nXi has associated probability information theta(Xi|parents(Xi)) that quantifies the\neffect of the parents on the node using a finite number of parameters. Once the topology\nof the Bayes net is laid out, we need only specify the local probability information for\neach variable, in the form of a conditional distribution given its parents. The full joint\ndistribution for all the variables is defined by the topology and the local probability information.\nThe local probability information attached to each node takes the form of aconditional\nprobability table (CPT). (CPTs can be used only for discrete variables). Each row in a CPT \ncontains the conditional probability of each node value for a conditioning case.\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n The semantics of Bayes nets defines each entry in the joint distribution\nas follows: equation . the Bayesian network is a correct representation of the domain\nonly if each node is conditionally independent of its other predecessors in the node\nordering, given its parents. 1) Nodes: determine the set of variables that are required\nto model the domain, now order them. 2) Links: for i = 1 to n do: choose a minimal set of parents\nfor Xi from X1 to Xi-1, for each paretn insert a link from the parent to Xi, CPts: write\ndown the cpt P(Xi|parents(Xi)). a Bayes net can often be far more compact than the full \njoint distribution. This property is what makes it feasible to handle domains with many variables.\nIf we stick to a causal model, we end up having to specify fewer numbers, and the numbers will\noften be easier to come up with.\n\n13.2.1 Conditional independence relations in Bayesian networks \nEach variable is conditionally independent of its non-descendants, given its parents.\ninstead of defining the full joint distribution as the product of conditional distributions,\nthe network defines a set of conditional independence properties. The full joint\ndistribution can be derived from those properties. Is a set of nodes X is conditionally \nindependent of another set Y, given a third set Z. This can be determined efficiently by \nexamining the Bayes net to see whether Z d-separates X and Y. The process works as follows:\n1) Consider just the ancestral subgraph consisting of X, Y, Z, and their ancestors. 2) Add \nlinks between any unlinked pair of nodes that share a common child; now we have the \nso-called moral graph. 3)Replace all directed links by undirected links. 4) If Z blocks \nall paths between X and Y in the resulting graph, then Z d-separates X and Y. In \nthat case, X is conditionally independent of Y, given Z.\n\n13.2.2 Efficient Representation of Conditional Distributions \nlling in the CPT for a node requires up to O(2^k) numbers and perhaps a great deal\nof experience with all the possible conditioning cases. Usually, such relationships are\ndescribable by a canonical distribution that fits some standard pattern. In such cases, the\ncomplete table can be specified just by naming the pattern and perhaps supplying a few parameters.\nUncertain relationships can often be characterized by so-called noisy logical relationships. \n\n13.2.3 Bayesian nets with continuous variables\ndiscretization. define a continuous variable using one of the standard families of probability\ndensity functions. Nonparametric representation.\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n The basic task for any probabilistic inference system is to compute the\nposterior probability distribution for a set of query variables, given some observed\nevent—usually, some assignment of values to a set of evidence variables. \n\n13.3.1 Inference by enumeration\na query can be answered using a Bayes net by computing sums of products of conditional\nprobabilities from the network. Equation (handwritten) with complexity O(2^n) after nesting\nsummations\n\n``` \nFunction enumeration-ask(X,e,bn) returns a distrobution over X\n  inputs: X, the query\n          e, observed values for variables E\n          bn, a bayes net with variables vars\n  Q(X) <- a distrobution over x, initially empty\n  for each value xi of X do\n    Q(xi) <- enumerate-all(vars, exi)\n      where exi is ectended with X = xi\n  return normalize(Q(X))\n\nfunction ennumerate-ALL(vars, e) returns a real number\n  if empty?(vars) then return 1\n  V<-First(vars)\n  if V is evidence variable with value v in e\n    then return P(v|parents(V))x enumerate-all(REST(vars),e)\n    else return sum P(v|parents(V))x enumerate-all(REST(vars),ev)\n      where ev is extended with V = v\n``` \n\n13.3.2 The variable elimination algorithm\nThe final question is, of course, how to do inference in the network to make predictions. \nWe turn now to this question. For each inference method that we describe, we will evaluate \nthe method on the insurance net to measure the time and space requirements of the method.\n\n\n## Module 11 Lecture Notes\n Bayes’ Nets: Big Picture\n- Two problems with using full joint distribution tables\nas our probabilistic models:\n- Unless there are only a few variables, the joint is WAY too\nbig to represent explicitly\n- Hard to learn (estimate) anything empirically about more\nthan a few variables at a time\n- Bayes’ nets: a technique for describing complex joint\ndistributions (models) using simple, local\ndistributions (conditional probabilities)\n- More properly called graphical models\n- We describe how variables locally interact\n- Local interactions chain together to give global, indirect\ninteractions\n\nGraphical Model Notation\n- Nodes: variables (with domains)\n- Can be assigned (observed) or unassigned\n(unobserved)\n- Arcs: interactions\n- Indicate “direct influence” between variables\n- Formally: encode conditional independence\n(more later)\n- For now: imagine that arrows mean\ndirect causation (in general, they don’t!)\n\nBayes’ Net Semantics\n- A set of nodes, one per variable X\n- A directed, acyclic graph\n- A conditional distribution for each node\n- A collection of distributions over X, one for each\ncombination of parents’ values\n- CPT: conditional probability table\n- Description of a noisy “causal” process\n- Bayes’ nets implicitly encode joint distributions\n- As a product of local conditional distributions\n- To see what probability a BN gives to a full assignment, multiply all the\nrelevant conditionals together:\n\nCausality?\n- When Bayes’ nets reflect the true causal patterns:\n- Often simpler (nodes have fewer parents)\n- Often easier to think about\n- Often easier to elicit from experts\n- BNs need not actually be causal\n- Sometimes no causal net exists over the domain\n(especially if variables are missing)\n- E.g. consider the variables Traffic and Drips\n- End up with arrows that reflect correlation, not causation\n- What do the arrows really mean?\n- Topology may happen to encode causal structure\n- Topology really encodes conditional independence\n\nActive / Inactive Paths\n- Question: Are X and Y conditionally independent given\nevidence variables {Z}?\n- Yes, if X and Y “d-separated” by Z\n- Consider all (undirected) paths from X to Y\n- No active paths = independence!\n- A path is active if each triple is active:\n- Causal chain A ® B ® C where B is unobserved (either direction)\n- Common cause A ¬ B ® C where B is unobserved\n- Common effect (aka v-structure)\nA ® B ¬ C where B or one of its descendents is observed\n- All it takes to block a path is a single inactive segment\n\nBayes Nets Representation Summary\n- Bayes nets compactly encode joint distributions\n- Guaranteed independencies of distributions can be\ndeduced from BN graph structure\n- D-separation gives precise conditional independence\nguarantees from graph alone\n- A Bayes’ Net’s joint distribution may have further\n(conditional) independence that is not detectable until\nyou inspect its specific distribution \n\n## Module 11 Definitions\n* **Bayesian network** - directed graph in which each node is annotated with quantitative probability information.\n* **Conditioning case** - a possible combination of values for the parent nodes—a miniature possible world\n* **Syntax of a bayes net** - onsists of a directed acyclic graph with some local probability information attached to each node\n* **Semantics** - defines how the syntax corresponds to a joint distribution over the variables of the network.\n* **Topological order** - any order consistent with the directed graph structure.\n* **Locally structured** - each subcomponent interacts directly with only a bounded number of other components\n* **Sparse** - locally structured\n* **Deterministic node** -  has its value specified exactly by the values of its parents, with no uncertainty.\n* **Context specific independence** - a variable is conditionally independent of some of its parents given certain values of others.\n* **Noisy or model** - allows for uncertainty about the ability of each parent to cause the child to be true\n* **Discretization** - dividing up the possible values into a fixed set of intervals.\n* **Nonparametric representation** - define the conditional distribution implicitly with a collection of instances, each containing specific values of the parent and child variables\n* **Hybrid bayseian network** - A network with both discrete and continuous variables\n* **Linear-gaussian condition distrobution** - child has a Gaussian distribution whose mean mu varies linearly with the value of the parent and whose standard deviation sigma is fixed\n* **Probit model ** - probability unit\n* **Expit or inverse logit model** - uses the logistic function 1/(1+e^-x) to produce a soft threshold—it maps any  to a value between 0 and 1\n* **Variable elimination algorithm** - evaluating expressions such as Equation in right-to-left order. Intermediate results are stored, and summations that depend on the variable are done.\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 19.1 Forms of Learning\n Any component of an agent program can be improved by machine learning.\nThe improvements, and the techniques used to make them, depend on these factors: 1) Which\ncomponent is to be improved. 2) What prior knowledge the agent has, which influences the\nmodel it builds. 3) What data and feedback on that data is available. The components of these agents include:\n1) A direct mapping from conditions on the current state to actions. 2) A means to \ninfer relevant properties of the world from the percept sequence. 3) Information about \nthe way the world evolves and about the results of possible actions the agent can take.\n4) Utility information indicating the desirability of world states. 5) Action-value \ninformation indicating the desirability of actions. 6) Goals that describe the most \ndesirable states. 7) A problem generator, critic, and learning element that enable\nthe system to improve. we consider transfer learning, in which knowledge from one domain\nis transferred to a new domain, so that learning can proceed faster with less data.\nWe do assume, however, that the designer of the system chooses a model framework that\ncan lead to effective learning. This chapter concentrates on problems where the input\nis a factored representation—a vector of attribute values. It is also possible for the\ninput to be any kind of data structure, including atomic and relational. We look at\nclassification and regression problems. Three types of feedback that accompany the inputs:\n1) supervised learning 2) unsupervised learning 3) reinforcement learning.\n\n\n## AIMA Chapter 19.2 Supervised Learning\n the task of supervised learning is this: Given a training set of N\nexample input–output pairs (x1,y1), ...(xN,yN), where each pair was generated by an unknown\nfunction y=f(x), discover a function h that approximates the true function f. The function \nh is called a hypothesis about the world. It is drawn from a hypothesis space H of possible\nfunctions. or we can say a function drawn from a function class. We call the output \nyi the ground truth—the true answer we are asking our model to predict. With continuous-valued\noutputs we can’t expect an exact match to the ground truth; instead we look for a best-fit\nfunction for which each h(xi) is close to yi. The true measure of a hypothesis is not how it\ndoes on the training set, but rather how well it handles inputs it has not yet seen.\nBias often results from restrictions imposed by the hypothesis space. Often there is\na bias–variance tradeoff: a choice between more complex, low-bias hypotheses that fit\nthe training data well and simpler, low-variance hypotheses that may generalize better. \nThe problem is that there is a tradeoff between the expressiveness of a hypothesis space\nand the computational complexity of finding a good hypothesis within that space. \n\n## AIMA Chapter 19.3 Learning Decision Trees\n Each internal node in the tree corresponds to a test of the value \nof one of the input attributes, the branches from the node are labeled with the possible\nvalues of the attribute, and the leaf nodes specify what value is to be returned by\nthe function.\n\n19.3.1 Expressiveness of decision trees \nA Boolean decision tree is equivalent to a logical statement of the form: Output <-> (path1 V path2 V...)\nAny function in propostional logic and be expressed as a decision tree. There is no representation\nthat is efficient for all kinds of functions.\n\n19.3.2 Learning decision trees from examples \nThere are four cases to consider for these recursive subproblems: 1) If the remaining examples \nare all positive (or all negative), then we are done: we can answer Yes or No.\n2) If there are some positive and some negative examples, then choose the best attribute\nto split them. 3) If there are no examples left, it means that no example has been observed\nfor this combination of attribute values, and we return the most common output value from \nthe set. 4) If there are no attributes left, but both positive and negative examples, \nit means that these examples have exactly the same description, but different classifications. \n``` \nfunction learn-decision-tree(examples, attributes, parent_examples) returns a tree\n  if examples is empty then return plurality-value(parent_examples)\n  else if all examples have the same classification then return the classification\n  else if attributes is empty then reutn plurality-value(examples)\n  else\n    A<- argmax_aEattributes Importance(a, examples)\n    tree <- a new decision tree wiht root test A\n    for each value v of A do \n      exs <-{e: e E ecamples and eA = v}\n      subtree <- learn-decision-tree(exs, attributes - A, examples)\n      add a branch to tree wiht label (A=v)and subtree subtree\n    return tree\n``` \nWe can evaluate the performance of a learning algorithm with a learning curve.\n\n19.3.3 Choosing attribute tests \nThe decision tree learning algorithm chooses the attribute with the highest Importance.\nThis is defined in terms of entropy.\n\n19.3.4 Generalization and overfitting \nOverfitting becomes more likely as the number of attributes grows, and less likely as \nwe increase the number of training examples. Larger hypothesis spaces have more capacity\nboth to fit and to overfit\n\n19.3.5 Broadening the applicability of decision trees \nDecision trees can be made more widely useful by handling the following complications:\nMissing data, continuous na dmultivalued input attributes, continuous-valued output \nattribute\n\n\n## AIMA Chapter 19.4 Model Selection and Optimization\n Our goal in machine learning is to select a hypothesis that will\noptimally fit future examples. To make that precise we need to define “future example”\nand “optimal fit.” the optimal fit is the hypothesis that minimizes the error rate.\nsplit the examples you have into two sets: a training set to create the hypothesis,\nand a test set to evaluate it. That means you need three data sets: 1) A training set \nto train candidate models. 2) A validation set, also known as a development set or dev\nset, to evaluate the candidate models and choose the best one. 3) A test set to do a \nfinal unbiased evaluation of the best model. K-fold cross-validation: plit the data into k\nequal subsets. We then perform k rounds of learning; on each round 1/k of the data \nare held out as a validation set and the remaining examples are used as the training set.\ntask of finding a good hypothesis as two subtasks: model selection and optimization.\n\n19.4.1 Model selection \n``` \nfunction model-selection(learner, examples, k) returns a (hypothesis, error rate) pair\n  err <- an array, indexed by size, storing validation-set error rates\n  training_set, test_set <- a partition of examples into two sets\n  for size = 1 to inf do\n    err[size] <- cross-validation(learner, size, training_set, k)\n    if err is starting to increase significantly then\n      best_size <- the value of size with the minnimum err[size]\n      h <- Learner(best_size, training_set)\n      return h, error-rate(h, test_set)\n\nfunction cross-validation(Learner,size,examples,k) returns error rate\n  N<- the number of examples\n  errs <- 0\n  for i = 1 to k do \n    validation_set <- examples[(i-1)xN/k:ixN/k]\n    training_set <- examples - validation_set\n    h <- Learner(size, training_set)\n    errs <- errs + ERROR-RATE(h, validation_set)\n  return errs/k\n``` \nModel classes typically start to overfit as the capacity approaches the point of interpolation\n\n19.4.2 From error rates to loss \nin machine learning it is traditional to express this as a negative: to minimize a loss \nfunction rather than maximize a utility function. Best hypothesis is the one with \nthe minimum expected generalization loss. Since P(x,Y) is not known in most cases, the\nlearning agent can only estimate generalization loss with empirical loss. h^* may differ\nfrom the true function because of unrealizability, variance, noise, and computational complexity.\n\n19.4.4 Hyperparameter tuning\nSimplest is hand tuning: Repeat until you have satisfactory performance.\nIf there are only a few hyperparameters a grid search is appropriate. try all combinations \nof values and see which performs best on the validation data. If there are too many combinations \nof possible values, then random search. Random sampling is also a good way to handle continuous values.\nBayesian optimization. Population based training: PBT, as well as by random mutation, \nas in genetic algorithms\n\n\n## AIMA Chapter 19.5 The Theory of Learning\n computational learning theory, which lies at the intersection of AI, \nstatistics, and theoretical computer science. The underlying principle is that any hypothesis \nthat is seriously wrong will almost certainly be “found out” with high probability after a \nsmall number of examples, because it will make an incorrect prediction. Must be probably \napproximately correct.\n\n19.5.1 PAC learning example: Learning decision lists \nIf a test succeeds when applied to an example description, the decision list specifies \nthe value to be returned. If the test fails, processing continues with the next test in \nthe list. Decision lists resemble decision trees, but their overall structure is simpler: \nthey branch only in one direction. \n\n``` \nfunction Decision-list-learning(examples) returns a decision list or failure\n  if examples is empty then return the trivial decision list nO\n  t <- a test that matches a nonempty subset examples_t are all positive or negative\n      such that all the members of examples_t are all positive or all negative\n  if there is no such t then return a failure\n  if the examples in examples_t are positive then o<- YES else o<- NO\n  return a decision list with initiial test t and outcome o adn remaining tests given by \n    Decision-List_learning(examples-examples_t)\n``` \n\n\n## AIMA Chapter 19.6 Linear Regression and Classification\n 19.6.1 Univariate linear regression \nh_w(x) = w1x +w0 use w bc coefficients are weights. The loss function is convex \nthis is true for every linear regression problem with an loss function, and implies \nthat there are no local minima. \n\n19.6.2 Gradient descent \nchoose any starting point in weight space—here, a point in the (\n \n) plane—and then compute an estimate of the gradient and move a small amount in the \nsteepest downhill direction, repeating until we converge on a point in weight space with \n(local) minimum loss. \n``` \nw <- any point in the parameter space\nwhile not converged do\n  for each wi in w do\n    w<- wi - alpha partial/partial wi Loss(w)\n``` \nSGD is widely applied to models other than linear regression, in particular neural networks. \nEven when the loss surface is not convex, the approach has proven effective in finding \ngood local minima that are close to the global minimum.\n\n19.6.3 Multivariable linear regression \nit is common to use regularization on multivariable linear functions to avoid overfitting. \nRecall that with regularization we minimize the total cost of a hypothesis, counting \nboth the empirical loss and the complexity of the hypothesis:\n\n19.6.4 Linear classifiers with a hard threshold \n we can think of h as the result of passing the linear function w*x through a threshold function:\n \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n multinomial naive Bayes classifier, so called because it is a Bayesian \nclassifier that makes a simplifying (naive) assumption about how the features interact.\nWe represent a text document bag-of-words as if it were a bag-of-words, that is, an \nunordered set of words with their position ignored, keeping only their frequency in the document\ncˆ = argmax_c∈C P(c|d) break down any conditional probability P(x|y) into\nthree other probabilities: P(x|y) = P(y|x)P(x)/P(y) then cˆ = argmax_c∈C P(c|d) = argmax_c∈C P(d|c)P(c)/P(d) \nTo return to classification: we compute the most probable class ˆc given some\ndocument d by choosing the class which has the highest product of two probabilities:\nthe prior probability of the class P(c) and the likelihood of the document P(d|c):\nClassifiers that use a linear combination of the inputs\nto make a classification decision —like naive Bayes and also logistic regression—\nare called linear classifiers.\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\nHow can we learn the probabilities P(c) and P(fi|c)?  Pˆ(c) = Nc/Ndoc\nPˆ(wi|c) = count(wi, c)/ summation w∈V count(w, c). \nWhat do we do about words that occur in our test data but are not in our vocabulary at all because they did not occur in any training document in any class? The\nunknown word solution for such unknown words is to ignore them—remove them from the test\ndocument and not include any probability for them at all. \nFinally, some systems choose to completely ignore another class of words: stop\nstop words words, very frequent words like the and a. This can be done by sorting the vocabulary by frequency in the training set, and defining the top 10–100 vocabulary entries\nas stop words, or alternatively by using one of the many predefined stop word lists\navailable online.\n\n``` \nfunction TRAIN NAIVE BAYES(D, C) returns log P(c) and log P(w|c)\nfor each class c ∈ C # Calculate P(c) terms\n  Ndoc = number of documents in D\n  Nc = number of documents from D in class c\n  logprior[c]← log Nc/Ndoc\n  V←vocabulary of D\n  bigdoc[c]←append(d) for d ∈ D with class c\n  for each word w in V # Calculate P(w|c) terms\n    count(w,c)←# of occurrences of w in bigdoc[c]\n    loglikelihood[w,c]← log (count(w, c) + 1)/summ_w' in V(count (w0, c) + 1)\nreturn logprior, loglikelihood, V\n\nfunction TEST NAIVE BAYES(testdoc, logprior, loglikelihood, C, V) returns best c\nfor each class c ∈ C\n  sum[c]← logprior[c]\n  for each position i in testdoc\n    word←testdoc[i]\n    if word ∈ V\n      sum[c]←sum[c]+ loglikelihood[word,c]\n  return argmaxc sum[c]\n``` \n\n \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n for sentiment classification and a number of other text classification tasks,\nwhether a word occurs or not seems to matter more than its frequency. Thus it\noften improves performance to clip the word counts in each document at 1. Binary\nmultinomial naive Bayes or binary NB. The variant uses the same Eq. 4.10 except\nthat for each document we remove all duplicate words before concatenating them\ninto the single big document. A second important addition commonly made when doing text classification for\nsentiment is to deal with negation. Consider the difference between I really like this\nmovie (positive) and I didn’t like this movie (negative). The negation expressed by\ndidn’t completely alters the inferences we draw from the predicate like. Similarly,\nnegation can modify a negative word to produce a positive review (don’t dismiss this\nfilm, doesn’t let us get bored). Finally, in some situations we might have insufficient labeled training data to\ntrain accurate naive Bayes classifiers using all words in the training set to estimate\npositive and negative sentiment. In such cases we can instead derive the positive \nand negative word features from sentiment lexicons, lists of words that are pre- sentiment\nlexicons annotated with positive or negative sentiment\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n A common solution here, rather than using all the words as individual features,\nis to predefine likely sets of words or phrases as features, combined with features\nthat are not purely linguistic. For example the open-source SpamAssassin tool1\npredefines features like the phrase “one hundred percent guaranteed”, or the feature\nmentions millions of dollars, which is a regular expression that matches suspiciously\nlarge sums of money. But it also includes features like HTML has a low ratio of text\nto image area, that aren’t purely linguistic and might require some sophisticated\ncomputation, or totally non-linguistic features about, say, the path that the email\ntook to arrive. For other tasks, like language id—determining what language a given piece\nof text is written in—the most effective naive Bayes features are not words at all,\nbut character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,\n‘thei’), or, even simpler byte n-grams, where instead of using the multibyte Unicode\ncharacter representations called codepoints, we just pretend everything is a string of\nraw bytes. Because spaces count as a byte, byte n-grams can model statistics about\nthe beginning or ending of words \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Bayes classifiers can use any sort of feature: dictionaries, URLs, email addresses, network features, phrases, and so on. But\nif, as in the previous section, we use only individual word features, and we use all\nof the words in the text (not a subset), then naive Bayes has an important similarity to language modeling. Specifically, a naive Bayes model can be viewed as a\nset of class-specific unigram language models, in which the model for each class\ninstantiates a unigram language model.\nSince the likelihood features from the naive Bayes model assign a probability to\neach word P(word|c), the model also assigns a probability to each sentence:\nP(s|c) = PI _ i∈positions P(wi|c)\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\nIn both cases, we need a metric for knowing how well our spam detector (or\npie-tweet-detector) is doing. To evaluate any system for detecting things, we start\nby building a confusion matrix. A confusion matrix confusion\nis a table for visualizing how an algorithm performs with respect to the human gold\nlabels, using two dimensions (system output and gold labels), and each cell labeling\na set of possible outcomes. precision and recall. Precision measures the percentage of the items that\nthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\nare positive according to the human gold labels). Precision is defined as\nPrecision = true positives/true positives + false positives\nRecall measures the percentage of items actually present in the input that were\ncorrectly identified by the system. Recall is defined as\nRecall = true positives/true positives + false negatives. There are many ways to define a single metric that incorporates aspects of both\nF-measure precision and recall. The simplest of these combinations is the F-measure, defined as:\nFβ = (β2 +1)PR/β^2P+R\nThe β parameter differentially weights the importance of recall and precision,\nbased perhaps on the needs of an application. Values of β > 1 favor recall, while\nvalues of β < 1 favor precision. When β = 1, precision and recall are equally balanced.\nIn macroaveraging, we compute the performance microaveraging for each class, and then \naverage over classes. In microaveraging, we collect the decisions for all classes into \na single confusion matrix, and then compute precision and recall from that table.\n \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n we use the training set to train the model, then use\nthe development test set (also called a devset) to perhaps tune some parameters,and \nin general decide what the best model is. In cross-validation, we choose a number k, and partition our data into k disjoint\nsubsets called folds. Now we choose one of those k folds as a test set, train our\nclassifier on the remaining k − 1 folds, and then compute the error rate on the test\nset. Then we repeat with another fold as the test set, again training on the other k−1\nfolds. We do this sampling process k times and average the test set error rate from\nthese k runs to get an average error rate.\n \n\n## SLP Chapter 4.9 Statistical Significance Testing\n We would like to know if δ(x) > 0, meaning that our logistic regression classifier\nhas a higher F1 than our naive Bayes classifier on X. δ(x) is called the effect size;\na bigger δ means that A seems to be way better than B; a small δ means A seems to\nbe only a little better. The hypothesis H0, called the null hypothesis, supposes that δ(x) is actually negative or zero, meaning that A is not better than B. We would like to know if we can\nconfidently rule out this hypothesis, and instead support H1, that A is better. We do this by creating a random variable X ranging over all test sets. Now we\nask how likely is it, if the null hypothesis H0 was correct, that among these test sets\nwe would encounter the value of δ(x) that we found. We formalize this likelihood\nas the p-value: the probability, assuming the null hypothesis H0 is true, of seeing\nthe δ(x) that we saw or one even greater P(δ(X) ≥ δ(x)|H0 is true).We say that a result \nis statistically significant if the δ we saw has a probability that is below the \nthreshold and we therefore reject this null hypothesis. There are two common non-parametric \ntests used in NLP: approximate randomization and the bootstrap test, showing the paired \nversion of the test, which again is most common in NLP. Paired tests are those in which we compare two sets of observations that are aligned:\neach observation in one set can be paired with an observation in another. This happens naturally when we are comparing the performance of two systems on the same\ntest set; we can pair the performance of system A on an individual observation xi\nwith the performance of system B on the same xi.\n\n``` \nfunction BOOTSTRAP(test set x, num of samples b) returns p-value(x)\nCalculate δ(x) # how much better does algorithm A do than B on x\ns = 0\nfor i = 1 to b do\nfor j = 1 to n do # Draw a bootstrap sample x^(i) of size n\n    Select a member of x at random and add it to x^(i)\n  Calculate δ(x^(i)) # how much better does algorithm A do than B on x^(i)\n  s←s + 1 if δ(x^(i)) ≥ 2δ(x)\np-value(x) ≈ s/b\n# on what % of the b samples did algorithm A beat expectations?\nreturn p-value(x) # if very few did, our observed δ is probably not accidental\n``` \n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n One class of harms is representational harms, \nharms caused by a system that demeans a social group, for example by perpetuating negative \nstereotypes about them. In other tasks classifiers may lead to both representational harms and other\nharms, such as censorship. For example the important text classification task of\ntoxicity detection is the task of detecting hate speech, abuse, harassment, or other toxicity\ndetection\nkinds of toxic language. While the goal of such classifiers is to help reduce societal harm,\ntoxicity classifiers can themselves cause harms.   \n\n## Module 12 Lecture Notes\n Model-Based Classification\nModel-based approach\n• Build a model (e.g. Bayes’ net) where both the label and\nfeatures are random variables\n• Instantiate any observed features\n• Query for the distribution of the\nlabel conditioned on the features\n\nWhat do we need in order to use Naïve Bayes?\n• Inference method (we just saw this part)\n• Start with a bunch of probabilities: P(Y) and the P(Fi\n|Y) tables\n• Use standard inference to compute P(Y|F1…Fn)\n• Nothing new here\n• Estimates of local conditional probability tables\n• P(Y), the prior over labels\n• P(Fi\n|Y) for each feature (evidence variable)\n• These probabilities are collectively called the parameters of the\nmodel and denoted by q\n• Up until now, we assumed these appeared by magic, but…\n• …they typically come from training data counts: we’ll look at this\nsoon\n\nBag-of-words Naïve Bayes:\n• Features: Wi is the word at positon i\n• As before: predict label conditioned on feature variables (spam vs. ham)\n• As before: assume features are conditionally independent given label\n• New: each Wi is identically distributed\nGenerative model:\n“Tied” distributions and bag-of-words\n• Usually, each variable gets its own conditional probability distribution P(F|Y)\n• In a bag-of-words model\n• Each position is identically distributed\n• All positions share the same conditional probs P(W|Y)\n• Why make this assumption?\n• Called “bag-of-words” because model is insensitive to word order or reordering\n\nRelative frequency parameters will overfit the training data!\n• Just because we never saw a 3 with pixel (15,15) on during training doesn’t mean we won’t\nsee it at test time\n• Unlikely that every occurrence of “minute” is 100% spam\n• Unlikely that every occurrence of “seriously” is 100% ham\n• What about all the words that don’t occur in the training set at all?\n• In general, we can’t go around giving unseen events zero probability\nAs an extreme case, imagine using the entire email as the only feature\n• Would get the training data perfect (if deterministic labeling)\n• Wouldn’t generalize at all\n• Just making the bag-of-words assumption gives us some generalization, but isn’t enough\nTo generalize better: we need to smooth or regularize the estimates\n\nWhat to Do About Errors?\nNeed more features– words aren’t enough!\n• Have you emailed the sender before?\n• Have 1K other people just gotten the same email?\n• Is the sending information consistent?\n• Is the email in ALL CAPS?\n• Do inline URLs point where they say they point?\n• Does the email address you by (your) name?\nCan add these information sources as new\nvariables in the NB model\nNext class we’ll talk about classifiers which let\nyou easily add arbitrary features more easily\n\nFirst step: get a baseline\n• Baselines are very simple “straw man” procedures\n• Help determine how hard the task is\n• Help know what a “good” accuracy is\nWeak baseline: most frequent label classifier\n• Gives all test instances whatever label was most common in the training set\n• E.g. for spam filtering, might label everything as ham\n• Accuracy might be very high if the problem is skewed\n• E.g. calling everything “ham” gets 66%, so a classifier that gets 70% isn’t very good…\nFor real research, usually use previous work as a (strong) baseline\n\nThe confidence of a probabilistic classifier:\n• Posterior over the top label\n• Represents how sure the classifier is of\nthe classification\n• Any probabilistic model will have\nconfidences\n• No guarantee confidence is correct\nCalibration\n• Weak calibration: higher confidences\nmean higher accuracy\n• Strong calibration: confidence predicts\naccuracy rate\n• What’s the value of calibration?\n\nSummary\nBayes rule lets us do diagnostic queries with causal probabilities\nThe naïve Bayes assumption takes all features to be independent given\nthe class label\nWe can build classifiers out of a naïve Bayes model using training data\nSmoothing estimates is important in real systems\nClassifier confidences are useful, when you can get them\n\nProblems with the Perceptron\nNoise: if the data isn’t separable,\nweights might thrash\n• Averaging weight vectors over\ntime can help (averaged\nperceptron)\nMediocre generalization: finds a\n“barely” separating solution\nOvertraining: test / held-out\naccuracy usually rises, then falls\n• Overtraining is a kind of\noverfitting\n\nFixing the Perceptron\nIdea: adjust the weight update to mitigate these effects\nMIRA*: choose an update size that fixes the current\nmistake…\n… but, minimizes the change to w\n\nIn practice, it’s also bad to make updates that are too large\n• Example may be labeled incorrectly\n• You may not have enough features\n• Solution: cap the maximum possible value of t with some\nconstant C\n• Corresponds to an optimization that assumes nonseparable data\n• Usually converges faster than perceptron\n• Usually better, especially on noisy data\n\nClassification: Comparison\nNaïve Bayes\n• Builds a model training data\n• Gives prediction probabilities\n• Strong assumptions about feature independence\n• One pass through data (counting)\nPerceptrons / MIRA:\n• Makes less assumptions about data\n• Mistake-driven learning\n• Multiple passes through data (prediction)\n• Often more accurate\n\nPerceptions can be used for all kinds of classification problems.\nThink of the inputs as features representing something we want to classify.\nThe feature values for inputs are fixed, but we can choose different weight vectors.\nDepending on the weight vector that we pick, we will get a different classifier. \n \n\n## Module 12 Definitions\n* **Induction** - Going from a specific set of observations to a general rule\n* **Deductive conclusions** - guaranteed to be correct if the premises are correct\n* **Factored representation** - vector of attribute values\n* **Classification** - when the output of the learning problem is a finite set of values\n* **Regression** - when the output of the learning problem is a number\n* **Supervised learning** - the agent observes input-output pairs and learns a function that maps from input to output\n* **Unsupervised learning** - the agent learns patterns in the input without any explicit feedback\n* **Clustering** - detecting potentially useful clusters of input examples\n* **Reinforcement learning** - the agent learns from a series of reinforcements: rewards and punishments\n* **Exploratory data analysis** - examining the data with statistical tests and visualizations—histograms, scatter plots, box plots—get some insight into what hypothesis space might be appropriate\n* **Consistent hypothesis** - h such that each xi in the training set has h(xi) = yi\n* **Test set** - second sample of (xi, yi) pairs\n* **Generalizes well** - if it accurately predicts the outpusts of the test set\n* **Bias** - the tendency of a predictive hypothesis to deviate from the expected value when averaged over different training sets\n* **Underfitting** - when it fails to find a pattern in the data.\n* **Variance** - the amount of change in the hypothesis due to fluctuation in the training data\n* **Decistion tree** - representation of a function that maps a vector of attribute values to a single output value—a “decision.”\n* **Boolean classification** - consider only inputs consisting of discrete values and outputs that are either true or false \n* **Happy graphs** - learning curves\n* **Entropy** -  measure of the uncertainty of a random variable\n* **Information gain** - from the attribute test on A is the expected reduction in entropy:\n* **Decision tree pruning** - combats overfitting by eliminating nodes that are not clearly relevant.\n* **Statistical significance test** - how large a gain should we require to split on an attribute?\n* **Early stopping** - have the decision tree algorithm stop generating nodes when there is no good attribute to split on\n* **Missing data** - in many domains not all the attribute values will be known for every example\n* **Continuous and multivalued input attributes** - may be that every example has a different attribute value. The information gain measure would give its highest score to such an attribute\n* **Split point test** - an inequality test on the value of an attribute\n* **Regression tree** - has at each leaf a linear function of some subset of numerical attributes, rather than a single output value\n* **Decision trees are unstable** - adding just one new example can change the test at the root\n* **Stationary assumption** - \n* **Independent and identically distributed or i.i.d** - each example has the same prior probabilityt distrobution adn is independent of the previous examples\n* **Error rate** - the proportion of times that h(x) != y for an (x,y) example\n* **Hyper parameters** - try different thresholds for x^2 pruning of decision trees, or different degrees for polynomials\n* **Leave-one-out cross-validation or loocv** - k-fold cross-validation when k=n\n* **Model selection** - choose a good hypothesis space\n* **Optimization (training)** - finds the best hypothesis for the space\n* **Interpolated** - a model that exactly fits all the training data\n* **Realizable** - hypothesis space H actually contains the true function f\n* **Varriance** - a learning algorithm will in general return different hypotheses for different sets of examples\n* **Computationally intractable** - a search can explore part of the space and return a reasonably good hypothesis, but can’t always guarantee the best one\n* **Small scale learning** - the number of training examples ranged from dozens to the low thousands\n* **Large scale learning** - millions of examples\n* **Regularization** - explicitly penalizing complex hypotheses is called\n* **Regularization function** - the complexity function\n* **Frature selection** - discard attributes that appear to be irrelevant\n* **Minimum description length or mdl hypothesis** - minimizes the total number of bits required\n* **Hand tuning** - guess based on past experience, train model,measure performance on validation data, analyze results, and use intuition to suggest new parameter values\n* **Random search** - samples uniformly from the set of all possible hyperparameter settings, repeating for as long as you are willing to spend the time and computational resources\n* **Bayesian optimization** - treats the task of choosing good hyperparameter values as a machine learning problem in itself\n* **Pbt** - trains in 2 waves, 2nd generation can choose based on the successful values from the previous generation.\n* **Computational learning theory** - which lies at the intersection of AI, statistics, and theoretical computer science.\n* **Pac learning algorithm** - Any learning algorithm that returns hypotheses that are probably approximately correct\n* **Approximately correct** - error(h) < epsilon which is a small constatne\n* **Sample complexity** - number of required examples as a function of epsilon and delta\n* **Decision lists** - onsists of a series of tests, each of which is a conjunction of literals\n* **Linear regression** - task of finding the hw that best fits the data\n* **Weight space** - the space defined by all possible settings of the weights\n* **Epoch** - a step that covers all the training examples\n* **Stochastic gradient descent or sgd** - randomly selects a small number of training examples at each step, and updates according to\n* **Decistion boundary** - is a line (or a surface, in higher dimensions) that separates the two classes.\n* **Linear separator** - linear decision boundary\n* **Training curve** - measures the classifier performance on a fixed training set as the learning process proceeds one update at a time on that training set.\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Consider a single input observation x, which we will represent by a vector of features [x1, x2,..., xn] (we’ll show sample features in the next subsection). The classifier\noutput y can be 1 (meaning the observation is a member of the class) or 0 (the observation is not a member of the class). We want to know the probability P(y = 1|x)\nthat this observation is a member of the class. So perhaps the decision is “positive \nsentiment” versus “negative sentiment”, the features represent counts of words in a\ndocument, P(y = 1|x) is the probability that the document has positive sentiment,\nand P(y = 0|x) is the probability that the document has negative sentiment.\nLogistic regression solves this task by learning, from a training set, a vector of\nweights and a bias term. Each weight wi is a real number, and is associated with one\nof the input features xi. The weight wi represents how important that input feature\nis to the classification decision, and can be positive or negative. The bias term, also called the intercept, is\nintercept another real number that’s added to the weighted inputs. To create a probability, we’ll pass z through the sigmoid function, σ(z). The\nsigmoid function (named because it looks like an s) is also called the logistic function, \nand gives logistic regression its name. The sigmoid has the following equation, σ(z) = 1/1+e^−z=1/1+exp(−z)\nThe sigmoid function has the property 1−σ(x) = σ(−x)  so we could also have expressed P(y = 0) as σ(−(w· x+b)).\nNow we have an algorithm that given an instance x computes the probability\nP(y = 1|x). How do we make a decision? For a test instance x, we say yes if the\nprobability P(y = 1|x) is more than .5, and no otherwise. We call .5 the decision\nboundary: decision(x) = \u001a1 if P(y = 1|x) > 0.5\n                        0 otherwise\nFor many tasks (especially when feature values can reference specific words)\nwe’ll need large numbers of features. Often these are created automatically via feature templates,\nIn order to avoid the extensive human effort of feature design, recent research in\nNLP has focused on representation learning: ways to learn features automatically\nin an unsupervised way from the input. Logistic regression has a number of advantages over naive\nBayes. Naive Bayes has overly strong conditional independence assumptions. Consider two features which are strongly correlated; in fact, imagine that we just add the\nsame feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate, multiplying them both in, overestimating the evidence. B\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n the distance between the system output and the gold output, and we call this distance\nloss the loss function or the cost function. The second thing we need is an optimization algorithm for iteratively updating\nthe weights so as to minimize this loss function. The standard algorithm for this is\ngradient descent; we’ll introduce the stochastic gradient descent algorithm in the\nfollowing section. \n\n## SLP Chapter 5.3 The cross-entropy loss function\n We need a loss function that expresses, for an observation x, how close the classifier\noutput ( ˆy = σ(w· x+b)) is to the correct output (y, which is 0 or 1). We’ll call this:\nL(yˆ, y) = How much ˆy differs from the true y This is called conditional maximum likelihood\nestimation: we choose the parameters w,b that maximize the log probability of\nthe true y labels in the training data given the observations x. The resulting loss\nfunction is the negative log likelihood loss, generally called the cross-entropy loss. \n\n## SLP Chapter 5.4 Gradient Descent\n So the goal is to find the set of weights which minimizes the loss function, averaged\nover all examples:θˆ = argmin_θ 1/m SUM_i=1 L_CE(f(x(i);θ), y(i)). For logistic regression, \nthis loss function is conveniently convex. A convex function has just one minimum; \nthere are no local minima to get stuck in, so gradient descent starting from any point \nis guaranteed to find the minimum. The gradient descent algorithm answers this question \nby finding the gradient of the loss function at the current point and moving in the \nopposite direction. The magnitude of the amount to move in gradient descent is the value of the\nslope d/dw L(f(x;w), y) weighted by a learning rate η. A higher (faster) learning\nrate means that we should move w more on each step.The gradient is just such a vector; it expresses the directional components of the\nsharpest slope along each of those N dimensions. If we’re just imagining two weight\ndimensions (say for one weight w and one bias b), the gradient might be a vector with\ntwo orthogonal components, each of which tells us how much the ground slopes in\nthe w dimension and in the b dimension. Logistic regression: gradient with respect to a single weight wj represents a\nvery intuitive value: the difference between the true y and our estimated \nˆy = σ(w·x+b) for that observation, multiplied by the corresponding input value xj.\n\n``` \nfunction STOCHASTIC GRADIENT DESCENT(L(), f(), x, y) returns θ\n# where: L is the loss function\n# f is a function parameterized by θ\n# x is the set of training inputs x^(1), x^(2),..., x^(m)\n# y is the set of training outputs (labels) y^(1), y^(2),..., y^(m)\nθ ←0\nrepeat til done # see caption\n  For each training tuple (x(i), y(i)) (in random order)\n    1. Optional (for reporting): # How are we doing on this tuple?\n    Compute ˆy(i) = f(x(i);θ) # What is our estimated output ˆy?\n    Compute the loss L(yˆ(i), y(i)) # How far off is ˆy(i)from the true output y(i)?\n    2. g←∇θ L(f(x(i);θ), y(i)) # How should we move θ to maximize loss?\n    3. θ ←θ − η g # Go the other way instead\nreturn θ\n``` \n\n\n\n## SLP Chapter 5.5 Regularization\n To avoid overfitting, a new regularization term R(θ) is added to the objective\nfunction θˆ = argmax_θ SUM^m_i=1 logP(y^(i)|x^(i))−αR(θ) \nThe L2 regularized objective function becomes:\nθˆ = argmax_θ [SUm_i=1 logP(y^(i)|x^(i))]−α SUM_j=1 θ^\nL1 regularization is a linear function of the weight values, named after the L1 norm L1\nregularization||W||1, the sum of the absolute values of the weights, or Manhattan distance (the\nManhattan distance is the distance you’d have to walk between two points in a city\nwith a street grid like New York):θˆ = argmax_θ [SUM_1=i logP(y^(i)|x^(i))]−αSUM_j=1|θj|\nL1 regularization is called lasso lasso regression and L2 regularization is called ridge regression,\nridge and both are commonly used in language processing\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n The multinomial logistic classifier uses a generalization of the sigmoid, called\nsoftmax the softmax function, to compute the probability p(y = c|x). The softmax function\ntakes a vector z = [z1,z2,...,zk] of k arbitrary values and maps them to a probability\ndistribution, with each value in the range (0,1), and all the values summing to 1.\nLike the sigmoid, it is an exponential function.\n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n Classifier components\nMachine learning classifiers require a training corpus of M observations\ninput/output pairs (x(i)\n,y(i)\n).\n1. A feature representation of the input. For each input observation\nx(i)\n, this will be a vector of features [x1,x2,...,xn].\n2. A classification function that computes the estimated class �! via\np(y|x).\n3. An objective function for learning, usually involving minimizing\nerror on training examples.\n4. An algorithm for optimizing the objective function.\n\nSentiment classifier\nFor sentiment classification, consider an input observation x,\nrepresented by a vector of features [x1,x2,...,xn]. The classifier output y\ncan be 1 (positive sentiment) or 0 (negative sentiment). We want to\nestimate P(y = 1|x).\nLogistic regression solves this task by learning, from a training set, a\nvector of weights and a bias term.\nz = sum_i wixi + b\nWe can also write this as a dot product:\nz = w*x+b\n\nNow we have an algorithm that given an instance x computes the\nprobability P(y = 1|x). How do we make a decision?\ny^ = {1 if P(y=1|x)>0.5   0 otherwise\nFor a test instance x, we say yes if the probability P(y = 1|x) is more than\n.5, and no otherwise. We call .5 the decision boundary\n\nLearning in logistic regression\nHow do we get the weights of the model? We learn the parameters\n(weights + bias) via learning. This requires 2 components:\n1. An objective function or loss function that tells us distance between\nthe system output and the gold output. We will use cross-entropy\nloss.\n2. An algorithm for optimizing the objective function. We will use\nstochastic gradient descent to minimize the loss function. \n\nWe need to determine for some observation x how close the classifier\noutput (y^!= σ (w · x + b)) is to the correct output (y, which is 0 or 1).\nL(y^,y) = how much y^! differs from the true y\nOne example is mean squared error\nL_MSE(y^,y) = 1/2(y^-y)^2 \nThe resulting loss function is the negative log likelihood loss, more\ncommonly called the cross entropy loss.\nThe result is cross-entropy loss:\nL_CE(y^,y)= −log(py|x) = −[y log y^! + (1 − y )log(1 − y^)]\nFinally, plug in the definition for y^<= σ (w · x) + b\nL_CE(y^,y) = −[y log σ(w·x+b) + (1 − y) log(1 − σ(w·x+b))]\nWe want the loss to be smaller if the model’s estimate is close to correct,\nand we want the loss to be bigger if it is confused.\n\nGradient descent is a method that finds a minimum of a function by figuring out in\nwhich direction (in the space of the parameters θ) the function’s slope is rising the most\nsteeply, and moving in the opposite direction. \n\nStochastic gradient descent chooses a single random example at a time and updates\nits weights on that example. As a result the updates can fluctuate.\nAn alternate is batch training, which computes the gradient over the entire dataset.\nThis gives a much better estimate of which direction to move the weights but takes a\nlong time to compute.\nA commonly used compromise is mini-batch training, where we train on a small\nbatch. The batch size can be 512 or 1024, often selected based on computational\nresources, so that all examples in the mini-batch can be processed in parallel. The loss\nis then accumulated.\n\nOverfitting is a problem with many machine learning models. Overfitting results in\npoor generalization and poor performance on unseen test set.\nIn logistic regression, if a feature only occurs in one class then it will get a high weight.\nSometimes we are just modelling noisy factors that just accidentally correlate with the\nclass.\nRegularization is a way to penalize large weights. A regularization term is added to\nthe loss function.\nLasso regression uses L1 regularization\nRidge regression uses L2 regularization \n\nInstead of directly outputting z = w·x+b, which is a linear function of x, neuron units\napply a non-linear function f to z.\nThe output of this function is called the activation value for the unit, represented by\nthe variable a. The output of a neural network is called y, so if the activation of a node\nis the final output of a network then\ny=a= f(z)\nThere are 3 commonly used non-linear functions used for f:\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\n\nThe simplest kind of NN is the Feed-Forward Neural Network\nMultilayer network, all units are usually fully-connected, and no cycles.\nThe outputs from each layer are passed to units in the next higher layer, and no\noutputs are passed back to lower layers. \n\nThe advantage of using a single matrix W for the weights of the entire layer is the\nhidden layer computation can be done efficiently with simple matrix operations.\nThe computation has three steps:\n1. multiplying the weight matrix by the input vector x,\n2. adding the bias vector b, and\n3. applying the activation function g (such as Sigmoid)\nThe output of the hidden layer, the vector h, is thus the following, using the sigmoid\nfunction σ:\nh = σ(Wx+b) \n\nLike logistic regression, we want to learn the best parameters for the neural net to\nmake its predictions �/ as close to possible as the gold standard labels in our training\ndata y.\nWhat do we need?\nA loss function – cross-entropy loss\nAn optimization algorithm – gradient descent\nA way of computing the gradient of the loss function – error propagation \nAlthough backpropagation was invented for neural nets, it is related to general\nprocedure called backward differentiation, which depends on the notion of\ncomputation graphs.\nA computation graph represents the process of computing a mathematical expression.\nThe computation is broken down into separate operations. Each operation is a node in\na graph.\n\n\n## Module 13 Definitions\n* **** - \n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 14 Lecture Notes\nHow should we represent the meaning of a word?\nIn N-gram LMs we represented words as a string of letters or as an index in a\nvocabulary list.\nIdeally, we want a meaning representation to encode:\n1. Synonyms – words that have similar meanings\n2. Antonyms – words that have opposite meanings\n3. Connotations – words that are positive or negative\n4. Semantic Roles – buy, sell, and pay are different parts of the same underlying\npurchasing event\n5. Support for entailment\nMost words don’t have many synonyms, but they do have a lot of similar words. Cat is\nnot a synonym of dog, but cats and dogs are certainly similar words. Context windows\nOccur in a window of +/- 2 words, in the same sentence, in the same document\nInstead of window of words use more complex contexts:\ndependency patters. Subj-of-verb, adj-mod, obj-of-verb\nLanguages have long distance dependencies.\nAlternates to counts\nRaw word frequency is not a great measure of association between words. It’s very\nskewed “the” and “of” are very frequent, but maybe not the most discriminative\nWe’d rather have a measure that asks whether a context word is particularly\ninformative about the target word.\nInstead of raw counts, it’s common to transform vectors using TF-IDF or PPMI\nSparse v. Dense Vectors\nCo-occurrence matrix (weighted by TF-IDF or mutual information)\n• Long (length |V| = 50,000+)\n• Sparse (most elements are zeros)\nAlternative: learn vectors that are\n• Short (length 200-1000)\n• Dense (most elements are non-zero)\nLanguage modeling Goal: Learn a function that returns the joint probability\nPrimary difficulty:\n1. There are too many parameters to accurately estimate. This is\nsometimes called the “curse of dimensionality”\n2. In n-gram-based models we fail to generalize to related words / word\nsequences that we have observed.\n\nIn LMs we user chain rule to get the conditional probability of the next\nword in the sequence given all of the previous words:\n\nWhat happens when we have a new (unseen) combination of n words?\n1. Back-off\n2. Smoothing / interpolation\nWe are basically just stitching together short sequences of observed\nwords.\n\nNeural Language models have several advantages over n-gram LMs:\n1. They don’t need smoothing\n2. They can handle much longer histories.\n3. They can generalize over contexts of similar words.\n4. Neural LMs tend to have much higher predictive accuracy than n-gram LMs.\nDisadvantage: slower to train than traditional n-gram LMs\n\nAdvantages of NN LMs\nBetter results. They achieve better preplexity scores than SOTA n-gram LMs.\nLarger N. NN LMs can scale to much larger orders of n. This is achievable because\nparameters are associated only with individual words, and not with n-grams.\nThey generalize across contexts. For example, by observing that the words blue,\ngreen, red, black, etc. appear in similar contexts, the model will be able to assign a\nreasonable score to the green car even though it never observed in training, because it\ndid observe blue car and red car.\nA by-product of training are word embeddings\n\nA Neural Probabilistic LM\nGiven:\nA training set w1 … wt where wt ∈V\nLearn:\nf(w1 … wt) = P(wt|w1 … wt-1)\nSubject to giving a high probability to an unseen text/dev set (e.g. minimizing\nthe perplexity)\nConstraint:\nCreate a proper probability distribution (e.g. sums to 1) so that we can take\nthe product of conditional probabilities to get the joint probability of a\nsentence\n\nSummary: How to learn word2vec (skip-gram) embeddings\nStart with V random 300-dimensional vectors as initial\nembeddings\nUse logistic regression\n• Take a corpus and take pairs of words that co-occur as positive\nexamples\n• Take pairs of words that don't co-occur as negative examples\n• Train the classifier to distinguish these by slowly adjusting all the\nembeddings to improve the classifier performance\n• Throw away the classifier code and keep the embeddings.\n\nIn a word-analogy task we are given two pairs of words that share a\nrelation (e.g. “man:woman”, “king:queen”).\nThe identity of the fourth word (“queen”) is hidden, and we need to infer\nit based on the other three by answering\n“man is to woman as king is to — ?”\nMore generally, we will say a:a∗ as b:b∗.\nCan we solve these with word vectors?\n\nImplicit Association test (Greenwald et al 1998): How associated are\n• concepts (flowers, insects) & attributes (pleasantness, unpleasantness)?\n• Studied by measuring timing latencies for categorization.\nPsychological findings on US participants:\n• African-American names are associated with unpleasant words (more\nthan European-American names)\n• Male names associated more with math, female names with arts\n• Old people's names with unpleasant words, young people with pleasant\nwords.\n\nConclusion\nEmbeddings = vector models of meaning\n• More fine-grained than just a string or index\n• Especially good at modeling similarity/analogy\n• Just download them and use cosines!!\n• Can use sparse models (tf-idf) or dense models (word2vec, GLoVE)\n• Useful in practice but know they encode cultural stereotypes\n\n\n## Module 14 Definitions\n* **** - \n\n\n# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n An makes a decision: has avaliable some actions a, assigns a probability\nP(s) to each possible state. The transition model is given by P(s'|s,a) the probability that \naction a in state s reaches state s'  $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$  An agent's preferences \nare captured by a utility function. Expected utility:  $EU(a)) = \\sum_s P(\\textnormal{RESULT}(a)=s')U(s')$  MEU:  $action = \\textnormal{argmax}_{a}EU(a)$  The MEU\nformalizes the geneearal notion that an intelligent agent should do the right thing but it \ndoes not operationalize that advice. An agent may not know how good a state is until it\nknows where it can go from that state. So if an agent acts so as to maximize a utility function\nthat correctly reflects the performance measure, then the agent will achieve the highest \npossible perfomace score. \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n An agent's preferences (handwritten). Lottery (handwritten). All formulas\nare also handwritten. The axioms of utility theory are really axioms about prefereneces. But \nwe can derrive the existence of a utility function, and the expected utility of a lottery.\nThe outcome of a nondeterministic action is a lottery, an agent can only act rationally by \nchoosing an action that maximizes expected utility. In a deterministic environment an agent\nneeds only a preference ranking on states- the numbers dont matter\n\n## AIMA Chapter 16.3 Utility Functions\n Utility functions map from lotteries to real numbers. \n\n16.3.1 Utility assessment and utility scales\nTo built decision-theoretic system that helps a human make decisions. We must find \nthe human's utility function, preference elicitation. We can assist the utility of \nany prize S by asking teh agentt to choose between S and a standard lottery. \n\n16.3.2 The utility of money\nIf an agent prefers more money the agent has a monotonic preference for \nmoney. The utility of money was almost exactly proportional to the logarithm of the amount. \n\n16.3.3 Expected utility and post-decision disappointment:\nThe rational way to choose the best action a*, is to maximize the expected utility: a* = argmax_aEU(a)\nProbability density function $P(x) = \\frac{d}{dx}((F(x)^{k})=kf(x)(F(x))^{k-1}$  With\nmore choices, extremely optimistic estimates are more likely to arise. We can avoid hte optimizer's curse\nwith a Bayesian approach that uses and explicit probability model P(EU^|EU). \n\n16.3.4 Human judgment and irrationality\nDecision theory is a normative theory. Humans are \"predicatably irrational. An explanation\nfor the apparently irrational preference is the certainty effect. People feel more comfortable\nmaking relative utility judgements rather than absolute ones. The very fact that different\nwordings of the same decision problem elicit different choices suggests that the decision problem itself is not getting through.\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n If the environment were deterministic, a solution would be easy.\nBut the environment is stochastic so it's actions are unreliable. The transition model\ndescribes the outcome of each action in each state. P(s'|s,a) is the probability of reaching\nstate s' if action a is done in state s. If a decision problem is sequential the utility\nfunction will depend on an environment history. For every transition from s to s' via a the\nthe agent recieves a reward R(s,a,s'), they may be + or - but are bounded by +/- Rmax.\nThe utility of an environment history is the sum of the rewards recieved. Methods for solving\nMDPs usually involve dynamic programming. No matter what the outcome of the action, the\nresulting state will be in the policy, and the agent will know what to do next. The quality\nof a policy is measured by the expected utility of the possiblke environment histories genterated\nby it. Optimal policy is denoted by pi*.\n\n17.1.1 Utilities over time\nFirst check if there is a finite horizon or an infinite horizon for decision making. \nWith a finite horizon, an optimal action in a given state may depend on the time left.\nWith no fixed time limit there is no reason to behave differently in the same state at differetn times\nWith additive discounted rewards the utility of a history is: $U_{h}([s_{0},a_{0},s_{1},a_{1},...]) = R(s_{0},a_{0},s_{1})+\\gamma R(s_{1},a_{1},s_{2})+\\gamma^{2} R(s_{1},a_{1},s_{2})$ three \nsolutions to equations that lead to infinite histories: 1. with discounted\nrewards the utility of an infinite sequence is finite 2. if the environment has terminal\nstates and we are guaranteed to get on eeventually then we will never need to compare infinite\nsequences 3. infinite sequences can be compared in terms of the average reward obtained \nper time step.\n\n17.1.2 Optimal policies and the utilities of states\nCompare policies by comparing their expected utilities them.  $U^{\\pi}(s) = E[\\sum_{t=0}^{\\infty} (\\gamma^{t}R(S_{t},\\pi(S_{t}),S_{t+1})]$ Optimal \npolicy is $\\pi_{s}^{*}=(\\textnormal{argmax}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ Consequence of using discounted utilities with infinite horizons is\nthe optimal policy being independent of the starting state. The utility of a state is the expected reward for the\nnext transition plus the discounted utility of the next state Bellman equation: $U(s)= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U(s')])$ Q \nfunction: $U(s)= \\textnormal{max}_{a}Q(s,a)$  We can extract the optimal policy from the q\nfunction $\\pi ^{*}= \\textnormal{argmax}_{a}Q(s,a)$  ``` \nfunction Q-value(mdp, s,a,U) returns a utility value\n  return sum_s' P(s'|s,a){[R(s,a,s')+gammaU[s']] ```\n  \n17.1.4 Representing MDPs\nDynamic decision networks, or DDNs. They typically have an exponential complexity advantage\nover atomic representations and can model quite substantial real-world problems.\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Bellman update:  $U(s)_{i+1}= \\textnormal{max}_{a \\epsilon A(s)}\\sum_{s'}P(s'|s,a)[R(s',s,a)+\\gamma U_{i}(s')])$ is to be applied simultaneously to \nall states at each iteration\n``` \n function Value-Iteration(mdp, e) returns a utility function\n    inputs mdp, with states S, actions A(s) transistion model P(s'|s,a), rewards R(s,a,a') discount gamma\n      e, the maximum error allowed in the utility of any state\n    local variables: U,U', vectors of utilities for stares in S, initially zero\n        delta, the maximum relative change in the utility of any state\n    repeat\n      U<- U'; delta <-0\n      for each state s in S do\n        U'[s]<-max_aeA(s) Q-Value(mdp,s,a,U)\n        if|U'[s]-U[s]| > delta then delta <-|U'[s] - U[s]|\n      until delta<= e(i-gamma)/gamma\n      return U\n``` \nValue iteration eventually converges to a unique set of solutions to the bellman equation. Contraction\nhas one fixed point, when applied the function must get closer to the fixed point.\n\nPolicy iteration stops when the policy improvemtn step yields no change in utilities\n``` \n function Policy-Iteration(mdp) returns a policy\n  inputs: mdp wiht stares, actions, transition model\n  local variables: U, a vector of utilites for states in S (initially 0)\n      pi, policy vector indexed by state, initially random\n\n  repeat\n    U<-Policy-Evaluation(pi, U, mdp)\n    unchanged <- true\n    for each state s in S do\n      a*<- argmax Q-value(mdp,s,a,U)\n      if Q-Value(mdp,s,a*,U)> Q-Value(mdp,s,pi[s],U) then\n        pi[s] <- a*; unchanged? <- false\n    until unchanged?\n    retunr pi\n``` \nAsynchronous policy iterstion is guaranteed to converege to an optimal policy. Linear Programming\nseeks to minimize U(s) for all s subject to the inequalities.\n\n\n## Module 8 Lecture Notes\n In our search algorithms so far, the\ntransition model was deterministic and\ndescribed the outcome of each action in\neach state.\nThe transition function is sometimes\nwritten as T(s, a, s’), or explicitly as a\nprobability:\n\nReward function\nWe will specify a utility or reward function for the agent.\nThe “rewards“ can be positive or negative but are bounded by some maximum value.\nBecause the decision process is sequential, we must specify the utility function on a\nsequence of states and actions.\nInstead of only giving a reward at the goal states, the agent can receive a reward at\neach time step, based on its transition from s to s’ via action a.\nThis is defined by a reward function\nR( s, a, s’ )\nFor example, we could give the Millennium Falcon a small negative reward of -0.04 for every\ntransition except for entering the terminal states (+1 for entering the planet’s orbit or -1 for\nsmashing into an asteroid).\nThe rewards are additive, so if the Millennium Falcon takes 4 steps before entering\nthe planet's orbit, it gets -0.04 + -0.04 + -0.04 + -0.04 + 1 = 0.84 for that solution\n\nMarkov Decision Process\nA Markov decision process or MDP is\n• a sequential decision problem\n• for a fully observable environment\n• with a stochastic transition model\n• that has additive rewards\nMDPs are non-deterministic search\nproblems. One way of solving them is via\nexpectimax search.\n\nMarkov Decision Process\nTo find a solution to an MDP, you need to define the following things:\n• A set of states s ∈ S\n• A set of actions a ∈ A\n• A transition function T(s, a, s’)\n• Probability that executing action a in s will lead to s’ P(s’ | s, a)\n• The probability is called the model\n• A reward function R(s, a, s’)\n• Sometimes just R(s) or R(s’)\n• An initial state s0\n• Optionally, one or more terminal states\n\nSolution == Policy\nIn search problems a solution was a\nsequence of action that corresponded to\nthe shortest path.\nBecause of the non-determinism in MDPs\nwe cannot simply give a sequence of\nactions.\nInstead, the solution to an MDP is a policy.\nA policy maps from a state onto the action\nto take if the agent is in that state.\n𝝿(s) = a\n\nOptimal Quantities\n▪ The value (utility) of a state s:\nV*(s) = expected utility starting in s and\nacting optimally\n▪ The value (utility) of a q-state (s,a): Q*(s,a) = expected utility starting out\nhaving taken action a from state s and\n(thereafter) acting optimally\n▪ The optimal policy:\nPi*(s) = optimal action from state s\n\nPolicy Iteration\nAlternative approach for optimal values:\n• Step 1: Policy evaluation: calculate utilities for some fixed policy (not\noptimal utilities!) until convergence\n• Step 2: Policy improvement: update policy using one-step lookahead with resulting converged (but not optimal!) utilities as future\nvalues\n• Repeat steps until policy converges\nThis is policy iteration\n• It’s still optimal!\n• Can converge (much) faster under some conditions\n\nComparison\nBoth value iteration and policy iteration compute the same thing (all optimal\nvalues)\nIn value iteration:\n• Every iteration updates both the values and (implicitly) the policy\n• We don’t track the policy, but taking the max over actions implicitly recomputes\nit\nIn policy iteration:\n• We do several passes that update utilities with fixed policy (each pass is fast\nbecause we consider only one action, not all of them)\n• After the policy is evaluated, a new policy is chosen (slow like a value iteration\npass)\n• The new policy will be better (or we’re done)\nBoth are dynamic programs for solving MDPs\n\nSummary: MDP Algorithms\nSo you want to….\n• Compute optimal values: use value iteration or policy iteration\n• Compute values for a particular policy: use policy evaluation\n• Turn your values into a policy: use policy extraction (one-step\nlookahead)\nThese all look the same!\n• They basically are – they are all variations of Bellman updates\n• They all use one-step lookahead expectimax fragments\n• They differ only in whether we plug in a fixed policy or max over\nactions\n \n\n## Module 8 Definitions\n* **Utility function** - assigns a single number to represend the desrability of a state\n* **Expected utility** - the average utility value of the outcomes, weighted by the probability that the outcome occurs\n* **Maximum expected utility (meu)** - a rational agent should choose the action that maximizes the agent's expected utility\n* **Lottery** - the set of outcomes for each action\n* **Orderability** - given 2 lotteries, a rational agent must either prefer\n* **Transitivity** - given any 3 lotteries if an agent preferes A to B and B to C then the agent must prefer A to C\n* **Continuity** - probability p the agent is indifferent between B and L that yields A with p and C with p-1\n* **Substitutibility** - if agent is indifferent between A and B then also indifferent between more complex lotteries with b sub a\n* **Monotonicity** - if an agent prefers A to B then the agent must prefer the lottery that has a higher prob for A\n* **Decomposability** - compound lotteries can be reduced to simpler ones using the laws of probability\n* **Existence of utility function** - if agent's preferences obey axioms of utility then there is func U that obeys\n* **Expected utility of a lottery** - the utility of a lottery is the sum of the probabilities of each outcome times their utilities\n* **Value function** - preference ranking on states\n* **Ordinal utility function** - value function\n* **Preference elicitation** - present choices to the human and using the observed preferences to find the utility function\n* **Normalized utilities us a scale with worst = 0 and best = 1** - \n* **Value of a statistical life** - determines the costs and benefits of regulations and interventions\n* **Micromort** - one in a million chance of death\n* **Qaly** - quality adjusted life year\n* **Risk-averse** - prefer a sure thing with a payoff that is less than the EMV of a gamble\n* **Risk-seeking** - the opposite of risk averse\n* **Certainty equivalent** - the value an agent will accept in lieu of a lottery\n* **Insurance premium** - the difference between the EMV of a lottery and its certanty equivalent\n* **Risk-neutral** - an agent that has a linear curve\n* **Unbiased** - the expected value of the error is zero\n* **Order statistic** - the distrobution of any particular ranked element of a sample\n* **Optimizer's curse** - the tendency for the estimated expected utility of the best choice to be too high\n* **Normative theory** - describes how a rational agent should act\n* **Descriptive theory** - describes how actual agents really do act\n* **Certainty effect** - people are strongly attracted to gains that are certain\n* **Ambiguity aversion** - most people elect the known proabability rather than the unknown unknowns\n* **Framing effect** - the exact wording of a decision problem can have a big impact on the agent's choices\n* **Anchoring effect** - proposing something outrageous so that settling with somthing slightly crazy isnt crazy\n* **Fully observable** - if the agent's sensors give it access to the complete state of the environment at every point in time\n* **Markovian** - the probability of reaching s' from s depends only on s not the history\n* **Environment history** - a sequence of states and action\n* **Markov decision process mdp** - sequential decision problem for a fully observable, stochastic environment with Markovian transition model and additive rewards\n* **Dynamic programming** - simplifying a problem by recursively breaking it into smaller pieces\n* **Policy** - what an agent should do for any state that the agent might reach\n* **Optimal policy** - yields the highest expected utility\n* **Finite horizon** - there is a fixed time N after which nothing matters-the game is over\n* **Nonstationary** - a policy that depends on the time\n* **Stationary** - optimal policy depends only on the current state\n* **Discount factor (gamma)** - [0,1] describes the agent's preferences for current rewards over future rewards\n* **Additive reward** - when gamma is 1\n* **Attribute** - each transition s_t -> s_t+1 is an attribute of the the history\n* **Propper policy** - a policy that is guaranteed to reach a terminal state\n* **Q-function (action-utility function)** - expected utility of taking a given action in a given state\n* **Sparse** - when most entries in MDP tables are zerso\n* **Dynamic decision network** - factored representations, \n* **Factored representations** - \n* **Value iteration** - n possible states then n bellman equations\n* **Contraction** - function of one argumentthat when applied to 2 inputs produces 2 closer together outputs\n* **Max norm** - measures the distance between utility vectors, measures the length of a vector by abs value of its biggest component.\n* **Policy loss** - ||U^pi_i-U||\n* **Policy iteration** - alternates  policy evaluation and policy improvent starting at an initial policy pi0\n* **Policy evaluation** - given pi_i calculate Ui=U^pi_i the utility of each state if pi_i were to be executed\n* **Policy improvement** - calculate a new MEU policy pi_i+1, using one step look ahead based on Ui\n* **Modified policy iteration** - uses simplified bellman equation\n* **Asynchronus policy iteration** - pick any subset ofof states and apply either kind of updating (policy improvement or value iteration)\n* **Linear programming** - general approach for formulating constrained optimization problems\n* **Real time dynamic programming** - like minimax, alternates max and chance nodes\n* **** - \n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n There is a tradeoff between exploitation of the current best option\nand exploration of previously unknown states to gain information. There are several\ndefinitions of bandit problems. The most general: each arm M is a Markov reward process. \nWith states Si, transition model Pi, and reward Ri. The arm defines a distrobution over\nsequences of rewards Ri0, Ri1,... A policy for any bandit problem: pull the arm that\nhas the highest gittens index then update the gittens indicies.\n\n17.3.2 The Bernoulli bandit\nThe state of arm Mi is defined by si and fi, the counts of successes and failures,\nthe transition probability predicts the next outcome to be 1 with probability (si)/(si+fi)\nand 0 with probability (fi)/(si+fi). There is an exploration bonus associated with arms\nthat have only been tried a few times\n\n17.3.3 Approximately optimal bandit policies\nFirst class of methods use the upper confidence bound.\nThompson sampling chooses an arm randomly according to the probability that the arm is\nin fact optimal\n\n17.3.4 Non-indexable variants\nSelection problem - making a decision in the midst of conditions. \nthe ordinary term for BSPs is multitaksing. We can take advantage of the loose nature\nof the interaction between the arms. This can be modeled by opportunity cost.\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n Reinforcement learning is an alternative to needing huge databases of \nmoves and their actions. Reinforcement learning differes from just solving the MDP because\nthe agent is not given the MDP to solve, the agent is in the MDP. If we can provide the\ncorrect reward signal to the agent reinforcement learning provides a very general way to build AI\nsystems. Especially in simulated environmentes when there is no shortage of opportunities\nto gain experience. Some key basic reinfrocement learning approaches are:\n\nModel-based reinforcement learning: agent uses a transition model of the environment to \nhelp interpret the reward signals and decide how to act. If the model is unknow the agent\nlearns it by observing the effects of its actions\n\nModel-free reinforcement learning: The agent never learns or knows the transition model.\nIt learns a more direct representation of how to behave. In one of 2 varieties: 1) action \nutility learning like Q learning. Where the agent learns a Q-function then chooses what to\ndo in s by finding the action with the highest q. 2) policy search: the agent learns a policy pi(s)\nthat maps directly from states to actions. this is a reflex agent\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Passive learning is similar to policy evaluation, except passive \nlearning agent does not know the reansistion model or the reward function. The expected \nutility associated with each state s: $U^{\\pi}(s) = E[\\sum_{t=0}^{\\infty} (\\gamma^{t}R(S_{t},\\pi(S_{t}),S_{t+1})]$ \n\n22.2.1 Direct utility estimation \nReduces reinfrocement learnign to a standard supervised learning problem in which\neach example is a (state, reward-to-go) pair. The utility of a state is determined by the\nreward and expected utility of the successor states-> the utility obeys the bellman equation.\nBy ignoring the connections between states, direct utility estimation misses opportunities\nfor learning.\n\n22.2.2 Adaptive dynamic programming\nTakes advantage of the constraints among the utilities of states by learning the transition\nmodel that connects them, then solving the corresponding Markov decision process using\ndynamic programming. For a passive learning agent we would plug the learned transition model and the\nobserved rewards into the bellman equation.\n```\nfunction Passibe-ADP-Learner(precept) returns an action\n  inputs: percept, a percept indicating the current state s' and reward signal r\n  persistent: pi, a fixed policy\n              mdp, an MDP with model P, rewards R, actions A, discount gamma\n              U, a table of utilities for states, initially empty\n              Ns'|sa a table of outcome count vectors indexed by state and action, initaially zero\n              s, a the previous state and action, initailly null\n  if s' new then U[s']<-0\n  if s not null then \n    increment Ns'|s,a[s,a][s']\n    R[s,a,s']<-r\n    add a to A[s]\n    P( |s,a)<-NORMALIZE(N_s'|s,a [s,a])\n    U<- POLICY EVALUATION(pi, U, mdp)\n    s,a <- s', pi[s']\n    return a\n```\n22.2.3 Temporal difference learning\nYou can also use the observed transistions to adjust the utilites of the oberved\ntransitions to adjust the utiliteis of the oberved states so that they agree with\nthe constraint equations. When the transition occurs apply this update\n $U^{\\pi}(s)\\leftarrow ^{\\pi} \\alpha[R(s',s,a)+\\gamma U^{\\pi}(s')-U^{\\pi}(s)]$  where alpha is the learning rate. This is know as the temporal-difference\nequation.\n```\nfunction Passive-TD-Learner(percept) returns an action\n  inputs: percept, a percept indicating the current state s' and reward signal r\n  persistent: pi, a fixed policy\n              s, the previous state, initially null\n              U, a table of utilities for states, initially empty\n              Ns, a table of frequencies for states, initially zero\n  if s' is new then U'[s']<-0\n  if s is not null then \n    increment Ns[s]\n    temporal difference equation\n  s<-s'\n  return pi[s']\n```\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n An active learning agent gets to decid what actions to take. Adaptive\ndynamic programming (ADP) agent: first needs to learn a complete transition model with \noutcome probabilities for all actions, next take into account the fact that hte agent \nhas a choice of actions. The utilities it needs to learn are those defined by the optimal\npolicy (obey the bellman equation). With U the agent can extract an optimal action by \none-step look-ahead to maximize the expected utility. If it uses policy iteration it could simply\nexecute the action the optimal policy recommends.\n\n22.3.1 Exploration\nSometimes greed pays off but it often does not. A learned model is not the same as\nthe true environment, what is learned could be suboptimal in the real environment. Greedy\nagent overlooks the fact that actions provide more than rewards they provide information\nin the form of percepts in the resulting state. An ADP using a GLIE scheme will eventually learn \nthe true transition model. A good approach is to give some weight to actions the agent has not\ntried very often while avoiding those that are believed to be of low utility.\n\n22.3.2 Safe exploration\nMany actions are irreversible, in the sense defined for online search agents: no set of \nactions can restore it to where it was before. Worst case the agent enters an absorbing state.\nA better idea to avoid catastrophe is to choose a plicy that works reasobably for a range of \nmodels that have a chance of being true, even if it is suboptimal for the maximum-likelihood model.\nThere are 3 mathematical approaches for this: 1) Bayesian reinforcement learning: posterior probablility P(h|e) is \nobtatined by bayes' rule given the observations to date. $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\sum_{h}P(h|e)U_{h}^{\\pi})$ 2) robust control theory allows for a\nset of possible models H wihtout assigning probabilities to them adn defines an optimal\nrobust policy as on that gives the best outcome in the worst case over H: $\\pi_{s}^{*}=(\\textnormal{argmax}_{\\pi}\\textnormal{min}_{h}U_{h}^{\\pi})$  This\nis often the set of models that exceed some likelihood threshold on P(h|e). The problem is that\nthis results in overly conservative behavior.\n\n22.3.3 Temporal-difference Q-learning\nThis avoids the need for a model by learning an action-utility function instead of a utility\nfunction. Q(s,a) denotes the expected total discounted reward if the agent takes action a in s\nand acts optimally after.\n\n``` \nfunction Q-learning-agent(percept) returns an action\n  inputs: percepts, a percept indicating teh current state s' and reward signal r\n  persistent: Q, a table of action values indexed by state adn action, initially zero\n              Nsa, a table of frequencies for state-action pairs, initially zero\n              s, a ther previous stat and action, initially null\n  if s is not null then\n    increment Nsa[s,a]\n    Q[s,a]<- Q[s,a] + alpha(Nsa[s,a])(r + gamma*max_a' Q[s',a']-Q[s,a]\n  s,a <- s', argmax_a' f(Q[s',a'], N_sa[s',a'])\n  return a\n``` \n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Evaluation functions are approximate utility functions.\nAdding look-ahead search means that effective behavior can be generated from a much simpler\nutility function approximator that is learnable from far fewer experiences. Allows for inductive\ngeneralization: the agent can generalize from states it has visited to states it has not. Backgamon!\n\n22.4.1 Approximating direct utility estimation\nThe state and the sum of rewards received constitute a training example for a supervised learning\nalgorithm. Reinforcement learning makes more sense to use an online learning algorithm that\nupdates after each trial. The rate of change of the error wrt each parameter (theta_i)\nis partial dEj/dthetai. Changing the parameters theta_i in response to an \nobserved transition between two states changes the values of U^theta for every other state!\nLinear function approximation:the function must be linear in the features. But these can\nbe arbitrary nonlinear functions of the state variables.\n\n22.4.2 Approximaating temporal-difference learning\nAdjust the parameters to try to reduce the temporal difference between successive states\nthe update rule can be shown to converge to the closest possible approximation to the\ntrue function when the function approximator is linear in the features. With active\nlearning and nonlinear functions such as neural networks, tricky.\nLearning a model for an observable environment is a supervised learning problem,\nbecause the next percept gives the outcome state. Partially observable environments have \na much harder learning problem. The next percept is no longer known.\n\n22.4.3 Deep reinforcement learning\nIt is always possible to represent U or Q as linear combinations of features, if they aren't\nin an efficiently computible form the linear function approximator\nmay be insufficient. The parameters are the layers of the network. Obstacles:\nit is often difficult to get good performance and the trained system may behave\nvery unpredictably if the environment differs even a little from the training data.\n \n22.4.4 Reward shaping\nReal world environments may have sparse rewards: many actions are required\nto achieve any nonzero reward. R'(s,a,s') = R(s,a,s') + gamma*phi(s') - phi(s)\nphi can be constructed to reflect any desirable aspects of the state.\n\n22.4.5 Hierarchical reinforcement learning \nDeal with long action sequences by breaking them into small pieces, then again until\nthey are short enough to make learning easy. Begins with a partial program. The theoretical\nfoundations of HRL are based on the concept of the joint spate space. This is all essentially\nlike solving an MDP with: states->choice states sigma of the joint space, the \nactions at sigma are the choices avaliable in sigma from the partial program, the \nreward functionp(sigma,c,sigma') is the expected sum of rewards for all physical transition, the\ntransition model t(sigma,c,sigma') -> if c invokes a physical\naction a, then t borrows from the physical model P(s'|s,a)\n\n## Module 9 Lecture Notes\n That wasn’t planning, it was learning!\nSpecifically, reinforcement learning\nThere was an MDP, but you couldn’t solve it with just computation\nYou needed to actually act to figure it out\nImportant ideas in reinforcement learning that came up\nExploration: you have to try unknown actions to get information\nExploitation: eventually, you have to use what you know\nRegret: even if you learn intelligently, you make mistakes\nSampling: because of chance, you have to try things repeatedly\nDifficulty: learning can be much harder than solving a known MDP\n\nReinforcement Learning\nStill assume a Markov decision process (MDP):\nA set of states s ∈ S\nA set of actions (per state) A\nA model T(s,a,s’)\nA reward function R(s,a,s’)\nStill looking for a policy π(s)\nNew twist: don’t know T or R\nI.e. we don’t know which states are good or what the actions do\nMust actually try actions and states out to learn\n\nModel-Based Learning\nModel-Based Idea:\nLearn an approximate model based on experiences\nSolve for values as if the learned model were correct\nStep 1: Learn empirical MDP model\nCount outcomes s’ for each s, a\nNormalize to give an estimate of T\nDiscover each R when we experience (s, a, s’)\nStep 2: Solve the learned MDP\nFor example, use value iteration, as before\n\nPassive Reinforcement Learning\nSimplified task: policy evaluation\nInput: a fixed policy π(s)\nYou don’t know the transitions T(s,a,s’)\nYou don’t know the rewards R(s,a,s’)\nGoal: learn the state values\nIn this case:\nLearner is “along for the ride”\nNo choice about what actions to take\nJust execute the policy and learn from experience\nThis is NOT offline planning! You actually take actions in the world.\n\nHow to Explore?\nSeveral schemes for forcing exploration\nSimplest: random actions (ε-greedy)\nEvery time step, flip a coin\nWith (small) probability ε, act randomly\nWith (large) probability 1-ε, act on current policy\nProblems with random actions?\nYou do eventually explore the space, but keep\nthrashing around once learning is done\nOne solution: lower ε over time\nAnother solution: exploration functions\n\nEven if you learn the optimal policy,\nyou still make mistakes along the way!\nRegret is a measure of your total\nmistake cost: the difference between\nyour (expected) rewards, including\nyouthful suboptimality, and optimal\n(expected) rewards\nMinimizing regret goes beyond\nlearning to be optimal – it requires\noptimally learning to be optimal\nExample: random exploration and\nexploration functions both end up\noptimal, but random exploration has\nhigher regret\n\n\n## Module 9 Definitions\n* **One armed bandit** - slot machine\n* **Exploitation** - taking advantage of the current best option\n* **Exploration** - an agent must explore lots of its terrain to learn how to behave in it\n* **Markov reward process** - mdp with only one possible action ai\n* **Stopping time** - if you pull a arm T times the stopping time is T\n* **Bernoulli bandit** - each arm Mi produces a reward of 0 or 1 with fixed but unknown probability mu_i\n* **Upper confidence bound** - use the samples from each arm to establish a confidence interfal for the value of the arm\n* **Confidence interval** - range within which the value can be estimate to lie with high confidence\n* **Bandit superprocess** - each arm is a full MDP in its own\n* **Opportunity cost** - how much utility is given up per time step by not devoting that time toanother arm\n* **Reinforcement learning** - an agent interacts with the world and periodically recieves rewards or reinforcements\n* **Rewards** - reflect how well the agent is doing\n* **Reinforcements** - psychology term for rewards\n* **Passive reinforcement learning** - the agent's policy is fixed and the task is to learn the utilities of the states\n* **Active reinforcement learning** - the agent must also figure out what to do: exploration\n* **Apprenticeship learning** - training a learning agent using demonstrations rather than reward signals\n* **Direct utility estimation** - the utility of a state is defined as the expected total rewardfrom that state on\n* **Reward-to-go** - expected total reward from that state onward\n* **Temporal difference methods** - adjust the utility estimate towards the ideal equilibrium\n* **Pseudoexperience** - generated by simulating the current transition model\n* **Prioritized sweeping** - heuristic prefers to make adjustments to states whose likely successors have just undergone a large adjustment in their own utility estimates.\n* **Active learning agent** - decides what actions to take\n* **Greedy agent** - takes the action that it currently believes to be optimal at each step\n* **Glie (greedy in the limit of infinite exploration)** - must try each action in each state an unbounded number of times to avoid missing an optimal action\n* **Exploration function** - f(u,n) bellman equation, number of times a state has been tried\n* **Absorbing stare** - no actions have any effect and no rewards are recieved\n* **Sarsa (state action reward state action)** - updates with the Q-value of the action a'\n* **Off-poliicy** - learns q values that answer what would this be worht in this state, assuming i stop using the current policy?\n* **On-policy** - what would this action be worth in the state, assuming i stick with my policy\n* **Bayesian reinforcement learning** - assumes prior probability P(h) over hypothesis h about what the model is\n* **Function approximation** - process of constructing a compact approximation of the true utility function or Q-function\n* **Widrow-hoff rule** - delta rule equation 22.10\n* **Catastrophic forgetting** - a nonzero weight on a inear feature can cause a car to slide bc it forgets danger\n* **Experience replay** - ensures the car relives its youthful crashing behavior so it doesnt forget\n* **Deep reinforcement learning system** - reinforcement learning system that uses a deep network as a function approximator\n* **Reward shaing** - supplying teh agent with additional rewards for making progress\n* **Pseudorewards** - extra rewards\n* **Partial program** - outlines a hierarchial structure for the agent's behavior\n* **Joint state space** - each state (s,m) is composed of a physical state and a machine state\n* **Choice state sigma=(s,m)** - program counter for m is at a choice point in the agent program\n* **Additive decomposition** - overall utility decomposes into several terms, depeding on a few variables\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Agents in the real world need to handle uncertainty. We have seeen\nagents handle this with a belief state. This works on simple problems but has drawbacks:\nthe agent must consider every possible explanation for its sensor observation, a correct\ncontingent plan must consdider arbitarily unlikely contigencies, sometimes there is no plan\nguaranted to achieve the goal- but the agent must act. The right thing to do—the rational\ndecision—therefore depends on both the relative importance of various goals and the\nlikelihood that, and degree to which, they will be achieved.\n\n12.1.1 Summarizing uncertainty\nTrying to use logic to cope wiht domain like medical diagnosis fails for 3 main reasons:\n1) laziness 2) theoretical ignorance 3) practical ignorance. Agent's knowledge can at best\nprovide only a degree of belief. This can be dealt with probability theory. The ontological \ncommitments of logic and probability theory are the same. The theory of probability provides\na way of summarizing the uncertainty that comes from our laziness and ignorance.\n\n12.1.2 Uncertainty and rational decisions\nTo make choices and agent must have preferences among the different possible outcomes of \nthe various plasn. Use utility theory to represent preferences and reason. A utility\nfunction can account for any set of preferences—quirky or typical, noble or perverse. \nDecision theory = probability theory + utility theory. An agent is rational if and only\nif it chooses the action that yields the highest expected utility, averaged over all\nthe possible outcomes of the action.\n\n``` \nfunction DT-agent(percept) returns and action\n  persistent: belief_state, probabilistic beliefs about the current state of the world\n              axtion, the agent's action\n  update belief_state based on action and percept\n  calculate outcome probabilities for actions\n    given action descriptions and current belief_state\n  select action with highest expected utility\n    given probabilites of outcomes and utility information\n  return action\n``` \n \n\n## AIMA Chapter 12.2 Basic Probability Notation\n For our agent to represent and use probabilistic information,\nwe need a formal language. But the language of probability theory has traditionally\nbeen informal.\n\n12.2.1 What probabilites are about \nprobabilistic assertions talk about how probable the various worlds are. A fully specified\nprobability model associates a numerical probability P(w) wiht each possibe word.\nevery possible world has a probability between 0 and 1 and that the total probability\nof the set of possible worlds is 1. In logic a set of words corresponds ot a proposition\nin formal language. Most of the time we have some evidence that has been uncovered.\n\n12.2.2 The language of propostitions in probability \npropositions describing sets of possible worlds are usually written in a notation\nthat combines elements of propositional logic and constraint satisfaction notation.\nIt is a factored represention in which a possibly word is represented by a set of variable/value \npairs. It can also be shown as a structure representation. Variables are called random variables\nFor continuous variables, it is not possible to write out the entire distribution as a vector,\nbecause there are infinitely many values. Instead, we can define the probability that a\nrandom variable takes on some value x as a parameterized function of x, usually called\na probability density function. A possible world is defined to be an assignment of values\nto all of the random variables under consideration.\n\n12.2.3 Probability axioms and their reasonableness\n\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\nSimple method for probabilittic inference. We use the full joint\ndistribution as the “knowledge base” from which answers to all questions may be derived. \nOne common practice is to extract the distrobution over some subset of variable(s).\nMarginalization adn conditioning are useful rules for all kinds of derivations involving\nprobability expressions \n\n## AIMA Chapter 12.4 Independence\n Independence assertions are usually based on knowledge of the domain.\nIf the complete set of variables can be divided into independent subsets, then the full\njoint distribution can be factored into separate joint distributions on those subsets.\nWhen they are available, then, independence assertions can help in reducing the size of\nthe domain representation and the complexity of the inference problem. Unfortunately,\nclean separation of entire sets of variables by independence is quite rare.  \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Bayes’ rule is useful in practice because there are many cases where we\ndo have good probability estimates for these three numbers and need to compute the fourth.\nOften, we perceive as evidence the effect of some unknown cause and we would like to\ndetermine that cause. a is cause b is effect. Causal direction P(effect|cause).\nDiagnostic direction P(cause|effect). diagnostic knowledge is often more fragile than\ncausal knowledge. If there is a sudden epidemic of meningitis, the unconditional\nprobability of meningitis, , will go up\n\n12.5.2 Using Bayes’ rule: Combining evidence\nTo make progress, we need to find some additional assertions about the domain that will\nenable us to simplify the expressions. The notion of independence in provides a clue,\nbut needs refining. These variables are independent, however, given the presence or\nthe absence of a cavity.expresses the conditional independence of toothache and catch given cavity. \nWe can plug it into  to obtain the probability of a cavity: (handwritten).\nIn general, for n symptoms that are all conditionally independent given cavity,\nthe size of the representation grows as O(n) instead of (O2^n). That means that conditional\nindependence assertions can allow probabilistic systems to scale up; moreover, they are\nmuch more commonly available than absolute independence assertions. Conceptually, \ncavity separates toothache and catch because it is a direct cause of both of them. \nThe decomposition of large probabilistic domains into weakly connected subsets through \nconditional independence is one of the most important developments in the recent history of AI.\n\n## AIMA Chapter 12.6 Naive Bayes Models\n Naieve bayes model, probability distrobution (written). \nIn practice, naive Bayes systems often work very well, even when the conditional independence\nassumption is not strictly true. The run time of this calculation is linear in the number\nof observed effects and does not depend on the number of unobserved effects.\n\n12.6.1 Text classification with naive Bayes \nHere the “cause” is the  variable, and the “effect” variables are the presence or\nabsence of certain key words, HasWordi. The naive Bayes model consists of the prior probabilities \nP(category) and the conditional probabilities P(HasWordi|Category). For each category c,\nP(category = c) is estimated as the fraction of all previously seen documents that are \nof category .\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n ure logical inference can conclude nothing about which square is most\nlikely to be safe, so a logical agent might have to choose randomly. We will see that a\nprobabilistic agent can do much better than the logical agent. The first term is the conditional \nprobability distribution of a breeze configuration, given a pit configuration; its values are \n1 if all the breezy squares are adjacent to the pits and 0 otherwise. The second term is the prior \nprobability of a pit configuration. Each square contains a pit with probability 0.2, \nindependently of the other squares. What this section has shown is that even seemingly complicated \nproblems can be formulated precisely in probability theory and solved with simple algorithms. \nTo get efficient solutions, independence and conditional independence relationships can be used \nto simplify the summations required. These relationships often correspond to our natural \nunderstanding of how the problem should be decomposed.\n\n## SLP Chapter 3.1 N-Grams\n P(w|h), the probability of a word w given some history h. One way\nis with relatice frequency counts. Take a large corpus, count the number of times we see\nh and count the number of times this is followed by w. even the web isn’t big enough to \ngive us good estimates in most cases. This is because language is creative; new sentences\nare created all the time, and we won’t always be able to count entire sentences.\nThe intuition of the n-gram model is that instead of computing the probability of\na word given its entire history, we can approximate the history by just the last few\nwords.The bigram model, for example, approximates the probability of a word given\nall the previous words P(wn|w1:n−1) by using only the conditional probability of the\npreceding word P(wn|wn−1). we are thus making the following approximation: P(wn|w1:n−1) ≈ P(wn|wn−1)\nThe assumption that the probability of a word depends only on the previous word is\nMarkov called a Markov assumption.An intuitive way to\nestimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood\nestimation the MLE estimate for the parameters of an n-gram model by getting counts from a\ncorpus, and normalizing the counts so that they lie between 0 and 1.\nFor the general case of MLE n-gram parameter estimation:\nP(wn|wn−N+1:n−1) = C(wn−N+1:n−1 wn)/C(wn−N+1:n−1) estimates the n-gram probability by dividing the\nobserved frequency of a particular sequence by the observed frequency of a prefix.\nThis ratio is called a relative frequency. Some of the bigram probabilities above encode some facts that we think of as strictly\nsyntactic in nature, like the fact that what comes after eat is usually a noun or an\nadjective, or that what comes after to is usually a verb. Others might be a fact about\nthe personal assistant task, like the high probability of sentences beginning with\nthe words I. And some might even be cultural rather than linguistic.\nWe always represent and compute language model probabilities in log format\nas log probabilities. Since probabilities are (by definition) less than or equal to log\nprobabilities 1, the more probabilities we multiply together, the smaller the product becomes\n\n## SLP Chapter 3.2 Evaluating Language Models\n The best way to evaluate the performance of a language model is to embed it in\nan application and measure how much the application improves (extrinsic evaluation).\nrunning big NLP systems end-to-end is often very expensive. Instead, it would be nice to have a metric \nthat can be used to quickly evaluate potential improvements in a language model. An \nintrinsic evaluation metric is one that measures the quality of a model independent of any application.\nwe need a test set. the probabilities of an n-gram model come from the corpus it is trained on, the training set or training corpus. We can then measure\nthe quality of an n-gram model by its performance on some unseen data called the\ntest set test set or test corpus. whichever model assigns a higher probability to the\ntest set—meaning it more accurately predicts the test set—is a better model.\nTraining on the test set introduces a bias that makes the probabilities all look too high, and causes huge\ninaccuracies in perplexity. The perplexity of a language model on a test set is the inverse probability of the test set, normalized\nby the number of words.PP(W) = P(w1w2 ...wN)^(-1/N). There is another way to think about\nperplexity: as the weighted average branching factor of a language. An (intrinsic) \nimprovement in perplexity does not guarantee an (extrinsic) improvement in the performance \nof a language processing task like speech recognition or machine translation. Nonetheless, \nbecause perplexity often correlates with such improvements, it is commonly used as a \nquick check on an algorithm. But a model’s improvement in perplexity should always be \nconfirmed by an end-to-end evaluation of a real task before concluding the evaluation of the model.\n\n## SLP Chapter 3.3 Sampling sentences from a language model\n One important way to visualize what kind of knowledge a language model embodies\nis to sample from it. Sampling from a distribution means to choose random points\naccording to their likelihood. Thus sampling from a language model—which represents \na distribution over sentences—means to generate some sentences, choosing\neach sentence according to its likelihood as defined by the model. We can use the \nsame technique to generate bigrams by first generating a random bigram that starts with \ns but in < > (according to its bigram probability). Let’s say the second word of that bigram is w. \nWe next choose a random bigram starting with w (again, drawn according to its bigram probability), and so on. \n\n## SLP Chapter 3.4 Generalization and Zeros\n The n-gram model, like many statistical models, is dependent on the training corpus.\nOne implication of this is that the probabilities often encode specific facts about a\ngiven training corpus. Another implication is that n-grams do a better and better job\nof modeling the training corpus as we increase the value of N. One step\nis to be sure to use a training corpus that has a similar genre to whatever task we are\ntrying to accomplish. It is equally important to get training data in the appropriate dialect or variety,\nespecially when processing social media posts or spoken transcripts. Our models may still be\nsubject to the problem of sparsity. For any n-gram that occurred a sufficient number\nof times, we might have a good estimate of its probability. But because any corpus is\nlimited, some perfectly acceptable English word sequences are bound to be missing\nfrom it. That is, we’ll have many cases of putative “zero probability n-grams” that\nshould really have some non-zero probability. The percentage of OOV\nwords that appear in the test set is called the OOV rate. An open vocabulary system open\nvocabulary is one in which we model these potential unknown words in the test set by adding a\npseudo-word called <UNK>. The first one is to turn the problem back into a closed vocabulary one\nby choosing a fixed vocabulary in advance: 1. Choose a vocabulary (word list) that is fixed in advance.\n2. Convert in the training set any word that is not in this set (any OOV word) to the unknown \nword token <UNK> in a text normalization step. 3. Estimate the probabilities for <UNK> from its \ncounts just like any other regular word in the training set. The second alternative, in \nsituations where we don’t have a prior vocabulary in advance, is to create such a vocabulary \nimplicitly, replacing words in the training data by <UNK> based on their frequency.\n\n## SLP Chapter 3.5 Smoothing\n To keep a language model from assigning zero probability to these unseen \nevents, we’ll have to shave off a bit of probability mass from some more frequent events \nand give it to the events we’ve never seen. This modification is called smoothing or discounting.\nLaplace smoothing:  All the counts that used to be zero will now\nhave a count of 1, the counts of 1 will be 2, and so on. This does not perform well but is \na good introduction, and its useful for other tasks like text classification.\nInstead of changing both the numerator and denominator, it is convenient to\ndescribe how a smoothing algorithm affects the numerator, by defining an adjusted\ncount c∗ . This adjusted count is easier to compare directly with the MLE counts and\ncan be turned into a probability like an MLE count by normalizing by N. \nA related way to view smoothing is as discounting (lowering) some non-zero\ncounts in order to get the probability mass that will be assigned to the zero counts.\nWe might describe a smoothing algorithm in terms of a relative discount dc=c*/c\nadd-k smoothing. P∗_Add-k(wn|wn−1) = C(wn−1wn) +k / C(wn−1) +kV\nAdd-k smoothing requires that we have a method for choosing k. Although add-k is useful for some\ntasks (including text classification), it turns out that it still doesn’t work well for\nlanguage modeling. \n\nBackoff and Interpolation:\nSometimes using less context is a good thing, helping to generalize more for contexts \nthat the model hasn’t learned much about. In backoff, we use the trigram if the evidence is\nsufficient, otherwise we use the bigram, otherwise the unigram. In other words, we\nonly “back off” to a lower-order n-gram if we have zero evidence for a higher-order\ninterpolation n-gram. By contrast, in interpolation, we always mix the probability estimates from\nall the n-gram estimators, weighing and combining the trigram, bigram, and unigram\ncounts.\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\nEfficiency considerations are important when building language models that use\nsuch large sets of n-grams. Rather than store each word as a string, it is generally\nrepresented in memory as a 64-bit hash number, with the words themselves stored\non disk. Probabilities are generally quantized using only 4-8 bits (instead of 8-byte\nfloats), and n-grams are stored in reverse tries. An n-gram language model can also be shrunk by pruning. \nStupid backoff gives up the idea of trying to make the language\nmodel a true probability distribution. There is no discounting of the higher-order\nprobabilities. If a higher-order n-gram has a zero count, we simply backoff to a\nlower order n-gram, weighed by a fixed (context-independent) weight\n\n## Module 10 Lecture Notes\n Uncertainty\nGeneral situation:\n• Observed variables (evidence): Agent knows certain things about the state of the world (e.g.,\nsensor readings or symptoms)\n• Unobserved variables (states): Agent needs to reason about other aspects (e.g. where an object\nis or what disease is present)\n• Model: Agent knows something about how the known variables relate to the unknown variables\nProbabilistic reasoning gives us a framework\nfor managing our beliefs and knowledge\nWhat Probabilities Are About\nLike logical assertions, probabilities are about possible worlds. Instead of strictly\nruling out possibilities (where a logical assertion is false), probabilities quantify how\nlikely a particular possible world is.\nIn probability theory, the possible worlds are called the sample space, and they\nmutually exclusive and exhaustive.\nA fully specified probability model associates a probability P(w) with each possible\nworld w. \n\nA random variable is some aspect of the world\nabout which we (may) have uncertainty\n• R = Is it raining?\n• U = Is the professor carrying an umbrella?\nWe denote random variables with capital letters\n\nProbabilistic Inference\nProbabilistic inference: compute a desired\nprobability from other known probabilities (e.g.\nconditional from joint)\nWe generally compute conditional probabilities\n• P(on time | no reported accidents) = 0.90\n• These represent the agent’s beliefs given the\nevidence\nProbabilities change with new evidence:\n• P(on time | no accidents, 5 a.m.) = 0.95\n• P(on time | no accidents, 5 a.m., raining) = 0.80\n• Observing new evidence causes beliefs to be updated\n\nProbabilistic Language Models\nOne goal: assign a probability to a sentence\n• Autocomplete for texting\n• Machine Translation\n• Spelling Correction\n• Speech Recognition\nOther Natural Language Generation tasks: summarization, question-answering, dialog\nsystems\n\n\n## Module 10 Definitions\n* **Belief state** - epresentation of the set of all possible world states that it might be in\n* **Qualification problem** - when we can't properly qualify a problem and therefore can't infer the proper plan\n* **Rational decision** - the right thing to do \n* **Laziness** - too much work to list the complete set of antecedents or consequents needed to ensure an exceptionless rule\n* **Theoretical ignorance** - Medical science has no complete theory for the domain.\n* **Practical ignorance** - if we know all rules we still might be uncertain if all necessary tests havent been run\n* **Ontological commitments** - the world is composed of facts that do or do not hold in any particular case\n* **Epistemological commitments** - logical agent believes each sentence to be true or false or has no opinion\n* **Utility theory** - very state (or state sequence) has a degree of usefulness, or utility, to an agent\n* **Decision theory** - Preferences, as expressed by utilities, are combined with probabilities\n* **Sample space** - the set of all possible words\n* **Unconditional or prior probabilities** - degrees of belief in propositions in the absence of any other information\n* **Conditional or posterior probability** - probability of doing something given that we have already done something\n* **Random variable** - is a function that maps from the domain of possible worlds omega to some range\n* **Bernoulli distrobution** - the set {0,1}\n* **Joint probability distrobution** - n x m table of probabilities based on the probability of P(n, m)\n* **Full joint probability distrobution** - probability model is completely determined by the joint distribution for all of the random variables\n* **Probabilistic inference** - computation of posterior probabilities for query propositions given observed evidence\n* **Marginalization or summing out** - sum up the probabilities for each possible value of the other variables, thereby taking them out of the equation\n* **Independence** - separating out probabilites when one cannot influence the others\n* **Text classification** - decide which of a predefined set of classes or categories it belongs to\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Bayesian networks can represent essentially any full joint probability distribution \nand in many cases can do so very concisely. 1) Each node corresponds to a random variable, \nwhich may be discrete or continuous. 2) Directed links or arrows connect pairs of nodes. \nIf there is an arrow from node X to node Y, X is said to be a parent of Y. 3) Each node\nXi has associated probability information theta(Xi|parents(Xi)) that quantifies the\neffect of the parents on the node using a finite number of parameters. Once the topology\nof the Bayes net is laid out, we need only specify the local probability information for\neach variable, in the form of a conditional distribution given its parents. The full joint\ndistribution for all the variables is defined by the topology and the local probability information.\nThe local probability information attached to each node takes the form of aconditional\nprobability table (CPT). (CPTs can be used only for discrete variables). Each row in a CPT \ncontains the conditional probability of each node value for a conditioning case.\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n The semantics of Bayes nets defines each entry in the joint distribution\nas follows: equation . the Bayesian network is a correct representation of the domain\nonly if each node is conditionally independent of its other predecessors in the node\nordering, given its parents. 1) Nodes: determine the set of variables that are required\nto model the domain, now order them. 2) Links: for i = 1 to n do: choose a minimal set of parents\nfor Xi from X1 to Xi-1, for each paretn insert a link from the parent to Xi, CPts: write\ndown the cpt P(Xi|parents(Xi)). a Bayes net can often be far more compact than the full \njoint distribution. This property is what makes it feasible to handle domains with many variables.\nIf we stick to a causal model, we end up having to specify fewer numbers, and the numbers will\noften be easier to come up with.\n\n13.2.1 Conditional independence relations in Bayesian networks \nEach variable is conditionally independent of its non-descendants, given its parents.\ninstead of defining the full joint distribution as the product of conditional distributions,\nthe network defines a set of conditional independence properties. The full joint\ndistribution can be derived from those properties. Is a set of nodes X is conditionally \nindependent of another set Y, given a third set Z. This can be determined efficiently by \nexamining the Bayes net to see whether Z d-separates X and Y. The process works as follows:\n1) Consider just the ancestral subgraph consisting of X, Y, Z, and their ancestors. 2) Add \nlinks between any unlinked pair of nodes that share a common child; now we have the \nso-called moral graph. 3)Replace all directed links by undirected links. 4) If Z blocks \nall paths between X and Y in the resulting graph, then Z d-separates X and Y. In \nthat case, X is conditionally independent of Y, given Z.\n\n13.2.2 Efficient Representation of Conditional Distributions \nlling in the CPT for a node requires up to O(2^k) numbers and perhaps a great deal\nof experience with all the possible conditioning cases. Usually, such relationships are\ndescribable by a canonical distribution that fits some standard pattern. In such cases, the\ncomplete table can be specified just by naming the pattern and perhaps supplying a few parameters.\nUncertain relationships can often be characterized by so-called noisy logical relationships. \n\n13.2.3 Bayesian nets with continuous variables\ndiscretization. define a continuous variable using one of the standard families of probability\ndensity functions. Nonparametric representation.\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n The basic task for any probabilistic inference system is to compute the\nposterior probability distribution for a set of query variables, given some observed\nevent—usually, some assignment of values to a set of evidence variables. \n\n13.3.1 Inference by enumeration\na query can be answered using a Bayes net by computing sums of products of conditional\nprobabilities from the network. Equation (handwritten) with complexity O(2^n) after nesting\nsummations\n\n``` \nFunction enumeration-ask(X,e,bn) returns a distrobution over X\n  inputs: X, the query\n          e, observed values for variables E\n          bn, a bayes net with variables vars\n  Q(X) <- a distrobution over x, initially empty\n  for each value xi of X do\n    Q(xi) <- enumerate-all(vars, exi)\n      where exi is ectended with X = xi\n  return normalize(Q(X))\n\nfunction ennumerate-ALL(vars, e) returns a real number\n  if empty?(vars) then return 1\n  V<-First(vars)\n  if V is evidence variable with value v in e\n    then return P(v|parents(V))x enumerate-all(REST(vars),e)\n    else return sum P(v|parents(V))x enumerate-all(REST(vars),ev)\n      where ev is extended with V = v\n``` \n\n13.3.2 The variable elimination algorithm\nThe final question is, of course, how to do inference in the network to make predictions. \nWe turn now to this question. For each inference method that we describe, we will evaluate \nthe method on the insurance net to measure the time and space requirements of the method.\n\n\n## Module 11 Lecture Notes\n Bayes’ Nets: Big Picture\n- Two problems with using full joint distribution tables\nas our probabilistic models:\n- Unless there are only a few variables, the joint is WAY too\nbig to represent explicitly\n- Hard to learn (estimate) anything empirically about more\nthan a few variables at a time\n- Bayes’ nets: a technique for describing complex joint\ndistributions (models) using simple, local\ndistributions (conditional probabilities)\n- More properly called graphical models\n- We describe how variables locally interact\n- Local interactions chain together to give global, indirect\ninteractions\n\nGraphical Model Notation\n- Nodes: variables (with domains)\n- Can be assigned (observed) or unassigned\n(unobserved)\n- Arcs: interactions\n- Indicate “direct influence” between variables\n- Formally: encode conditional independence\n(more later)\n- For now: imagine that arrows mean\ndirect causation (in general, they don’t!)\n\nBayes’ Net Semantics\n- A set of nodes, one per variable X\n- A directed, acyclic graph\n- A conditional distribution for each node\n- A collection of distributions over X, one for each\ncombination of parents’ values\n- CPT: conditional probability table\n- Description of a noisy “causal” process\n- Bayes’ nets implicitly encode joint distributions\n- As a product of local conditional distributions\n- To see what probability a BN gives to a full assignment, multiply all the\nrelevant conditionals together:\n\nCausality?\n- When Bayes’ nets reflect the true causal patterns:\n- Often simpler (nodes have fewer parents)\n- Often easier to think about\n- Often easier to elicit from experts\n- BNs need not actually be causal\n- Sometimes no causal net exists over the domain\n(especially if variables are missing)\n- E.g. consider the variables Traffic and Drips\n- End up with arrows that reflect correlation, not causation\n- What do the arrows really mean?\n- Topology may happen to encode causal structure\n- Topology really encodes conditional independence\n\nActive / Inactive Paths\n- Question: Are X and Y conditionally independent given\nevidence variables {Z}?\n- Yes, if X and Y “d-separated” by Z\n- Consider all (undirected) paths from X to Y\n- No active paths = independence!\n- A path is active if each triple is active:\n- Causal chain A ® B ® C where B is unobserved (either direction)\n- Common cause A ¬ B ® C where B is unobserved\n- Common effect (aka v-structure)\nA ® B ¬ C where B or one of its descendents is observed\n- All it takes to block a path is a single inactive segment\n\nBayes Nets Representation Summary\n- Bayes nets compactly encode joint distributions\n- Guaranteed independencies of distributions can be\ndeduced from BN graph structure\n- D-separation gives precise conditional independence\nguarantees from graph alone\n- A Bayes’ Net’s joint distribution may have further\n(conditional) independence that is not detectable until\nyou inspect its specific distribution \n\n## Module 11 Definitions\n* **Bayesian network** - directed graph in which each node is annotated with quantitative probability information.\n* **Conditioning case** - a possible combination of values for the parent nodes—a miniature possible world\n* **Syntax of a bayes net** - onsists of a directed acyclic graph with some local probability information attached to each node\n* **Semantics** - defines how the syntax corresponds to a joint distribution over the variables of the network.\n* **Topological order** - any order consistent with the directed graph structure.\n* **Locally structured** - each subcomponent interacts directly with only a bounded number of other components\n* **Sparse** - locally structured\n* **Deterministic node** -  has its value specified exactly by the values of its parents, with no uncertainty.\n* **Context specific independence** - a variable is conditionally independent of some of its parents given certain values of others.\n* **Noisy or model** - allows for uncertainty about the ability of each parent to cause the child to be true\n* **Discretization** - dividing up the possible values into a fixed set of intervals.\n* **Nonparametric representation** - define the conditional distribution implicitly with a collection of instances, each containing specific values of the parent and child variables\n* **Hybrid bayseian network** - A network with both discrete and continuous variables\n* **Linear-gaussian condition distrobution** - child has a Gaussian distribution whose mean mu varies linearly with the value of the parent and whose standard deviation sigma is fixed\n* **Probit model ** - probability unit\n* **Expit or inverse logit model** - uses the logistic function 1/(1+e^-x) to produce a soft threshold—it maps any  to a value between 0 and 1\n* **Variable elimination algorithm** - evaluating expressions such as Equation in right-to-left order. Intermediate results are stored, and summations that depend on the variable are done.\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 19.1 Forms of Learning\n Any component of an agent program can be improved by machine learning.\nThe improvements, and the techniques used to make them, depend on these factors: 1) Which\ncomponent is to be improved. 2) What prior knowledge the agent has, which influences the\nmodel it builds. 3) What data and feedback on that data is available. The components of these agents include:\n1) A direct mapping from conditions on the current state to actions. 2) A means to \ninfer relevant properties of the world from the percept sequence. 3) Information about \nthe way the world evolves and about the results of possible actions the agent can take.\n4) Utility information indicating the desirability of world states. 5) Action-value \ninformation indicating the desirability of actions. 6) Goals that describe the most \ndesirable states. 7) A problem generator, critic, and learning element that enable\nthe system to improve. we consider transfer learning, in which knowledge from one domain\nis transferred to a new domain, so that learning can proceed faster with less data.\nWe do assume, however, that the designer of the system chooses a model framework that\ncan lead to effective learning. This chapter concentrates on problems where the input\nis a factored representation—a vector of attribute values. It is also possible for the\ninput to be any kind of data structure, including atomic and relational. We look at\nclassification and regression problems. Three types of feedback that accompany the inputs:\n1) supervised learning 2) unsupervised learning 3) reinforcement learning.\n\n\n## AIMA Chapter 19.2 Supervised Learning\n the task of supervised learning is this: Given a training set of N\nexample input–output pairs (x1,y1), ...(xN,yN), where each pair was generated by an unknown\nfunction y=f(x), discover a function h that approximates the true function f. The function \nh is called a hypothesis about the world. It is drawn from a hypothesis space H of possible\nfunctions. or we can say a function drawn from a function class. We call the output \nyi the ground truth—the true answer we are asking our model to predict. With continuous-valued\noutputs we can’t expect an exact match to the ground truth; instead we look for a best-fit\nfunction for which each h(xi) is close to yi. The true measure of a hypothesis is not how it\ndoes on the training set, but rather how well it handles inputs it has not yet seen.\nBias often results from restrictions imposed by the hypothesis space. Often there is\na bias–variance tradeoff: a choice between more complex, low-bias hypotheses that fit\nthe training data well and simpler, low-variance hypotheses that may generalize better. \nThe problem is that there is a tradeoff between the expressiveness of a hypothesis space\nand the computational complexity of finding a good hypothesis within that space. \n\n## AIMA Chapter 19.3 Learning Decision Trees\n Each internal node in the tree corresponds to a test of the value \nof one of the input attributes, the branches from the node are labeled with the possible\nvalues of the attribute, and the leaf nodes specify what value is to be returned by\nthe function.\n\n19.3.1 Expressiveness of decision trees \nA Boolean decision tree is equivalent to a logical statement of the form: Output <-> (path1 V path2 V...)\nAny function in propostional logic and be expressed as a decision tree. There is no representation\nthat is efficient for all kinds of functions.\n\n19.3.2 Learning decision trees from examples \nThere are four cases to consider for these recursive subproblems: 1) If the remaining examples \nare all positive (or all negative), then we are done: we can answer Yes or No.\n2) If there are some positive and some negative examples, then choose the best attribute\nto split them. 3) If there are no examples left, it means that no example has been observed\nfor this combination of attribute values, and we return the most common output value from \nthe set. 4) If there are no attributes left, but both positive and negative examples, \nit means that these examples have exactly the same description, but different classifications. \n``` \nfunction learn-decision-tree(examples, attributes, parent_examples) returns a tree\n  if examples is empty then return plurality-value(parent_examples)\n  else if all examples have the same classification then return the classification\n  else if attributes is empty then reutn plurality-value(examples)\n  else\n    A<- argmax_aEattributes Importance(a, examples)\n    tree <- a new decision tree wiht root test A\n    for each value v of A do \n      exs <-{e: e E ecamples and eA = v}\n      subtree <- learn-decision-tree(exs, attributes - A, examples)\n      add a branch to tree wiht label (A=v)and subtree subtree\n    return tree\n``` \nWe can evaluate the performance of a learning algorithm with a learning curve.\n\n19.3.3 Choosing attribute tests \nThe decision tree learning algorithm chooses the attribute with the highest Importance.\nThis is defined in terms of entropy.\n\n19.3.4 Generalization and overfitting \nOverfitting becomes more likely as the number of attributes grows, and less likely as \nwe increase the number of training examples. Larger hypothesis spaces have more capacity\nboth to fit and to overfit\n\n19.3.5 Broadening the applicability of decision trees \nDecision trees can be made more widely useful by handling the following complications:\nMissing data, continuous na dmultivalued input attributes, continuous-valued output \nattribute\n\n\n## AIMA Chapter 19.4 Model Selection and Optimization\n Our goal in machine learning is to select a hypothesis that will\noptimally fit future examples. To make that precise we need to define “future example”\nand “optimal fit.” the optimal fit is the hypothesis that minimizes the error rate.\nsplit the examples you have into two sets: a training set to create the hypothesis,\nand a test set to evaluate it. That means you need three data sets: 1) A training set \nto train candidate models. 2) A validation set, also known as a development set or dev\nset, to evaluate the candidate models and choose the best one. 3) A test set to do a \nfinal unbiased evaluation of the best model. K-fold cross-validation: plit the data into k\nequal subsets. We then perform k rounds of learning; on each round 1/k of the data \nare held out as a validation set and the remaining examples are used as the training set.\ntask of finding a good hypothesis as two subtasks: model selection and optimization.\n\n19.4.1 Model selection \n``` \nfunction model-selection(learner, examples, k) returns a (hypothesis, error rate) pair\n  err <- an array, indexed by size, storing validation-set error rates\n  training_set, test_set <- a partition of examples into two sets\n  for size = 1 to inf do\n    err[size] <- cross-validation(learner, size, training_set, k)\n    if err is starting to increase significantly then\n      best_size <- the value of size with the minnimum err[size]\n      h <- Learner(best_size, training_set)\n      return h, error-rate(h, test_set)\n\nfunction cross-validation(Learner,size,examples,k) returns error rate\n  N<- the number of examples\n  errs <- 0\n  for i = 1 to k do \n    validation_set <- examples[(i-1)xN/k:ixN/k]\n    training_set <- examples - validation_set\n    h <- Learner(size, training_set)\n    errs <- errs + ERROR-RATE(h, validation_set)\n  return errs/k\n``` \nModel classes typically start to overfit as the capacity approaches the point of interpolation\n\n19.4.2 From error rates to loss \nin machine learning it is traditional to express this as a negative: to minimize a loss \nfunction rather than maximize a utility function. Best hypothesis is the one with \nthe minimum expected generalization loss. Since P(x,Y) is not known in most cases, the\nlearning agent can only estimate generalization loss with empirical loss. h^* may differ\nfrom the true function because of unrealizability, variance, noise, and computational complexity.\n\n19.4.4 Hyperparameter tuning\nSimplest is hand tuning: Repeat until you have satisfactory performance.\nIf there are only a few hyperparameters a grid search is appropriate. try all combinations \nof values and see which performs best on the validation data. If there are too many combinations \nof possible values, then random search. Random sampling is also a good way to handle continuous values.\nBayesian optimization. Population based training: PBT, as well as by random mutation, \nas in genetic algorithms\n\n\n## AIMA Chapter 19.5 The Theory of Learning\n computational learning theory, which lies at the intersection of AI, \nstatistics, and theoretical computer science. The underlying principle is that any hypothesis \nthat is seriously wrong will almost certainly be “found out” with high probability after a \nsmall number of examples, because it will make an incorrect prediction. Must be probably \napproximately correct.\n\n19.5.1 PAC learning example: Learning decision lists \nIf a test succeeds when applied to an example description, the decision list specifies \nthe value to be returned. If the test fails, processing continues with the next test in \nthe list. Decision lists resemble decision trees, but their overall structure is simpler: \nthey branch only in one direction. \n\n``` \nfunction Decision-list-learning(examples) returns a decision list or failure\n  if examples is empty then return the trivial decision list nO\n  t <- a test that matches a nonempty subset examples_t are all positive or negative\n      such that all the members of examples_t are all positive or all negative\n  if there is no such t then return a failure\n  if the examples in examples_t are positive then o<- YES else o<- NO\n  return a decision list with initiial test t and outcome o adn remaining tests given by \n    Decision-List_learning(examples-examples_t)\n``` \n\n\n## AIMA Chapter 19.6 Linear Regression and Classification\n 19.6.1 Univariate linear regression \nh_w(x) = w1x +w0 use w bc coefficients are weights. The loss function is convex \nthis is true for every linear regression problem with an loss function, and implies \nthat there are no local minima. \n\n19.6.2 Gradient descent \nchoose any starting point in weight space—here, a point in the (\n \n) plane—and then compute an estimate of the gradient and move a small amount in the \nsteepest downhill direction, repeating until we converge on a point in weight space with \n(local) minimum loss. \n``` \nw <- any point in the parameter space\nwhile not converged do\n  for each wi in w do\n    w<- wi - alpha partial/partial wi Loss(w)\n``` \nSGD is widely applied to models other than linear regression, in particular neural networks. \nEven when the loss surface is not convex, the approach has proven effective in finding \ngood local minima that are close to the global minimum.\n\n19.6.3 Multivariable linear regression \nit is common to use regularization on multivariable linear functions to avoid overfitting. \nRecall that with regularization we minimize the total cost of a hypothesis, counting \nboth the empirical loss and the complexity of the hypothesis:\n\n19.6.4 Linear classifiers with a hard threshold \n we can think of h as the result of passing the linear function w*x through a threshold function:\n \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n multinomial naive Bayes classifier, so called because it is a Bayesian \nclassifier that makes a simplifying (naive) assumption about how the features interact.\nWe represent a text document bag-of-words as if it were a bag-of-words, that is, an \nunordered set of words with their position ignored, keeping only their frequency in the document\ncˆ = argmax_c∈C P(c|d) break down any conditional probability P(x|y) into\nthree other probabilities: P(x|y) = P(y|x)P(x)/P(y) then cˆ = argmax_c∈C P(c|d) = argmax_c∈C P(d|c)P(c)/P(d) \nTo return to classification: we compute the most probable class ˆc given some\ndocument d by choosing the class which has the highest product of two probabilities:\nthe prior probability of the class P(c) and the likelihood of the document P(d|c):\nClassifiers that use a linear combination of the inputs\nto make a classification decision —like naive Bayes and also logistic regression—\nare called linear classifiers.\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\nHow can we learn the probabilities P(c) and P(fi|c)?  Pˆ(c) = Nc/Ndoc\nPˆ(wi|c) = count(wi, c)/ summation w∈V count(w, c). \nWhat do we do about words that occur in our test data but are not in our vocabulary at all because they did not occur in any training document in any class? The\nunknown word solution for such unknown words is to ignore them—remove them from the test\ndocument and not include any probability for them at all. \nFinally, some systems choose to completely ignore another class of words: stop\nstop words words, very frequent words like the and a. This can be done by sorting the vocabulary by frequency in the training set, and defining the top 10–100 vocabulary entries\nas stop words, or alternatively by using one of the many predefined stop word lists\navailable online.\n\n``` \nfunction TRAIN NAIVE BAYES(D, C) returns log P(c) and log P(w|c)\nfor each class c ∈ C # Calculate P(c) terms\n  Ndoc = number of documents in D\n  Nc = number of documents from D in class c\n  logprior[c]← log Nc/Ndoc\n  V←vocabulary of D\n  bigdoc[c]←append(d) for d ∈ D with class c\n  for each word w in V # Calculate P(w|c) terms\n    count(w,c)←# of occurrences of w in bigdoc[c]\n    loglikelihood[w,c]← log (count(w, c) + 1)/summ_w' in V(count (w0, c) + 1)\nreturn logprior, loglikelihood, V\n\nfunction TEST NAIVE BAYES(testdoc, logprior, loglikelihood, C, V) returns best c\nfor each class c ∈ C\n  sum[c]← logprior[c]\n  for each position i in testdoc\n    word←testdoc[i]\n    if word ∈ V\n      sum[c]←sum[c]+ loglikelihood[word,c]\n  return argmaxc sum[c]\n``` \n\n \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n for sentiment classification and a number of other text classification tasks,\nwhether a word occurs or not seems to matter more than its frequency. Thus it\noften improves performance to clip the word counts in each document at 1. Binary\nmultinomial naive Bayes or binary NB. The variant uses the same Eq. 4.10 except\nthat for each document we remove all duplicate words before concatenating them\ninto the single big document. A second important addition commonly made when doing text classification for\nsentiment is to deal with negation. Consider the difference between I really like this\nmovie (positive) and I didn’t like this movie (negative). The negation expressed by\ndidn’t completely alters the inferences we draw from the predicate like. Similarly,\nnegation can modify a negative word to produce a positive review (don’t dismiss this\nfilm, doesn’t let us get bored). Finally, in some situations we might have insufficient labeled training data to\ntrain accurate naive Bayes classifiers using all words in the training set to estimate\npositive and negative sentiment. In such cases we can instead derive the positive \nand negative word features from sentiment lexicons, lists of words that are pre- sentiment\nlexicons annotated with positive or negative sentiment\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n A common solution here, rather than using all the words as individual features,\nis to predefine likely sets of words or phrases as features, combined with features\nthat are not purely linguistic. For example the open-source SpamAssassin tool1\npredefines features like the phrase “one hundred percent guaranteed”, or the feature\nmentions millions of dollars, which is a regular expression that matches suspiciously\nlarge sums of money. But it also includes features like HTML has a low ratio of text\nto image area, that aren’t purely linguistic and might require some sophisticated\ncomputation, or totally non-linguistic features about, say, the path that the email\ntook to arrive. For other tasks, like language id—determining what language a given piece\nof text is written in—the most effective naive Bayes features are not words at all,\nbut character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,\n‘thei’), or, even simpler byte n-grams, where instead of using the multibyte Unicode\ncharacter representations called codepoints, we just pretend everything is a string of\nraw bytes. Because spaces count as a byte, byte n-grams can model statistics about\nthe beginning or ending of words \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Bayes classifiers can use any sort of feature: dictionaries, URLs, email addresses, network features, phrases, and so on. But\nif, as in the previous section, we use only individual word features, and we use all\nof the words in the text (not a subset), then naive Bayes has an important similarity to language modeling. Specifically, a naive Bayes model can be viewed as a\nset of class-specific unigram language models, in which the model for each class\ninstantiates a unigram language model.\nSince the likelihood features from the naive Bayes model assign a probability to\neach word P(word|c), the model also assigns a probability to each sentence:\nP(s|c) = PI _ i∈positions P(wi|c)\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\nIn both cases, we need a metric for knowing how well our spam detector (or\npie-tweet-detector) is doing. To evaluate any system for detecting things, we start\nby building a confusion matrix. A confusion matrix confusion\nis a table for visualizing how an algorithm performs with respect to the human gold\nlabels, using two dimensions (system output and gold labels), and each cell labeling\na set of possible outcomes. precision and recall. Precision measures the percentage of the items that\nthe system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\nare positive according to the human gold labels). Precision is defined as\nPrecision = true positives/true positives + false positives\nRecall measures the percentage of items actually present in the input that were\ncorrectly identified by the system. Recall is defined as\nRecall = true positives/true positives + false negatives. There are many ways to define a single metric that incorporates aspects of both\nF-measure precision and recall. The simplest of these combinations is the F-measure, defined as:\nFβ = (β2 +1)PR/β^2P+R\nThe β parameter differentially weights the importance of recall and precision,\nbased perhaps on the needs of an application. Values of β > 1 favor recall, while\nvalues of β < 1 favor precision. When β = 1, precision and recall are equally balanced.\nIn macroaveraging, we compute the performance microaveraging for each class, and then \naverage over classes. In microaveraging, we collect the decisions for all classes into \na single confusion matrix, and then compute precision and recall from that table.\n \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n we use the training set to train the model, then use\nthe development test set (also called a devset) to perhaps tune some parameters,and \nin general decide what the best model is. In cross-validation, we choose a number k, and partition our data into k disjoint\nsubsets called folds. Now we choose one of those k folds as a test set, train our\nclassifier on the remaining k − 1 folds, and then compute the error rate on the test\nset. Then we repeat with another fold as the test set, again training on the other k−1\nfolds. We do this sampling process k times and average the test set error rate from\nthese k runs to get an average error rate.\n \n\n## SLP Chapter 4.9 Statistical Significance Testing\n We would like to know if δ(x) > 0, meaning that our logistic regression classifier\nhas a higher F1 than our naive Bayes classifier on X. δ(x) is called the effect size;\na bigger δ means that A seems to be way better than B; a small δ means A seems to\nbe only a little better. The hypothesis H0, called the null hypothesis, supposes that δ(x) is actually negative or zero, meaning that A is not better than B. We would like to know if we can\nconfidently rule out this hypothesis, and instead support H1, that A is better. We do this by creating a random variable X ranging over all test sets. Now we\nask how likely is it, if the null hypothesis H0 was correct, that among these test sets\nwe would encounter the value of δ(x) that we found. We formalize this likelihood\nas the p-value: the probability, assuming the null hypothesis H0 is true, of seeing\nthe δ(x) that we saw or one even greater P(δ(X) ≥ δ(x)|H0 is true).We say that a result \nis statistically significant if the δ we saw has a probability that is below the \nthreshold and we therefore reject this null hypothesis. There are two common non-parametric \ntests used in NLP: approximate randomization and the bootstrap test, showing the paired \nversion of the test, which again is most common in NLP. Paired tests are those in which we compare two sets of observations that are aligned:\neach observation in one set can be paired with an observation in another. This happens naturally when we are comparing the performance of two systems on the same\ntest set; we can pair the performance of system A on an individual observation xi\nwith the performance of system B on the same xi.\n\n``` \nfunction BOOTSTRAP(test set x, num of samples b) returns p-value(x)\nCalculate δ(x) # how much better does algorithm A do than B on x\ns = 0\nfor i = 1 to b do\nfor j = 1 to n do # Draw a bootstrap sample x^(i) of size n\n    Select a member of x at random and add it to x^(i)\n  Calculate δ(x^(i)) # how much better does algorithm A do than B on x^(i)\n  s←s + 1 if δ(x^(i)) ≥ 2δ(x)\np-value(x) ≈ s/b\n# on what % of the b samples did algorithm A beat expectations?\nreturn p-value(x) # if very few did, our observed δ is probably not accidental\n``` \n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n One class of harms is representational harms, \nharms caused by a system that demeans a social group, for example by perpetuating negative \nstereotypes about them. In other tasks classifiers may lead to both representational harms and other\nharms, such as censorship. For example the important text classification task of\ntoxicity detection is the task of detecting hate speech, abuse, harassment, or other toxicity\ndetection\nkinds of toxic language. While the goal of such classifiers is to help reduce societal harm,\ntoxicity classifiers can themselves cause harms.   \n\n## Module 12 Lecture Notes\n Model-Based Classification\nModel-based approach\n• Build a model (e.g. Bayes’ net) where both the label and\nfeatures are random variables\n• Instantiate any observed features\n• Query for the distribution of the\nlabel conditioned on the features\n\nWhat do we need in order to use Naïve Bayes?\n• Inference method (we just saw this part)\n• Start with a bunch of probabilities: P(Y) and the P(Fi\n|Y) tables\n• Use standard inference to compute P(Y|F1…Fn)\n• Nothing new here\n• Estimates of local conditional probability tables\n• P(Y), the prior over labels\n• P(Fi\n|Y) for each feature (evidence variable)\n• These probabilities are collectively called the parameters of the\nmodel and denoted by q\n• Up until now, we assumed these appeared by magic, but…\n• …they typically come from training data counts: we’ll look at this\nsoon\n\nBag-of-words Naïve Bayes:\n• Features: Wi is the word at positon i\n• As before: predict label conditioned on feature variables (spam vs. ham)\n• As before: assume features are conditionally independent given label\n• New: each Wi is identically distributed\nGenerative model:\n“Tied” distributions and bag-of-words\n• Usually, each variable gets its own conditional probability distribution P(F|Y)\n• In a bag-of-words model\n• Each position is identically distributed\n• All positions share the same conditional probs P(W|Y)\n• Why make this assumption?\n• Called “bag-of-words” because model is insensitive to word order or reordering\n\nRelative frequency parameters will overfit the training data!\n• Just because we never saw a 3 with pixel (15,15) on during training doesn’t mean we won’t\nsee it at test time\n• Unlikely that every occurrence of “minute” is 100% spam\n• Unlikely that every occurrence of “seriously” is 100% ham\n• What about all the words that don’t occur in the training set at all?\n• In general, we can’t go around giving unseen events zero probability\nAs an extreme case, imagine using the entire email as the only feature\n• Would get the training data perfect (if deterministic labeling)\n• Wouldn’t generalize at all\n• Just making the bag-of-words assumption gives us some generalization, but isn’t enough\nTo generalize better: we need to smooth or regularize the estimates\n\nWhat to Do About Errors?\nNeed more features– words aren’t enough!\n• Have you emailed the sender before?\n• Have 1K other people just gotten the same email?\n• Is the sending information consistent?\n• Is the email in ALL CAPS?\n• Do inline URLs point where they say they point?\n• Does the email address you by (your) name?\nCan add these information sources as new\nvariables in the NB model\nNext class we’ll talk about classifiers which let\nyou easily add arbitrary features more easily\n\nFirst step: get a baseline\n• Baselines are very simple “straw man” procedures\n• Help determine how hard the task is\n• Help know what a “good” accuracy is\nWeak baseline: most frequent label classifier\n• Gives all test instances whatever label was most common in the training set\n• E.g. for spam filtering, might label everything as ham\n• Accuracy might be very high if the problem is skewed\n• E.g. calling everything “ham” gets 66%, so a classifier that gets 70% isn’t very good…\nFor real research, usually use previous work as a (strong) baseline\n\nThe confidence of a probabilistic classifier:\n• Posterior over the top label\n• Represents how sure the classifier is of\nthe classification\n• Any probabilistic model will have\nconfidences\n• No guarantee confidence is correct\nCalibration\n• Weak calibration: higher confidences\nmean higher accuracy\n• Strong calibration: confidence predicts\naccuracy rate\n• What’s the value of calibration?\n\nSummary\nBayes rule lets us do diagnostic queries with causal probabilities\nThe naïve Bayes assumption takes all features to be independent given\nthe class label\nWe can build classifiers out of a naïve Bayes model using training data\nSmoothing estimates is important in real systems\nClassifier confidences are useful, when you can get them\n\nProblems with the Perceptron\nNoise: if the data isn’t separable,\nweights might thrash\n• Averaging weight vectors over\ntime can help (averaged\nperceptron)\nMediocre generalization: finds a\n“barely” separating solution\nOvertraining: test / held-out\naccuracy usually rises, then falls\n• Overtraining is a kind of\noverfitting\n\nFixing the Perceptron\nIdea: adjust the weight update to mitigate these effects\nMIRA*: choose an update size that fixes the current\nmistake…\n… but, minimizes the change to w\n\nIn practice, it’s also bad to make updates that are too large\n• Example may be labeled incorrectly\n• You may not have enough features\n• Solution: cap the maximum possible value of t with some\nconstant C\n• Corresponds to an optimization that assumes nonseparable data\n• Usually converges faster than perceptron\n• Usually better, especially on noisy data\n\nClassification: Comparison\nNaïve Bayes\n• Builds a model training data\n• Gives prediction probabilities\n• Strong assumptions about feature independence\n• One pass through data (counting)\nPerceptrons / MIRA:\n• Makes less assumptions about data\n• Mistake-driven learning\n• Multiple passes through data (prediction)\n• Often more accurate\n\nPerceptions can be used for all kinds of classification problems.\nThink of the inputs as features representing something we want to classify.\nThe feature values for inputs are fixed, but we can choose different weight vectors.\nDepending on the weight vector that we pick, we will get a different classifier. \n \n\n## Module 12 Definitions\n* **Induction** - Going from a specific set of observations to a general rule\n* **Deductive conclusions** - guaranteed to be correct if the premises are correct\n* **Factored representation** - vector of attribute values\n* **Classification** - when the output of the learning problem is a finite set of values\n* **Regression** - when the output of the learning problem is a number\n* **Supervised learning** - the agent observes input-output pairs and learns a function that maps from input to output\n* **Unsupervised learning** - the agent learns patterns in the input without any explicit feedback\n* **Clustering** - detecting potentially useful clusters of input examples\n* **Reinforcement learning** - the agent learns from a series of reinforcements: rewards and punishments\n* **Exploratory data analysis** - examining the data with statistical tests and visualizations—histograms, scatter plots, box plots—get some insight into what hypothesis space might be appropriate\n* **Consistent hypothesis** - h such that each xi in the training set has h(xi) = yi\n* **Test set** - second sample of (xi, yi) pairs\n* **Generalizes well** - if it accurately predicts the outpusts of the test set\n* **Bias** - the tendency of a predictive hypothesis to deviate from the expected value when averaged over different training sets\n* **Underfitting** - when it fails to find a pattern in the data.\n* **Variance** - the amount of change in the hypothesis due to fluctuation in the training data\n* **Decistion tree** - representation of a function that maps a vector of attribute values to a single output value—a “decision.”\n* **Boolean classification** - consider only inputs consisting of discrete values and outputs that are either true or false \n* **Happy graphs** - learning curves\n* **Entropy** -  measure of the uncertainty of a random variable\n* **Information gain** - from the attribute test on A is the expected reduction in entropy:\n* **Decision tree pruning** - combats overfitting by eliminating nodes that are not clearly relevant.\n* **Statistical significance test** - how large a gain should we require to split on an attribute?\n* **Early stopping** - have the decision tree algorithm stop generating nodes when there is no good attribute to split on\n* **Missing data** - in many domains not all the attribute values will be known for every example\n* **Continuous and multivalued input attributes** - may be that every example has a different attribute value. The information gain measure would give its highest score to such an attribute\n* **Split point test** - an inequality test on the value of an attribute\n* **Regression tree** - has at each leaf a linear function of some subset of numerical attributes, rather than a single output value\n* **Decision trees are unstable** - adding just one new example can change the test at the root\n* **Stationary assumption** - \n* **Independent and identically distributed or i.i.d** - each example has the same prior probabilityt distrobution adn is independent of the previous examples\n* **Error rate** - the proportion of times that h(x) != y for an (x,y) example\n* **Hyper parameters** - try different thresholds for x^2 pruning of decision trees, or different degrees for polynomials\n* **Leave-one-out cross-validation or loocv** - k-fold cross-validation when k=n\n* **Model selection** - choose a good hypothesis space\n* **Optimization (training)** - finds the best hypothesis for the space\n* **Interpolated** - a model that exactly fits all the training data\n* **Realizable** - hypothesis space H actually contains the true function f\n* **Varriance** - a learning algorithm will in general return different hypotheses for different sets of examples\n* **Computationally intractable** - a search can explore part of the space and return a reasonably good hypothesis, but can’t always guarantee the best one\n* **Small scale learning** - the number of training examples ranged from dozens to the low thousands\n* **Large scale learning** - millions of examples\n* **Regularization** - explicitly penalizing complex hypotheses is called\n* **Regularization function** - the complexity function\n* **Frature selection** - discard attributes that appear to be irrelevant\n* **Minimum description length or mdl hypothesis** - minimizes the total number of bits required\n* **Hand tuning** - guess based on past experience, train model,measure performance on validation data, analyze results, and use intuition to suggest new parameter values\n* **Random search** - samples uniformly from the set of all possible hyperparameter settings, repeating for as long as you are willing to spend the time and computational resources\n* **Bayesian optimization** - treats the task of choosing good hyperparameter values as a machine learning problem in itself\n* **Pbt** - trains in 2 waves, 2nd generation can choose based on the successful values from the previous generation.\n* **Computational learning theory** - which lies at the intersection of AI, statistics, and theoretical computer science.\n* **Pac learning algorithm** - Any learning algorithm that returns hypotheses that are probably approximately correct\n* **Approximately correct** - error(h) < epsilon which is a small constatne\n* **Sample complexity** - number of required examples as a function of epsilon and delta\n* **Decision lists** - onsists of a series of tests, each of which is a conjunction of literals\n* **Linear regression** - task of finding the hw that best fits the data\n* **Weight space** - the space defined by all possible settings of the weights\n* **Epoch** - a step that covers all the training examples\n* **Stochastic gradient descent or sgd** - randomly selects a small number of training examples at each step, and updates according to\n* **Decistion boundary** - is a line (or a surface, in higher dimensions) that separates the two classes.\n* **Linear separator** - linear decision boundary\n* **Training curve** - measures the classifier performance on a fixed training set as the learning process proceeds one update at a time on that training set.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}