{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1709918f-3ce7-41d8-a14e-5068b2752e7b"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "289b1f1d-6d48-4c02-a31d-88a7aaf2e4d6"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = True"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\"\n",
        "* Agent makes actions a, has a probability P(s) to each state s\n",
        "* Transition model for the uncertain outcomes is P(s' | s,a), probability that action a in state s yields states s'\n",
        "* related by the equation $P(RESULT(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n",
        "* make decisions based on immedaite outcomes, preferences represented by utility function U(s)\n",
        "* utility assigns a number to express the value of a state\n",
        "* expected utility EU(a) is the average utility of the outcomes weighed by probability: $EU(a) = \\sum_{s'} P(RESULT(a) = s')U(s')$ \n",
        "* maximum expected utility principle says the agent should pick the action that mazimizes the expected utility if it is rational\n",
        "* action = argmax(a) EU(a)\n",
        "* formalizes what the agent should do but doesnt operate on the advice, lots of other factors to consider\n",
        "* if the agent acts to maximize utility function that is correctly representative of the performance, agent will receive the highest possible performance score over all environments averaged\n",
        "\n",
        "\n",
        "\"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\"\n",
        "A >- B - agent prefers A over B\n",
        "\n",
        "A ~ B - agent is indifferent between A and B\n",
        "\n",
        "A >-_ B agent prefers A over B or is indifferent\n",
        "\n",
        "* lottery is a set of outcomes for each action, and each action we assume to be a lottery ticket\n",
        "* lottery with possible outcomes S1, S2, ... with probabilityies p1, p2, ... is expressed as L = [p1, S1; p2, S2; ... pn, Sn]\n",
        "* each outcome of a lottery can be an atomic state or anotehr lottery\n",
        "\n",
        "Constraints we expect lotteries to obey\n",
        "* Orderability - rational agent must prefer one lottery, the other, or rate them as equally preferrable, cannot avoid a decision\n",
        "* A >- B, B >- A, or A ~ B\n",
        "\n",
        "* Transitivity - if we have lotteries A, B, and C, and the agent prefers A to B and B to C, then it prefers A to C\n",
        "* A >- B and B >-C -> A >- C\n",
        "\n",
        "* Continuity - if a lottery B is between A and C, then there exists a probability p where the agent is indifferent between B guarenteed and lottery A with probability p and C with probability 1 - p\n",
        "* A >- B >- C -> exists p [p,A; 1-p, C] ~ B\n",
        "\n",
        "* Substitutability - agent is indifferent between A and B, then also indifferent for two more complex lotteries that are equal except for B substituted for B\n",
        "* A ~ B -> [p, A; 1 - p, C] ~ [p, B; 1 - p, C]\n",
        "\n",
        "* Monotonicity - agent prefers A to B, must prefer the lottery that has a higher probability for A\n",
        "* A >- B -> (p > q <-> [p,A; 1 - p, B] >- [q,A; 1 - q, B])\n",
        "\n",
        "* Decomposability - compound probabilities can be reduced with the laws of probability, e.g.\n",
        "* [p,A; 1 - p, [q,B; 1 - q, C]] ~ [p,A; (1 - p)q,B; (1 - p)(1 - q), C]\n",
        "\n",
        "* if an agent violates one of these it exhibits irrational behavior\n",
        "* Existance of a utility function: if agent behaves all these, then there exists a function U such that U(a) > U(b) iff A is preferred to B, and U(a) = U(b) iff the agent is indifferent\n",
        "* expected utility of a lottery is the sum of the probability of each outcome multipled by the utility of the outcome - U([p1,S1; ... ; pn,Sn]) = $\\sum_i piU(Si)$ \n",
        "* utility of a compound lottery becomes determined completely\n",
        "* agent only needs preference of states, the numbers don't matter, referred to as value function or ordinal utility function\n",
        "\n",
        "\"\"\"\n",
        "chapter16_3 =  \"\"\"\n",
        "* agent allowed any preferences it likes so long as it follows the axioms\n",
        "* preference elicitation, presents choices to humans and uses the preferences to determine the utility function\n",
        "* need to know what the human utility function is to make decisions on our behalf\n",
        "* want to create a fixed scale fo preference, normalized utilities scale from 0 to 1\n",
        "* using this normalized utility scale, we can determine the utility of a prize S and by asking the agent to pick between S and a standard lottery\n",
        "* probability is adjusted until the agent is indifferent, then S is assigned that utility, do this for each prize then have a lookup table of values\n",
        "* tradeoffs of life and death made all the time\n",
        "* refusal to put monetary value on life can mean its undervalued\n",
        "* value of a statistical life determines costs and benefits of regulation/intervention\n",
        "* micromort is 1 in a million chance of death, QALY is quality adjusted life years for patients\n",
        "* if an agent prefers more money than less, it has a monotonic preference for more money\n",
        "* expected monetary value is probabilities multiplied by monetary amounts\n",
        "* utility of money generally found to be proportional to the logarithm of the amount\n",
        "* positive curve, agents are risk-averse and prefer a sure thing, negative curve, risk-seeking\n",
        "* certainty equivalent is the amount someone will except instead of doing the lottery\n",
        "* difference between the EMV of the lottery and the certainty equivalent is the insurance premium\n",
        "* an agent is risk-neutral if the linear curve is observed\n",
        "* often working with estimated expected values EU(a) instead of the true expected utility, assumed to be unbiased when not always the case\n",
        "* order statistic is distribution of ranked element of a sample\n",
        "* optimizer's curse, tendency for expected utility of the best choice to be too high\n",
        "* normative theory describes how rational agents should act, descriptive theory is how actual agents really act\n",
        "* certainty effect, people often prefer the sure thing in gains, reduce computational burden, distrust the stated probabilities, regret if it doesnt pan out\n",
        "* people also have ambiguity aversion, go with the sure probability\n",
        "* framing effect, how decision problems are worded\n",
        "* anchoring effect, making relative utiltiy judgements rather than absolute ones, cheaper product compared to more expensive one\n",
        "* evolutionary psychology says our decision making processes were not evolved to solve word problems with probabilities\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\"\n",
        "* fully observable environment allows agent to always know where it is\n",
        "* transition model describes the outcome of each action in the state, stocastic in our maze so P(s'|s,a) as before, or T(s,a,s')\n",
        "* utility function here depends on sequence of states and actions (environmental history), rather than single state\n",
        "* agent will receive a reward R(s,a,s') for moving to state s'\n",
        "* Markov Decision process (MDP) consists of set of states, actions, transtiion model, reward function\n",
        "* involve dynamic programming often, simplifying a problem by breaking it down and solving optimal solutions to subproblems\n",
        "* policy solution pi is action recommended for a given state so agent knows what to do, quality measured by expected utility of environment histories generated\n",
        "* optimal policy pi* yields highest expected utility\n",
        "* reward a big factor in how the agent moves and if/when it chooses to exit\n",
        "* with a finite horizon (meaing theres a fixed time after which nothing matters), an optimal action may depend on time left\n",
        "* nonstationary policy depends on the time\n",
        "* with infinite time (horizon), optimal action only depends on current state, policy is stationary, these are simpler\n",
        "* discount factor between 0 and 1, preference of agent for current vs. future rewards. Close to 0, future rewards are insignificant, close to 1 more willing to wait for them. If exactly 1 then purely additive rewards\n",
        "* makes infinities go away, rewards not always guarenteed, valued higher\n",
        "* utility of infinite sequence now finite\n",
        "* if agent contains terminal states and guarenteed to reach it then never need to compare infinite sequences, proper policy if it exits\n",
        "* infinite sequences can be compared by average reward obtained per time step\n",
        "* expected utility of executing pi starting in s is Upi(s) = E * $\\sum_t->inf y^t R(S_t, pi(S_t), S_{t+1})$\n",
        "* optimal policy pi* = argmax of pi U^pi (s)\n",
        "* in infinite horizons with discouted utilities, optimal policy independent of starting state\n",
        "* utility function allows agent to select actions with highest expected utility: $pi^*(s) = argmax_{a exist A(s)} \\sum_s' P(s'|s,a)[R(s,a,s') + yU(s')]$\n",
        "* utility of state is expected reward for next transtion plus discounted utility of next state, so utility of state is: $U(s) = max_{a exist A(s)} \\sum_s' P(s'|s,a)[R(s,a,s') + yU(s')]$, bellman equation\n",
        "* action-utility function or Q function, Q(s,a) is expected utility of taking action in given state: U(s) = max_a Q(s,a), optimal policy takes max action that yields highest Q state\n",
        "* can develop equation for Q-funtcion, $Q(s,a) = \\sum_s' P(s'|s,a)[R(s,a,s') + ymax_{a'}Q(s',a')]$\n",
        "* all policies optimal with a reward function that is always zero\n",
        "* represent P(s'|s,a) and R(s,a,s') using 3D tables of size |S|^2|A|, often sparse tables because bounded transitions between states\n",
        "* can represent MDPs by extending dynamic bayesian networks to create dynamic decision etwrosk, factored representations with exponential complexity advatnage over atomic representation\n",
        "\n",
        "\"\"\"\n",
        "chapter17_2 =  \"\"\"\n",
        "* if we have n possible states, n Bellman equations, one per state\n",
        "* n equations have n unknowns, utilities of each state\n",
        "* equations nonlinear because the max operation isn't linear, try iterative approach instead\n",
        "* start with arbitrary values for the utilities, then update each utility until we hit equilibrium\n",
        "* update assumed to be applied to all states simultaneously, will converge and give unique solutions and the policy\n",
        "* value iteration, think of as propagating information through state space to update each state \n",
        "* value teration converges due to contraction, function of one argument that produces two output values that are close together by some constant factor, applied to two inputs\n",
        "* only has one fixed point, and the value must get closer to the fixed poiint upon every application\n",
        "* distance between the utility vectors measured with max norm, measures the length of the vector by value of biggest component\n",
        "* Bellman update is contraction by factor of discount on space of utility vectors\n",
        "* agent only cares about how well it does if it makes decision using utility function\n",
        "* define policy loss as the most an agent can lose by executing a policy instead of the optimal one\n",
        "* policy iteration alternates between evaluation and improvement\n",
        "* policy evaluation: given policy pi_i, calculate U_i = Upi_i, utility of each step if current policy is executed\n",
        "* policy improvement: calculate new MEU policy pi_i + 1 using look ahead based on U_i\n",
        "* terminates when policy improvement doesnt update the utilities\n",
        "* use q value function, get a simplified version of Bellman equation relating utility of s to neighbor utilities\n",
        "* equations now linear since max is removed, so we can solve a n state n linear equation n unknown in O(n^3) time\n",
        "* often the most efficient in small state spaces\n",
        "* don't have to do exact policy evaluation, can do number of simplified value iteratoin steps to give estimate\n",
        "* repeat this many times to get the next utility estimate, modified policy iteration\n",
        "* can also update any subset of states and apply policy iteration or simplified value iteration to the subset, asynchronous policy iteration, guarenteed to converge to optimal policy\n",
        "* Linear progreamming used to formulate constrained optimization problems\n",
        "* can reduce solving MDP to solving formulated linear program\n",
        "* want to minimize U(s) for all s\n",
        "* MDPs can be solved in polynomial time in number of states, actions, bits to specify model\n",
        "* Value and policy iteration both offline algorithms to solve states then agent acts\n",
        "* for online, use expectiminimax simplification, evaluation function applied at nonterminal leaves given default value\n",
        "* decision taken by backing up the utility values and averaging at the chance nodes\n",
        "* limit depth to avoid high branching factor, focus on most likely outcomes\n",
        "* real time dynamic programming, explored states create a sub-MDP of MDP, and can be solved to yield decision of current state\n",
        "* state space may be too large, or difficult to make heuristic\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\"\n",
        "Asteroid Field:\n",
        "* 4x3 environment with goal states, pick an action at each time step, terminates when goal state reached, stochastic environment\n",
        "* pick a sequence of actions, can get there correctly with no errors or with \"correct\" errors\n",
        "* Transition model written as T(s,a,s'), or p(s',s,a) - probability of reaching s' given s and action a\n",
        "* Markovian transitions, meaning probability of arriving in s' depends only on s, no earlier states\n",
        "* utility/reward function can be positive or negative but bounded by some maximum value\n",
        "* can receive this reward R(s,a,s') at any time step since the decisions are sequential, additive as well\n",
        "\n",
        "Markov Decision Process (MDPs):\n",
        "* sequential decision process, fully observable environment, stochastic transition model, additive rewards\n",
        "* non-deterministic search problems so we can solve using expectimax search to calculated expected utilities\n",
        "* to solve an MDP, need 1) a set of states s in S, 2) a set of actions a in A, 3) a transition function T(s,a,s') or the probabilit that executing action a in s leads to s', 4) a reward function R(s,a,s'), 5) an initial state s0, 6) one or more terminal states (optional)\n",
        "* solution is a policy since there is non-determinism, not guaranteed we end up where we want to go, pi(s) = a. Stochastic so may be correct but not happen, optimal policy is pi*\n",
        "* same policy executed multiple times may lead to different states and actions and different scores, so need expected utility of all paths generated\n",
        "* performance of agent is the sum of the rewards of the transitions it takes, can cause the agent to exit early/late/never\n",
        "* Policy is choice of action for each state, utility is sum of the potentially discounted rewards\n",
        "\n",
        "Discounting:\n",
        "* Want to maximize sum of rewards, prefer now versus later\n",
        "* each time we go down a level, multiply discount once, allows algorithms to converge\n",
        "* if we assume stationary preferences then we can only define utilities as additive or discounted\n",
        "\n",
        "Infinite utilities problem:\n",
        "* apply finite horizon (similar to depth limited search) to terminate the search after some number of steps to avoid infinite rewards\n",
        "* this will yield nonstationary policies since the policy depends on the time left\n",
        "* Alternatively, use discounting between 0 and 1 to lower future rewards, smaller y means smaller horizon, shorter term focus\n",
        "* or absorbing state, force that for each policy a terminal state will be reached eventually\n",
        "\n",
        "Star Wars hyperdrive MDP:\n",
        "* need to travel far away quickly, can cruise, hyperspace, punch it, faster is more reward\n",
        "* each MDP is like an expectimax search tree, has a state s, (s,a) as a q-state, (s,a,s') as a transition\n",
        "\n",
        "Optimal Qualities:\n",
        "* value or utility of a state V*(s) is expected utility starting in s and acting optimally\n",
        "* value or utility of a q-state (s,a) Q*(s,a) is expected utility starting out having taken action a from s, from optimal action\n",
        "* optimal policy pi*(s) is optimal action from state s\n",
        "* want to compute expectimax value of a state, average sum of discounted rewards over expected utility under optimal action\n",
        "\n",
        "Recursive definition of value (Bellman equations):\n",
        "* V*(s) = max_A Q*(s,a)\n",
        "* Q*(s,a) = $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$ \n",
        "* V*(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$\n",
        "* optimal utility via expectimax recurrence has these one step lookahead equations to get optimal values\n",
        "\n",
        "* can do too much work with expectimax in serach tree, states can be repeated and tree can go on forever\n",
        "* instead do depth limited search with increasing depths, depending on gamma depth past a certain point is negligable\n",
        "* time limited values, can define Vk(s) to be optimal value if game ends in k more time steps, or what depth k expectimax would yield\n",
        "\n",
        "Value iteration:\n",
        "* start with V0(s) = 0, then do one ply of expectimax from each state, repeat until convergence, yields complexity of O(S^2A)\n",
        "* this will converge to unique optimal values, the policy may converge way before the values which is fine\n",
        "* Vk+1(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s)]$\n",
        "* two cases for convergence: 1) if tree has maximum depth m, then VM holds the untruncated values\n",
        "* if discount < 1, will have to converge as k increases since rewards are shrinking\n",
        "\n",
        "Policy evaluation:\n",
        "* perform expectimax over all actions to get optimal values\n",
        "* motivation, if we fix policy, we only have to consider one action per state\n",
        "* compute the utility under this fixed policy, not optimal usually\n",
        "* perform one step look ahead with this policy: V_pi(s) = $\\sum_s' T(s,pi(s),s')[R(s,pi(s),s') + yV_pi(s')]$\n",
        "* turn bellman equations into updates like value iteration\n",
        "* V_0 pi(s) = 0, V_k+1 pi(s) = $\\sum_s' T(s,pi(s),s')[R(s,pi(s),s') + yV_pi(s')]$\n",
        "* yields O(S^2) efficiency per iteration\n",
        "* alternatively could solve with linear systems since there are no maxes\n",
        "\n",
        "* if we have optimal values V*(s), need to do one step expectimax to get optimal action, pi*(s) = argmax_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$\n",
        "* if we have the optimal q-values, very trivial, just pick the best one, pi*(s) = argmax_a Q*(s,a)\n",
        "\n",
        "Value iteration issues:\n",
        "* slow O(S^2A) per iteation, max at each state rarely changes, policy converges long before the values\n",
        "\n",
        "Policy iteration approach:\n",
        "* 1) policy evaluation - calculate utilities for fixed policy until they converge\n",
        "* 2) policy improvement - update policies using one step look with converged utilities as future values\n",
        "* repeat until converges, still optimal and can converge much faster\n",
        "* Evaluation, find find values V_k+1 pi_i(s) = $\\sum_s' T(s,pi_i(s),s')[R(s,pi_i(s),s') + yVk pi_i(s')]$\n",
        "* improvement, get better policy with lookahead, pi_i+1(s) = argmax_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV_pi_i(s')]$\n",
        "\n",
        "* value and policy iteration compute the same thing\n",
        "* value iteration updates values and policy implicitply, don't track the policy and instead take the max over the actions to figure it out\n",
        "* policy iteration does several passes to update utility with the same policy, evauate the policy then pick a new one, go until the policy doesn't change\n",
        "* both dynamic operations for solving MDPS\n",
        "* basically the same, variants of Bellman equations, only difference comes in fixed policy or maximum of actions\n",
        "* if you want to compute optimal values, use either, compute values for one policy use policy evaluation, turn values into policy use policy extraction\n",
        "\n",
        "Maximum expected utility:\n",
        "* reational agent should amximize expected utility given knowledge, not using minimax here and averaging instead\n",
        "* terminal function scale doesnt matter with worse case, want better states to have higher priorities, insensitivity to monotonic transformations\n",
        "* magnitudes are meaningful for average case reasoning\n",
        "* utilities have to be functions from outcomes to real numbers, can be simple or more complex, describe preferences\n",
        "* agents must have prefernce among prizes, either one to the other or indifferent\n",
        "* need constraints on preferences to assign rationality, or agent with intransitive preference can give everything up\n",
        "\n",
        "5 axioms of rationality:\n",
        "* Orderability: A >- B or B >- A, or A ~ B\n",
        "* Transitivity: A >- B and B >-C -> A >- C\n",
        "* Continuity: A >- B >- C -> exists p [p,A; 1-p, C] ~ B\n",
        "* Substitutability: A ~ B -> [p, A; 1 - p, C] ~ [p, B; 1 - p, C]\n",
        "* Monotonicity: A >- B -> (p >_ q <-> [p,A; 1 - p, B] >-_ [q,A; 1 - q, B])\n",
        "\n",
        "* if all of these satisifed, then ther exists utility function that prefers one lottery to the other and gives appropriate utilities\n",
        "* choose action that maximizes expected utility\n",
        "* normalized utilities between 0 and 1, Micromorts, QALYs, etc.\n",
        "* deterministic prizes only, ordinal utility can be determined only, meaning ordering on all the prizes\n",
        "* compare prize A to a lottery, when there is indifference we assign that probability to the utility\n",
        "* expected monetary value exists but money itself not a utility function\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"assigns a single number to express state value\",\n",
        "\"expected utility\" : \"average utility value of all outcomes weighed by their probability\",\n",
        "\"maximum expected utility\": \"agent should pick the action that maximizes their expected utility\",\n",
        "\"value function\" : \"ordinal utility function, agent only needs preference of states\",\n",
        "\"Markovian\" : \"probability of arriving in new state depends only on the current state\",\n",
        "\"Environment history\": \"the different states and actions visited by the agent following a policy\",\n",
        "\"stationary preferences\": \"if path a > b, path a with a reward > path b with the same reward\",\n",
        "\"Additive utility\": \"reward over the path is the sum of the reward with no discount\",\n",
        "\"Discounted utility\": \"reward over the path is the sum of the rewards multiplied by discounts\",\n",
        "\"Expected monetary value\": \"amount of money expected in terms of probability\",\n",
        "\"Certainty equivalent\": \"amount of money accepted instead of doing the lottery\",\n",
        "\"Insurance premium\": \"Difference between EMV and certainty equivalent\",\n",
        "\"modified policy iteration\": \"perform small number of simpliefied value iteration with fixed policy\",\n",
        "\"Asynchronous policy iteration\": \"pick a subset of states and apply policy improvement or simplified value iteration\"\n",
        "\n",
        "\n",
        "}\n",
        "\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "55fee2e6-4e4f-4a62-bd59-8ec69f0d9291"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "      print(len(summary))\n",
        "      print(len(summary.split(' ')))\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\n* Agent makes actions a, has a probability P(s) to each state s\n* Transition model for the uncertain outcomes is P(s' | s,a), probability that action a in state s yields states s'\n* related by the equation $P(RESULT(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n* make decisions based on immedaite outcomes, preferences represented by utility function U(s)\n* utility assigns a number to express the value of a state\n* expected utility EU(a) is the average utility of the outcomes weighed by probability: $EU(a) = \\sum_{s'} P(RESULT(a) = s')U(s')$ \n* maximum expected utility principle says the agent should pick the action that mazimizes the expected utility if it is rational\n* action = argmax(a) EU(a)\n* formalizes what the agent should do but doesnt operate on the advice, lots of other factors to consider\n* if the agent acts to maximize utility function that is correctly representative of the performance, agent will receive the highest possible performance score over all environments averaged\n\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nA >- B - agent prefers A over B\n\nA ~ B - agent is indifferent between A and B\n\nA >-_ B agent prefers A over B or is indifferent\n\n* lottery is a set of outcomes for each action, and each action we assume to be a lottery ticket\n* lottery with possible outcomes S1, S2, ... with probabilityies p1, p2, ... is expressed as L = [p1, S1; p2, S2; ... pn, Sn]\n* each outcome of a lottery can be an atomic state or anotehr lottery\n\nConstraints we expect lotteries to obey\n* Orderability - rational agent must prefer one lottery, the other, or rate them as equally preferrable, cannot avoid a decision\n* A >- B, B >- A, or A ~ B\n\n* Transitivity - if we have lotteries A, B, and C, and the agent prefers A to B and B to C, then it prefers A to C\n* A >- B and B >-C -> A >- C\n\n* Continuity - if a lottery B is between A and C, then there exists a probability p where the agent is indifferent between B guarenteed and lottery A with probability p and C with probability 1 - p\n* A >- B >- C -> exists p [p,A; 1-p, C] ~ B\n\n* Substitutability - agent is indifferent between A and B, then also indifferent for two more complex lotteries that are equal except for B substituted for B\n* A ~ B -> [p, A; 1 - p, C] ~ [p, B; 1 - p, C]\n\n* Monotonicity - agent prefers A to B, must prefer the lottery that has a higher probability for A\n* A >- B -> (p > q <-> [p,A; 1 - p, B] >- [q,A; 1 - q, B])\n\n* Decomposability - compound probabilities can be reduced with the laws of probability, e.g.\n* [p,A; 1 - p, [q,B; 1 - q, C]] ~ [p,A; (1 - p)q,B; (1 - p)(1 - q), C]\n\n* if an agent violates one of these it exhibits irrational behavior\n* Existance of a utility function: if agent behaves all these, then there exists a function U such that U(a) > U(b) iff A is preferred to B, and U(a) = U(b) iff the agent is indifferent\n* expected utility of a lottery is the sum of the probability of each outcome multipled by the utility of the outcome - U([p1,S1; ... ; pn,Sn]) = $\\sum_i piU(Si)$ \n* utility of a compound lottery becomes determined completely\n* agent only needs preference of states, the numbers don't matter, referred to as value function or ordinal utility function\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n\n* agent allowed any preferences it likes so long as it follows the axioms\n* preference elicitation, presents choices to humans and uses the preferences to determine the utility function\n* need to know what the human utility function is to make decisions on our behalf\n* want to create a fixed scale fo preference, normalized utilities scale from 0 to 1\n* using this normalized utility scale, we can determine the utility of a prize S and by asking the agent to pick between S and a standard lottery\n* probability is adjusted until the agent is indifferent, then S is assigned that utility, do this for each prize then have a lookup table of values\n* tradeoffs of life and death made all the time\n* refusal to put monetary value on life can mean its undervalued\n* value of a statistical life determines costs and benefits of regulation/intervention\n* micromort is 1 in a million chance of death, QALY is quality adjusted life years for patients\n* if an agent prefers more money than less, it has a monotonic preference for more money\n* expected monetary value is probabilities multiplied by monetary amounts\n* utility of money generally found to be proportional to the logarithm of the amount\n* positive curve, agents are risk-averse and prefer a sure thing, negative curve, risk-seeking\n* certainty equivalent is the amount someone will except instead of doing the lottery\n* difference between the EMV of the lottery and the certainty equivalent is the insurance premium\n* an agent is risk-neutral if the linear curve is observed\n* often working with estimated expected values EU(a) instead of the true expected utility, assumed to be unbiased when not always the case\n* order statistic is distribution of ranked element of a sample\n* optimizer's curse, tendency for expected utility of the best choice to be too high\n* normative theory describes how rational agents should act, descriptive theory is how actual agents really act\n* certainty effect, people often prefer the sure thing in gains, reduce computational burden, distrust the stated probabilities, regret if it doesnt pan out\n* people also have ambiguity aversion, go with the sure probability\n* framing effect, how decision problems are worded\n* anchoring effect, making relative utiltiy judgements rather than absolute ones, cheaper product compared to more expensive one\n* evolutionary psychology says our decision making processes were not evolved to solve word problems with probabilities\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\n* fully observable environment allows agent to always know where it is\n* transition model describes the outcome of each action in the state, stocastic in our maze so P(s'|s,a) as before, or T(s,a,s')\n* utility function here depends on sequence of states and actions (environmental history), rather than single state\n* agent will receive a reward R(s,a,s') for moving to state s'\n* Markov Decision process (MDP) consists of set of states, actions, transtiion model, reward function\n* involve dynamic programming often, simplifying a problem by breaking it down and solving optimal solutions to subproblems\n* policy solution pi is action recommended for a given state so agent knows what to do, quality measured by expected utility of environment histories generated\n* optimal policy pi* yields highest expected utility\n* reward a big factor in how the agent moves and if/when it chooses to exit\n* with a finite horizon (meaing theres a fixed time after which nothing matters), an optimal action may depend on time left\n* nonstationary policy depends on the time\n* with infinite time (horizon), optimal action only depends on current state, policy is stationary, these are simpler\n* discount factor between 0 and 1, preference of agent for current vs. future rewards. Close to 0, future rewards are insignificant, close to 1 more willing to wait for them. If exactly 1 then purely additive rewards\n* makes infinities go away, rewards not always guarenteed, valued higher\n* utility of infinite sequence now finite\n* if agent contains terminal states and guarenteed to reach it then never need to compare infinite sequences, proper policy if it exits\n* infinite sequences can be compared by average reward obtained per time step\n* expected utility of executing pi starting in s is Upi(s) = E * $\\sum_t->inf y^t R(S_t, pi(S_t), S_{t+1})$\n* optimal policy pi* = argmax of pi U^pi (s)\n* in infinite horizons with discouted utilities, optimal policy independent of starting state\n* utility function allows agent to select actions with highest expected utility: $pi^*(s) = argmax_{a exist A(s)} \\sum_s' P(s'|s,a)[R(s,a,s') + yU(s')]$\n* utility of state is expected reward for next transtion plus discounted utility of next state, so utility of state is: $U(s) = max_{a exist A(s)} \\sum_s' P(s'|s,a)[R(s,a,s') + yU(s')]$, bellman equation\n* action-utility function or Q function, Q(s,a) is expected utility of taking action in given state: U(s) = max_a Q(s,a), optimal policy takes max action that yields highest Q state\n* can develop equation for Q-funtcion, $Q(s,a) = \\sum_s' P(s'|s,a)[R(s,a,s') + ymax_{a'}Q(s',a')]$\n* all policies optimal with a reward function that is always zero\n* represent P(s'|s,a) and R(s,a,s') using 3D tables of size |S|^2|A|, often sparse tables because bounded transitions between states\n* can represent MDPs by extending dynamic bayesian networks to create dynamic decision etwrosk, factored representations with exponential complexity advatnage over atomic representation\n\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\n* if we have n possible states, n Bellman equations, one per state\n* n equations have n unknowns, utilities of each state\n* equations nonlinear because the max operation isn't linear, try iterative approach instead\n* start with arbitrary values for the utilities, then update each utility until we hit equilibrium\n* update assumed to be applied to all states simultaneously, will converge and give unique solutions and the policy\n* value iteration, think of as propagating information through state space to update each state \n* value teration converges due to contraction, function of one argument that produces two output values that are close together by some constant factor, applied to two inputs\n* only has one fixed point, and the value must get closer to the fixed poiint upon every application\n* distance between the utility vectors measured with max norm, measures the length of the vector by value of biggest component\n* Bellman update is contraction by factor of discount on space of utility vectors\n* agent only cares about how well it does if it makes decision using utility function\n* define policy loss as the most an agent can lose by executing a policy instead of the optimal one\n* policy iteration alternates between evaluation and improvement\n* policy evaluation: given policy pi_i, calculate U_i = Upi_i, utility of each step if current policy is executed\n* policy improvement: calculate new MEU policy pi_i + 1 using look ahead based on U_i\n* terminates when policy improvement doesnt update the utilities\n* use q value function, get a simplified version of Bellman equation relating utility of s to neighbor utilities\n* equations now linear since max is removed, so we can solve a n state n linear equation n unknown in O(n^3) time\n* often the most efficient in small state spaces\n* don't have to do exact policy evaluation, can do number of simplified value iteratoin steps to give estimate\n* repeat this many times to get the next utility estimate, modified policy iteration\n* can also update any subset of states and apply policy iteration or simplified value iteration to the subset, asynchronous policy iteration, guarenteed to converge to optimal policy\n* Linear progreamming used to formulate constrained optimization problems\n* can reduce solving MDP to solving formulated linear program\n* want to minimize U(s) for all s\n* MDPs can be solved in polynomial time in number of states, actions, bits to specify model\n* Value and policy iteration both offline algorithms to solve states then agent acts\n* for online, use expectiminimax simplification, evaluation function applied at nonterminal leaves given default value\n* decision taken by backing up the utility values and averaging at the chance nodes\n* limit depth to avoid high branching factor, focus on most likely outcomes\n* real time dynamic programming, explored states create a sub-MDP of MDP, and can be solved to yield decision of current state\n* state space may be too large, or difficult to make heuristic\n\n\n## Module 8 Lecture Notes\n\nAsteroid Field:\n* 4x3 environment with goal states, pick an action at each time step, terminates when goal state reached, stochastic environment\n* pick a sequence of actions, can get there correctly with no errors or with \"correct\" errors\n* Transition model written as T(s,a,s'), or p(s',s,a) - probability of reaching s' given s and action a\n* Markovian transitions, meaning probability of arriving in s' depends only on s, no earlier states\n* utility/reward function can be positive or negative but bounded by some maximum value\n* can receive this reward R(s,a,s') at any time step since the decisions are sequential, additive as well\n\nMarkov Decision Process (MDPs):\n* sequential decision process, fully observable environment, stochastic transition model, additive rewards\n* non-deterministic search problems so we can solve using expectimax search to calculated expected utilities\n* to solve an MDP, need 1) a set of states s in S, 2) a set of actions a in A, 3) a transition function T(s,a,s') or the probabilit that executing action a in s leads to s', 4) a reward function R(s,a,s'), 5) an initial state s0, 6) one or more terminal states (optional)\n* solution is a policy since there is non-determinism, not guaranteed we end up where we want to go, pi(s) = a. Stochastic so may be correct but not happen, optimal policy is pi*\n* same policy executed multiple times may lead to different states and actions and different scores, so need expected utility of all paths generated\n* performance of agent is the sum of the rewards of the transitions it takes, can cause the agent to exit early/late/never\n* Policy is choice of action for each state, utility is sum of the potentially discounted rewards\n\nDiscounting:\n* Want to maximize sum of rewards, prefer now versus later\n* each time we go down a level, multiply discount once, allows algorithms to converge\n* if we assume stationary preferences then we can only define utilities as additive or discounted\n\nInfinite utilities problem:\n* apply finite horizon (similar to depth limited search) to terminate the search after some number of steps to avoid infinite rewards\n* this will yield nonstationary policies since the policy depends on the time left\n* Alternatively, use discounting between 0 and 1 to lower future rewards, smaller y means smaller horizon, shorter term focus\n* or absorbing state, force that for each policy a terminal state will be reached eventually\n\nStar Wars hyperdrive MDP:\n* need to travel far away quickly, can cruise, hyperspace, punch it, faster is more reward\n* each MDP is like an expectimax search tree, has a state s, (s,a) as a q-state, (s,a,s') as a transition\n\nOptimal Qualities:\n* value or utility of a state V*(s) is expected utility starting in s and acting optimally\n* value or utility of a q-state (s,a) Q*(s,a) is expected utility starting out having taken action a from s, from optimal action\n* optimal policy pi*(s) is optimal action from state s\n* want to compute expectimax value of a state, average sum of discounted rewards over expected utility under optimal action\n\nRecursive definition of value (Bellman equations):\n* V*(s) = max_A Q*(s,a)\n* Q*(s,a) = $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$ \n* V*(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$\n* optimal utility via expectimax recurrence has these one step lookahead equations to get optimal values\n\n* can do too much work with expectimax in serach tree, states can be repeated and tree can go on forever\n* instead do depth limited search with increasing depths, depending on gamma depth past a certain point is negligable\n* time limited values, can define Vk(s) to be optimal value if game ends in k more time steps, or what depth k expectimax would yield\n\nValue iteration:\n* start with V0(s) = 0, then do one ply of expectimax from each state, repeat until convergence, yields complexity of O(S^2A)\n* this will converge to unique optimal values, the policy may converge way before the values which is fine\n* Vk+1(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s)]$\n* two cases for convergence: 1) if tree has maximum depth m, then VM holds the untruncated values\n* if discount < 1, will have to converge as k increases since rewards are shrinking\n\nPolicy evaluation:\n* perform expectimax over all actions to get optimal values\n* motivation, if we fix policy, we only have to consider one action per state\n* compute the utility under this fixed policy, not optimal usually\n* perform one step look ahead with this policy: V_pi(s) = $\\sum_s' T(s,pi(s),s')[R(s,pi(s),s') + yV_pi(s')]$\n* turn bellman equations into updates like value iteration\n* V_0 pi(s) = 0, V_k+1 pi(s) = $\\sum_s' T(s,pi(s),s')[R(s,pi(s),s') + yV_pi(s')]$\n* yields O(S^2) efficiency per iteration\n* alternatively could solve with linear systems since there are no maxes\n\n* if we have optimal values V*(s), need to do one step expectimax to get optimal action, pi*(s) = argmax_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$\n* if we have the optimal q-values, very trivial, just pick the best one, pi*(s) = argmax_a Q*(s,a)\n\nValue iteration issues:\n* slow O(S^2A) per iteation, max at each state rarely changes, policy converges long before the values\n\nPolicy iteration approach:\n* 1) policy evaluation - calculate utilities for fixed policy until they converge\n* 2) policy improvement - update policies using one step look with converged utilities as future values\n* repeat until converges, still optimal and can converge much faster\n* Evaluation, find find values V_k+1 pi_i(s) = $\\sum_s' T(s,pi_i(s),s')[R(s,pi_i(s),s') + yVk pi_i(s')]$\n* improvement, get better policy with lookahead, pi_i+1(s) = argmax_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV_pi_i(s')]$\n\n* value and policy iteration compute the same thing\n* value iteration updates values and policy implicitply, don't track the policy and instead take the max over the actions to figure it out\n* policy iteration does several passes to update utility with the same policy, evauate the policy then pick a new one, go until the policy doesn't change\n* both dynamic operations for solving MDPS\n* basically the same, variants of Bellman equations, only difference comes in fixed policy or maximum of actions\n* if you want to compute optimal values, use either, compute values for one policy use policy evaluation, turn values into policy use policy extraction\n\nMaximum expected utility:\n* reational agent should amximize expected utility given knowledge, not using minimax here and averaging instead\n* terminal function scale doesnt matter with worse case, want better states to have higher priorities, insensitivity to monotonic transformations\n* magnitudes are meaningful for average case reasoning\n* utilities have to be functions from outcomes to real numbers, can be simple or more complex, describe preferences\n* agents must have prefernce among prizes, either one to the other or indifferent\n* need constraints on preferences to assign rationality, or agent with intransitive preference can give everything up\n\n5 axioms of rationality:\n* Orderability: A >- B or B >- A, or A ~ B\n* Transitivity: A >- B and B >-C -> A >- C\n* Continuity: A >- B >- C -> exists p [p,A; 1-p, C] ~ B\n* Substitutability: A ~ B -> [p, A; 1 - p, C] ~ [p, B; 1 - p, C]\n* Monotonicity: A >- B -> (p >_ q <-> [p,A; 1 - p, B] >-_ [q,A; 1 - q, B])\n\n* if all of these satisifed, then ther exists utility function that prefers one lottery to the other and gives appropriate utilities\n* choose action that maximizes expected utility\n* normalized utilities between 0 and 1, Micromorts, QALYs, etc.\n* deterministic prizes only, ordinal utility can be determined only, meaning ordering on all the prizes\n* compare prize A to a lottery, when there is indifference we assign that probability to the utility\n* expected monetary value exists but money itself not a utility function\n\n\n## Module 8 Definitions\n* **Utility function** - assigns a single number to express state value\n* **Expected utility** - average utility value of all outcomes weighed by their probability\n* **Maximum expected utility** - agent should pick the action that maximizes their expected utility\n* **Value function** - ordinal utility function, agent only needs preference of states\n* **Markovian** - probability of arriving in new state depends only on the current state\n* **Environment history** - the different states and actions visited by the agent following a policy\n* **Stationary preferences** - if path a > b, path a with a reward > path b with the same reward\n* **Additive utility** - reward over the path is the sum of the reward with no discount\n* **Discounted utility** - reward over the path is the sum of the rewards multiplied by discounts\n* **Expected monetary value** - amount of money expected in terms of probability\n* **Certainty equivalent** - amount of money accepted instead of doing the lottery\n* **Insurance premium** - Difference between EMV and certainty equivalent\n* **Modified policy iteration** - perform small number of simpliefied value iteration with fixed policy\n* **Asynchronous policy iteration** - pick a subset of states and apply policy improvement or simplified value iteration\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\"\n",
        "* N armed bandit is slot machined with n levels\n",
        "* Tradeoff of exploitation of current best action and exploration of previously unknown states and actions to learn information\n",
        "* In bandit problems, each arm Mi is a Markov decision process that is an MDP with only one action\n",
        "* The overall bandit problem is an MDP with all the same variables as before \n",
        "* Important to note that arms are independent, only can work one at a time\n",
        "* If we pull a one armed bandit T times, say the stopping time is T\n",
        "* Getting index defines the kind of value for M in terms of ability to deliver timely rewards\n",
        "* Pull the arm with the highest getting index then update the indices, each decision can be calculated in O(1) time\n",
        "* Bernoulli bandit has each arm Mi with a reward of 0 or 1 with a fixed unknown probability, infinitely many states exist in it, but can approximate\n",
        "* Arms with higher payoff probabilities are preferred but there is an exploration bonus for those only tried a few times\n",
        "* Getting indices for realistic problems is very difficult\n",
        "* Upper confidence bound, a heuristic that samples each arm to establish confidence interval for the arm, then choose the arm with then highest upper bound on the confidence interval\n",
        "* Just picks the arm with highest UCB value\n",
        "* Determines the regression relative to the clairvoyant policy, picks the best arm and averages the reward\n",
        "* No algorithm can have regret that grows more slowly than O(logN)\n",
        "* Thompson sampling chooses an arm randomly according to the probability that it is optimal given the samples so far, regret of O(logN)\n",
        "* No index function exists for a selection problem\n",
        "\"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\"\n",
        "* Agent gets rewards/reinforcements for how it’s doing\n",
        "* Same goal as MDPs, maximize the expected sum of the rewards\n",
        "* Agent is the MDP instead of being handed one, has to learn transition model and reward function\n",
        "* Model based reinforcement learning, agent uses a transition model of environment to help interpret rewards and make decisions on how to act\n",
        "* Transition model useful for state estimation, and RL learns utility function\n",
        "* Model free reinforcement learning, agent doesn’t know or learn transition model, has to learn direct representation of how to behave\n",
        "* Action utility learning is utilized, agent need to learn Q functions and choose actions based on that\n",
        "* Learns a policy that maps to the states and has to act accordingly, a reflex agent\n",
        "\"\"\"\n",
        "chapter22_2 =  \"\"\"\n",
        "* Passive reinforcement learning, agent has fixed policy that determines its actions and it wants to learn the utility function when executing the actions\n",
        "* Similar to policy evaluation, but doesn’t know transition model that specifies probability of reaching next state, nor the reward function\n",
        "* Usually include discount factor to encourage it to prioritize nearer rewards\n",
        "* Direct utility estimation, utility of the state is defined as expected total reward from the state onwards, and each trial is a sample of this quantity for each state visited\n",
        "* This approach ignores that utility is determined by reward and expected utility of successor states, so it misses learning and can converge very slowly\n",
        "* Adaptive dynamic programming agent learns transitions model that connects states and solves MDPs using dynamic programming\n",
        "* Plugging in learned transition model and observed rewards to calculate utilities of states\n",
        "* Can also use modified policy iteration, simplified value iteration process to update utility estimates after each change to the model\n",
        "* Temporal difference learning, use observed transitions to adjust utilities of the observed states so they align with constraint equations\n",
        "* Include a learning rate parameter, update rule uses difference in utilities between successive states\n",
        "* Adjust the utility estimates towards ideal equilibrium that holds locally when all estimates are correct\n",
        "* Average value of the optimal utility converges to correct quantity even if values continue to move around\n",
        "* TD doesn’t need a transition model to perform the updates, environment supplies the connections instead\n",
        "* TD adjusts state to agree with observed successors, whereas ADP adjusts state to agree with all successors that might occur weighed by probability\n",
        "* Prioritized sweeping heuristic adjusts states whose likely successors have undergone large adjustment in utility estimates first\n",
        "\"\"\"\n",
        "chapter22_3 =  \"\"\"\n",
        "* Active reinforcement learning agent gets to decide what actions to take instead\n",
        "* Needs to learn complete transition model with outcome probabilities for all actions rather than just the policy \n",
        "* Sometimes acting greedy will converge to optimal solution, sometimes not\n",
        "* Agent still must make a tradeoff between exploration of current best action and exploration of unknown states like in bandit problems\n",
        "* Greedy in limit of infinite exploration (GLIE) tries teach action in each state unbounded number of times to avoid having a finite probability that the optimal action is missed\n",
        "* Can have it randomly choose an action every so often, or give weights to actions it has not tried often, avoid actions with low believed utilities\n",
        "* Have an exploration function then that determines how greed is traded off against curiosity\n",
        "* Many actions are irreversible, nothing can restore it to the state you just left from\n",
        "* Worst case agent entered an absorbing state where no actions have any effect and no rewards are received \n",
        "* Agent in real settings needs to avoid both of these\n",
        "* Can end up in a bad state because our model is known or because its incorrect\n",
        "* Bayesian reinforcement learning assumes prior probability over hypotheses h about what the true model is, agent can decide to stop learning and get highest expected utility path\n",
        "* This becomes an exploration POMDP where belief states are distributions over models\n",
        "* Second approach from robust control theory allows set of possible models without assigning probability, then defines optimal robust policy as one that has the best outcome in worst case\n",
        "* This can lead to overly conservative behavior\n",
        "* Temporal difference Q learning, has to construct active temporal difference learning agent to learn transition model so it can choose an action based on utility function and look ahead\n",
        "* Q-learning method avoids need for the model by learning the action utility function instead of utility function, doesn’t need a transition model either for learning or for action selection\n",
        "* SARSA is closely related, state action reward state action, update rule is very similar except it updates with Q value of action that is actually taken\n",
        "* Q learning is off policy because it learns Q values to whether actions would be worth in this state if you stopped whatever policy you are using and act according to policy that chose best action \n",
        "* SARSA is on policy since it learns Q values that ask whether action would be worth it if you stick to the policy, Q learning more flexible as a result\n",
        "* Both learn optimal policy but slower than ADP agent\n",
        "\"\"\"\n",
        "chapter22_4 =  \"\"\"\n",
        "* Generalization in reinforcement learning\n",
        "* Look ahead search means behavior can be generated from much simpler utility function approximation learnable from fewer experiences\n",
        "* State and sum of rewards are a training example for supervised learning algorithm\n",
        "* For RL, makes more sense to use online learning algorithm that updates parameters after each trial\n",
        "* Changing the parameters theta_i in response to observed transition between two states changes value of U_theta for every other state in Widrow-Hoff rule for online least squares\n",
        "* For passive TD learning, update rule can be shown to converge to closest possible approximation to true function when function approximate is linear in features, when nonlinear anything can happen\n",
        "* Need to go beyond linear function approximations because may be no good linear function that comes close to approximating utility function, and may not be able to invent features in new domain\n",
        "* Use deep reinforcement learning to try and explore nonlinear function approximations \n",
        "* Try to figure out what went wrong in long series of actions, credit assignment\n",
        "* Reward shaping, supplying agent with psuedorewards for making progress, can speed up learning quite a bit\n",
        "* Hierarchical reinforcement learning, break up long action sequences into smaller pieces, until action sequences are short enough to make learning easy\n",
        "* Begins with partial program that outlines hierarchical structure for agents behavior, give it a choose operator to allow agent to do anything from specific set\n",
        "* Based on concept of joint state space, each state composite of physical and machine state\n",
        "* Choice state is one where program counter for m is at choice point in agent program\n",
        "* Can be effective for learning complex behaviors\n",
        "* Provides additive decomposition of overall utility function as well\n",
        "\"\"\"\n",
        "chapter22_5 =  \"\"\"\n",
        "* Policy search, keep adjusting the policy as long as it changes and performance improves, then stop\n",
        "* Interested in parameterized representation of policy that has fewer parameters than states in state space\n",
        "* Each q function could be linear or nonlinear function, policy search adjusts parameters to improve policy, not the same as q learning\n",
        "* Policy search finds value of theta that results in good performance, whereas Q learning trying to optimize Q function\n",
        "* Often use stochastic policy representations that specify the probability of selection action a in state s, like the softmax function\n",
        "* Policy value is expected reward to go when current policy is executed\n",
        "* Follow policy gradient vector provide that it is differentiable\n",
        "* Then follow the empirical gradient by evaluating change in policy value for small increments in each parameter, converges to local optimum in policy space\n",
        "* Gets more difficult when environment is nondeterministic\n",
        "* Correlated sampling, have each program play the same set of actions an eliminate measurement error due to difference in states received\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\"\n",
        "Double bandits\n",
        "* slot machines, can create MDP when you pull the lever and either win or lose from each state, no discount, both states have same value\n",
        "* solving MDPs if offline, all quantities calculated out and you know the details of the MDP, dont need to actually play\n",
        "* Online planning suddently different, have to figure out what is going on with the states\n",
        "* learning instead of planning, still and MDP but you have to figure it out instead of being able to do so ahead of time\n",
        "* Exploration - tried unknown actions to get information\n",
        "* Exploitation - eventually use whatever you learn/know already\n",
        "* Regret - even if you learn well, still make mistakes\n",
        "* Sampling - have to try things many times because of chance\n",
        "* Difficulty - learning can be harder than solving MDP computationally\n",
        "\n",
        "Reinforcement learning\n",
        "* have an agent that provides action to environment, that returns state s and reward r\n",
        "* get feedback in form of rewards, utility of the agent defined by these reward functions, so must maximize the expected rewards\n",
        "* all fo the learning is done based on samples of the outcomes\n",
        "* still have set of states s in S, actions a in A, model T(s,a,s'), reward function R(s,a,s'), policy pi(s)\n",
        "* now we don't know T or R so must try actions to learn what to do and which states to prefer\n",
        "* offline is MDPs, online is reinforcement learning\n",
        "\n",
        "Model-Based learning\n",
        "* learn approximate model based on observations, solve for the valuess as if what you learned is correct\n",
        "* 1) learn MDP model from experience by counting outcomes s' for each state and action pair, normalize to get estimate of T(s,a,s'), discover each R(s,a,s') when (s,a,s') happens\n",
        "* 2) solve the learned MDP, can use value iteration as before\n",
        "\n",
        "Model Free learning:\n",
        "Passive Reinforcement learning\n",
        "* now do policy evaluation, given a fixed policy pi(s) and unknown T and R functions, want to learn the state values\n",
        "* learner has no choice what actions to take, just execute the policy and learn from what happens\n",
        "* actions actually taken, no offline planning involved\n",
        "\n",
        "Direction Evaluation\n",
        "* want to compute values for every state given the current policy\n",
        "* do what the policy says, write down the sum of discounted rewards each time you visit a state, and average them all together to get a decent estimate\n",
        "* easy to understand, don't need to know T or R, eventually gets the correct averages using transitions\n",
        "* but can waste information and state connection s, takes a long time since each state is learned separately\n",
        "* in policy evaluation, updates calculaed V for the fixed policy by replacing it with the lookahead step\n",
        "* we need T and R to do it though, so not possible here, how can we update the weighted average without knowing the weights\n",
        "\n",
        "Sample Based Policy Evaluation\n",
        "* want to calculate the bellman equation of V_k+1 pi_i(s) = $\\sum_s' T(s,pi_i(s),s')[R(s,pi_i(s),s') + yVk pi_i(s')]$, take sames of outcomes s', take action and average\n",
        "* sample_1 = R(s, pi(s), s1') + yV_k pi(s1')\n",
        "* sample_2 = R(s, pi(s), s2') + yV_k pi(s2')  \n",
        "* V_k+1 pi_i(s) = 1/n $\\sum_i sample_i$\n",
        "\n",
        "Temporal difference learning\n",
        "* want to learn from every experience, update V(s) every time we have a transition, so likely s' will contribute more often\n",
        "* still a fixed policy with the evaluation, keep a running average fo the successor values\n",
        "* samples of V(s): sample = R(s,pi_i(s),s') + yVpi(s')]\n",
        "* updates to V(s): Vpi(s) = (1 - a)Vpi(s) + a(sample)\n",
        "* same update: Vpi(s) = Vpi(s) + a(sample - Vpi(s))\n",
        "* keep an exponential moving average with running interpoliation uupdate: x_n = (1 - a) * x_n-1 + a * x_n\n",
        "* makes the recent samples more important and forgets about the distance past values that were less accurate anyway\n",
        "* x_n = x_n + (1 - a) * x_n-1 + (1 - a)^2 * x_n-2 + ... / (1 + (1 - a) + (1 - a)^2 + ...)\n",
        "* if we decrease the learning rate, can get converging averages\n",
        "* mimicks Bellman updates with these running averages, but if we want to turn values into new policy, can't do it because we need Q values for policy\n",
        "* so learn Q-values instead of values, makes the actino selection model free as well\n",
        "\n",
        "Active reinforcement learning\n",
        "* full reinforcement learning has optimal policies where you don't know transitions or rewards but can choose actions, learn optimal policy/values\n",
        "* tradeoff exploration and explotation\n",
        "* Q learning, find the successive depth limited valeus with value iteration: Vk+1(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yVk(s')]$\n",
        "* Q-values more useful so we use them instead: Qk+1(s,a) = $\\sum_s' T(s,a,s')[R(s,a,s') + yMax_a'Qk(s',a')]$\n",
        "* start with both terms as 0 which we know is right\n",
        "* receive a sample (s,a,s',r), consider old esimate Q(s,a)\n",
        "* look at new sample estimate: sample = R(s,a,s') + ymax_a' Q(s',a'), incorporate into running average Q(s,a) = (1 - a)Q(s,a) + a[sample]\n",
        "* acting suboptimally, but q learning converges to optimal policy, called off-policy learning\n",
        "* to do so, need to explore enough, make learning rate small enough but not too quickly, doesn't matter how you select actions\n",
        "\n",
        "Exploration\n",
        "* can pick random actions, epsilon greedy, flip a coin and act randomkly with some probability epsilon, otherwise act on current policy\n",
        "* if we do this, can thrash when learning is done, could also lower epsilon over time or look at exploration functions\n",
        "* explore areas where we are not sure how bad they are, can eventually stop exploring\n",
        "* take a value estimate u and visit count n, return optimistic utility like f(u,n) = u + k/n\n",
        "* instead of regular q update Q(s,a) = a R(s,a,s') + y max_a' Q(s',a'), do modified q update: Q(s,a) = a R(s,a,s') + y max_a' f(Q(s',a'), N(s',a'))\n",
        "* propagates bonus back to states that lead to unknown ones\n",
        "\n",
        "Regret - measure of total mistake cost, difference between expected rewards (with suboptimal moves) and optimal expected rewards\n",
        "* requres optimal learning to be optimal in minimizing regret, not possible often\n",
        "\n",
        "Approximate Q-learning\n",
        "* basic q-learning keeps a table of all q-values, can't do this often and learn about each state, too many to visit in training/hold in memory\n",
        "* generalize instead from small number of training states from experience to new similar situations\n",
        "* from evaluation functions, score non-terminals in depth limited search and wanted to return actual minimax function but had to do linear sum of features eval(s) = w1f1(s) + ... wnfn(s)\n",
        "* here we can write a q function for any state with a few weights:\n",
        "* V(s) = w1f1(s) + ... wnfn(s)\n",
        "* Q(s,a) = w1f1(s,a) + ... wnfn(s,a)\n",
        "* good since our experience is summer up in a few numbers, but states may share features but be very different in value\n",
        "* have our transition function (s,a,s',r), difference = [r + y max_a' Q(s',a')] - Q(s,a)\n",
        "* exact Qs are Q(s,a) = Q(s,a) + a * difference, approximate Qs are wi = wi + a * difference * fi(s,a)\n",
        "* we adjust the weights of the active features, disprefer bad states if something bad happens, can justify with online least squares\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"off-policy learning\" : \"Q learning in active reinforcement learning converges to optimal policy\",\n",
        "\"n-armed bandit\": \"Slot machine with n levers\",\n",
        "\"one armed bandit\": \"slot machine with only one arm M that produces rewards and has fixed cost for each pull\",\n",
        "\"Restart MDP\": \"learn about the MDP through Gittins index and go back to the initial state\",\n",
        "\"selection problem\": \"choosing the best option of a bandit environment under certain conditions\",\n",
        "\"Bandit superprocess\": \"each arm is full MDP rather than a Markov deicion prosess with one action\",\n",
        "\"opportunity cost\": \"how much utility is given up by not devoting time to another arm\",\n",
        "\"dominating policy\": \"An optimal policy if it exists for ann MDP\",\n",
        "\"reinforcement learning\": \"agent interacts with world and periodically gets rewards based on how its doing\",\n",
        "\"sparse rewards\": \"agent given no informative reward signal for most of the game\",\n",
        "\"passive reinforcement learning\": \"fixed policy and need to learn utility of states\",\n",
        "\"active reinforcement learning\": \"agent must also figure out what to do and learn utilities\",\n",
        "\"exploration\": \"agent must learn as much about its environment as possible\",\n",
        "\"trial\": \"executing the policy in passive reinforcement learning\",\n",
        "\"pseudoexperience\": \"simulating the current transition model in TD and ADP\",\n",
        "\"greedy agent\": \"greedily takes the action it believes to be optimal at each step in ARL\",\n",
        "\"evaluation function\": \"compact measure of desirability for vast state spaces\",\n",
        "\"function approximation\": \"process of constructing compact approximation of true utility or Q function\",\n",
        "\"catastrophic forgetting\": \"assign zero weights to features when model learns too well\",\n",
        "\"experience replay\": \"replay experience perioidcally to avoid catastophic forgetting\",\n",
        "\"credit assignment\": \"figuring out where agent went wrong in long string of actions/states\"\n",
        "\n",
        "\n",
        "\n",
        "}"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "547647ae-7f62-4d6f-946c-ad63e463988b"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n\n* N armed bandit is slot machined with n levels\n* Tradeoff of exploitation of current best action and exploration of previously unknown states and actions to learn information\n* In bandit problems, each arm Mi is a Markov decision process that is an MDP with only one action\n* The overall bandit problem is an MDP with all the same variables as before \n* Important to note that arms are independent, only can work one at a time\n* If we pull a one armed bandit T times, say the stopping time is T\n* Getting index defines the kind of value for M in terms of ability to deliver timely rewards\n* Pull the arm with the highest getting index then update the indices, each decision can be calculated in O(1) time\n* Bernoulli bandit has each arm Mi with a reward of 0 or 1 with a fixed unknown probability, infinitely many states exist in it, but can approximate\n* Arms with higher payoff probabilities are preferred but there is an exploration bonus for those only tried a few times\n* Getting indices for realistic problems is very difficult\n* Upper confidence bound, a heuristic that samples each arm to establish confidence interval for the arm, then choose the arm with then highest upper bound on the confidence interval\n* Just picks the arm with highest UCB value\n* Determines the regression relative to the clairvoyant policy, picks the best arm and averages the reward\n* No algorithm can have regret that grows more slowly than O(logN)\n* Thompson sampling chooses an arm randomly according to the probability that it is optimal given the samples so far, regret of O(logN)\n* No index function exists for a selection problem\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n\n* Agent gets rewards/reinforcements for how it’s doing\n* Same goal as MDPs, maximize the expected sum of the rewards\n* Agent is the MDP instead of being handed one, has to learn transition model and reward function\n* Model based reinforcement learning, agent uses a transition model of environment to help interpret rewards and make decisions on how to act\n* Transition model useful for state estimation, and RL learns utility function\n* Model free reinforcement learning, agent doesn’t know or learn transition model, has to learn direct representation of how to behave\n* Action utility learning is utilized, agent need to learn Q functions and choose actions based on that\n* Learns a policy that maps to the states and has to act accordingly, a reflex agent\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\n* Passive reinforcement learning, agent has fixed policy that determines its actions and it wants to learn the utility function when executing the actions\n* Similar to policy evaluation, but doesn’t know transition model that specifies probability of reaching next state, nor the reward function\n* Usually include discount factor to encourage it to prioritize nearer rewards\n* Direct utility estimation, utility of the state is defined as expected total reward from the state onwards, and each trial is a sample of this quantity for each state visited\n* This approach ignores that utility is determined by reward and expected utility of successor states, so it misses learning and can converge very slowly\n* Adaptive dynamic programming agent learns transitions model that connects states and solves MDPs using dynamic programming\n* Plugging in learned transition model and observed rewards to calculate utilities of states\n* Can also use modified policy iteration, simplified value iteration process to update utility estimates after each change to the model\n* Temporal difference learning, use observed transitions to adjust utilities of the observed states so they align with constraint equations\n* Include a learning rate parameter, update rule uses difference in utilities between successive states\n* Adjust the utility estimates towards ideal equilibrium that holds locally when all estimates are correct\n* Average value of the optimal utility converges to correct quantity even if values continue to move around\n* TD doesn’t need a transition model to perform the updates, environment supplies the connections instead\n* TD adjusts state to agree with observed successors, whereas ADP adjusts state to agree with all successors that might occur weighed by probability\n* Prioritized sweeping heuristic adjusts states whose likely successors have undergone large adjustment in utility estimates first\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\n* Active reinforcement learning agent gets to decide what actions to take instead\n* Needs to learn complete transition model with outcome probabilities for all actions rather than just the policy \n* Sometimes acting greedy will converge to optimal solution, sometimes not\n* Agent still must make a tradeoff between exploration of current best action and exploration of unknown states like in bandit problems\n* Greedy in limit of infinite exploration (GLIE) tries teach action in each state unbounded number of times to avoid having a finite probability that the optimal action is missed\n* Can have it randomly choose an action every so often, or give weights to actions it has not tried often, avoid actions with low believed utilities\n* Have an exploration function then that determines how greed is traded off against curiosity\n* Many actions are irreversible, nothing can restore it to the state you just left from\n* Worst case agent entered an absorbing state where no actions have any effect and no rewards are received \n* Agent in real settings needs to avoid both of these\n* Can end up in a bad state because our model is known or because its incorrect\n* Bayesian reinforcement learning assumes prior probability over hypotheses h about what the true model is, agent can decide to stop learning and get highest expected utility path\n* This becomes an exploration POMDP where belief states are distributions over models\n* Second approach from robust control theory allows set of possible models without assigning probability, then defines optimal robust policy as one that has the best outcome in worst case\n* This can lead to overly conservative behavior\n* Temporal difference Q learning, has to construct active temporal difference learning agent to learn transition model so it can choose an action based on utility function and look ahead\n* Q-learning method avoids need for the model by learning the action utility function instead of utility function, doesn’t need a transition model either for learning or for action selection\n* SARSA is closely related, state action reward state action, update rule is very similar except it updates with Q value of action that is actually taken\n* Q learning is off policy because it learns Q values to whether actions would be worth in this state if you stopped whatever policy you are using and act according to policy that chose best action \n* SARSA is on policy since it learns Q values that ask whether action would be worth it if you stick to the policy, Q learning more flexible as a result\n* Both learn optimal policy but slower than ADP agent\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\n* Generalization in reinforcement learning\n* Look ahead search means behavior can be generated from much simpler utility function approximation learnable from fewer experiences\n* State and sum of rewards are a training example for supervised learning algorithm\n* For RL, makes more sense to use online learning algorithm that updates parameters after each trial\n* Changing the parameters theta_i in response to observed transition between two states changes value of U_theta for every other state in Widrow-Hoff rule for online least squares\n* For passive TD learning, update rule can be shown to converge to closest possible approximation to true function when function approximate is linear in features, when nonlinear anything can happen\n* Need to go beyond linear function approximations because may be no good linear function that comes close to approximating utility function, and may not be able to invent features in new domain\n* Use deep reinforcement learning to try and explore nonlinear function approximations \n* Try to figure out what went wrong in long series of actions, credit assignment\n* Reward shaping, supplying agent with psuedorewards for making progress, can speed up learning quite a bit\n* Hierarchical reinforcement learning, break up long action sequences into smaller pieces, until action sequences are short enough to make learning easy\n* Begins with partial program that outlines hierarchical structure for agents behavior, give it a choose operator to allow agent to do anything from specific set\n* Based on concept of joint state space, each state composite of physical and machine state\n* Choice state is one where program counter for m is at choice point in agent program\n* Can be effective for learning complex behaviors\n* Provides additive decomposition of overall utility function as well\n\n\n## Module 9 Lecture Notes\n\nDouble bandits\n* slot machines, can create MDP when you pull the lever and either win or lose from each state, no discount, both states have same value\n* solving MDPs if offline, all quantities calculated out and you know the details of the MDP, dont need to actually play\n* Online planning suddently different, have to figure out what is going on with the states\n* learning instead of planning, still and MDP but you have to figure it out instead of being able to do so ahead of time\n* Exploration - tried unknown actions to get information\n* Exploitation - eventually use whatever you learn/know already\n* Regret - even if you learn well, still make mistakes\n* Sampling - have to try things many times because of chance\n* Difficulty - learning can be harder than solving MDP computationally\n\nReinforcement learning\n* have an agent that provides action to environment, that returns state s and reward r\n* get feedback in form of rewards, utility of the agent defined by these reward functions, so must maximize the expected rewards\n* all fo the learning is done based on samples of the outcomes\n* still have set of states s in S, actions a in A, model T(s,a,s'), reward function R(s,a,s'), policy pi(s)\n* now we don't know T or R so must try actions to learn what to do and which states to prefer\n* offline is MDPs, online is reinforcement learning\n\nModel-Based learning\n* learn approximate model based on observations, solve for the valuess as if what you learned is correct\n* 1) learn MDP model from experience by counting outcomes s' for each state and action pair, normalize to get estimate of T(s,a,s'), discover each R(s,a,s') when (s,a,s') happens\n* 2) solve the learned MDP, can use value iteration as before\n\nModel Free learning:\nPassive Reinforcement learning\n* now do policy evaluation, given a fixed policy pi(s) and unknown T and R functions, want to learn the state values\n* learner has no choice what actions to take, just execute the policy and learn from what happens\n* actions actually taken, no offline planning involved\n\nDirection Evaluation\n* want to compute values for every state given the current policy\n* do what the policy says, write down the sum of discounted rewards each time you visit a state, and average them all together to get a decent estimate\n* easy to understand, don't need to know T or R, eventually gets the correct averages using transitions\n* but can waste information and state connection s, takes a long time since each state is learned separately\n* in policy evaluation, updates calculaed V for the fixed policy by replacing it with the lookahead step\n* we need T and R to do it though, so not possible here, how can we update the weighted average without knowing the weights\n\nSample Based Policy Evaluation\n* want to calculate the bellman equation of V_k+1 pi_i(s) = $\\sum_s' T(s,pi_i(s),s')[R(s,pi_i(s),s') + yVk pi_i(s')]$, take sames of outcomes s', take action and average\n* sample_1 = R(s, pi(s), s1') + yV_k pi(s1')\n* sample_2 = R(s, pi(s), s2') + yV_k pi(s2')  \n* V_k+1 pi_i(s) = 1/n $\\sum_i sample_i$\n\nTemporal difference learning\n* want to learn from every experience, update V(s) every time we have a transition, so likely s' will contribute more often\n* still a fixed policy with the evaluation, keep a running average fo the successor values\n* samples of V(s): sample = R(s,pi_i(s),s') + yVpi(s')]\n* updates to V(s): Vpi(s) = (1 - a)Vpi(s) + a(sample)\n* same update: Vpi(s) = Vpi(s) + a(sample - Vpi(s))\n* keep an exponential moving average with running interpoliation uupdate: x_n = (1 - a) * x_n-1 + a * x_n\n* makes the recent samples more important and forgets about the distance past values that were less accurate anyway\n* x_n = x_n + (1 - a) * x_n-1 + (1 - a)^2 * x_n-2 + ... / (1 + (1 - a) + (1 - a)^2 + ...)\n* if we decrease the learning rate, can get converging averages\n* mimicks Bellman updates with these running averages, but if we want to turn values into new policy, can't do it because we need Q values for policy\n* so learn Q-values instead of values, makes the actino selection model free as well\n\nActive reinforcement learning\n* full reinforcement learning has optimal policies where you don't know transitions or rewards but can choose actions, learn optimal policy/values\n* tradeoff exploration and explotation\n* Q learning, find the successive depth limited valeus with value iteration: Vk+1(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yVk(s')]$\n* Q-values more useful so we use them instead: Qk+1(s,a) = $\\sum_s' T(s,a,s')[R(s,a,s') + yMax_a'Qk(s',a')]$\n* start with both terms as 0 which we know is right\n* receive a sample (s,a,s',r), consider old esimate Q(s,a)\n* look at new sample estimate: sample = R(s,a,s') + ymax_a' Q(s',a'), incorporate into running average Q(s,a) = (1 - a)Q(s,a) + a[sample]\n* acting suboptimally, but q learning converges to optimal policy, called off-policy learning\n* to do so, need to explore enough, make learning rate small enough but not too quickly, doesn't matter how you select actions\n\nExploration\n* can pick random actions, epsilon greedy, flip a coin and act randomkly with some probability epsilon, otherwise act on current policy\n* if we do this, can thrash when learning is done, could also lower epsilon over time or look at exploration functions\n* explore areas where we are not sure how bad they are, can eventually stop exploring\n* take a value estimate u and visit count n, return optimistic utility like f(u,n) = u + k/n\n* instead of regular q update Q(s,a) = a R(s,a,s') + y max_a' Q(s',a'), do modified q update: Q(s,a) = a R(s,a,s') + y max_a' f(Q(s',a'), N(s',a'))\n* propagates bonus back to states that lead to unknown ones\n\nRegret - measure of total mistake cost, difference between expected rewards (with suboptimal moves) and optimal expected rewards\n* requres optimal learning to be optimal in minimizing regret, not possible often\n\nApproximate Q-learning\n* basic q-learning keeps a table of all q-values, can't do this often and learn about each state, too many to visit in training/hold in memory\n* generalize instead from small number of training states from experience to new similar situations\n* from evaluation functions, score non-terminals in depth limited search and wanted to return actual minimax function but had to do linear sum of features eval(s) = w1f1(s) + ... wnfn(s)\n* here we can write a q function for any state with a few weights:\n* V(s) = w1f1(s) + ... wnfn(s)\n* Q(s,a) = w1f1(s,a) + ... wnfn(s,a)\n* good since our experience is summer up in a few numbers, but states may share features but be very different in value\n* have our transition function (s,a,s',r), difference = [r + y max_a' Q(s',a')] - Q(s,a)\n* exact Qs are Q(s,a) = Q(s,a) + a * difference, approximate Qs are wi = wi + a * difference * fi(s,a)\n* we adjust the weights of the active features, disprefer bad states if something bad happens, can justify with online least squares\n\n\n\n## Module 9 Definitions\n* **Off-policy learning** - Q learning in active reinforcement learning converges to optimal policy\n* **N-armed bandit** - Slot machine with n levers\n* **One armed bandit** - slot machine with only one arm M that produces rewards and has fixed cost for each pull\n* **Restart mdp** - learn about the MDP through Gittins index and go back to the initial state\n* **Selection problem** - choosing the best option of a bandit environment under certain conditions\n* **Bandit superprocess** - each arm is full MDP rather than a Markov deicion prosess with one action\n* **Opportunity cost** - how much utility is given up by not devoting time to another arm\n* **Dominating policy** - An optimal policy if it exists for ann MDP\n* **Reinforcement learning** - agent interacts with world and periodically gets rewards based on how its doing\n* **Sparse rewards** - agent given no informative reward signal for most of the game\n* **Passive reinforcement learning** - fixed policy and need to learn utility of states\n* **Active reinforcement learning** - agent must also figure out what to do and learn utilities\n* **Exploration** - agent must learn as much about its environment as possible\n* **Trial** - executing the policy in passive reinforcement learning\n* **Pseudoexperience** - simulating the current transition model in TD and ADP\n* **Greedy agent** - greedily takes the action it believes to be optimal at each step in ARL\n* **Evaluation function** - compact measure of desirability for vast state spaces\n* **Function approximation** - process of constructing compact approximation of true utility or Q function\n* **Catastrophic forgetting** - assign zero weights to features when model learns too well\n* **Experience replay** - replay experience perioidcally to avoid catastophic forgetting\n* **Credit assignment** - figuring out where agent went wrong in long string of actions/states\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\"\n",
        "* Agents need to handle uncertainty due to partial observability, nondeterminism, or adversaries\n",
        "* Belief state has drawbacks, agent must consider every possible explanation for sensor observations\n",
        "* Also must handle every eventuality can grow arbitrarily large and have unlikely contingencies, and sometimes there is no plan guaranteed to achieve the goal, hut still has to act\n",
        "* Logical qualification problem, may have a plan but still never arrive due to many reasons, can still be the right thing to do\n",
        "* Rational decision depends on relative importances of various goals and likelihood that they will be achieved\n",
        "* Trying to use logic to cope with domain like medical diagnosis fails for several reasons, laziness (too much work to list everything), theoretical ignorance, and practical ignorance \n",
        "* Agent’s knowledge can provide only degree of belief in relevant sentences\n",
        "* Main tool for dealing with these is probability theory, theory of probability provides way of summarizing uncertainty that comes from laziness and ignorance\n",
        "* Agent must have preferences among different outcomes of various plans, use utility theory to represent preferences and reason quantitatively\n",
        "* Utility of the state is relative to the agent\n",
        "* Decision theory = probability theory + utility theory\n",
        "* Agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all possible outcomes, principle of maximum expected utility\n",
        "* Represent possibilities and probabilities\n",
        "\"\"\"\n",
        "chapter12_2 = \"\"\"\n",
        "* Possible set of all worlds in probability theory is the sample space, worlds are mutually exclusive and exhaustive \n",
        "* Probability model associates numerical probability with each possible world\n",
        "* Events are sets of actions, sets of worlds correspond to proposition in formal language\n",
        "* Unconditional or prior probabilities refer to degree of belief in absence of other information\n",
        "* Usually have some information called evidence, in which case look at conditional or posterior probabilities \n",
        "* Product rule: P(a and b) = P(a|b)P(b)\n",
        "* Variables in probability theory are random variables\n",
        "* P can be probability distribution for each random variable, or assignment of probability for each possible value\n",
        "* Categorial distribution if there is a finite discrete range\n",
        "* Joint probability distributions for two or more variables probability of occurring at the same time\n",
        "* Possible world defined to be an assignment of values to all random variables under consideration\n",
        "* Could have full joint probability distribution if joint probabilities are known for every variable\n",
        "* Inclusion-exclusion principle: P(a or b) = P(a) + P(b) - P(a and b)\n",
        "* If agent 1 expresses set of degrees of belief that violate axioms of probability theory then combination of bets by agent 2 guarantees agent 1 loses money every time\n",
        "\"\"\"\n",
        "chapter12_3 = \"\"\"\n",
        "* Probabilistic inference: computation of posterior probabilities given query propositions with observed evidence\n",
        "* Full joint distribution used as the knowledge base to answer questions\n",
        "* Can isolate one probability by adding all entries in a given row to get marginal probability\n",
        "* Marginalization of summing out is summing all possible values of other variables to take them out of the equation\n",
        "* Most interested in computing conditional probabilities of some variables given evidence about others\n",
        "\"\"\"\n",
        "chapter12_4 = \"\"\"\n",
        "* Independence between propositions a and b can be written as \n",
        "* P(a|b) = P(a) or P(b|a) = P(b) or P(a and b) = P(a)P(b)\n",
        "* Independence assertions usually based on knowledge of the domain\n",
        "* If complete set of variables can be divide into independent subsets, then full joint distributions can be factored into separate joint distributions based on them\n",
        "* Can reduce size of domain representation and complexity of inference problem, but rare to actually find\n",
        "\"\"\"\n",
        "chapter12_5 = \"\"\"\n",
        "* Bayes rule underlies most modern AI systems for probabilistic inference\n",
        "* Can compute single term P(b|a) with all other terms P(a|b)P(b)P(a)\n",
        "* Often have good probabilistic estimates for some values but not all\n",
        "* Diagnostic knowledge in bayes is often more fragile than causal knowledge\n",
        "* If dependent on multiple variables, need independence to refine our statements a bit\n",
        "* May not be independent, but can be independent given the presence or absence of another variable\n",
        "* Creates conditional independence if P(a and b|c) = P(a|c)P(b|c)\n",
        "* This leads us to P(X|Y,Z) = P(X|Z) and P(X|Y,Z) = P(Y|Z)\n",
        "* Conditional independence assertions can let probabilistic models to scale up and much more commonly available than absolute independence assertions\n",
        "* The middle variable then separates the other two since they are a direct cause of both of them\n",
        "\"\"\"\n",
        "chapter12_6 = \"\"\"\n",
        "* Bayesian classifiers can work well even when conditions independence is not strictly true\n",
        "* Can obtain probability of the cause given some observed events\n",
        "* For each cause, multiply the prior probability of the cause by the product of the conditional probabilities of the overspeed effects given the cause, then normalize\n",
        "* Linear in number of observed events, doesn’t depend on number of unobserved effects\n",
        "* Can be used for text classification, given text decides which predefined set of classes it belongs to\n",
        "* Classify them each into a category\n",
        "* Model has prior probabilities and conditional probability, estimated as fraction of all previously seen documents that are of category c\n",
        "* Check which key words appear in the document then obtain posterior probability distribution over the categories\n",
        "* If predicting one class, take the highest posterior probability\n",
        "* Assumes words occur independently in documents and determine frequencies by document category\n",
        "* Model is overconfident in predictions, but rankings can still be very accurate\n",
        "\"\"\"\n",
        "chapter12_7 = \"\"\"\n",
        "* Wumpus world again\n",
        "* Agents sensors only give partial information about the world\n",
        "* Pure logical inferences can’t conclude which squares are the most safe, so may have to randomly choose\n",
        "* Use the probabilistic agent instead\n",
        "* Calculate the probability that each square is a pit\n",
        "* As before, sum over entries from the full joint distribution\n",
        "* To get efficient solutions, independence and conditional independence relationships can be used to simplify the summations required\n",
        "* Axioms of probability constrain the probabilities of logically related propositions, if an agent violates them they may behave irrationally\n",
        "* Bayes rules allows unknown probabilities to be computed from known conditional probabilities, often in causal direction\n",
        "* Naive babes models assumes conditional independence of all effect variables given a single cause variable, and grows linearly in number of effects\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\"\n",
        "* N-gram language model\n",
        "* Assign probability to next possible word, language models assign probabilities to sequences of words\n",
        "* N-gram is sqeucen of n words that assigns probability to sentences and sequences of words, refers to either the word sequence itself or the predictive model that assigns it a probability\n",
        "* Probability of word given some history, could do so with relative frequency counts \n",
        "* With a large enough corpus, could compute counts and estimate the probability, but most of the time isn’t big enough to give good estimations\n",
        "* Can decompose the probability of entire sequences using the chain rule of probability and make them dependent on all the words that came before the current word of interest\n",
        "* Illustrates the link between computing joint probability of sequence and computing conditional probability of word given previous words\n",
        "* Instead of computing probability of word given entire history, can approximate history using just the last few words\n",
        "* Bigram looks only at the previous word, making a Markov assumption that the word only depends on the previous word probability\n",
        "* Can generalize this to tri-gram and n-gram as well\n",
        "* Estimate probabilities using maximum likelihood estimate or MLE, get it by getting counts from corpus and normalizing the counts so they’re between 0 and 1\n",
        "* Gets the relative frequency by diving the observed frequency of a particular sequence by the observed frequency of a prefix\n",
        "* Resulting parameter maximizes the likelihood of the training set given the model\n",
        "* Some bigram statistics encode facts that are strictly syntactic is nature, like parts of speech combinations\n",
        "* More common to use trigram models or 4/5-gram models when sufficient training data exists \n",
        "* Always compute language model probabilities in log format as log probabilities \n",
        "\"\"\"\n",
        "SLP_chapter3_2 = \"\"\"\n",
        "* Evaluate performance of model by embedding it in application and measuring how the application improves, called extrinsic evaluation\n",
        "* Only way to know if improvement in component is going to help the task at hand\n",
        "* Intrinsic evaluation measure the quality of a model independent of any application, requires a test set\n",
        "* Probabilities of the n-gram model come from the training set, test performance on test set and held out set, want to assign higher probabilities to the test set to get better model\n",
        "* Training on test set introduces bias that makes probabilities too high and has inaccuracies in perplexity\n",
        "* Divide data into training, development, and test set\n",
        "* Use perplexity as our metric instead of raw probability, inverse probability of the test set normalize by the number of words\n",
        "* The higher the conditional probability of the word sequence, the lower the perplexity, so minimizing it is equivalent to maximizing test set probability\n",
        "* Can also look at weighed average branching factor of a language, or number of possible next words that can follow any word\n",
        "* In computing perplexities, n-gram model must be constructed without any knowledge of the test set of vocabulary of the test set, or will be artificially low perplexity\n",
        "* Perplexities of two language models are only comparable if they use identical vocabularies\n",
        "* Intrinsic improvement in perplexity doesn’t guarantee extrinsic improvement in performance of language processing task like speech recognition\n",
        "* Should always be confide by an end to end evaluation of a real task\n",
        "\"\"\"\n",
        "SLP_chapter3_3 = \"\"\"\n",
        "* Sampling from distribution chooses random points according to their likelihood\n",
        "* Sampling from language model generates sentences, choosing each sentence according to likelihood as defined by model, so likely to generate sentences the model thinks has high probability over ones with low probability \n",
        "* Bigrams done by generating random bigrams that start with s, them choose random bigrams stating with next word and so on until sentence end token is hit\n",
        "\"\"\"\n",
        "SLP_chapter3_4 = \"\"\"\n",
        "* Dependent on training corpus, so probabilities often encode specific facts about training corpus, also tend to do better as N increases\n",
        "* Can be very similar to trained corpus depending on size and how accurate they become, and size of N\n",
        "* Statistical models likely to be useless as predictors if training and test sets are extremely different\n",
        "* One solution is to have the training corpus be a similar genre of text to the test set, give it appropriate dialect and variety as well when processing different genres like social media and spoken transcripts versus official documents\n",
        "* Matching genres and dialects not sufficient, may be too sparse and missing perfectly good English word sequences from the training set\n",
        "* Zeros that are never seen in the training set but in test set mean we are underestimating probability of all sorts of words that may occur, and if probability of any word inn the test set is 0 the entire probability of the test set is 0\n",
        "* Closed vocabulary systems are language tasks where we know this won’t happen since we know all the words that can occur, can be reasonable in some domains but not in others\n",
        "* Other contexts we have unknown of out of vocabulary words we have never seen before, OOV rate is percentage that occur in test set\n",
        "* Open vocabulary system is one where we model potential unknown words in test set by adding a pseudo word\n",
        "* Two common ways to train probabilities of unknown word model, one is to turn problem back into closed vocabulary one by choosing fixed vocabulary in advance\n",
        "* In this way, we choose a vocabulary that is fixed in advance, convert the training set any word that is not in the set to unknown word token and estimate the probabilities for it like any other word in the set\n",
        "* Second alternative is to create a vocabulary implicitly and replace words with pseudo word based on their frequency\n",
        "\"\"\"\n",
        "SLP_chapter3_5 = \"\"\"\n",
        "* To keep language model from assigning zero probability to unseen events, have to do smoothing or discounting to shave off probability from more seen events and give them to this unseen event\n",
        "* Laplace smoothing, add-k smoothing, stupid backoff, and Kneser-Ney smoothing all methods\n",
        "* In Laplace, add one to all n-gram counts before normalizing them into probabilities, doesn’t perform well enough to be used in model n-gram models but important conceptually\n",
        "* Also called add-one smoothing, just adds one to each count, define an adjusted count to be this added terms and can compare it to the standard MLE count\n",
        "* Discounting smoothing, lowering some non-zero counts in order to get probability mass that is assigned to zero counts. Instead of referring to discounted counts, may have a smoothing algorithm in terms of relative discount\n",
        "* Add-k smoothing, move a bit less of the probability mass to the unseen events, instead of adding one we add a fractional count k. This requires that we have a method for choosing k, like optimizing on dev set\n",
        "* Sometimes using less context is a good thing to generalize more for contexts we have not learned much about\n",
        "* In backoff, we use trigram if evidence is sufficient, otherwise the bigram, otherwise unigram, only back off to lower order n-gram if we have zero evidence for a higher order n gram\n",
        "* Interpolation, mix the probability estimates from all n-gram estimators, weigh and combine the trigram, bigram, unigram counts\n",
        "* Both simple interpolation and conditional interpolation lambdas learned from held out corpus\n",
        "* In backoff, continue backing off until we reach a history that has some counts, have to discount the higher order n-grams to save some probability for lower order ones\n",
        "* Katz backoff we rely on discounted probability if we have seen n-gram before, otherwise recurveiyl back off to probability for shoter history n gram, good with good-tuning smoothing method \n",
        "\"\"\"\n",
        "SLP_chapter3_7 = \"\"\"\n",
        "* Can build very large language models from web\n",
        "* Efficiency considerations important when building language models that use large sets of n-grams\n",
        "* Can be shrunk by pruning or stored as a hash number instead of a string\n",
        "* Could also build approximate language models with bloom filter techniques, or use sorted arrays to build probability tables in minimal number of passes through large corpus \n",
        "* Very large language models can have much simpler algorithms that are sufficient, called stupid backoff\n",
        "* Gives up trim to make language model true probability distribution, no discounting of higher-oder probabilities, just backoff to lower order n-gram weighed by fixed weight if there is a zero count\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\"\n",
        "* observed variables (evidence): agent knows things about the state of the world through sensors \n",
        "* unobserved variables (states): agent needs to figure out other aspects, what/where something is \n",
        "* model: agent knows some information about the how known variables ralte to unknown variables\n",
        "* probabilistic reasoning creates framework for dealing with all of these\n",
        "* probabilities about possible worlds, quantify how likely a possible world is\n",
        "* all possible worlds are the sample space, mutually exclusive and exhaustive in probability theory\n",
        "* a correct model associates a probability P(w) with each world w\n",
        "\n",
        "Random Variables - some aspect of the world where we may have uncertainty, denoted as capital letters\n",
        "* probability of any possible world has to be between 0 and 1, and total probability of all possible worlds has to equal 1\n",
        "\n",
        "Probability distributions\n",
        "* unobserved random variables have distributions, or table of probabilities\n",
        "* all probabilities must be greater than or equal to 0, sum of all probabilities of variable equal 1\n",
        "\n",
        "Joint distributions\n",
        "* specifies a real number for each assignment or outcome, over a set of random variables\n",
        "* probability of two random variables having some value, same principles old\n",
        "* the size of the distribution of n variables with domain size d is impractical to calculate except for small distributions\n",
        "\n",
        "Probabilistic models\n",
        "* joint distribution over set of random variables\n",
        "* include random variables with domains, assignments called outcomes, joint distributions that assign probability, normalized (sum to 1), hopefully only certain variables directly interact\n",
        "* event is a set E of outcomes, sum across however many questions we have or that fit the description\n",
        "* e.g. prob that its hot, prob that its hot and sunny, or sunny, etc.\n",
        "* events we care about usually are partial assignments\n",
        "\n",
        "Marginal Distributions\n",
        "* sub-tables that eliminate variables, can combine collapsed rows by adding probabilities together\n",
        "\n",
        "Conditional Probabilities:\n",
        "* relation between joint and conditional probabilities, taken as the definition\n",
        "* p(a|b) = P(a,b) / p(b)\n",
        "* use relationships to calculate probability of an event given another event\n",
        "* conditional distributions are probability distributions over variables given some other variable values\n",
        "\n",
        "Probabilistic Inference:\n",
        "* compute desired probability from other known probabilities\n",
        "* usually interested in conditional probabilities, correspond to agent's beliefs given the evidence\n",
        "* these probabilites change with new evidence, beliefs of the agent update\n",
        "\n",
        "Inference by Enumeration\n",
        "* generally have evidence variables E with some value e, query variable Q, hidden variable H, want P(Q| e1...ek)\n",
        "* 1) select entries consistent with evidence, 2) sum out H to get joint of Query and evidence, 3) Normalize\n",
        "* P(Q, e1, .. ek) = sum (h1..hr) P(Q, h1 .. hr, e1 .. ek)\n",
        "* normalize by taking Z = sum_z P(Q, e1...ek), multiply 1/Z by P\n",
        "\n",
        "Product rule: P(y)P(x|y) = P(x,y)\n",
        "\n",
        "Chain rule\n",
        "* can write joint distributions as incremental product of conditional distributions\n",
        "* P(x1, x2, x3) = P(x1)P(x2|x1)P(x3|x1,x2)\n",
        "* P(x1, x2, ... xn) = mult_i (P(x_i|x1...x_i-1))\n",
        "\n",
        "Bayes Rule\n",
        "* P(x,y) = P(x|y)P(y) = P(y|x)(x)\n",
        "* we can divide this out to get P(x|y) = (P(y|x) / P(y)) * P(x)\n",
        "* can build a conditional from reverse, one often more difficult to find than the other\n",
        "\n",
        "Probabilistic Language Models\n",
        "* want to assign a probability to a sentence\n",
        "* autocomplete, machine translation, spell correction, speech recognition, summarization, Q&A, dialog systems\n",
        "* complete the probability of a sentence or sequece of words (P(W) = P(w1,w2,..wn), and probability of upcoming word P(w3|w1,w2)\n",
        "* model that does either of these is called a language model P(W) or P(wn | w1,..wn-1)\n",
        "* P(W) calculated with chain rule of probability\n",
        "* P(B|A) = P(A,B)/ P(A), can rewrite as P(A,B) = P(A)P(B|A)\n",
        "* P(A,B,C,D) = P(A)P(B|A)P(C|A,B)P(D|A,B,C)\n",
        "* joint probability of words in a sentence: P(w1,w2,..wn) = mult_i P(wi|w1w2..wi-1)\n",
        "* P(the eagles won) = P(the) * P(Eagles|the) * P(won|the Eagles)\n",
        "* Estimate probabilities using maximum likelihood estimate (MLE), doesnt work\n",
        "* only depends on the previous k words, not entire context, K is number of context words we take into account\n",
        "* unigram (no history), bigram (1 word history), trigram (2 words), 4-gram (3 words)\n",
        "\n",
        "Unigram model:\n",
        "* P(w1w2...wn) = mult_i P(wi)\n",
        "* only look at the word itself, no history\n",
        "Bigram model:\n",
        "* condition on previous word: P(wi|w1w2...wi-1) = P(wi|wi-1)\n",
        "\n",
        "N-gram models\n",
        "* can extend this to trigram, 4,5-grams\n",
        "* geerally insufficient model of language because of long distance dependenceis, but often can get away with using N-gram models\n",
        "\n",
        "Language Modeling\n",
        "* Estimating bigram probabilities\n",
        "* maximum likelihood estimate: P(wi|wi-1) = count(wi-1,wi) / count(wi-1)\n",
        "* MLE has problems when zeros exist, assign 0 probability to all sentences containing it if it isnt close enough\n",
        "* out of vocabulary items (use term <unk> to normalize training data by replace word not in L with it), fixed lexicon L of size V, try to avoid zeros with smoothing\n",
        "* practical issues: do everything in log space to avoid underflow: log(p1 * p2 * p3 * p4) = log(p1) + log(p2) + log(p3) + log(p4)\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"sample space\" : \"the possible worlds, mutually exclusive and exhaustive\",\n",
        "\"Marginalization\": \"summing out, combining rows in distributions by adding probabilities\",\n",
        "\"belief state\": \"representation of the set of all possible world states agent might be in\",\n",
        "\"factored representation\": \"possible world is represented as set of variable/value pairs\"\n",
        "\n",
        "}"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "9cfc4c4c-4961-4581-8178-45ab7c35aa69"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\n* Agents need to handle uncertainty due to partial observability, nondeterminism, or adversaries\n* Belief state has drawbacks, agent must consider every possible explanation for sensor observations\n* Also must handle every eventuality can grow arbitrarily large and have unlikely contingencies, and sometimes there is no plan guaranteed to achieve the goal, hut still has to act\n* Logical qualification problem, may have a plan but still never arrive due to many reasons, can still be the right thing to do\n* Rational decision depends on relative importances of various goals and likelihood that they will be achieved\n* Trying to use logic to cope with domain like medical diagnosis fails for several reasons, laziness (too much work to list everything), theoretical ignorance, and practical ignorance \n* Agent’s knowledge can provide only degree of belief in relevant sentences\n* Main tool for dealing with these is probability theory, theory of probability provides way of summarizing uncertainty that comes from laziness and ignorance\n* Agent must have preferences among different outcomes of various plans, use utility theory to represent preferences and reason quantitatively\n* Utility of the state is relative to the agent\n* Decision theory = probability theory + utility theory\n* Agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all possible outcomes, principle of maximum expected utility\n* Represent possibilities and probabilities\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n\n* Possible set of all worlds in probability theory is the sample space, worlds are mutually exclusive and exhaustive \n* Probability model associates numerical probability with each possible world\n* Events are sets of actions, sets of worlds correspond to proposition in formal language\n* Unconditional or prior probabilities refer to degree of belief in absence of other information\n* Usually have some information called evidence, in which case look at conditional or posterior probabilities \n* Product rule: P(a and b) = P(a|b)P(b)\n* Variables in probability theory are random variables\n* P can be probability distribution for each random variable, or assignment of probability for each possible value\n* Categorial distribution if there is a finite discrete range\n* Joint probability distributions for two or more variables probability of occurring at the same time\n* Possible world defined to be an assignment of values to all random variables under consideration\n* Could have full joint probability distribution if joint probabilities are known for every variable\n* Inclusion-exclusion principle: P(a or b) = P(a) + P(b) - P(a and b)\n* If agent 1 expresses set of degrees of belief that violate axioms of probability theory then combination of bets by agent 2 guarantees agent 1 loses money every time\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n\n* Probabilistic inference: computation of posterior probabilities given query propositions with observed evidence\n* Full joint distribution used as the knowledge base to answer questions\n* Can isolate one probability by adding all entries in a given row to get marginal probability\n* Marginalization of summing out is summing all possible values of other variables to take them out of the equation\n* Most interested in computing conditional probabilities of some variables given evidence about others\n\n\n## AIMA Chapter 12.4 Independence\n\n* Independence between propositions a and b can be written as \n* P(a|b) = P(a) or P(b|a) = P(b) or P(a and b) = P(a)P(b)\n* Independence assertions usually based on knowledge of the domain\n* If complete set of variables can be divide into independent subsets, then full joint distributions can be factored into separate joint distributions based on them\n* Can reduce size of domain representation and complexity of inference problem, but rare to actually find\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\n* Bayes rule underlies most modern AI systems for probabilistic inference\n* Can compute single term P(b|a) with all other terms P(a|b)P(b)P(a)\n* Often have good probabilistic estimates for some values but not all\n* Diagnostic knowledge in bayes is often more fragile than causal knowledge\n* If dependent on multiple variables, need independence to refine our statements a bit\n* May not be independent, but can be independent given the presence or absence of another variable\n* Creates conditional independence if P(a and b|c) = P(a|c)P(b|c)\n* This leads us to P(X|Y,Z) = P(X|Z) and P(X|Y,Z) = P(Y|Z)\n* Conditional independence assertions can let probabilistic models to scale up and much more commonly available than absolute independence assertions\n* The middle variable then separates the other two since they are a direct cause of both of them\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\n* Bayesian classifiers can work well even when conditions independence is not strictly true\n* Can obtain probability of the cause given some observed events\n* For each cause, multiply the prior probability of the cause by the product of the conditional probabilities of the overspeed effects given the cause, then normalize\n* Linear in number of observed events, doesn’t depend on number of unobserved effects\n* Can be used for text classification, given text decides which predefined set of classes it belongs to\n* Classify them each into a category\n* Model has prior probabilities and conditional probability, estimated as fraction of all previously seen documents that are of category c\n* Check which key words appear in the document then obtain posterior probability distribution over the categories\n* If predicting one class, take the highest posterior probability\n* Assumes words occur independently in documents and determine frequencies by document category\n* Model is overconfident in predictions, but rankings can still be very accurate\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n\n* Wumpus world again\n* Agents sensors only give partial information about the world\n* Pure logical inferences can’t conclude which squares are the most safe, so may have to randomly choose\n* Use the probabilistic agent instead\n* Calculate the probability that each square is a pit\n* As before, sum over entries from the full joint distribution\n* To get efficient solutions, independence and conditional independence relationships can be used to simplify the summations required\n* Axioms of probability constrain the probabilities of logically related propositions, if an agent violates them they may behave irrationally\n* Bayes rules allows unknown probabilities to be computed from known conditional probabilities, often in causal direction\n* Naive babes models assumes conditional independence of all effect variables given a single cause variable, and grows linearly in number of effects\n\n\n## SLP Chapter 3.1 N-Grams\n\n* N-gram language model\n* Assign probability to next possible word, language models assign probabilities to sequences of words\n* N-gram is sqeucen of n words that assigns probability to sentences and sequences of words, refers to either the word sequence itself or the predictive model that assigns it a probability\n* Probability of word given some history, could do so with relative frequency counts \n* With a large enough corpus, could compute counts and estimate the probability, but most of the time isn’t big enough to give good estimations\n* Can decompose the probability of entire sequences using the chain rule of probability and make them dependent on all the words that came before the current word of interest\n* Illustrates the link between computing joint probability of sequence and computing conditional probability of word given previous words\n* Instead of computing probability of word given entire history, can approximate history using just the last few words\n* Bigram looks only at the previous word, making a Markov assumption that the word only depends on the previous word probability\n* Can generalize this to tri-gram and n-gram as well\n* Estimate probabilities using maximum likelihood estimate or MLE, get it by getting counts from corpus and normalizing the counts so they’re between 0 and 1\n* Gets the relative frequency by diving the observed frequency of a particular sequence by the observed frequency of a prefix\n* Resulting parameter maximizes the likelihood of the training set given the model\n* Some bigram statistics encode facts that are strictly syntactic is nature, like parts of speech combinations\n* More common to use trigram models or 4/5-gram models when sufficient training data exists \n* Always compute language model probabilities in log format as log probabilities \n\n\n## SLP Chapter 3.2 Evaluating Language Models\n\n* Evaluate performance of model by embedding it in application and measuring how the application improves, called extrinsic evaluation\n* Only way to know if improvement in component is going to help the task at hand\n* Intrinsic evaluation measure the quality of a model independent of any application, requires a test set\n* Probabilities of the n-gram model come from the training set, test performance on test set and held out set, want to assign higher probabilities to the test set to get better model\n* Training on test set introduces bias that makes probabilities too high and has inaccuracies in perplexity\n* Divide data into training, development, and test set\n* Use perplexity as our metric instead of raw probability, inverse probability of the test set normalize by the number of words\n* The higher the conditional probability of the word sequence, the lower the perplexity, so minimizing it is equivalent to maximizing test set probability\n* Can also look at weighed average branching factor of a language, or number of possible next words that can follow any word\n* In computing perplexities, n-gram model must be constructed without any knowledge of the test set of vocabulary of the test set, or will be artificially low perplexity\n* Perplexities of two language models are only comparable if they use identical vocabularies\n* Intrinsic improvement in perplexity doesn’t guarantee extrinsic improvement in performance of language processing task like speech recognition\n* Should always be confide by an end to end evaluation of a real task\n\n\n## SLP Chapter 3.3 Sampling sentences from a language model\n\n* Sampling from distribution chooses random points according to their likelihood\n* Sampling from language model generates sentences, choosing each sentence according to likelihood as defined by model, so likely to generate sentences the model thinks has high probability over ones with low probability \n* Bigrams done by generating random bigrams that start with s, them choose random bigrams stating with next word and so on until sentence end token is hit\n\n\n## SLP Chapter 3.4 Generalization and Zeros\n\n* Dependent on training corpus, so probabilities often encode specific facts about training corpus, also tend to do better as N increases\n* Can be very similar to trained corpus depending on size and how accurate they become, and size of N\n* Statistical models likely to be useless as predictors if training and test sets are extremely different\n* One solution is to have the training corpus be a similar genre of text to the test set, give it appropriate dialect and variety as well when processing different genres like social media and spoken transcripts versus official documents\n* Matching genres and dialects not sufficient, may be too sparse and missing perfectly good English word sequences from the training set\n* Zeros that are never seen in the training set but in test set mean we are underestimating probability of all sorts of words that may occur, and if probability of any word inn the test set is 0 the entire probability of the test set is 0\n* Closed vocabulary systems are language tasks where we know this won’t happen since we know all the words that can occur, can be reasonable in some domains but not in others\n* Other contexts we have unknown of out of vocabulary words we have never seen before, OOV rate is percentage that occur in test set\n* Open vocabulary system is one where we model potential unknown words in test set by adding a pseudo word\n* Two common ways to train probabilities of unknown word model, one is to turn problem back into closed vocabulary one by choosing fixed vocabulary in advance\n* In this way, we choose a vocabulary that is fixed in advance, convert the training set any word that is not in the set to unknown word token and estimate the probabilities for it like any other word in the set\n* Second alternative is to create a vocabulary implicitly and replace words with pseudo word based on their frequency\n\n\n## SLP Chapter 3.5 Smoothing\n\n* To keep language model from assigning zero probability to unseen events, have to do smoothing or discounting to shave off probability from more seen events and give them to this unseen event\n* Laplace smoothing, add-k smoothing, stupid backoff, and Kneser-Ney smoothing all methods\n* In Laplace, add one to all n-gram counts before normalizing them into probabilities, doesn’t perform well enough to be used in model n-gram models but important conceptually\n* Also called add-one smoothing, just adds one to each count, define an adjusted count to be this added terms and can compare it to the standard MLE count\n* Discounting smoothing, lowering some non-zero counts in order to get probability mass that is assigned to zero counts. Instead of referring to discounted counts, may have a smoothing algorithm in terms of relative discount\n* Add-k smoothing, move a bit less of the probability mass to the unseen events, instead of adding one we add a fractional count k. This requires that we have a method for choosing k, like optimizing on dev set\n* Sometimes using less context is a good thing to generalize more for contexts we have not learned much about\n* In backoff, we use trigram if evidence is sufficient, otherwise the bigram, otherwise unigram, only back off to lower order n-gram if we have zero evidence for a higher order n gram\n* Interpolation, mix the probability estimates from all n-gram estimators, weigh and combine the trigram, bigram, unigram counts\n* Both simple interpolation and conditional interpolation lambdas learned from held out corpus\n* In backoff, continue backing off until we reach a history that has some counts, have to discount the higher order n-grams to save some probability for lower order ones\n* Katz backoff we rely on discounted probability if we have seen n-gram before, otherwise recurveiyl back off to probability for shoter history n gram, good with good-tuning smoothing method \n\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n\n* Can build very large language models from web\n* Efficiency considerations important when building language models that use large sets of n-grams\n* Can be shrunk by pruning or stored as a hash number instead of a string\n* Could also build approximate language models with bloom filter techniques, or use sorted arrays to build probability tables in minimal number of passes through large corpus \n* Very large language models can have much simpler algorithms that are sufficient, called stupid backoff\n* Gives up trim to make language model true probability distribution, no discounting of higher-oder probabilities, just backoff to lower order n-gram weighed by fixed weight if there is a zero count\n\n\n## Module 10 Lecture Notes\n\n* observed variables (evidence): agent knows things about the state of the world through sensors \n* unobserved variables (states): agent needs to figure out other aspects, what/where something is \n* model: agent knows some information about the how known variables ralte to unknown variables\n* probabilistic reasoning creates framework for dealing with all of these\n* probabilities about possible worlds, quantify how likely a possible world is\n* all possible worlds are the sample space, mutually exclusive and exhaustive in probability theory\n* a correct model associates a probability P(w) with each world w\n\nRandom Variables - some aspect of the world where we may have uncertainty, denoted as capital letters\n* probability of any possible world has to be between 0 and 1, and total probability of all possible worlds has to equal 1\n\nProbability distributions\n* unobserved random variables have distributions, or table of probabilities\n* all probabilities must be greater than or equal to 0, sum of all probabilities of variable equal 1\n\nJoint distributions\n* specifies a real number for each assignment or outcome, over a set of random variables\n* probability of two random variables having some value, same principles old\n* the size of the distribution of n variables with domain size d is impractical to calculate except for small distributions\n\nProbabilistic models\n* joint distribution over set of random variables\n* include random variables with domains, assignments called outcomes, joint distributions that assign probability, normalized (sum to 1), hopefully only certain variables directly interact\n* event is a set E of outcomes, sum across however many questions we have or that fit the description\n* e.g. prob that its hot, prob that its hot and sunny, or sunny, etc.\n* events we care about usually are partial assignments\n\nMarginal Distributions\n* sub-tables that eliminate variables, can combine collapsed rows by adding probabilities together\n\nConditional Probabilities:\n* relation between joint and conditional probabilities, taken as the definition\n* p(a|b) = P(a,b) / p(b)\n* use relationships to calculate probability of an event given another event\n* conditional distributions are probability distributions over variables given some other variable values\n\nProbabilistic Inference:\n* compute desired probability from other known probabilities\n* usually interested in conditional probabilities, correspond to agent's beliefs given the evidence\n* these probabilites change with new evidence, beliefs of the agent update\n\nInference by Enumeration\n* generally have evidence variables E with some value e, query variable Q, hidden variable H, want P(Q| e1...ek)\n* 1) select entries consistent with evidence, 2) sum out H to get joint of Query and evidence, 3) Normalize\n* P(Q, e1, .. ek) = sum (h1..hr) P(Q, h1 .. hr, e1 .. ek)\n* normalize by taking Z = sum_z P(Q, e1...ek), multiply 1/Z by P\n\nProduct rule: P(y)P(x|y) = P(x,y)\n\nChain rule\n* can write joint distributions as incremental product of conditional distributions\n* P(x1, x2, x3) = P(x1)P(x2|x1)P(x3|x1,x2)\n* P(x1, x2, ... xn) = mult_i (P(x_i|x1...x_i-1))\n\nBayes Rule\n* P(x,y) = P(x|y)P(y) = P(y|x)(x)\n* we can divide this out to get P(x|y) = (P(y|x) / P(y)) * P(x)\n* can build a conditional from reverse, one often more difficult to find than the other\n\nProbabilistic Language Models\n* want to assign a probability to a sentence\n* autocomplete, machine translation, spell correction, speech recognition, summarization, Q&A, dialog systems\n* complete the probability of a sentence or sequece of words (P(W) = P(w1,w2,..wn), and probability of upcoming word P(w3|w1,w2)\n* model that does either of these is called a language model P(W) or P(wn | w1,..wn-1)\n* P(W) calculated with chain rule of probability\n* P(B|A) = P(A,B)/ P(A), can rewrite as P(A,B) = P(A)P(B|A)\n* P(A,B,C,D) = P(A)P(B|A)P(C|A,B)P(D|A,B,C)\n* joint probability of words in a sentence: P(w1,w2,..wn) = mult_i P(wi|w1w2..wi-1)\n* P(the eagles won) = P(the) * P(Eagles|the) * P(won|the Eagles)\n* Estimate probabilities using maximum likelihood estimate (MLE), doesnt work\n* only depends on the previous k words, not entire context, K is number of context words we take into account\n* unigram (no history), bigram (1 word history), trigram (2 words), 4-gram (3 words)\n\nUnigram model:\n* P(w1w2...wn) = mult_i P(wi)\n* only look at the word itself, no history\nBigram model:\n* condition on previous word: P(wi|w1w2...wi-1) = P(wi|wi-1)\n\nN-gram models\n* can extend this to trigram, 4,5-grams\n* geerally insufficient model of language because of long distance dependenceis, but often can get away with using N-gram models\n\nLanguage Modeling\n* Estimating bigram probabilities\n* maximum likelihood estimate: P(wi|wi-1) = count(wi-1,wi) / count(wi-1)\n* MLE has problems when zeros exist, assign 0 probability to all sentences containing it if it isnt close enough\n* out of vocabulary items (use term <unk> to normalize training data by replace word not in L with it), fixed lexicon L of size V, try to avoid zeros with smoothing\n* practical issues: do everything in log space to avoid underflow: log(p1 * p2 * p3 * p4) = log(p1) + log(p2) + log(p3) + log(p4)\n\n\n## Module 10 Definitions\n* **Sample space** - the possible worlds, mutually exclusive and exhaustive\n* **Marginalization** - summing out, combining rows in distributions by adding probabilities\n* **Belief state** - representation of the set of all possible world states agent might be in\n* **Factored representation** - possible world is represented as set of variable/value pairs\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\"\n",
        "* Bayesian network expressions, can represent dependencies among variables\n",
        "* Can represent any full joint probability distributions often very concisely\n",
        "* Each node corresponds to continuous or discrete random variable, directed links or arrows connect pairs of nodes, if there is an arrow from X to Y say that X it the parent of Y\n",
        "* No directed cycles and is a directed acyclic graph as a result, and each node has associated probability information that quantifies effect of the parents using number of parameters\n",
        "* X has a direct influence on Y, easier to specify than the probabilities that they influence the others\n",
        "* Only need local probability information once the map is laid out in form of conditional distribution given the parents\n",
        "* Each row in CPT contains conditional probability of each node value for conditioning case, every row must sum to one\n",
        "* Node with no parents only has one row, boolean parents have 2^k independently specifiable probabilities\n",
        "* Probabilities summarize potentially infinite set of circumstances in the environment\n",
        "* Syntax is directed acyclic graph with some local probability information tied to each node\n",
        "\"\"\"\n",
        "chapter13_2 = \"\"\"\n",
        "* Semantics, how the syntax corresponds to joint distribution over the variables in the network\n",
        "* If a representation of joint distribution, can be used to answer any query by summing relevant joint probability values\n",
        "* When one estimates values for conditional distributions, need to be actual conditional probabilities for the variable given its parents\n",
        "* First rewrite entries in joint distribution in terms of conditional probability using product rule\n",
        "* Repeat the process reducing to join smaller sets of variables to get one big product, the chain rule\n",
        "* Determine the set of variables required to model the domain, order them somehow\n",
        "* For links, choose minimal set of parents and insert link from parent to child, write down conditional probability in the table\n",
        "* Parents should contain all nodes that directly influence them\n",
        "* No redundant probability values, so no change of inconsistency so impossible to create a Bayesian network that violates the axioms of probability\n",
        "* Can be specified by 2^K * n variables, k is other nodes it is influence by and n is number of Boolean variables, 2^n in joint distribution\n",
        "* Can leave out links even if slight dependencies exist, slight accuracy gain not worth complexity\n",
        "* Order of the network can affect the number of links and complexity \n",
        "* Each variable is conditionally independent of its non-descendants given its parents\n",
        "* Network can be viewed as instead defining a set of conditional independence properties and full joint distribution can be derived from the properties\n",
        "* Variable is conditionally independent of all other nodes in network given its parents children and Childrens parents - Markov blanket\n",
        "* D-separation: consider just ancestral subgraph of X,Y,Z and ancestors\n",
        "* Add links between unlinked pairs of nodes that share common child and now have a moral graph\n",
        "* Replace all directed links with undirected links\n",
        "* If Z blocks all paths between X and Y in the resulting graph then Z d-separates X and Y, X is conditionally independent of Y given Z, otherwise Pringle Bayesian net doesn’t require conditional independence\n",
        "* Worst case scenario where parents and child relationship is arbitrary described by canonical distribution to fit some pattern\n",
        "* Simplest example provided by deterministic nodes\n",
        "* Context specific independence, in conditional distribution if variable is conditionally independent of some of the parents given certain values of others\n",
        "* Uncertain relationships can be characterized by noisy logical relationships, O(k) parameters for k parents instead of O(2^K) for full conditional probability table\n",
        "* For continuous quantities, do discretization, define continuous variable using family of density functions, or nonparametric representation to define conditional distribution with collection of instances\n",
        "* Bayes network includes hidden variables that are neither input nor output by important for structuring network so it is reasonably sparse with manageable number of parameters\n",
        "\"\"\"\n",
        "chapter13_3 = \"\"\"\n",
        "* Need to compute posterior probability for a set of query variables given event and assignment of values to evidence variables\n",
        "* Inference by enumeration: query can be answered using a Bayesian net by computing sums of products of conditional probabilities from the network\n",
        "* Add terms and multiply the numbers together, worse case have O(2^n) terms in sum, each a product of O(n) probability values so we have O(n2^n) complexity, can be reduced to O(2^n) by nesting computation\n",
        "* Enumeration:ask evaluates expressing trees using depth first recursion, similar to backtracking algorithm for solving CSPs, linear in number of variables, time complexity is always O(2^n)\n",
        "* Contains repeated subexpressions that we are trying to avoid\n",
        "* Do the calculation once and save the expression for later use, like dynamic programming\n",
        "* Variable elimination algorithm does this, evaluates expressions right to left or bottom up, then stores intermediate results, summations over each variable that depend on the value\n",
        "* Have factors that are matrices indexed by values of argument variables\n",
        "* Can sum out variables from product of factors by adding up sub matrices formed by fixing variable to each of values, matrices not multiplied until we need to sum out a variable from the product\n",
        "* Time and space complexity of variable elimination dominated by size of largest factor during algorithm runs\n",
        "* Find that every variable that is not an ancestor of a query variable or an evidence variable is irrelevant to the query\n",
        "* Complexity of Bayesian net depends on structure of network\n",
        "* Singly connected trees of polytrees have at most one undirected path between any two nodes in network, time and space complexity is linear in size of network\n",
        "* For multiply connected networks, variable elimination can have exponential time/space complexity in worst case even when parents are bounded\n",
        "* Includes inference in propositional logic as special case so Bayes net inference is NP-hard\n",
        "* Can reduce satisfiability problems to Bayesian net inference problems\n",
        "* Similarities between complexity of Bayes net inference and complexity of CPSs, tree width important in both\n",
        "* Can also reduced Bayes net inference to satisfiability\n",
        "* Using clustering algorithms or join Tre algorithms to solve posterior probabilities for all variable in the network can take it from O(n^2) time to O(n)\n",
        "* Join individual nodes to form cluster nodes then replace with meganodes \n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\"\n",
        "Probabilistic models\n",
        "* describe how a portion of the world workds, always a simplification and may not account for every variable/actions between them\n",
        "* with these models, our agents need to know what to do about unknown variables given evidence\n",
        "\n",
        "* Two variabels are independent if for all x, y P(x,y) = P(x)P(y), or for all x,y P(x|y) = P(x)\n",
        "* Simplifying modeling assumption, usually empirically find they are close to independent at best\n",
        "* N independent coin flips will all come out independently, doens't depend on previous trial\n",
        "\n",
        "Conditional independence\n",
        "* toothache, cavity, and detection\n",
        "* if cavity, probability it is detected doesnt depend on toothache, P(detect|toothache, cavity) = P(detect|cavity)\n",
        "* same thing in reverse case: P(detect|toothace, no cavity) = P(detect| no cavity)\n",
        "* detect therefore conditionally independt of toothache given cabity\n",
        "* P(toothche|detect, cavity) = P(toothache|cavity), P(toothache, detect|cavity) = P(toothache|cavity)P(detect|cavity)\n",
        "* using the chain rule here again to derive these probabilities\n",
        "* unconditional independence is rare, doesnt allow us to say anything about other variables\n",
        "* most basic/robust form of knowledge about unknown environments\n",
        "* x conditionally independent of y given z if for all x,y,z P(x,y|z) = P(x|z)P(y|z) or P(x|z,y) = P(x|z)\n",
        "\n",
        "Chain rule traffic rain umbrella\n",
        "* P(x1,x2,..xn) = P(x1)P(x2|x1)P(x3|x1,x2)...\n",
        "* P(traffic, rain, umbrella) = P(rain)P(traffic|rain)P(umbrella|rain,traffic) - decomposition\n",
        "* P(traffic, rain, umbrella) = P(rain)P(traffic|rain)P(umbrella|rain) - conditional independence\n",
        "* can express these assumptions with Bayes nets\n",
        "\n",
        "Bayes Nets\n",
        "* problems with full joint distribution tables as probabilistic models: unless few variables way to big to represent all entries, hard to learn anything for more than a few variables ach time\n",
        "* Bayes nets have a technique for describing complex disjoint distributions using local distributions (models using conditional probabilities)\n",
        "* graphical models with local interactions chained together to get global indirect interactions\n",
        "* independence assumptions required to define the bayes net, can involve more conditional indepdences, crucial for modeling \n",
        "\n",
        "Graphical Model\n",
        "* Nodes are variables with domains that can be assigned or unassigned\n",
        "* Arcs are interactions that have direct influence between variables, encode conditional independence\n",
        "* picture that arrows imply direct causation even though generally not the case\n",
        "* N coin flips, N nodes with no interactions since they are all independent\n",
        "* Traffic: 1) rain and traffic independent, 2) rain causes traffic (more accurate)\n",
        "* traffic, rain, low pressure, roof drips, ballgame, cavity: L -> R -> B,T,D, B->T, C alone\n",
        "* alarm network, bulgary, alarm, mary calls, john calls, earthquake: B,E -> A -> M,J\n",
        "* have a set of nodes, one per variable and a directed acyclic graph\n",
        "* also have conditional distribution foe each node, or a collection of distributions over X, one for each parents values combinations\n",
        "* get a description of noisy process for causation\n",
        "* altogether is a graph and local conditional probabilities\n",
        "* implicitly encode join distributions as product of local conditional distributions\n",
        "\n",
        "Probabilties in Bayes Nets\n",
        "* probability bayes net gives to a full assignment, multiply conditionals together P(x1,x2,..xn) = mult(i->n) P(xi|parents(Xi))\n",
        "* guarenteed to result in propoer joint distribution, chain rule valid for all distributions P(x1,x2,..xn) = mult(i->n) P(xi|x1..xi-1)\n",
        "* assuming conditional independences P(xi|x1,x2..xi-1) = P(xi|parents(Xi))\n",
        "* not every bayes net can represent every joint dsitrbution, graph structure enforces conditional independences\n",
        "* only distributions with absolutely independent variables can be represented by Bayes net with no arcs\n",
        "\n",
        "Causality in Bayes Nets\n",
        "* when they reflect true causal patterns, often simpler with fewer parents, easier to think about and elicit from experts\n",
        "* don't actually need to be causal, sometimes not possible and get correlation arrows instead of causation\n",
        "* arrows really referring to conditional independence and may also be causal but not always\n",
        "* Bayes nets have joint distribution over N boolean variables of size 2^N\n",
        "* N-node net with k parents is O(N * 2^k+1)\n",
        "* can calculate these joint probabilties, sae space, get Conditional probability tables easier, answer queries faster\n",
        "* conditional independence is property of distribution\n",
        "\n",
        "D-separation\n",
        "* Causal chain, x->y->z, e.g. low pressure -> rain -> traffic\n",
        "* P(x,y,z) = P(x)P(y|x)P(z|y)\n",
        "* no guarenteed x independent of z, one set where this is not the case shows its not a guarentee, e.g. low pressure causes rain causes traffic, high pressure casues no rain causes no traffic\n",
        "* guarenteed x independent of z given y? Yes - P(z|x,y) = P(x,y,z) / P(x,y) = (P(x)P(y|x)P(z|y)) / (P(x)P(y|x)) = P(z|y)\n",
        "* evidence along the chain blocks the influence\n",
        "\n",
        "Common Cause\n",
        "* y-> x,z, P(x,y,z) = P(y)P(x|y)P(z|y)\n",
        "* no guarentee x is independent of z, but yes guarenteed x and z indepedent given y\n",
        "* P(z|x,y) = P(x,y,z) / P(x,y) = (P(y)P(x|y)P(z|y)) / (P(y)P(x|y)) = P(z|y)\n",
        "\n",
        "Common Effect\n",
        "* x,y->z, x and y are independent, but not x and y are not independent given z, backwards from all the other cases, now it activates influence between possible causes\n",
        "* now we can look at any given bayes net and see if two variables are independent, break down complex examples into simpler ones of these three cases\n",
        "\n",
        "Reachability\n",
        "* want to hide evidence nodes and look for paths in the graph\n",
        "* identify active/inactve paths\n",
        "* x and y are conditionally independent given evidence variabes Z if x and y are d-separated by Z, have to consider all undirected paths from x to y, if no active paths then they are independent\n",
        "* path is active if each triple is active, causal chain (a->b->c where b is unobserved), common cause (a <- b -> c where b is unobserved), or common effect (a -> b <- c where b or one descendent is unobserved), if we have one inactive segment then the entire path is blocked\n",
        "\n",
        "* can now query if two variables are independent, check all undirected paths between the two inputs\n",
        "* if one ore more are active than the independence is not guarenteed, but if all paths are inactive then independence is guarenteed\n",
        "* can now run d-separation algorithm to build complete list of conditional independences that are true in the form: Xi indep. Xj | {Xk1..Xkn}\n",
        "* list determines set of probability distributions that can be represented, now calculate everything\n",
        "* can deduce guarenteed independencies of distributions from graphs, joint distribution may have more conditional independence not detectable until you look at specific distributions\n",
        "\n",
        "Inference\n",
        "* calculate a useful quantity from the joint probability distributions\n",
        "* posterior probability P(Q|E1=e1..Ek=ek), most likely explanation argmax_q P(Q=q|E1=e1..)\n",
        "* same steps as before, have evidence variables, query variable, hidden variables\n",
        "* 1) select entries that are consistent with the evidence, 2) sum out H to get query and evidence, 3) normalize by Z, want to find P(Q|e1..ek)\n",
        "* in bayes nets, easy if we have unlimited time\n",
        "* inference by enumeration very slow, have to join up entire joint distribution before sum out hidden variables\n",
        "* variable elimination conversely you interleave joining and marginalizing, still Np-hard but much faster than inference by enumeration\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"conditional probability table\" : \"local probability information attached to each node in the bayesian network, only for discrete variables\", \n",
        "\"conditioning case\": \"possible combination of values for the parent nodes in bayesian network\",\n",
        "\"topological order\": \"numbering nodes in any order consistent with directed graph structure\",\n",
        "\"deterministic nodes\": \"Values specified exactly by the value of the parents with no uncertainty\",\n",
        "\"leak node\": \"can add to model to cover miscellaneous causes\",\n",
        "\"discretization\": \"splitting up possible values into fixed set of intervals\",\n",
        "\"hybrid Bayesian network\": \"network with both discrete and continuous variables\",\n",
        "}\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "1075413b-529e-42f9-d095-69024649c1f6"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n\n* Bayesian network expressions, can represent dependencies among variables\n* Can represent any full joint probability distributions often very concisely\n* Each node corresponds to continuous or discrete random variable, directed links or arrows connect pairs of nodes, if there is an arrow from X to Y say that X it the parent of Y\n* No directed cycles and is a directed acyclic graph as a result, and each node has associated probability information that quantifies effect of the parents using number of parameters\n* X has a direct influence on Y, easier to specify than the probabilities that they influence the others\n* Only need local probability information once the map is laid out in form of conditional distribution given the parents\n* Each row in CPT contains conditional probability of each node value for conditioning case, every row must sum to one\n* Node with no parents only has one row, boolean parents have 2^k independently specifiable probabilities\n* Probabilities summarize potentially infinite set of circumstances in the environment\n* Syntax is directed acyclic graph with some local probability information tied to each node\n\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n\n* Semantics, how the syntax corresponds to joint distribution over the variables in the network\n* If a representation of joint distribution, can be used to answer any query by summing relevant joint probability values\n* When one estimates values for conditional distributions, need to be actual conditional probabilities for the variable given its parents\n* First rewrite entries in joint distribution in terms of conditional probability using product rule\n* Repeat the process reducing to join smaller sets of variables to get one big product, the chain rule\n* Determine the set of variables required to model the domain, order them somehow\n* For links, choose minimal set of parents and insert link from parent to child, write down conditional probability in the table\n* Parents should contain all nodes that directly influence them\n* No redundant probability values, so no change of inconsistency so impossible to create a Bayesian network that violates the axioms of probability\n* Can be specified by 2^K * n variables, k is other nodes it is influence by and n is number of Boolean variables, 2^n in joint distribution\n* Can leave out links even if slight dependencies exist, slight accuracy gain not worth complexity\n* Order of the network can affect the number of links and complexity \n* Each variable is conditionally independent of its non-descendants given its parents\n* Network can be viewed as instead defining a set of conditional independence properties and full joint distribution can be derived from the properties\n* Variable is conditionally independent of all other nodes in network given its parents children and Childrens parents - Markov blanket\n* D-separation: consider just ancestral subgraph of X,Y,Z and ancestors\n* Add links between unlinked pairs of nodes that share common child and now have a moral graph\n* Replace all directed links with undirected links\n* If Z blocks all paths between X and Y in the resulting graph then Z d-separates X and Y, X is conditionally independent of Y given Z, otherwise Pringle Bayesian net doesn’t require conditional independence\n* Worst case scenario where parents and child relationship is arbitrary described by canonical distribution to fit some pattern\n* Simplest example provided by deterministic nodes\n* Context specific independence, in conditional distribution if variable is conditionally independent of some of the parents given certain values of others\n* Uncertain relationships can be characterized by noisy logical relationships, O(k) parameters for k parents instead of O(2^K) for full conditional probability table\n* For continuous quantities, do discretization, define continuous variable using family of density functions, or nonparametric representation to define conditional distribution with collection of instances\n* Bayes network includes hidden variables that are neither input nor output by important for structuring network so it is reasonably sparse with manageable number of parameters\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n\n* Need to compute posterior probability for a set of query variables given event and assignment of values to evidence variables\n* Inference by enumeration: query can be answered using a Bayesian net by computing sums of products of conditional probabilities from the network\n* Add terms and multiply the numbers together, worse case have O(2^n) terms in sum, each a product of O(n) probability values so we have O(n2^n) complexity, can be reduced to O(2^n) by nesting computation\n* Enumeration:ask evaluates expressing trees using depth first recursion, similar to backtracking algorithm for solving CSPs, linear in number of variables, time complexity is always O(2^n)\n* Contains repeated subexpressions that we are trying to avoid\n* Do the calculation once and save the expression for later use, like dynamic programming\n* Variable elimination algorithm does this, evaluates expressions right to left or bottom up, then stores intermediate results, summations over each variable that depend on the value\n* Have factors that are matrices indexed by values of argument variables\n* Can sum out variables from product of factors by adding up sub matrices formed by fixing variable to each of values, matrices not multiplied until we need to sum out a variable from the product\n* Time and space complexity of variable elimination dominated by size of largest factor during algorithm runs\n* Find that every variable that is not an ancestor of a query variable or an evidence variable is irrelevant to the query\n* Complexity of Bayesian net depends on structure of network\n* Singly connected trees of polytrees have at most one undirected path between any two nodes in network, time and space complexity is linear in size of network\n* For multiply connected networks, variable elimination can have exponential time/space complexity in worst case even when parents are bounded\n* Includes inference in propositional logic as special case so Bayes net inference is NP-hard\n* Can reduce satisfiability problems to Bayesian net inference problems\n* Similarities between complexity of Bayes net inference and complexity of CPSs, tree width important in both\n* Can also reduced Bayes net inference to satisfiability\n* Using clustering algorithms or join Tre algorithms to solve posterior probabilities for all variable in the network can take it from O(n^2) time to O(n)\n* Join individual nodes to form cluster nodes then replace with meganodes \n\n\n## Module 11 Lecture Notes\n\nProbabilistic models\n* describe how a portion of the world workds, always a simplification and may not account for every variable/actions between them\n* with these models, our agents need to know what to do about unknown variables given evidence\n\n* Two variabels are independent if for all x, y P(x,y) = P(x)P(y), or for all x,y P(x|y) = P(x)\n* Simplifying modeling assumption, usually empirically find they are close to independent at best\n* N independent coin flips will all come out independently, doens't depend on previous trial\n\nConditional independence\n* toothache, cavity, and detection\n* if cavity, probability it is detected doesnt depend on toothache, P(detect|toothache, cavity) = P(detect|cavity)\n* same thing in reverse case: P(detect|toothace, no cavity) = P(detect| no cavity)\n* detect therefore conditionally independt of toothache given cabity\n* P(toothche|detect, cavity) = P(toothache|cavity), P(toothache, detect|cavity) = P(toothache|cavity)P(detect|cavity)\n* using the chain rule here again to derive these probabilities\n* unconditional independence is rare, doesnt allow us to say anything about other variables\n* most basic/robust form of knowledge about unknown environments\n* x conditionally independent of y given z if for all x,y,z P(x,y|z) = P(x|z)P(y|z) or P(x|z,y) = P(x|z)\n\nChain rule traffic rain umbrella\n* P(x1,x2,..xn) = P(x1)P(x2|x1)P(x3|x1,x2)...\n* P(traffic, rain, umbrella) = P(rain)P(traffic|rain)P(umbrella|rain,traffic) - decomposition\n* P(traffic, rain, umbrella) = P(rain)P(traffic|rain)P(umbrella|rain) - conditional independence\n* can express these assumptions with Bayes nets\n\nBayes Nets\n* problems with full joint distribution tables as probabilistic models: unless few variables way to big to represent all entries, hard to learn anything for more than a few variables ach time\n* Bayes nets have a technique for describing complex disjoint distributions using local distributions (models using conditional probabilities)\n* graphical models with local interactions chained together to get global indirect interactions\n* independence assumptions required to define the bayes net, can involve more conditional indepdences, crucial for modeling \n\nGraphical Model\n* Nodes are variables with domains that can be assigned or unassigned\n* Arcs are interactions that have direct influence between variables, encode conditional independence\n* picture that arrows imply direct causation even though generally not the case\n* N coin flips, N nodes with no interactions since they are all independent\n* Traffic: 1) rain and traffic independent, 2) rain causes traffic (more accurate)\n* traffic, rain, low pressure, roof drips, ballgame, cavity: L -> R -> B,T,D, B->T, C alone\n* alarm network, bulgary, alarm, mary calls, john calls, earthquake: B,E -> A -> M,J\n* have a set of nodes, one per variable and a directed acyclic graph\n* also have conditional distribution foe each node, or a collection of distributions over X, one for each parents values combinations\n* get a description of noisy process for causation\n* altogether is a graph and local conditional probabilities\n* implicitly encode join distributions as product of local conditional distributions\n\nProbabilties in Bayes Nets\n* probability bayes net gives to a full assignment, multiply conditionals together P(x1,x2,..xn) = mult(i->n) P(xi|parents(Xi))\n* guarenteed to result in propoer joint distribution, chain rule valid for all distributions P(x1,x2,..xn) = mult(i->n) P(xi|x1..xi-1)\n* assuming conditional independences P(xi|x1,x2..xi-1) = P(xi|parents(Xi))\n* not every bayes net can represent every joint dsitrbution, graph structure enforces conditional independences\n* only distributions with absolutely independent variables can be represented by Bayes net with no arcs\n\nCausality in Bayes Nets\n* when they reflect true causal patterns, often simpler with fewer parents, easier to think about and elicit from experts\n* don't actually need to be causal, sometimes not possible and get correlation arrows instead of causation\n* arrows really referring to conditional independence and may also be causal but not always\n* Bayes nets have joint distribution over N boolean variables of size 2^N\n* N-node net with k parents is O(N * 2^k+1)\n* can calculate these joint probabilties, sae space, get Conditional probability tables easier, answer queries faster\n* conditional independence is property of distribution\n\nD-separation\n* Causal chain, x->y->z, e.g. low pressure -> rain -> traffic\n* P(x,y,z) = P(x)P(y|x)P(z|y)\n* no guarenteed x independent of z, one set where this is not the case shows its not a guarentee, e.g. low pressure causes rain causes traffic, high pressure casues no rain causes no traffic\n* guarenteed x independent of z given y? Yes - P(z|x,y) = P(x,y,z) / P(x,y) = (P(x)P(y|x)P(z|y)) / (P(x)P(y|x)) = P(z|y)\n* evidence along the chain blocks the influence\n\nCommon Cause\n* y-> x,z, P(x,y,z) = P(y)P(x|y)P(z|y)\n* no guarentee x is independent of z, but yes guarenteed x and z indepedent given y\n* P(z|x,y) = P(x,y,z) / P(x,y) = (P(y)P(x|y)P(z|y)) / (P(y)P(x|y)) = P(z|y)\n\nCommon Effect\n* x,y->z, x and y are independent, but not x and y are not independent given z, backwards from all the other cases, now it activates influence between possible causes\n* now we can look at any given bayes net and see if two variables are independent, break down complex examples into simpler ones of these three cases\n\nReachability\n* want to hide evidence nodes and look for paths in the graph\n* identify active/inactve paths\n* x and y are conditionally independent given evidence variabes Z if x and y are d-separated by Z, have to consider all undirected paths from x to y, if no active paths then they are independent\n* path is active if each triple is active, causal chain (a->b->c where b is unobserved), common cause (a <- b -> c where b is unobserved), or common effect (a -> b <- c where b or one descendent is unobserved), if we have one inactive segment then the entire path is blocked\n\n* can now query if two variables are independent, check all undirected paths between the two inputs\n* if one ore more are active than the independence is not guarenteed, but if all paths are inactive then independence is guarenteed\n* can now run d-separation algorithm to build complete list of conditional independences that are true in the form: Xi indep. Xj | {Xk1..Xkn}\n* list determines set of probability distributions that can be represented, now calculate everything\n* can deduce guarenteed independencies of distributions from graphs, joint distribution may have more conditional independence not detectable until you look at specific distributions\n\nInference\n* calculate a useful quantity from the joint probability distributions\n* posterior probability P(Q|E1=e1..Ek=ek), most likely explanation argmax_q P(Q=q|E1=e1..)\n* same steps as before, have evidence variables, query variable, hidden variables\n* 1) select entries that are consistent with the evidence, 2) sum out H to get query and evidence, 3) normalize by Z, want to find P(Q|e1..ek)\n* in bayes nets, easy if we have unlimited time\n* inference by enumeration very slow, have to join up entire joint distribution before sum out hidden variables\n* variable elimination conversely you interleave joining and marginalizing, still Np-hard but much faster than inference by enumeration\n\n\n## Module 11 Definitions\n* **Conditional probability table** - local probability information attached to each node in the bayesian network, only for discrete variables\n* **Conditioning case** - possible combination of values for the parent nodes in bayesian network\n* **Topological order** - numbering nodes in any order consistent with directed graph structure\n* **Deterministic nodes** - Values specified exactly by the value of the parents with no uncertainty\n* **Leak node** - can add to model to cover miscellaneous causes\n* **Discretization** - splitting up possible values into fixed set of intervals\n* **Hybrid bayesian network** - network with both discrete and continuous variables\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\"\n",
        "* Components of agent include direct mapping from conditions on current state to action, means to infer relevant properties of the world, and information about the way world evolves and the results of actions an agent takes\n",
        "* Also utility information, action-value information, goal that describes most desirable states, problem generator/critic\n",
        "* Assumes little prior knowledge of the agent, transfer learning is when knowledge from one domain transferred to another\n",
        "* Induction is going from specific set of observations to general rule, may be incorrect whereas deductive had to be correct\n",
        "* Factored representation is vector of attribute values, dealing with classification and regression problems\n",
        "* Supervised learning: agent observed input-output pairs and learns a function that maps from input to output\n",
        "* Unsupervised learning: agent learns patterns in the input without explicit feedback, most commonly does clustering \n",
        "* Reinforcement learning agent learns from series of reinforcements, rewards and punishments\n",
        "\"\"\"\n",
        "chapter19_2 = \"\"\"\n",
        "* Giving training set of n example inputs, discover function that approximates true function f\n",
        "* Hypothesis drawn from hypothesis space, h is a model drawn from model class, or function from function class, ground truth is the true label\n",
        "* If we dont have prior knowledge about process that generated data perform exploratory data analysis\n",
        "* Look for best fit function for each function close to y\n",
        "* Evaluate performance on test set, generalizes well if it accurately predicts outcomes of test set\n",
        "* Bias is tendency of predictive hypothesis to deviate from expected value when averaged over different training sets\n",
        "* Underfiting when it fails to find pattern in data\n",
        "* Variance is amount of change in hypothesis due to fluctuation in training data\n",
        "* Overfitting when data pays too much attention to particular data set and performs poorly on unseen data\n",
        "* Bias-variance tradeoff between complexity, more complex low bias and low variance generalized better\n",
        "* Tradeoff between expressiveness of hypothesis space and computational complexity of finding good hypothesis in space\n",
        "\"\"\"\n",
        "chapter19_3 = \"\"\"\n",
        "* Decision tree is representation of a function that maps vector of attributes values to single output value decision\n",
        "* Performs actions until a leaf is reached, boolean classification if it only true or false\n",
        "* Can often be very intuitive with interpretable results\n",
        "* Want to be consistent with examples and small as possible\n",
        "* Can generate error or noise in the data if nondeterministic results\n",
        "* Can evaluate performance with learning curve, proportion correct on test set with training set size\n",
        "* Importance measured by entropy, fundamental quantity in information theory, measure of uncertainty of a random variable, no uncertainty is 0\n",
        "* Information gain from attribute test is expected reduction in entropy\n",
        "* Larger hypothesis spaces have more capacity to fit and overfit\n",
        "* Can do decision tree pruning to combat overfitting, eliminate irrelevant nodes, can do a significance test to figure out when to cut the tree\n",
        "* Early stopping, have decision tree algorithm stop generating nodes when there is no good attribute to split on\n",
        "* Split point test, inequality test on value of an attribute can be used to deal with continuous values\n",
        "* Information gain ratio tells you which variables to split the data on\n",
        "* If trying to predict numerical output value then need regression tree instead of classification tree\n",
        "* Decision trees are unstable, adding one new example can change test at the root, random forest model can fix some of the issues\n",
        "\"\"\"\n",
        "chapter19_4 = \"\"\"\n",
        "* Stationarity: the assumption that all future examples will be like the past, without it we are doomed\n",
        "* Assume each example has the same prior probability distribution and is independent of previous examples\n",
        "* Independent and identically distributed\n",
        "* Optimal fit is hypothesis that minimizes the error rate, measure this on the testt set of examples\n",
        "* Split data into training and test set, can try different hyper parameter classes\n",
        "* Need validation set as well to evaluate candidate models and choose the best one\n",
        "* Can do k-fold cross validation if we need more data than we have\n",
        "* Leave one out cross validation is extreme example of this\n",
        "* Model selection chooses good hypothesis class, optimization finds best hypothesis within the space\n",
        "* Complexity measured by number of decision tree nodes and by number of neural network parameters\n",
        "* Often training set error reaches 0 as complexity increases\n",
        "* Model that exactly fits all the training data has interpolated the data, start to overfit once we approach this\n",
        "* Want to minimize loss function instead of maximizing utility function in NN\n",
        "* Define prior probabilities and get generalization losses for hypothesis h\n",
        "* Agent can often only estimate empirical loss on set of examples\n",
        "* Learning problem is realizable if hypothesis space contains true function\n",
        "* Can be computationally intractable to search all possibilities to get f\n",
        "* Penalize hypotheses with regularization, L1 or L2 regularization\n",
        "* Can also reduce number of dimensions that model works with through feature selection\n",
        "\"\"\"\n",
        "chapter19_5 = \"\"\"\n",
        "* Computational learning theory, any hypothesis that is seriously wrong will almost certainly be discovered after only a few examples because of inaccurate predictions\n",
        "* Therefore any hypothesis consistent with sufficiently large set of training examples is unlikely to be seriously wrong, probably approximately correct (PAC)\n",
        "* Decision lists outline how PAC learning should be applied to new decision space\n",
        "* Branch only in one direction to test PAC \n",
        "\"\"\"\n",
        "chapter19_6 = \"\"\"\n",
        "* Linear functions of continuous valued inputs\n",
        "* Coefficients thought of as weights in linear regression, loss function always convex\n",
        "* Minimizing loss through gradient descent, estimate the gradient and move in steepest downhill direction\n",
        "* Learning rate or step size tells us how quickly to move\n",
        "* Loss is quadratic so partial derivative will be linear\n",
        "* Batch gradient descent learning rule is over entire dataset at once, an epoch\n",
        "* Stochastic gradient descent randomly selects small number at each step, usually with minibatc\n",
        "* Can easily extend to multivariable linear regression problems, solve for the data matrices of weight and bias instead since have multiple regressions\n",
        "* Usually dont worry about overfitting in univariate linear regression, now need regularization to penalize large weights and avoid overfitting\n",
        "* L1 regularization tends to produce sparse models with many weights of 0\n",
        "* L2 does not tend to produce zero weights, doesn’t that dimensional axes seriously like l1 does\n",
        "* Can do classification as well as regression\n",
        "* Decision boundary is line that separates two classes, straight line, linearly separable if this can be done\n",
        "* Perceptron learning rule is simple weight update that coverages to solution provided they are linearly separable\n",
        "* Correct output, nothing changes\n",
        "* If y is 1 but w(x) is 0, wi increased when xi is positive and decreased when xi is negative\n",
        "* If y is 0 but w(x) is 1, wi decreased when xi is positive and increased when xi is negative\n",
        "* Use sigmoidal output function to make things easier with predictions\n",
        "* Process of fitting weights to minimize loss on dataset is logistic regression\n",
        "\"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\"\n",
        "* Multinomial naive Bayes classifier, makes simplifying assumption about how features interact\n",
        "* Represent text document as bag of words, unordered set and frequency only kept\n",
        "* Returns the maximum class based on probability\n",
        "* Use Bayes rule to transform this expression into probabilities that have useful properties, can break down conditionals into three other probabilities\n",
        "* Naive bayes is a generative model because we can have implicit assumption about how a document is generated, sample a class, then generate words by sampling from conditional probability\n",
        "* Compute most probable class given document with the prior probability P(c) of the class and the likelihood of the document P(d|c), c = argmax c P(D|C)P(C)\n",
        "* Need some simplifying assumptions to calculate, bag of words assumption says the position doesn’t matter in the document\n",
        "* Naive Bayes assumption is second, conditional independence is that probabilities are independent given the class c and can be naively computed as P(f1|c) * P(F2|c) * …\n",
        "* Consider word position by walking an index through each word position in the document\n",
        "* Done in log space to avoid underflow and increase speed\n",
        "* Classifiers that use linear combination of inputs to make classification decision are linear classifiers\n",
        "\"\"\"\n",
        "slp_chapter4_2 = \"\"\"\n",
        "* Use frequencies in data to get estimates of the two terms\n",
        "* Maximum likelihood estimate, sum all of the instances divided by all of the words\n",
        "* But if we get a word we’ve never seen before can have zero probability and get error\n",
        "* Laplace smoothing, add one to pretend we’ve seen every word more than we actually have\n",
        "* If we encounter words never before seen, we usually ignore them, can also ignore stop words completely, which are a class of very common words\n",
        "\"\"\"\n",
        "slp_chapter4_3 = \"\"\"\n",
        "* Sentiment analysis, + is positive sentiment, - if negative\n",
        "* Word doesn’t occur in training set, drop it completely\n",
        "* Get a higher probability for the negative sentiment than the positive sentiment, so we predict negative class for the test sentence\n",
        "\"\"\"\n",
        "slp_chapter4_4 = \"\"\"\n",
        "* Whether a word appears or not seems to matter more than frequency, can clip word counts in each document at one, binary multinomial naive Bayes\n",
        "* For each document we remove all duplicate words before concatenating into single document\n",
        "* Also have to deal with negation when looking at sentiment classifiers\n",
        "* May have insufficient labeled training data to train accurate naive Bayes classifiers using all words in training set, so have to derive positive and negative word features from sentiment lexicons\n",
        "* Add a feature that is counted whenever a word from the lexicon appears\n",
        "* If training data is sparse or not representative of test set, dense lexicon features instead of sparse individual word features may generalize better\n",
        "\"\"\"\n",
        "slp_chapter4_5 = \"\"\"\n",
        "* Naive Bayes doesn’t require our classifier use all words in training data as features\n",
        "* Spam detection, deciding if email is spam or appropriate piece of mail\n",
        "* Rather than use all words as features, predefine likely sets of words or phrases as features combined with features that really not purely linguistic\n",
        "* Language id - determining what language a piece of text is written in, most effective to use character n-grams or byte n-grams isnstead of word n-grams\n",
        "* Often trained on multilingual texts\n",
        "\"\"\"\n",
        "slp_chapter4_6 = \"\"\"\n",
        "* Naive bayes classifiers can use any sort of feature\n",
        "* We only use individual word features and use all words in the text, then similar to language modeling\n",
        "* Naive bayes can be viewed as a set of class-specific unigram language models in which the model for each class instantiates a unigram language model\n",
        "* Model assigns probability to each sentence as well, positive model will assign higher probabilities to sentences given a positive sentiment\n",
        "* Once we multiply in prior of full naive bayes model a different classification decision may be made\n",
        "\"\"\"\n",
        "slp_chapter4_7 = \"\"\"\n",
        "* Gold labels are human defined labels that we are trying to match up to\n",
        "* Can build a confusion matrix for visualizing how an algorithm performs with respect to human gold labels using system output and gold labels and label each cell as positive outcomes\n",
        "* Can also calculate accuracy, percentage of observations labeled correctly, generally not used for text classification tasks\n",
        "* Precision measures percentage of items that system detected that are in fact positive, true positives / true positives + false positives\n",
        "* Recall measures the percentage of items actually present that were correctly identified by the system, true positives / true positives + false negatives\n",
        "* F-measure incorporates both aspects of precision and recall\n",
        "* Precision/recall has to be adjusted slightly if we have more than 2 classes\n",
        "* Macro averaging, compute performance for each class then average over all classes \n",
        "* Micro averaging, collect decisions for all classes into single confusion matrix then compute precision/recall from a table\n",
        "\"\"\"\n",
        "slp_chapter4_8 = \"\"\"\n",
        "* Use development set to tune some parameters, decide what the best model is in general\n",
        "* Use of a dev set avoids overfitting the test set, but it may not be large enough to be representative\n",
        "* Do cross-validation instead, k disjoint partitions called fold made, each one used as a test set in a run of the models\n",
        "* Pick a fold, remaining k-1 folds used in training, then fold used as the test set\n",
        "* Average it out, can do 10 times for 10-fold cross validation\n",
        "* Ned the whole corpus to be blind, can’t examine what is going on because then would be looking at the test set\n",
        "* Common to Crete fixed training set and test set, then do 10-fold cross validation inside the training set\n",
        "\"\"\"\n",
        "slp_chapter4_9 = \"\"\"\n",
        "* Do statistical significance hypothesis testing to see if one system is better at describing something than another\n",
        "* Effect size, bigger value means model A is doing way better than B, smaller says only doing a little bit better\n",
        "* H0 is null hypothesis, says value s actually negative or 0 so A is not better than B\n",
        "* Calculate the p-value, can reject null hypothesis if below a certain threshold value and statistically significant\n",
        "* Two common non-parametric tests in NLP is approximate randomization and bootstrap test\n",
        "* Paired tests are those where we compare two sets of observations that are aligned\n",
        "* Bootstrapping refers to repeatedly drawing large number of smaller samples with replacement from original larger sample\n",
        "* Acts as one-sided empirical p-value\n",
        "\"\"\"\n",
        "slp_chapter4_10 = \"\"\"\n",
        "* Representational harms caused by a system that demeans a social group and perpetuates negative stereotypes\n",
        "* Toxicity detection task of detecting hate speech and other kinds of toxic language, want to reduce societal harm but can backfire\n",
        "* False positives can lead to censoring of important issues\n",
        "* Ml models replicate and amplify biases they find in their training data, can be cause by labels, resources used, model architecture \n",
        "* Can release a model card that details the model to see if any issues are immediately apparent\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\"\n",
        "* want to acquire model from data/experience, learning the parameters, structure, hidden concepts\n",
        "* spam classification, get a large collection of emails labeled by hand, want to predict labels of future emails. Features are attributes that make decision, words, text patterns, non-text details\n",
        "* digit recognition, get labeled collection of images, predict new digit images, attributes are pixels and shape patterns\n",
        "* have inputs x, predict labels y, spam detection, medical diagnosis, essay grading, fraud detection, etc.\n",
        "\n",
        "Model-based classification\n",
        "* build a model (like a Bayes net) where label and features are random variables, fill in observed features and query for distribution fo labels conditions on features\n",
        "* Naive bayes for ditigs assumes all features are independent effects of the labels, one variable for each pixel position, on/off, maps to feature vector\n",
        "* general naive bayes learns for y labels, P(Y, F1, ...Fn) = P(Y) mult_i P(F_i|Y), have to specify how features depend on the class\n",
        "* number of parameters linear in number of features\n",
        "* inference for naive bayes wants to compute posterior distribution over label variable, so we do the following:\n",
        "* 1) get joint probability of label and evidence for each of the labels Y, 2) sum them together to get probability of evidence, 3) normalize by taking step 1 / step 2\n",
        "* to use naive bayes, need inference method (probabilities P(Y) and P(Fi|Y), calculate P(Y|F1..Fn)), and estimates of local contiional probability tables (P(Y) over labels, P(Fi|Y) for each feature, called parameters of model)\n",
        "* Bag of words Naive bayes has word features Wi at position i, predict label conditioned on feature variables, assume they are conditionally independent, but now each Wi is identically distributed\n",
        "* in BOW, each position identically distributed, all positions share same contioinal probabilitiy, model is insensitive to word order/reordering\n",
        "* spam filtering model e.g. P(Y, W1 ... Wn) = P(Y) mult_i P(Wi|Y) \n",
        "\n",
        "Concepts/Definitions\n",
        "* Data is labeled instances, separated into training set, held out set, test set\n",
        "* Features: attributes and values characterizing each x input\n",
        "* experimentation cycle: learn the parameters on training set, tune hyperparameters on held out set, compute accuracy of test set (don't look at it ever)\n",
        "* evaluation, calculate accuracy as fraction of instances predicted correctly\n",
        "* overfitting, fitting the training data very closely but not generalizing well to the test data set. Posterios determined by relative probabilities, if overfit can go to infinite\n",
        "\n",
        "Generalization and Overfitting\n",
        "* relative frequency parameters overfit training data, don't want to assign unseen events zero probability\n",
        "* want to generalize better to the test set, so we smooth or regularize the estimates\n",
        "\n",
        "Parameter estimation\n",
        "* need to estimate the distribution of a random variable, difficult to decide so use training data\n",
        "* for each outcome x, look at the empirical rate of the value as P(x) = count(x) / total samples, then this is the estimate that maxmizes the likelihood of the data L(x, theta) = mult_i P_theta(xi)\n",
        "* relative frequencies give mazimum likelihood estimates, theta_ML = argmax_theta P(X|theta) = argmax_theta mult_i P_theta(Xi)\n",
        "* can do Laplace smoothing, pretend you saw every outcome one more time than you did\n",
        "* P_lap(x) = c(x) + 1 / sum_x [c(x) + 1] = c(x) + 1 / N + |X|\n",
        "* or could pretend you say it k extra times, P_lap(x) = c(x) + k / N + k|X|, k is the strength of the prior\n",
        "* for conditionals, smooth each condition separately: P_lap(x) = c(x,y) + k / c(y) + k|X|\n",
        "* could also do linear interpolation, get empirical P(x) from the data, make sure estimate P(X|Y) isn't too different from P(x) by P_lin(x|y) = a P_hat(x|y) + (1 - a)P_hat(x)\n",
        "* smoothing critical in real classification problems\n",
        "\n",
        "Held out data tuning\n",
        "* we get two unknowns now, the parameter probabilities P(x,y), P(y) and the hyperparameters (type of smoothing, k, a, etc.)\n",
        "* learn paramters from training data, tune hyperparameters on this set, for each value of hyperparameters, train and test on held out set, choose best values and do final run on test data\n",
        "\n",
        "Feature errors\n",
        "* if we get errors, can sometimes need more features as words are not enough\n",
        "* add these new information sources as new variables in Naive bayes model\n",
        "* naive bayes can incorporate many features, but generally do better with homogenous cases of features\n",
        "\n",
        "Baselines\n",
        "* need a baseline as first step, see how hard task is and what a good accuracy entails\n",
        "* a weak baseline gives the most frequent label classifier, give the test set whatever label was most common, accuracy may be high or low, real research uses previous work as baselines\n",
        "* confidence of classifier comes from posterior over top label, confidence(x) = max_y P(y,x)\n",
        "* says hos sure the classifier is of a prediction, comes with any probabilistic model but no guarentee it is correct\n",
        "* weak calibration - higher confidences mean higher accuracy, strong calibration - confidence predicts the accuracy rates\n",
        "* bayes lets us do queries with causal probabilities, naive bayes takes all features to be independent, need smoothing and confidences helpful when you can get them\n",
        "\n",
        "Perceptrons\n",
        "* Loosely inspired by human neurons\n",
        "* linear classifiers, inputs are feature values, each feature with a weight and the sum if the activation\n",
        "* activation_w(x) = sum_i wi * fi(x) = w * f(x), if >= 0 output +1, < 0 output -1\n",
        "* in binary case, compare features to weight vector, trying to learn weight vectors from examples\n",
        "* have inputs, weights, weighted sum, activation function with theshold, output\n",
        "* output is 0 if sum_j wjxj <= theshold, 1 if sum_j wjxj > theshold\n",
        "* by varying weights and threshold we get different models of decisions\n",
        "* write sum_j wjxj as dot product of input vector and weight vector, w * x\n",
        "* move threshold to other side of inequality, define bias as the -1 * theshold b = -theshold\n",
        "* output is 0 if wx + b <= 0, 1 if wx + b > 0\n",
        "* have complex network of perceptrsont that make subtle decisions\n",
        "\n",
        "Binary Decision rule\n",
        "* in space of feature vectors, each example is a point, weight vector is hyperplane, one side +1, other -1\n",
        "* start with weights = 0, for each training instance classify with weights, if correct don't change anything, if wrong adjust the weight vector\n",
        "* y = +1 if w * f(x) >= 0, -1 if w * f(x) < 0, adjust w by w = w + y* * f, subtract if y* is -1, separable case very intuitive\n",
        "\n",
        "Multiclass decition rule\n",
        "* now we have weight vector for each class w_y, score activation of a class y is w_y * f(x), highest prediction score wins y = argmax_y w_y * f(X)\n",
        "* weights start at 0, for each training instance predict with current weights, if wrong lower score of wrong answer and raise score of right answer\n",
        "* w_y = w_y - f(x), w_y* = w_y* + f(x)\n",
        "\n",
        "Perceptron properties\n",
        "* separability: can be true if training set is perfectly separated but not always possible\n",
        "* convergence, if training is separable, perceptron eventually converges in binary case\n",
        "* maximum number of mistakes in binary case relatest to margin or degree of separability, mistakes < k / d^2\n",
        "\n",
        "Issues:\n",
        "* if data is not separable, weights may thrash, averaging weight vectors over time can help\n",
        "* can create mediocre generalization if a barely separating solution found\n",
        "* can overtrain if test/held out set accuracy rises then eventually falls, training accuracy keeps increasing\n",
        "\n",
        "Fixing perceptrion:\n",
        "* adjust the weight update to lessen effects\n",
        "* MIRA* - choose update size thatfixes current mistake but minimzies change to w\n",
        "* min_w 0.5 * sum_y ||w_y - w'_y||^2, wy* * f(x) >= wy * f(x) + 1, +1 helps to generalize\n",
        "* have some value tao (t) to help fix. if we guess y instead of y* on example x with features f(x), wy = w'y - t(f(x)), wy* = w'y* + t(f(x))\n",
        "* minimum correcting update -> min_t ||tf||^2, end up with tao = ((w'y - w'y*) * f + 1) / (2f * f)\n",
        "* in practice, bad to make updates that are too large, example may be labeled incorrecrly or you may not have enough features\n",
        "* so we cap the maximum value of tao with a constant C, tao* = min(((w'y - w'y*) * f + 1) / (2f * f), C)\n",
        "* this will correspond to optimization assuming non-separable data, converges faster than perceptron and better on noisy data\n",
        "\n",
        "Support Vector Machines\n",
        "* maximize the margin between the support vectors (only ones that matter), all other training examples ignored\n",
        "* find separator with max margin, optimize over all examples at once, good according to intuition, theory, pratice\n",
        "* min_w 0.5 ||w||^2, for all i, y - wy* * f(xi) >= wy * f(xi) + 1\n",
        "* data that is linearly separable is great for linear decision rules, but if its too hard, can map to higher dimensional space\n",
        "\n",
        "* Naive Bayes builds model training data, gives prediction probabiltiies, has assumptions about feature independence and passes through data once\n",
        "* perceptrons/MIRA make less assumptions about data, mistake-driven learning, multiple passes through data, and are often more accurate\n",
        "* original feature space can be mapped to higher dimensional feature space where training set is separable\n",
        "\n",
        "Logical functions:\n",
        "* networks of perceptons can compute any logical functions\n",
        "* build up these networks with NAND gates\n",
        "* XOR cannot be solved with one neural unit, but can be raise dto higher dimensional space to be solved linearly\n",
        "\n",
        "Activation functions\n",
        "* insstead of outputting wx + b as linear function, apply non-linear function to output called activation value\n",
        "* activation value is a, output of entire network is y, so we get y = a = f(wx + b)\n",
        "* three most commonly used ones are sigmoid function (1/1+e^-z), tanh, and ReLU (x if > 0, 0 if < 0)\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "b153dfa7-ee25-408a-ed5d-facab0f80252"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\n* Components of agent include direct mapping from conditions on current state to action, means to infer relevant properties of the world, and information about the way world evolves and the results of actions an agent takes\n* Also utility information, action-value information, goal that describes most desirable states, problem generator/critic\n* Assumes little prior knowledge of the agent, transfer learning is when knowledge from one domain transferred to another\n* Induction is going from specific set of observations to general rule, may be incorrect whereas deductive had to be correct\n* Factored representation is vector of attribute values, dealing with classification and regression problems\n* Supervised learning: agent observed input-output pairs and learns a function that maps from input to output\n* Unsupervised learning: agent learns patterns in the input without explicit feedback, most commonly does clustering \n* Reinforcement learning agent learns from series of reinforcements, rewards and punishments\n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n\n* Giving training set of n example inputs, discover function that approximates true function f\n* Hypothesis drawn from hypothesis space, h is a model drawn from model class, or function from function class, ground truth is the true label\n* If we dont have prior knowledge about process that generated data perform exploratory data analysis\n* Look for best fit function for each function close to y\n* Evaluate performance on test set, generalizes well if it accurately predicts outcomes of test set\n* Bias is tendency of predictive hypothesis to deviate from expected value when averaged over different training sets\n* Underfiting when it fails to find pattern in data\n* Variance is amount of change in hypothesis due to fluctuation in training data\n* Overfitting when data pays too much attention to particular data set and performs poorly on unseen data\n* Bias-variance tradeoff between complexity, more complex low bias and low variance generalized better\n* Tradeoff between expressiveness of hypothesis space and computational complexity of finding good hypothesis in space\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n\n* Decision tree is representation of a function that maps vector of attributes values to single output value decision\n* Performs actions until a leaf is reached, boolean classification if it only true or false\n* Can often be very intuitive with interpretable results\n* Want to be consistent with examples and small as possible\n* Can generate error or noise in the data if nondeterministic results\n* Can evaluate performance with learning curve, proportion correct on test set with training set size\n* Importance measured by entropy, fundamental quantity in information theory, measure of uncertainty of a random variable, no uncertainty is 0\n* Information gain from attribute test is expected reduction in entropy\n* Larger hypothesis spaces have more capacity to fit and overfit\n* Can do decision tree pruning to combat overfitting, eliminate irrelevant nodes, can do a significance test to figure out when to cut the tree\n* Early stopping, have decision tree algorithm stop generating nodes when there is no good attribute to split on\n* Split point test, inequality test on value of an attribute can be used to deal with continuous values\n* Information gain ratio tells you which variables to split the data on\n* If trying to predict numerical output value then need regression tree instead of classification tree\n* Decision trees are unstable, adding one new example can change test at the root, random forest model can fix some of the issues\n\n\n## AIMA Chapter 5.5 Stochastic Games\n\n* Stationarity: the assumption that all future examples will be like the past, without it we are doomed\n* Assume each example has the same prior probability distribution and is independent of previous examples\n* Independent and identically distributed\n* Optimal fit is hypothesis that minimizes the error rate, measure this on the testt set of examples\n* Split data into training and test set, can try different hyper parameter classes\n* Need validation set as well to evaluate candidate models and choose the best one\n* Can do k-fold cross validation if we need more data than we have\n* Leave one out cross validation is extreme example of this\n* Model selection chooses good hypothesis class, optimization finds best hypothesis within the space\n* Complexity measured by number of decision tree nodes and by number of neural network parameters\n* Often training set error reaches 0 as complexity increases\n* Model that exactly fits all the training data has interpolated the data, start to overfit once we approach this\n* Want to minimize loss function instead of maximizing utility function in NN\n* Define prior probabilities and get generalization losses for hypothesis h\n* Agent can often only estimate empirical loss on set of examples\n* Learning problem is realizable if hypothesis space contains true function\n* Can be computationally intractable to search all possibilities to get f\n* Penalize hypotheses with regularization, L1 or L2 regularization\n* Can also reduce number of dimensions that model works with through feature selection\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\n* Computational learning theory, any hypothesis that is seriously wrong will almost certainly be discovered after only a few examples because of inaccurate predictions\n* Therefore any hypothesis consistent with sufficiently large set of training examples is unlikely to be seriously wrong, probably approximately correct (PAC)\n* Decision lists outline how PAC learning should be applied to new decision space\n* Branch only in one direction to test PAC \n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\n* Linear functions of continuous valued inputs\n* Coefficients thought of as weights in linear regression, loss function always convex\n* Minimizing loss through gradient descent, estimate the gradient and move in steepest downhill direction\n* Learning rate or step size tells us how quickly to move\n* Loss is quadratic so partial derivative will be linear\n* Batch gradient descent learning rule is over entire dataset at once, an epoch\n* Stochastic gradient descent randomly selects small number at each step, usually with minibatc\n* Can easily extend to multivariable linear regression problems, solve for the data matrices of weight and bias instead since have multiple regressions\n* Usually dont worry about overfitting in univariate linear regression, now need regularization to penalize large weights and avoid overfitting\n* L1 regularization tends to produce sparse models with many weights of 0\n* L2 does not tend to produce zero weights, doesn’t that dimensional axes seriously like l1 does\n* Can do classification as well as regression\n* Decision boundary is line that separates two classes, straight line, linearly separable if this can be done\n* Perceptron learning rule is simple weight update that coverages to solution provided they are linearly separable\n* Correct output, nothing changes\n* If y is 1 but w(x) is 0, wi increased when xi is positive and decreased when xi is negative\n* If y is 0 but w(x) is 1, wi decreased when xi is positive and increased when xi is negative\n* Use sigmoidal output function to make things easier with predictions\n* Process of fitting weights to minimize loss on dataset is logistic regression\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\n* Multinomial naive Bayes classifier, makes simplifying assumption about how features interact\n* Represent text document as bag of words, unordered set and frequency only kept\n* Returns the maximum class based on probability\n* Use Bayes rule to transform this expression into probabilities that have useful properties, can break down conditionals into three other probabilities\n* Naive bayes is a generative model because we can have implicit assumption about how a document is generated, sample a class, then generate words by sampling from conditional probability\n* Compute most probable class given document with the prior probability P(c) of the class and the likelihood of the document P(d|c), c = argmax c P(D|C)P(C)\n* Need some simplifying assumptions to calculate, bag of words assumption says the position doesn’t matter in the document\n* Naive Bayes assumption is second, conditional independence is that probabilities are independent given the class c and can be naively computed as P(f1|c) * P(F2|c) * …\n* Consider word position by walking an index through each word position in the document\n* Done in log space to avoid underflow and increase speed\n* Classifiers that use linear combination of inputs to make classification decision are linear classifiers\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n\n* Use frequencies in data to get estimates of the two terms\n* Maximum likelihood estimate, sum all of the instances divided by all of the words\n* But if we get a word we’ve never seen before can have zero probability and get error\n* Laplace smoothing, add one to pretend we’ve seen every word more than we actually have\n* If we encounter words never before seen, we usually ignore them, can also ignore stop words completely, which are a class of very common words\n\n\n## SLP Chapter 4.3 Worked Example\n\n* Sentiment analysis, + is positive sentiment, - if negative\n* Word doesn’t occur in training set, drop it completely\n* Get a higher probability for the negative sentiment than the positive sentiment, so we predict negative class for the test sentence\n\n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n\n* Whether a word appears or not seems to matter more than frequency, can clip word counts in each document at one, binary multinomial naive Bayes\n* For each document we remove all duplicate words before concatenating into single document\n* Also have to deal with negation when looking at sentiment classifiers\n* May have insufficient labeled training data to train accurate naive Bayes classifiers using all words in training set, so have to derive positive and negative word features from sentiment lexicons\n* Add a feature that is counted whenever a word from the lexicon appears\n* If training data is sparse or not representative of test set, dense lexicon features instead of sparse individual word features may generalize better\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n\n* Naive Bayes doesn’t require our classifier use all words in training data as features\n* Spam detection, deciding if email is spam or appropriate piece of mail\n* Rather than use all words as features, predefine likely sets of words or phrases as features combined with features that really not purely linguistic\n* Language id - determining what language a piece of text is written in, most effective to use character n-grams or byte n-grams isnstead of word n-grams\n* Often trained on multilingual texts\n\n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n\n* Naive bayes classifiers can use any sort of feature\n* We only use individual word features and use all words in the text, then similar to language modeling\n* Naive bayes can be viewed as a set of class-specific unigram language models in which the model for each class instantiates a unigram language model\n* Model assigns probability to each sentence as well, positive model will assign higher probabilities to sentences given a positive sentiment\n* Once we multiply in prior of full naive bayes model a different classification decision may be made\n\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n\n* Gold labels are human defined labels that we are trying to match up to\n* Can build a confusion matrix for visualizing how an algorithm performs with respect to human gold labels using system output and gold labels and label each cell as positive outcomes\n* Can also calculate accuracy, percentage of observations labeled correctly, generally not used for text classification tasks\n* Precision measures percentage of items that system detected that are in fact positive, true positives / true positives + false positives\n* Recall measures the percentage of items actually present that were correctly identified by the system, true positives / true positives + false negatives\n* F-measure incorporates both aspects of precision and recall\n* Precision/recall has to be adjusted slightly if we have more than 2 classes\n* Macro averaging, compute performance for each class then average over all classes \n* Micro averaging, collect decisions for all classes into single confusion matrix then compute precision/recall from a table\n\n\n## SLP Chapter 4.8 Test sets and Cross-validation\n\n* Use development set to tune some parameters, decide what the best model is in general\n* Use of a dev set avoids overfitting the test set, but it may not be large enough to be representative\n* Do cross-validation instead, k disjoint partitions called fold made, each one used as a test set in a run of the models\n* Pick a fold, remaining k-1 folds used in training, then fold used as the test set\n* Average it out, can do 10 times for 10-fold cross validation\n* Ned the whole corpus to be blind, can’t examine what is going on because then would be looking at the test set\n* Common to Crete fixed training set and test set, then do 10-fold cross validation inside the training set\n\n\n## SLP Chapter 4.9 Statistical Significance Testing\n\n* Do statistical significance hypothesis testing to see if one system is better at describing something than another\n* Effect size, bigger value means model A is doing way better than B, smaller says only doing a little bit better\n* H0 is null hypothesis, says value s actually negative or 0 so A is not better than B\n* Calculate the p-value, can reject null hypothesis if below a certain threshold value and statistically significant\n* Two common non-parametric tests in NLP is approximate randomization and bootstrap test\n* Paired tests are those where we compare two sets of observations that are aligned\n* Bootstrapping refers to repeatedly drawing large number of smaller samples with replacement from original larger sample\n* Acts as one-sided empirical p-value\n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n\n* Representational harms caused by a system that demeans a social group and perpetuates negative stereotypes\n* Toxicity detection task of detecting hate speech and other kinds of toxic language, want to reduce societal harm but can backfire\n* False positives can lead to censoring of important issues\n* Ml models replicate and amplify biases they find in their training data, can be cause by labels, resources used, model architecture \n* Can release a model card that details the model to see if any issues are immediately apparent\n\n\n## Module 12 Lecture Notes\n\n* want to acquire model from data/experience, learning the parameters, structure, hidden concepts\n* spam classification, get a large collection of emails labeled by hand, want to predict labels of future emails. Features are attributes that make decision, words, text patterns, non-text details\n* digit recognition, get labeled collection of images, predict new digit images, attributes are pixels and shape patterns\n* have inputs x, predict labels y, spam detection, medical diagnosis, essay grading, fraud detection, etc.\n\nModel-based classification\n* build a model (like a Bayes net) where label and features are random variables, fill in observed features and query for distribution fo labels conditions on features\n* Naive bayes for ditigs assumes all features are independent effects of the labels, one variable for each pixel position, on/off, maps to feature vector\n* general naive bayes learns for y labels, P(Y, F1, ...Fn) = P(Y) mult_i P(F_i|Y), have to specify how features depend on the class\n* number of parameters linear in number of features\n* inference for naive bayes wants to compute posterior distribution over label variable, so we do the following:\n* 1) get joint probability of label and evidence for each of the labels Y, 2) sum them together to get probability of evidence, 3) normalize by taking step 1 / step 2\n* to use naive bayes, need inference method (probabilities P(Y) and P(Fi|Y), calculate P(Y|F1..Fn)), and estimates of local contiional probability tables (P(Y) over labels, P(Fi|Y) for each feature, called parameters of model)\n* Bag of words Naive bayes has word features Wi at position i, predict label conditioned on feature variables, assume they are conditionally independent, but now each Wi is identically distributed\n* in BOW, each position identically distributed, all positions share same contioinal probabilitiy, model is insensitive to word order/reordering\n* spam filtering model e.g. P(Y, W1 ... Wn) = P(Y) mult_i P(Wi|Y) \n\nConcepts/Definitions\n* Data is labeled instances, separated into training set, held out set, test set\n* Features: attributes and values characterizing each x input\n* experimentation cycle: learn the parameters on training set, tune hyperparameters on held out set, compute accuracy of test set (don't look at it ever)\n* evaluation, calculate accuracy as fraction of instances predicted correctly\n* overfitting, fitting the training data very closely but not generalizing well to the test data set. Posterios determined by relative probabilities, if overfit can go to infinite\n\nGeneralization and Overfitting\n* relative frequency parameters overfit training data, don't want to assign unseen events zero probability\n* want to generalize better to the test set, so we smooth or regularize the estimates\n\nParameter estimation\n* need to estimate the distribution of a random variable, difficult to decide so use training data\n* for each outcome x, look at the empirical rate of the value as P(x) = count(x) / total samples, then this is the estimate that maxmizes the likelihood of the data L(x, theta) = mult_i P_theta(xi)\n* relative frequencies give mazimum likelihood estimates, theta_ML = argmax_theta P(X|theta) = argmax_theta mult_i P_theta(Xi)\n* can do Laplace smoothing, pretend you saw every outcome one more time than you did\n* P_lap(x) = c(x) + 1 / sum_x [c(x) + 1] = c(x) + 1 / N + |X|\n* or could pretend you say it k extra times, P_lap(x) = c(x) + k / N + k|X|, k is the strength of the prior\n* for conditionals, smooth each condition separately: P_lap(x) = c(x,y) + k / c(y) + k|X|\n* could also do linear interpolation, get empirical P(x) from the data, make sure estimate P(X|Y) isn't too different from P(x) by P_lin(x|y) = a P_hat(x|y) + (1 - a)P_hat(x)\n* smoothing critical in real classification problems\n\nHeld out data tuning\n* we get two unknowns now, the parameter probabilities P(x,y), P(y) and the hyperparameters (type of smoothing, k, a, etc.)\n* learn paramters from training data, tune hyperparameters on this set, for each value of hyperparameters, train and test on held out set, choose best values and do final run on test data\n\nFeature errors\n* if we get errors, can sometimes need more features as words are not enough\n* add these new information sources as new variables in Naive bayes model\n* naive bayes can incorporate many features, but generally do better with homogenous cases of features\n\nBaselines\n* need a baseline as first step, see how hard task is and what a good accuracy entails\n* a weak baseline gives the most frequent label classifier, give the test set whatever label was most common, accuracy may be high or low, real research uses previous work as baselines\n* confidence of classifier comes from posterior over top label, confidence(x) = max_y P(y,x)\n* says hos sure the classifier is of a prediction, comes with any probabilistic model but no guarentee it is correct\n* weak calibration - higher confidences mean higher accuracy, strong calibration - confidence predicts the accuracy rates\n* bayes lets us do queries with causal probabilities, naive bayes takes all features to be independent, need smoothing and confidences helpful when you can get them\n\nPerceptrons\n* Loosely inspired by human neurons\n* linear classifiers, inputs are feature values, each feature with a weight and the sum if the activation\n* activation_w(x) = sum_i wi * fi(x) = w * f(x), if >= 0 output +1, < 0 output -1\n* in binary case, compare features to weight vector, trying to learn weight vectors from examples\n* have inputs, weights, weighted sum, activation function with theshold, output\n* output is 0 if sum_j wjxj <= theshold, 1 if sum_j wjxj > theshold\n* by varying weights and threshold we get different models of decisions\n* write sum_j wjxj as dot product of input vector and weight vector, w * x\n* move threshold to other side of inequality, define bias as the -1 * theshold b = -theshold\n* output is 0 if wx + b <= 0, 1 if wx + b > 0\n* have complex network of perceptrsont that make subtle decisions\n\nBinary Decision rule\n* in space of feature vectors, each example is a point, weight vector is hyperplane, one side +1, other -1\n* start with weights = 0, for each training instance classify with weights, if correct don't change anything, if wrong adjust the weight vector\n* y = +1 if w * f(x) >= 0, -1 if w * f(x) < 0, adjust w by w = w + y* * f, subtract if y* is -1, separable case very intuitive\n\nMulticlass decition rule\n* now we have weight vector for each class w_y, score activation of a class y is w_y * f(x), highest prediction score wins y = argmax_y w_y * f(X)\n* weights start at 0, for each training instance predict with current weights, if wrong lower score of wrong answer and raise score of right answer\n* w_y = w_y - f(x), w_y* = w_y* + f(x)\n\nPerceptron properties\n* separability: can be true if training set is perfectly separated but not always possible\n* convergence, if training is separable, perceptron eventually converges in binary case\n* maximum number of mistakes in binary case relatest to margin or degree of separability, mistakes < k / d^2\n\nIssues:\n* if data is not separable, weights may thrash, averaging weight vectors over time can help\n* can create mediocre generalization if a barely separating solution found\n* can overtrain if test/held out set accuracy rises then eventually falls, training accuracy keeps increasing\n\nFixing perceptrion:\n* adjust the weight update to lessen effects\n* MIRA* - choose update size thatfixes current mistake but minimzies change to w\n* min_w 0.5 * sum_y ||w_y - w'_y||^2, wy* * f(x) >= wy * f(x) + 1, +1 helps to generalize\n* have some value tao (t) to help fix. if we guess y instead of y* on example x with features f(x), wy = w'y - t(f(x)), wy* = w'y* + t(f(x))\n* minimum correcting update -> min_t ||tf||^2, end up with tao = ((w'y - w'y*) * f + 1) / (2f * f)\n* in practice, bad to make updates that are too large, example may be labeled incorrecrly or you may not have enough features\n* so we cap the maximum value of tao with a constant C, tao* = min(((w'y - w'y*) * f + 1) / (2f * f), C)\n* this will correspond to optimization assuming non-separable data, converges faster than perceptron and better on noisy data\n\nSupport Vector Machines\n* maximize the margin between the support vectors (only ones that matter), all other training examples ignored\n* find separator with max margin, optimize over all examples at once, good according to intuition, theory, pratice\n* min_w 0.5 ||w||^2, for all i, y - wy* * f(xi) >= wy * f(xi) + 1\n* data that is linearly separable is great for linear decision rules, but if its too hard, can map to higher dimensional space\n\n* Naive Bayes builds model training data, gives prediction probabiltiies, has assumptions about feature independence and passes through data once\n* perceptrons/MIRA make less assumptions about data, mistake-driven learning, multiple passes through data, and are often more accurate\n* original feature space can be mapped to higher dimensional feature space where training set is separable\n\nLogical functions:\n* networks of perceptons can compute any logical functions\n* build up these networks with NAND gates\n* XOR cannot be solved with one neural unit, but can be raise dto higher dimensional space to be solved linearly\n\nActivation functions\n* insstead of outputting wx + b as linear function, apply non-linear function to output called activation value\n* activation value is a, output of entire network is y, so we get y = a = f(wx + b)\n* three most commonly used ones are sigmoid function (1/1+e^-z), tanh, and ReLU (x if > 0, 0 if < 0)\n\n\n## Module 12 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\"\n",
        "* Train a classifier to make binary decision about class of new input observation \n",
        "* Logistic regression applies weights and bias term, bias term is also called the intercept, another real number added to the weighed inputs\n",
        "* Have the dot product of the logistic regression result z = wx + b\n",
        "* Need them between 0 and 1 so we apply the sigmoid function\n",
        "* Pushes the outliers between 0 and 1, and it is differentiable which helps in learning\n",
        "* P(y = 1) = sig(wx + b), P(y= 0) is 1 minus this\n",
        "* Logistic regression applied to all sorts of NLP tasks, any property of the input can be a feature\n",
        "* Period disambiguation, whether a period at the end of a sentence is part of a word or end of a sentence\n",
        "* Features generally designed by examine training set, helpful to build complex features that are combinations of more primitive ones\n",
        "* Feature interactions are designed by hand in logistic regression and Naive Bayes\n",
        "* Feature templates are often needed when we have large numbers of features\n",
        "* Representation learning, ways to learn features automatically in unsupervised way from input\n",
        "* Naive Bayes has overly strong conditional independence assumptions, logistic regression much more robust to correlated features \n",
        "* Many correlated features, logistic regression assigns more accurate probability, so works better on larger documents/databases\n",
        "* Nice Bayes will still often make correct classification decision and can work well on small datasets and short documents\n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\"\n",
        "* Know the correct labels in each instance\n",
        "* Want to learn parameters that make prediction as close as possible to true label\n",
        "* Need a metric for how close current label is to the goal, call it the loss of cost function, cross-entropy loss\n",
        "* Second is optimization algorithm for updating the weights to minimize loss function, gradient descent (stochastic) in particular\n",
        "\"\"\"\n",
        "slp_chapter5_3 = \"\"\"\n",
        "* Need loss function that prefers correct class labels of training examples to be more likely\n",
        "* Conditional maximum likelihood estimation, choose parameters web, that maximize log probability of true y labels in training data given observations x, leads to cross entropy loss\n",
        "* Want loss to be smaller if models estimate is close to correct, bigger if the model is confused\n",
        "* Higher predicted y, the better the classifier, lower the worse the classifier\n",
        "* Negative log of probability is a good loss metric since it goes from 0 to infinity, ensures correct answer is maximized, probability of incorrect answer is minimized since two sum to one\n",
        "\"\"\"\n",
        "slp_chapter5_4 = \"\"\"\n",
        "* Gradient descent, find the optimal weights and minimize the loss function e just defined\n",
        "* Loss function is parameterized by the weights, minimize loss averaged over all examples\n",
        "* Figures out in which direction the functions slope is rising the most steeply and go the other direction\n",
        "* Logistic regression the loss function is convex, so there is only a global minimum and can start from anywhere\n",
        "* Finds gradient of loss function at current point and moves in opposite direction\n",
        "* Gradient is multi-variable generalization of the slope, just a vector that expresses directional components of sharpest slope\n",
        "* Magnitude of amount to move is value of slope weighed by the learning rate\n",
        "* Higher learning rate means that we should move more on each step\n",
        "* Have a weight for each dimension\n",
        "* Derivative of cross entropy function can be used to estimate gradient for a single variable\n",
        "* Stochastic gradient descent minimizes loss function by computing gradient after each training example and moving parameters in right direction\n",
        "* If learning rate is too high we overshoot minus function, if too low it takes too small of steps and takes a long time\n",
        "* Usually have higher learning rate and slowly decrease it, hyper parameter chosen by us\n",
        "* Stochastic chooses single random example at each time point\n",
        "* In batch training compute gradient over entire dataset \n",
        "* Mini-batch trains on group of examples from the entire dataset, batch gradient descent then, computationally efficient \n",
        "\"\"\"\n",
        "slp_chapter5_5 = \"\"\"\n",
        "* Regularization, used to avoid overfitting, want model to generalize well from training data to unseen test set\n",
        "* Regularization term added to objective function, used to penalize large weights, setting where weights match training data perfectly but uses weights with high values will be penalized more than setting that matches data less well with smaller weights\n",
        "* L2 regularization is quadratic of weight values, uses square of L2 norm of weight values, same as Euclidean distance, ridge regression\n",
        "* L1 regularization is linear function of weight values, Manhattan distance, just sum of weights, lasso regression\n",
        "* Both commonly used in language processing, l2 easier to optimizer because of its simple derivative compared to l1 \n",
        "* L2 weights prefer weight vectors with many small weights, l1 prefers sparse solutions with larger weights but more weights set to 0\n",
        "* L1 viewed as Laplace prior on weights, l2 assuming weights are distributed acorns go Gaussian distribution with mean 0\n",
        "\"\"\"\n",
        "slp_chapter5_6 = \"\"\"\n",
        "* Multinomial logistic regression, if we need more than two classes, also called softmax regression\n",
        "* Target y can range over two classes, want to know probability of it belonging to a certain class\n",
        "* Uses generalization of sigmoid called softmax that compute probability between 0 and 1 for each class\n",
        "* Input to softmax is dot product between weight vector and input vector\n",
        "* Need separate weight vectors and biases for each of the K classes\n",
        "* Feature can be evidence for or against each individual class now\n",
        "* Sometimes make dependence explicitly and represent features themselves as function of input and class\n",
        "* Loss function for multinomial logistic regression generalizes loss function from 2 to K classes\n",
        "* Takes input and represents it as a one-hot vector, all values 0 except for term corresponding to true class\n",
        "* Cross entropy loss is log of output probability corresponding to correct class, call it negative log likelihood loss\n",
        "* Gradient is difference between value for true class k and probability the classifier outputs for class k weighted by value of input correspond to element of the weight\n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\"\n",
        "* Building block of neural network is single computational unit, takes real valued numbers as input and performs computation on them, produces output\n",
        "* Bias term and weighted sum of inputs, express as vector dot product\n",
        "* Apply nonlinear transformation f to z, referred to as activation\n",
        "* Sigmoid function, tanh, ReLU rectified linear unit all used as activatio functions\n",
        "* Sigmoid not used commonly as activation function, tanh almost always better, smoothly differentiable and mapping outlier values towards the mean\n",
        "* ReLU takes max of the value and 0, negative values set to zero then, nice properties since its close to linear\n",
        "* Gradients that are almost 0 cause error signal to get smaller and smaller until useless, vanishing gradient problem\n",
        "\"\"\"\n",
        "slp_chapter7_2 = \"\"\"\n",
        "* Single neural unit can’t compute XOR function\n",
        "* Perceptron is simple neural unit with binary output and does not have nonlinear activation function\n",
        "* Cannot linearly classify the output of XOR, decision boundary cannot cover every output, not linearly separable\n",
        "* Can be calculated with layered network of units\n",
        "* Weights learned automatically using error backpropagation algorithm\n",
        "* Hidden layers learn to form useful representations\n",
        "\"\"\"\n",
        "slp_chapter7_3 = \"\"\"\n",
        "* Feedforward network, multilayer network with connected units and no cycles, output of each layer passed to units in next layer with no passes back to previous layer\n",
        "* Also called multi-layer perceptrons\n",
        "* Input nodes, hidden nodes, output nodes\n",
        "* Hidden layer is formed of hidden units, take weighed sum of inputs and apply non-linearity\n",
        "* Each layer fully connected meaning takes inputs as outputs of previous layers and link between every pair of units in two adjacent layers\n",
        "* Multiply weight matrix by input vector, add bias vector, apply activation function\n",
        "* Output layer has a weight matrix, some models don’t include bias in output layer\n",
        "* Softmax function normalizing the vector of real values, convert to vector that encodes probability distribution\n",
        "* If we dont use nonlinear activation layers then resulting network is exactly equivalent to single layer network\n",
        "\"\"\"\n",
        "slp_chapter7_4 = \"\"\"\n",
        "* Use feedforward networks for NLP\n",
        "* Have two layer sentiment classifier, could be scalar feature inputs, output layer could have 2 or 3 nodes\n",
        "* Adding hidden layer allows network to represent non-linear interactions between features, may give better sentiment classifier\n",
        "* Representing words as word2vec embeddings as input representation\n",
        "* Use another algorithm to pretrain the input words before input feeding\n",
        "\"\"\"\n",
        "slp_chapter7_5 = \"\"\"\n",
        "* Language modeling: predicting upcoming words from prior word context, very important NLP task\n",
        "* Compared to n-grams, neural language models can handle much longer histories, generalize better over contexts of similar words, more accurate at word prediction\n",
        "* But much more complex, slower to train, less interpretable, so small tasks n-gram often still used more often\n",
        "* Takes input at time t a representation of some number of previous words and outputs probability distribution over next possible words\n",
        "* Then approximates probability of a word given entire prior context by approximating based on N previous words\n",
        "* Words in prior context represented as embeddings rather than just word identity as in n-gram models, allows them to generalize better to new data\n",
        "* Forward inference, given an input runs forward pass on network to produce probability distribution over possible outputs, next words in this case\n",
        "* One hot vector representation used, one dimension for each word in vocabulary and mostly zeros\n",
        "* Moving window can see N words into the past, multiply vectors by embedding matrix, concatenated to produce embedding layer, followed by hidden layer then output layer to produce probability distribution\n",
        "* Select three embeddings for E, multiply by W, multiply by U, apply softmax\n",
        "\"\"\"\n",
        "slp_chapter7_6 = \"\"\"\n",
        "* Want to learn parameters W and b for each layer to get as close as possible to true y\n",
        "* Use cross entropy loss for logistic regression, use gradient descent optimization algorithm\n",
        "* Need to know gradient of loss function, need to do error backpropagation or backward differentiation\n",
        "* Cross entropy loss in neural networks the same as logistic regression\n",
        "* Still use one hot vectors, can determine loss as negative log likelihood loss\n",
        "* Calculating gradient require partial derivative of the loss function with respect to each parameter\n",
        "* Backprop needed to update all weights, not just the ones in the final layer\n",
        "* Depends on notion. Of computation graphs, representations of process of computing mathematical expression where broken down into separate operations each as a node\n",
        "* Can easily compute the forward pass as before\n",
        "* Makes use of the chain rule to calculate backwards differentiation, pass gradients back from final node to all nodes in the graph\n",
        "* Takes an upstream gradient and computes local gradient then computes downstream gradient to pass on to earlier node\n",
        "* Also take derivatives of the activation functions\n",
        "* Optimization in neural networks is non-convex optimization problem, more complex than logistic regression\n",
        "* Dropout used as a form of regularization, also tuning hyper parameters is crucial, learning rate, mini batch size, model architecture, need to parallelize onto vector-based GPUs\n",
        "\"\"\"\n",
        "slp_chapter7_7 = \"\"\"\n",
        "* Training neural language model\n",
        "* For some tasks ok to freeze embedding layer with initial word2vec values, hold it constant and only modify W, U, and b, dont update embeddings E\n",
        "* Often would still like to refine embeddings while training network, important when taks places strong constraints on what makes a good representation for words\n",
        "* Want one single embedding dictionary shared among the words\n",
        "* Input as very long text, concatenating all sentences, starting with random weights and iteratively moving through text predicting each word, use cross entropy loss for each word\n",
        "* Classes are words in vocabulary, pick the highest predicted one\n",
        "* Gradient can then be calculated with backpropagation \n",
        "* Results in language omodelling and new set of embeddings E to be used as word representations in other tasks\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\"\n",
        "Logistic Regression\n",
        "* for the classifier we need training samples of M observations, input/output pairs (x,y)\n",
        "* also need 1) feature representation of the input as a vector of features [x1,x2,..xn], 2) classification function that computes estimated class y_hat with p(y|x), 3) objective function for learning with minimization of training examples, 4) algorithm for optimizing objective function\n",
        "\n",
        "Sentiment classifier\n",
        "* consider input observation x represented as vector of features, output can be 1 (positive) or 0 (negative), and we want to esimate the P(y = 1|x)\n",
        "* can use logistic regression to solve this by learning from training set with weights and bias term\n",
        "* same as before, z = sum_i wi*xi + b, or z = wx + b\n",
        "* apply the sigmoid function y = 1/(1+e^-z), so we get P(y = 1) = sigmoid(wx + b) = 1/(1+e^-(wx + b))\n",
        "* we have the probability from this, so if this P(y=1|x) > 0.5 we label it 1, otherwise 0, call 0.5 the decision boundary\n",
        "\n",
        "Learning in logistic regression\n",
        "* learn the parameters (weights and bias) with learning, which requires 1) objective or loss function that tells us distance between system and gold ouput, use cross-entropy loss, and 2) algorithm for optimizing objective function, we will use stochastic gradient descent to minimize the loss function\n",
        "* loss function needs to determine how close the classifier output is to the correct output, probabilities are to true 0 and 1\n",
        "* we want L(y_hat, y), how much y_hat differs from the true y, could use mean squared error L(y_hat, y) = 0.5(y_hat - y)^2\n",
        "\n",
        "Probabilistic classification\n",
        "* want a loss function that prefers correct class labels of training data\n",
        "* choose parameters w and b that maximize the log probabilities of the true labels in training data, loss function is negative log likelihood loss or cross entropy loss\n",
        "* for one observation x, want to maximize the probabilitiy of correct label p(y|x), p(y|x) = y_hat^y(1-y_hat)^(1-y)\n",
        "* if y = 1, then p(y|x) = y_hat, if y = 0 then p(y|x) = 1 - y_hat\n",
        "* change to logs stil mazimizing, log(p(y|x)) = log(y_hat^y(1-y_hat)^(1-y)) = ylog(y_hat) + (1 - y)log(1 - y_hat)\n",
        "* tells us what log likelihood is maximized, flip the sign since we want to minimize it\n",
        "* so we get cross entropy loss as Lce(y_hat, y) = -log(p(y|x)) = - [ylog(y_hat) + (1 - y)log(1 - y_hat)]\n",
        "* plug in definition of y_hat to get Lce(y_hat, y) = - [ylog(sig(wx + b)) + (1 - y)log(1 - sig(wx + b))]\n",
        "* want losses to be smaller if model's estimate is close to correct, and loss to be bigger if it gets confused\n",
        "* loss on all training examples: log p(training labels) = log mult_i->m p(yi|xi) = sum_i->m log p(yi|xi) = - sum_i->m Lce(y_hat_i|y_i)\n",
        "* use gradient descent to find good weight and bias settings by minimizing loss function\n",
        "\n",
        "Gradient descent\n",
        "* finds minimum of function by figuring out in which direction in parameter space the function's slope is rising the most and moving in oppostie direction\n",
        "* theta_hat = argmin_that 1/m * sum_i->m Lce(yi, xi; theta)\n",
        "* for logistic regression, loss function is convex, has just one minimum so no local minima to get stuck on, guarenteed to find the min\n",
        "* iteratively find minimum, take steps of learning rate until min reached\n",
        "* magnitude of the amount to move in GD is slope weight learned by learning rate\n",
        "* higher LR means we move w more on each step\n",
        "* w_t+1 = w_t - LR * d/dw f(x;w): new weight = old_weight - Lr * derivative of fn = slope\n",
        "* update each dimension wi, by taking partial derivatives with respect to L(f(x;theta), y)\n",
        "* final equation for updating theta based on gradient is theta_t+1 = theta_t + LR * L(f(x;theta), y)\n",
        "* to update theta, need definition for gradeitn L(f(x;theta), y)\n",
        "* for logistric regression, cross entropy loss is Lce(w,b) = -[ylog(sig(wx + b)) + (1 - y)log(1 - sig(wx + b))]\n",
        "* derivative of this function for one observation vector and a single weight is [sig(wx + b) - y]xj\n",
        "* difference between true y and estimate for x multiplied by corresponding input value xj\n",
        "* want to minimze the average loss, so have - 1/m sum_i->m yi log(sig(wxi + b)) + (1 - yi)log(1 - sig(wxi + b))\n",
        "* loss for batch or entire dataset is average loss over m examples, gradient fo multiple points is the sum of each individual gradient sum_i->m [sig(wxi + b) - yi]xj_i\n",
        "\n",
        "Stochastic gradient descent algorithm\n",
        "* for each training tupe (xi, yi) compute y_hat_i = f(xi, thetha)\n",
        "* compute the loss L(y_hat_i, y_u), calculate the gradient as g = L(f(xi;theta, yi)), then move other way, theta = theta - L.R. * g\n",
        "* derivative of function for single training example for single weight is [s(sx + b) - y]xj, get the gradient, move the opposite direction\n",
        "\n",
        "Mini-batch training\n",
        "* stochastic GD chooses a single random example and updates the weights on the example\n",
        "* can also do batch training which computes gradient over the entire dataset which can give a better estimate but takes longer to computer\n",
        "* can also do mini-batch training where the model is trained on small batches so all of them can be computed in parallel, then accumulate all the loss\n",
        "\n",
        "* overfitting can result in poor generaliztion and poor performance on the test set\n",
        "* if a feature only occurs in one class in logistic regression then it gets a high weight, can sometimes model noisy factors\n",
        "* add regularization to penalize large weights, added to the loss function\n",
        "* Lasso regression uses L1 regularization, ridge regression uses L2 regularization\n",
        "\n",
        "Multinomial logistric regression\n",
        "* if we want more than two classes, can extend to be several, for sentiment e.g. can extend to be positive, negative, or neutral\n",
        "* want to know the probability of y for each class c in C, p(y = c|x)\n",
        "* to get probabilities, have to do a generalization of the sigmoid function called softmax function\n",
        "* softmax(zi) = e^zj / sum_i->k e^zj for 1 <= i <= k\n",
        "* softmax takes input vector z = [z1,z2..zk] and outputs vector of values normalized to probabilities\n",
        "\n",
        "Perceptrons\n",
        "* developed in 50s/60s inspired by neuron\n",
        "* building block of neural networks is single computational unit, takes in real numbers as input and performs computation\n",
        "* cell body/dendrites like input, axon like neuron, output like terminal branches of axon\n",
        "* neural networks combine input layer, several hidden layers, output layer\n",
        "* logistic regression uses weight vecor and bias term like perceptrons\n",
        "* z = sum_i = wixi + b, or z = wx + b, transform with sigmoid function\n",
        "* have a step function, 1 if wx+ b > 0, 0 is wx + b <= 0\n",
        "* sigmoid neurons in place to prevent the weights in the network to suddenly flip output from 0 to 1\n",
        "* smoothness of sigmoid means small changes in weights and bias produce small changes in neuron output, can then choose small changes to achieve the desired output\n",
        "\n",
        "Feed forward neural network\n",
        "* simplest kind of NN, multilayer network where all units are fully connectwed with no cycles, outputs from previous layer are passed to the next layer and nothing is bassed back up\n",
        "* single hidden unit has parameters w and b, represent parameters for entire hidden layer by combining the weight vector wi and bias bi for each unit i into single W and b terms\n",
        "* hidden layer operations can now be done easily for the entire layer\n",
        "* 1) multiply weight matrix by input vector x, add bias vector b, apply sigmoid activation function, get h = sig(Wx + b)\n",
        "* output layer has weight matrix U, so intermediate output z is z = Uh, then get final output when the softmaxx is applied as y = softmax(z), normalizes output and turns it into probability distribution\n",
        "\n",
        "Backpropagation\n",
        "* derivatives only give correct updates for last layers, need to pass it back to earlier layers as well\n",
        "* backward differentiation which depends on computation graphs\n",
        "* these graphs represetn computing mathematical expressions by breaking them down into separate operations, each one as a node in a graph\n",
        "* calculate everything on the forward pass as usual, then use chain rule to do backwards differentiation, dL/da', dL/db, dL/dc\n",
        "* for composite function f(x) = u(v(x)), derivative of f(x) is df/dx = du/dv * dv/dx\n",
        "* for f(x) = u(v(w(x))), derivative of f(x) is df/dx = du/dv * dv/dw * dw/dx\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "12d7848b-4228-47f0-c210-39b1e5a96a47"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\n* Train a classifier to make binary decision about class of new input observation \n* Logistic regression applies weights and bias term, bias term is also called the intercept, another real number added to the weighed inputs\n* Have the dot product of the logistic regression result z = wx + b\n* Need them between 0 and 1 so we apply the sigmoid function\n* Pushes the outliers between 0 and 1, and it is differentiable which helps in learning\n* P(y = 1) = sig(wx + b), P(y= 0) is 1 minus this\n* Logistic regression applied to all sorts of NLP tasks, any property of the input can be a feature\n* Period disambiguation, whether a period at the end of a sentence is part of a word or end of a sentence\n* Features generally designed by examine training set, helpful to build complex features that are combinations of more primitive ones\n* Feature interactions are designed by hand in logistic regression and Naive Bayes\n* Feature templates are often needed when we have large numbers of features\n* Representation learning, ways to learn features automatically in unsupervised way from input\n* Naive Bayes has overly strong conditional independence assumptions, logistic regression much more robust to correlated features \n* Many correlated features, logistic regression assigns more accurate probability, so works better on larger documents/databases\n* Nice Bayes will still often make correct classification decision and can work well on small datasets and short documents\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n\n* Know the correct labels in each instance\n* Want to learn parameters that make prediction as close as possible to true label\n* Need a metric for how close current label is to the goal, call it the loss of cost function, cross-entropy loss\n* Second is optimization algorithm for updating the weights to minimize loss function, gradient descent (stochastic) in particular\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\n* Need loss function that prefers correct class labels of training examples to be more likely\n* Conditional maximum likelihood estimation, choose parameters web, that maximize log probability of true y labels in training data given observations x, leads to cross entropy loss\n* Want loss to be smaller if models estimate is close to correct, bigger if the model is confused\n* Higher predicted y, the better the classifier, lower the worse the classifier\n* Negative log of probability is a good loss metric since it goes from 0 to infinity, ensures correct answer is maximized, probability of incorrect answer is minimized since two sum to one\n\n\n## SLP Chapter 5.4 Gradient Descent\n\n* Gradient descent, find the optimal weights and minimize the loss function e just defined\n* Loss function is parameterized by the weights, minimize loss averaged over all examples\n* Figures out in which direction the functions slope is rising the most steeply and go the other direction\n* Logistic regression the loss function is convex, so there is only a global minimum and can start from anywhere\n* Finds gradient of loss function at current point and moves in opposite direction\n* Gradient is multi-variable generalization of the slope, just a vector that expresses directional components of sharpest slope\n* Magnitude of amount to move is value of slope weighed by the learning rate\n* Higher learning rate means that we should move more on each step\n* Have a weight for each dimension\n* Derivative of cross entropy function can be used to estimate gradient for a single variable\n* Stochastic gradient descent minimizes loss function by computing gradient after each training example and moving parameters in right direction\n* If learning rate is too high we overshoot minus function, if too low it takes too small of steps and takes a long time\n* Usually have higher learning rate and slowly decrease it, hyper parameter chosen by us\n* Stochastic chooses single random example at each time point\n* In batch training compute gradient over entire dataset \n* Mini-batch trains on group of examples from the entire dataset, batch gradient descent then, computationally efficient \n\n\n## SLP Chapter 5.5 Regularization\n\n* Regularization, used to avoid overfitting, want model to generalize well from training data to unseen test set\n* Regularization term added to objective function, used to penalize large weights, setting where weights match training data perfectly but uses weights with high values will be penalized more than setting that matches data less well with smaller weights\n* L2 regularization is quadratic of weight values, uses square of L2 norm of weight values, same as Euclidean distance, ridge regression\n* L1 regularization is linear function of weight values, Manhattan distance, just sum of weights, lasso regression\n* Both commonly used in language processing, l2 easier to optimizer because of its simple derivative compared to l1 \n* L2 weights prefer weight vectors with many small weights, l1 prefers sparse solutions with larger weights but more weights set to 0\n* L1 viewed as Laplace prior on weights, l2 assuming weights are distributed acorns go Gaussian distribution with mean 0\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n\n* Multinomial logistic regression, if we need more than two classes, also called softmax regression\n* Target y can range over two classes, want to know probability of it belonging to a certain class\n* Uses generalization of sigmoid called softmax that compute probability between 0 and 1 for each class\n* Input to softmax is dot product between weight vector and input vector\n* Need separate weight vectors and biases for each of the K classes\n* Feature can be evidence for or against each individual class now\n* Sometimes make dependence explicitly and represent features themselves as function of input and class\n* Loss function for multinomial logistic regression generalizes loss function from 2 to K classes\n* Takes input and represents it as a one-hot vector, all values 0 except for term corresponding to true class\n* Cross entropy loss is log of output probability corresponding to correct class, call it negative log likelihood loss\n* Gradient is difference between value for true class k and probability the classifier outputs for class k weighted by value of input correspond to element of the weight\n\n\n## SLP Chapter 7.1 Units\n\n* Building block of neural network is single computational unit, takes real valued numbers as input and performs computation on them, produces output\n* Bias term and weighted sum of inputs, express as vector dot product\n* Apply nonlinear transformation f to z, referred to as activation\n* Sigmoid function, tanh, ReLU rectified linear unit all used as activatio functions\n* Sigmoid not used commonly as activation function, tanh almost always better, smoothly differentiable and mapping outlier values towards the mean\n* ReLU takes max of the value and 0, negative values set to zero then, nice properties since its close to linear\n* Gradients that are almost 0 cause error signal to get smaller and smaller until useless, vanishing gradient problem\n\n\n## SLP Chapter 7.2 The XOR problem\n\n* Single neural unit can’t compute XOR function\n* Perceptron is simple neural unit with binary output and does not have nonlinear activation function\n* Cannot linearly classify the output of XOR, decision boundary cannot cover every output, not linearly separable\n* Can be calculated with layered network of units\n* Weights learned automatically using error backpropagation algorithm\n* Hidden layers learn to form useful representations\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n\n* Feedforward network, multilayer network with connected units and no cycles, output of each layer passed to units in next layer with no passes back to previous layer\n* Also called multi-layer perceptrons\n* Input nodes, hidden nodes, output nodes\n* Hidden layer is formed of hidden units, take weighed sum of inputs and apply non-linearity\n* Each layer fully connected meaning takes inputs as outputs of previous layers and link between every pair of units in two adjacent layers\n* Multiply weight matrix by input vector, add bias vector, apply activation function\n* Output layer has a weight matrix, some models don’t include bias in output layer\n* Softmax function normalizing the vector of real values, convert to vector that encodes probability distribution\n* If we dont use nonlinear activation layers then resulting network is exactly equivalent to single layer network\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n\n* Use feedforward networks for NLP\n* Have two layer sentiment classifier, could be scalar feature inputs, output layer could have 2 or 3 nodes\n* Adding hidden layer allows network to represent non-linear interactions between features, may give better sentiment classifier\n* Representing words as word2vec embeddings as input representation\n* Use another algorithm to pretrain the input words before input feeding\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n\n* Language modeling: predicting upcoming words from prior word context, very important NLP task\n* Compared to n-grams, neural language models can handle much longer histories, generalize better over contexts of similar words, more accurate at word prediction\n* But much more complex, slower to train, less interpretable, so small tasks n-gram often still used more often\n* Takes input at time t a representation of some number of previous words and outputs probability distribution over next possible words\n* Then approximates probability of a word given entire prior context by approximating based on N previous words\n* Words in prior context represented as embeddings rather than just word identity as in n-gram models, allows them to generalize better to new data\n* Forward inference, given an input runs forward pass on network to produce probability distribution over possible outputs, next words in this case\n* One hot vector representation used, one dimension for each word in vocabulary and mostly zeros\n* Moving window can see N words into the past, multiply vectors by embedding matrix, concatenated to produce embedding layer, followed by hidden layer then output layer to produce probability distribution\n* Select three embeddings for E, multiply by W, multiply by U, apply softmax\n\n\n## SLP Chapter 7.6 Training Neural Nets\n\n* Want to learn parameters W and b for each layer to get as close as possible to true y\n* Use cross entropy loss for logistic regression, use gradient descent optimization algorithm\n* Need to know gradient of loss function, need to do error backpropagation or backward differentiation\n* Cross entropy loss in neural networks the same as logistic regression\n* Still use one hot vectors, can determine loss as negative log likelihood loss\n* Calculating gradient require partial derivative of the loss function with respect to each parameter\n* Backprop needed to update all weights, not just the ones in the final layer\n* Depends on notion. Of computation graphs, representations of process of computing mathematical expression where broken down into separate operations each as a node\n* Can easily compute the forward pass as before\n* Makes use of the chain rule to calculate backwards differentiation, pass gradients back from final node to all nodes in the graph\n* Takes an upstream gradient and computes local gradient then computes downstream gradient to pass on to earlier node\n* Also take derivatives of the activation functions\n* Optimization in neural networks is non-convex optimization problem, more complex than logistic regression\n* Dropout used as a form of regularization, also tuning hyper parameters is crucial, learning rate, mini batch size, model architecture, need to parallelize onto vector-based GPUs\n\n\n## SLP Chapter 7.7 Training the neural language model\n\n* Training neural language model\n* For some tasks ok to freeze embedding layer with initial word2vec values, hold it constant and only modify W, U, and b, dont update embeddings E\n* Often would still like to refine embeddings while training network, important when taks places strong constraints on what makes a good representation for words\n* Want one single embedding dictionary shared among the words\n* Input as very long text, concatenating all sentences, starting with random weights and iteratively moving through text predicting each word, use cross entropy loss for each word\n* Classes are words in vocabulary, pick the highest predicted one\n* Gradient can then be calculated with backpropagation \n* Results in language omodelling and new set of embeddings E to be used as word representations in other tasks\n\n\n## Module 13 Lecture Notes\n\nLogistic Regression\n* for the classifier we need training samples of M observations, input/output pairs (x,y)\n* also need 1) feature representation of the input as a vector of features [x1,x2,..xn], 2) classification function that computes estimated class y_hat with p(y|x), 3) objective function for learning with minimization of training examples, 4) algorithm for optimizing objective function\n\nSentiment classifier\n* consider input observation x represented as vector of features, output can be 1 (positive) or 0 (negative), and we want to esimate the P(y = 1|x)\n* can use logistic regression to solve this by learning from training set with weights and bias term\n* same as before, z = sum_i wi*xi + b, or z = wx + b\n* apply the sigmoid function y = 1/(1+e^-z), so we get P(y = 1) = sigmoid(wx + b) = 1/(1+e^-(wx + b))\n* we have the probability from this, so if this P(y=1|x) > 0.5 we label it 1, otherwise 0, call 0.5 the decision boundary\n\nLearning in logistic regression\n* learn the parameters (weights and bias) with learning, which requires 1) objective or loss function that tells us distance between system and gold ouput, use cross-entropy loss, and 2) algorithm for optimizing objective function, we will use stochastic gradient descent to minimize the loss function\n* loss function needs to determine how close the classifier output is to the correct output, probabilities are to true 0 and 1\n* we want L(y_hat, y), how much y_hat differs from the true y, could use mean squared error L(y_hat, y) = 0.5(y_hat - y)^2\n\nProbabilistic classification\n* want a loss function that prefers correct class labels of training data\n* choose parameters w and b that maximize the log probabilities of the true labels in training data, loss function is negative log likelihood loss or cross entropy loss\n* for one observation x, want to maximize the probabilitiy of correct label p(y|x), p(y|x) = y_hat^y(1-y_hat)^(1-y)\n* if y = 1, then p(y|x) = y_hat, if y = 0 then p(y|x) = 1 - y_hat\n* change to logs stil mazimizing, log(p(y|x)) = log(y_hat^y(1-y_hat)^(1-y)) = ylog(y_hat) + (1 - y)log(1 - y_hat)\n* tells us what log likelihood is maximized, flip the sign since we want to minimize it\n* so we get cross entropy loss as Lce(y_hat, y) = -log(p(y|x)) = - [ylog(y_hat) + (1 - y)log(1 - y_hat)]\n* plug in definition of y_hat to get Lce(y_hat, y) = - [ylog(sig(wx + b)) + (1 - y)log(1 - sig(wx + b))]\n* want losses to be smaller if model's estimate is close to correct, and loss to be bigger if it gets confused\n* loss on all training examples: log p(training labels) = log mult_i->m p(yi|xi) = sum_i->m log p(yi|xi) = - sum_i->m Lce(y_hat_i|y_i)\n* use gradient descent to find good weight and bias settings by minimizing loss function\n\nGradient descent\n* finds minimum of function by figuring out in which direction in parameter space the function's slope is rising the most and moving in oppostie direction\n* theta_hat = argmin_that 1/m * sum_i->m Lce(yi, xi; theta)\n* for logistic regression, loss function is convex, has just one minimum so no local minima to get stuck on, guarenteed to find the min\n* iteratively find minimum, take steps of learning rate until min reached\n* magnitude of the amount to move in GD is slope weight learned by learning rate\n* higher LR means we move w more on each step\n* w_t+1 = w_t - LR * d/dw f(x;w): new weight = old_weight - Lr * derivative of fn = slope\n* update each dimension wi, by taking partial derivatives with respect to L(f(x;theta), y)\n* final equation for updating theta based on gradient is theta_t+1 = theta_t + LR * L(f(x;theta), y)\n* to update theta, need definition for gradeitn L(f(x;theta), y)\n* for logistric regression, cross entropy loss is Lce(w,b) = -[ylog(sig(wx + b)) + (1 - y)log(1 - sig(wx + b))]\n* derivative of this function for one observation vector and a single weight is [sig(wx + b) - y]xj\n* difference between true y and estimate for x multiplied by corresponding input value xj\n* want to minimze the average loss, so have - 1/m sum_i->m yi log(sig(wxi + b)) + (1 - yi)log(1 - sig(wxi + b))\n* loss for batch or entire dataset is average loss over m examples, gradient fo multiple points is the sum of each individual gradient sum_i->m [sig(wxi + b) - yi]xj_i\n\nStochastic gradient descent algorithm\n* for each training tupe (xi, yi) compute y_hat_i = f(xi, thetha)\n* compute the loss L(y_hat_i, y_u), calculate the gradient as g = L(f(xi;theta, yi)), then move other way, theta = theta - L.R. * g\n* derivative of function for single training example for single weight is [s(sx + b) - y]xj, get the gradient, move the opposite direction\n\nMini-batch training\n* stochastic GD chooses a single random example and updates the weights on the example\n* can also do batch training which computes gradient over the entire dataset which can give a better estimate but takes longer to computer\n* can also do mini-batch training where the model is trained on small batches so all of them can be computed in parallel, then accumulate all the loss\n\n* overfitting can result in poor generaliztion and poor performance on the test set\n* if a feature only occurs in one class in logistic regression then it gets a high weight, can sometimes model noisy factors\n* add regularization to penalize large weights, added to the loss function\n* Lasso regression uses L1 regularization, ridge regression uses L2 regularization\n\nMultinomial logistric regression\n* if we want more than two classes, can extend to be several, for sentiment e.g. can extend to be positive, negative, or neutral\n* want to know the probability of y for each class c in C, p(y = c|x)\n* to get probabilities, have to do a generalization of the sigmoid function called softmax function\n* softmax(zi) = e^zj / sum_i->k e^zj for 1 <= i <= k\n* softmax takes input vector z = [z1,z2..zk] and outputs vector of values normalized to probabilities\n\nPerceptrons\n* developed in 50s/60s inspired by neuron\n* building block of neural networks is single computational unit, takes in real numbers as input and performs computation\n* cell body/dendrites like input, axon like neuron, output like terminal branches of axon\n* neural networks combine input layer, several hidden layers, output layer\n* logistic regression uses weight vecor and bias term like perceptrons\n* z = sum_i = wixi + b, or z = wx + b, transform with sigmoid function\n* have a step function, 1 if wx+ b > 0, 0 is wx + b <= 0\n* sigmoid neurons in place to prevent the weights in the network to suddenly flip output from 0 to 1\n* smoothness of sigmoid means small changes in weights and bias produce small changes in neuron output, can then choose small changes to achieve the desired output\n\nFeed forward neural network\n* simplest kind of NN, multilayer network where all units are fully connectwed with no cycles, outputs from previous layer are passed to the next layer and nothing is bassed back up\n* single hidden unit has parameters w and b, represent parameters for entire hidden layer by combining the weight vector wi and bias bi for each unit i into single W and b terms\n* hidden layer operations can now be done easily for the entire layer\n* 1) multiply weight matrix by input vector x, add bias vector b, apply sigmoid activation function, get h = sig(Wx + b)\n* output layer has weight matrix U, so intermediate output z is z = Uh, then get final output when the softmaxx is applied as y = softmax(z), normalizes output and turns it into probability distribution\n\nBackpropagation\n* derivatives only give correct updates for last layers, need to pass it back to earlier layers as well\n* backward differentiation which depends on computation graphs\n* these graphs represetn computing mathematical expressions by breaking them down into separate operations, each one as a node in a graph\n* calculate everything on the forward pass as usual, then use chain rule to do backwards differentiation, dL/da', dL/db, dL/dc\n* for composite function f(x) = u(v(x)), derivative of f(x) is df/dx = du/dv * dv/dx\n* for f(x) = u(v(w(x))), derivative of f(x) is df/dx = du/dv * dv/dw * dw/dx\n\n\n\n## Module 13 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\"\n",
        "* Predict next word in sequence given preceding context\n",
        "* Can assign conditional probability to every possible next word, get distributions over entire vocabulary, can also assign probabilities to entire sequences by using conditional probabilities with chain rule\n",
        "* Evaluate on how well they do on unseen text, good models assign higher probabilities to unseen data\n",
        "* Use perplexity to measure quality of a language model, inverse probability that theta assigns to the test set normalized by test set length\n",
        "* Can use the chain rule again to expand computation of probability of test set\n",
        "\"\"\"\n",
        "slp_chapter9_2 = \"\"\"\n",
        "* Recurrent neural network is any network that contains cycle within network connections, so values of some unit are directly or indirectly dependent on earlier outputs as input\n",
        "* Difficult to reason and train, very effective when applied to language\n",
        "* Simple RNNs lead to LSTM networks\n",
        "* Input multiplied by weight matrix and passed through non-linear activation function to compute values for hidden unit layer, then used to calculate output \n",
        "* Sequences are processed by presenting one item to the network at a time\n",
        "* Link augments input to computation at hidden layer with value of hidden layer from preceding point in time\n",
        "* Hidden layer from previous step is like a memory unit, informs later decisions, context in previous hidden layer can include information extending to beginning of sequence\n",
        "* Can appear more complex than non-recurrent architectures but are not that different, still feed forward but new set of weights tell what to do with previous context\n",
        "* Trained with back propagation still\n",
        "* Nearly identical to feedforward networks with forward inference\n",
        "* Use training set, loss function, back propagation to get gradients to adjust weights\n",
        "* Have extra weights from previous hidden layer\n",
        "* Back-propagation requires two passes for training of weights, second pass we process sequences in reverse, back propagation through time\n",
        "\"\"\"\n",
        "slp_chapter9_3 = \"\"\"\n",
        "* As language models, RNN process input sequence one word at a time to predict next word from current word and previous hidden state\n",
        "* Dot babe limited context problem as n-grams since they can represent information in hidden states\n",
        "* Forward inference the same, input sequence is series of word embeddings represented as one-hot vector and output is vector that represents probability distribution over vocabulary\n",
        "* Model uses embedding matrix to retrieve embedding for current word and combines with hidden layer to get new hidden layer, generates output layer fed through softmax to get probability\n",
        "* Output is set of scores over vocabulary given evidence in hidden layer\n",
        "* Probability of entire sequence is product of probabilities of each item in sequence\n",
        "* Train to minimize error in predicting true next word in training sequence using cross entropy loss\n",
        "* Always give the model the correct history sequence to predict the next word, teacher forcing\n",
        "* weights adjusted to minimize CE loss ver training sequence with gradient descent\n",
        "* Final layer provides way to score likelihood of each word given evidence present in final hidden layer of network\n",
        "* Weight tying is method that uses single set of embeddings at input and softmax layer to reduce redundancy, lower dimensionality\n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\"\n",
        "* Representation of word is string of letters or index in vocabulary list\n",
        "* Lexical semantics define some of the ways we set up words for our NLP\n",
        "* Lemma is the focus word, also called citation form, mouse is lemma for mice e.g.\n",
        "* Each lemma can have multiple meanings, call each one a word sense, can be polysemous and have multiple meanings, makes interpretation difficult\n",
        "* Word sense disambiguation is process of deducing word from context\n",
        "* Synonyms are important to capture, if they have the same propositional meaning\n",
        "* Principle of contrast says difference in linguistic form always associated with difference in meaning\n",
        "* Words can have lots of similar words that are not similes \n",
        "* Words can also be related in ways other than similarity, word association\n",
        "* Another relation is if two words belong to the same semantic field, have structured relations between them\n",
        "* Semantic frames are set of words that denote perspectives or participants in types of event\n",
        "* Words can have different meanings or connotations as well, mean different things with same word, sentiment can reflect positive or negative words\n",
        "\"\"\"\n",
        "slp_chapter6_2 = \"\"\"\n",
        "* Vector semantics are the standard way to represent word meaning in NLP\n",
        "* Define meaning of word by distribution in language use, meaning neighboring words and grammatical environments \n",
        "* Represent a word in multidimensional semantic space that is derived from distributions of word neighbors, vectors for representing these words called embeddings, sometimes only applied to word2vec dense vectors\n",
        "* Mapping of from one space or structure to another, can project down. To 2d space to see the relationships between words\n",
        "* Classifiers can assign sentiment as long as it sees words with similar meanings\n",
        "* Tf-idf model has meaning of word defined by simple function of counts of nearby words, gives a sparse representation\n",
        "* Word2vec has short dense vectors with useful semantic properties\n",
        "* Can also use cosine to compute semantic similarity between too words/sentences\n",
        "\"\"\"\n",
        "slp_chapter6_3 = \"\"\"\n",
        "* Vector or distributional models are based on co-occurence matrix often, how often words co-occur\n",
        "* Term-document matrix, each row represents a word in vocabulary, each column a document from collection of documents\n",
        "* Each cell is number of times particular word occurs in particular document\n",
        "* First defined as part of vector space model of information retrieval\n",
        "* Vector space is collection of vectors characterized by dimension\n",
        "* Ordering of numbers in vector space indicate different meaningful dimension where documents vary\n",
        "* Vector for a document is a point in V dimension space\n",
        "* Two similar documents have similar words, column vectors will be similar as well\n",
        "* V rows for number of words, and D columns for each document in collection\n",
        "* Information retrieval is process of finding document from documents in some collection that matche a query\n",
        "* Represent query as vector, node to compare two vectors for similarity\n",
        "* Could use term-term matrix or word-word matrix or term-context matrix, columns are labeled by words instead of documents so V x V, each cell records number of times the row and column word co-occur In some context in training set\n",
        "* Context could also be document, most common to use smaller contexts to get neighboring words\n",
        "* Size of vocabulary can be tens of thousands of words, most of them are zero so we have sparse vector representation, effective algorithms exist to store them\n",
        "\"\"\"\n",
        "slp_chapter6_4 = \"\"\"\n",
        "* Measure similarity between two target words that takes in two vectors, use cosine of the angle between the vectors\n",
        "* Cosine based on dot product operator, called inner product\n",
        "* Raw dot product favors long vectors, higher is vector is longer with higher values in each direction\n",
        "* We can normalize it by diving the dot product by the lengths of each of the two vectors, normalized dot product is the same as the cosine of the angle between the two vectors\n",
        "* Can pre-normalize the vector, divide by its length, create a unit vector of length 1\n",
        "* Then compute unit vector and take cosine which is equivalent to dot product\n",
        "* Cosine ranges from 1 for vectors pointing in same direction, 0 for orthogonal vectors, -1 for pointing in opposite direction\n",
        "\"\"\"\n",
        "slp_chapter6_8 = \"\"\"\n",
        "* Word2vec, represent words as short dense embeddings, work better in every NLP task than dense vectors\n",
        "* Skip-gram with negative sampling is one of two algorithms in word2vec, static embeddings meaning method learns one fixed embedding for each word in the vocabulary\n",
        "* Dynamic contextual embeddings like BERT are separate class\n",
        "* Can use running text as implicit supervised training data for a classifier, word that appears near the target is the gold standard, call this self supervision since we don’t need to label anything \n",
        "* Simplifies task compared to neural networks, and simplifies architecture\n",
        "* Treat word and neighboring context as positive examples, randomly sample other words as negative samples, use logistic regression to train a classifier to distinguish, and use weights as embeddings\n",
        "* Given tuple of target word and candidate context word, get probability that it is a real context word, base probability on embedding similarity\n",
        "* Word likely to occur near target if embedding vector is similar to target embedding\n",
        "* Two vectors are similar if they have a high dot product, sue sigmoid function to turn it into a probability\n",
        "* Says all context words are independent so we can multiply their probabilities, need embeddings for each target word and context word in vocabulary \n",
        "* Stores two embeddings for each word, one for word as target and one as context\n",
        "* Takes input paragraph of text and vocabulary size B, assigns random embedding vector to each word then shifts embeddings to be more like embeddings in training set\n",
        "* Noise words can be chosen accord to weighted unigram frequency with alpha as a weight\n",
        "* Maximize similarity of target/context word pairs from positive examples and minimize similarity of negative example pairs\n",
        "* Minimize loss with stochastic gradient descent\n",
        "* Learns target embedding and context embedding, can add them together or throw away C matrix and look at target matrix\n",
        "* Fasttext does better with unknown words by using subword models and generalizes better to unknown words\n",
        "\"\"\"\n",
        "slp_chapter6_10 = \"\"\"\n",
        "* Size of context window used to collect counts is relevant to both tf-idf vectors and dense word2vec vectors\n",
        "* Shorter context dings lead to representations that are more syntactic since coming from nearby words, semantically similar with same parts of speech\n",
        "* Long context windows, highest cosine words to target word are topically related but not similar\n",
        "* First order co-occurrence if they are typically nearby each other, also called syntagmatic association\n",
        "* Two words have second-order co-occurrence if they have similar neighbors, called paradigmatic associated\n",
        "* Ability to capture relational meanings also important\n",
        "* Parallelogram model for solving simple analogy problems can be constructed \n",
        "* Sparse vector models of meaning could solve such analogy problems, better success with word2vec or GLoVe vectors\n",
        "* Want to find the argmin distance x of a* - a + b\n",
        "* Closest value returned by parallelogram model is Sally not b* but one of the 3 inputs words or morphological variants, so much be explicitly excluded\n",
        "* Doesn’t work well for other relations besides frequent words, small instances, and certain relations\n",
        "* Can also use embeddings to study how meaning changes over time\n",
        "\"\"\"\n",
        "slp_chapter6_11 = \"\"\"\n",
        "* Embeddings can produce implicit biases and stereotypes latent in text\n",
        "* Embedding analogies can exhibit gender stereotypes in word2vec embeddings trained on news text\n",
        "* Allocations harm, system allocates resources unfairly to different groups\n",
        "* Embeddings also amplify bias, gendered terms become even more gendered in space compared to input text statistics\n",
        "* Can also encode implicit associations that are property of human reasoning\n",
        "* Representational harm is harm caused by system demeaning or ignoring some social groups\n",
        "* Recent research focused on transforming embedding space to remove gender stereotypes, debasing may reduce bias but hard to eliminate since is it in training text\n",
        "* Historical embeddings also being used to measure biases in the bast\n",
        "* Can document historical gender biases and racial stereotypes\n",
        "\"\"\"\n",
        "slp_chapter6_12 = \"\"\"\n",
        "* Most important evaluation metric is extrinsic evaluation on tasks, seeing if performance improves over some other model\n",
        "* Most common metric is similarity, correlation between algorithms word similarity and ratings assigned by humans\n",
        "* Slightly more realistic metrics are intrinsic similarity tasks that include context, they offer richer evaluation scenarios\n",
        "* Analogy task is another task used for evaluation\n",
        "* All embeddings suffer from variability, may produce different results from the same dataset, individual documents may strongly impact embeddings\n",
        "* Train multiple embeddings with bootstrap sampling over documents and average results\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\"\n",
        "Language models\n",
        "* estimate the probability of a sentence containing a word sequence w1:n as P(w1:n) + mult_i->n P(wi|wi-k:i-1)\n",
        "* need to estimate probability of P(wi+1|wk-1;1) from large paragraph, p_hat(MLE)(wi+1 = m|wi-k:i) = #(wi-k;i+1)/(wi-k:i), p_hat(add - a)(wi+1 = m|wi-k:i) = #(wi-k;i+1) + a/(wi-k:i) + a|V|\n",
        "* neural language models have advantages over nn-gram LMs, 1) don't need smomthing, 2) can handle much longer histories, 3) can generalize over contexts of similar words, 4) tend to have higher prediction accuracy\n",
        "* can be slower to train than traditional n-gram Lms though\n",
        "\n",
        "Neural Language models\n",
        "* first associate each word in vocabulary with vector-representation and create notion of similarity as a result\n",
        "* then express joint probability function of word sequence in terms of word vectors for words in sequewnce\n",
        "* then learn the word vectors and the parameters of the function\n",
        "* word vectors will be low dimensional dense vectors, probability function expressed as product of conditional probabilites of next word given predvious word using feed forward neural network described earlier\n",
        "* input is k-gram of words w1:k, output is probability distribution over next word, k context words treated as word window with each word as an embedded vector\n",
        "* input vector x concatenates v(w) for each of k words, v(w) in R^dw, x = [v(w1);v(w2);..v(wk)]\n",
        "* fed into a neural network with one ore more hidden layers\n",
        "* input layer is one hot vectors with word as 1, then projection layer, hidden layer, output layer\n",
        "* training examples are word k-grams from paragraph, identities of first k-1 words used as features, last word is target label, trained using cross-entropy loss\n",
        "* better results with better preplexity scores than n-gram LMs, larger N to scale to since parameters are associated with individual words, and they generalize across contexts well, by-product are word embeddings\n",
        "\n",
        "Language modeling\n",
        "* want to find a function that returns the joint probability again\n",
        "* this is difficult because of the curse of dimensionality (too many parameters to estimate) and we can fail to generalize related words/word sequences that we have observed in n-gram-based models\n",
        "* sparse statistics just too high to estimate from the data\n",
        "* in LMs we use chain rule to get conditional probability of next word in sequence given all of the previous words, P(w1w2w3..wt) = sum_t->T P(wt|w1..wt-1)\n",
        "* probability of the next word only depends on previous n - 1 words, smaller n makes it easier to get a probability from data\n",
        "* can construct probability tables to look up probs of words given histories of observed sequences\n",
        "* but if we have unseen combination of n words, either back off or smooth/interpolate to stitch together short sequences of observed words\n",
        "* try generalizing instead, take a sentence and assign probabilities to similar sentence\n",
        "\n",
        "Neural probabilistic LM\n",
        "* use vector space model where words are vectors with real values, define a function that returns join probability of words in sequence based on previous seqeunce, learn word representations and probability function from data\n",
        "* given a training set w1..wt, learn f(w1..wt) = P(wt|w1..wt-1), subject to giving high probability to unseen text/dev set, create probability distribution so we can take product of conditonal probabilities to get joint ones\n",
        "* create mapping function C from any word in V onto R^M, store in VxM matrix and initialize with singular value decomposition\n",
        "* neural architecture is a function g tha mapes sequence of word vectors onto probability distribution over vocab V, g(C(wt-n) ..C(wt-1)) p P(wt|wt-n ..wt-1)\n",
        "* when 50 dimenstional vectors result from training LM prohected to 2 dimensions, see similar words close together\n",
        "* ELMo, GPT, BERT, GPT-2 the current state of the art models\n",
        "\n",
        "Word meaning\n",
        "* how to represent meaning of word, in Ngram LMs did so as string of letters or index in vocabulary list\n",
        "* want them to encode synonyms (similar meaning), antonyms (opposite meaning), connotations (positive or negative words), semantic roles, support for entailment (e.g. dog is proper subset of animal)\n",
        "* leximal semantics used to be trees like wordnet, now use more vector space models\n",
        "* most words don't have many synonyms but can use similar words instead for many applications\n",
        "* from a lot of context words humans can guess what words mean, so algorithm calls two words similar if their word contexts are similar\n",
        "* initially, vector space models were developed with each document as point in space, user query is pseudo document and represented point in same space, retrieved documents that were close to it\n",
        "\n",
        "Term-Document matrix\n",
        "* each column is a document, each row is a term, value in a cell is how often that word appeared in the document\n",
        "* length of document vector is size of vocabulary, document vectors can be sparse, then can measure similarity between two documents by comparing columns\n",
        "* word similarity lets you do plagiarism detection\n",
        "\n",
        "Vector comparisons\n",
        "* can look at two documents and terms in each, calculate euclidean distance between them, want to use L2 length normalization (||u|| = sqrt(sum_i->n ui^2)) and divde each vector by its L2 length\n",
        "* can also do cosine difference, does the L2 normalization as well. 1 - sqrt(sum_i->n ui * vi) / (sqrt(sum_i->n ui^2) * sqrt(sum_i->n vi^2))\n",
        "* the cosine angle will tell us the similarity between two terms\n",
        "\n",
        "Term-term matrix\n",
        "* aka term-context matrix, length of vector is now |V| instead of number of documents, value in a cell indicates how often word appears in context window surrounding other word\n",
        "* context windows are w-2, w-1, target_word, w+1, w+2, list how many times this takes place\n",
        "* can occur in window of +/- 2 words in same sentence in same document, instead of windows can use more complext contexts like dependency patterns, languages often have long distance dependencies\n",
        "* use the syntax to define the word's context\n",
        "* raw word frequency not a great measure of association, skewed with more common words like the, of, or, rather have measure of context word as particularly informative about target word\n",
        "\n",
        "TF-IDF\n",
        "* term frequency * inverse document frequency (how often word occured in document * 1 over number of documents it occurred in)\n",
        "* can have spare and dense vectors, co-occurrence matrices weighed by Tf-IDF can be long and sparse, would rather have short and dense vectors\n",
        "\n",
        "Dense vectors\n",
        "* to get them, train a classifier\n",
        "* treat targed word and neighboring context as positive examples, randomly sample other words to get negative samples, use logistic regression to train classifiedr to distinguish and weights as the embeddings\n",
        "* short vectors easier to use as features in ML, may generalize better than strong explicit counts\n",
        "* may do better at capturing synonyms, work better in practice\n",
        "\n",
        "Word2Vec\n",
        "* learn embeddings as part of word prediction process, train classifier to predict neighboring words, learn dense embeddings for words in paragraphs\n",
        "* fast and easy to train, availabe online with pretrained embeddings\n",
        "* predict neighboring word in context window of 2c of surrounding words, [wt-2, wt-1, wt+1, wt+2], use negative sampling for training - want prediction of target words to be high and irrelevant words to be low\n",
        "* inspired by neural nets, nearest neighbors are pretty good, embeddings can capture relational meanings\n",
        "* many possible models, matrix types (term-document, term-context, pattern-pair), reweighting (length norm, TF-IDF, PPMI, probabilities), comparisons (cosine, manhattan, jaccard, KL divergence, JS distance, DICE), dim. reduction (word2vec, GloVe, PCA, LDA, LSA) \n",
        "* predict rather than count, instead of looking at how often a word occurs, train classifier on binary prediction task, take learned weights as word embeddings but don't care about task\n",
        "* use running text as implicitly supervised trainign data, word near target word, no need for supervision, comes from neural language modeling\n",
        "\n",
        "Word2Vec Skip-gram with negative sampling (SGNS)\n",
        "* 1) treat target word and neighboring context as positive examples\n",
        "* 2) randomly sample other words to get negative examples\n",
        "* 3) use logistic regression to train classifier for the two\n",
        "* 4) use weights as embeddings\n",
        "* give a tuple (target, context), return the probability that the context is a real context word P(+|t,c) or P(-|t,c) = 1 - P(+|t,c)\n",
        "* words likely to appear close to similar words, to module simularity with dot product, similarity(t,c) = t * c\n",
        "* but dot product (and cosine) is not a probability, turn it into a probability using the sigmoid function (1/1+e^-x)\n",
        "* p(+|t,c) = 1/(1+e^-(t*c)); P(-|t,c) = 1 - p(+|t,c) = e^(-t*c)/(1+e^-(t*c))\n",
        "* assume all context words are independent so we have P(+|t,c1:k) = mult_i->k 1/(1+e^-(t*ci)); log(P(+|t,c1:k)) = sum_i->k log(1/(1+e^-(t*ci))\n",
        "* can compute probability of these values, learn the embeddings from Word2vec automatically\n",
        "* starts with initial set of embedding vectors, shifts the embedding to be more like embedding of words in nearby texts\n",
        "* for each positive example, create k negative ones with noise words and random words that are not term\n",
        "* choosing noise words, could pick according to unigram frequency P(w), more common to pick according to Pa(w), where Pa(w) = count(w)^a / sum_w count(w)^a\n",
        "* give rare noise words higher probability\n",
        "* start with random weights or 0, adjust to make positive pairs more likely and negative pairs less likely\n",
        "\n",
        "SGNS example\n",
        "* start with 300 * V random parameters, words as vectors of length 300\n",
        "* maximize similarity of target word, context word pairs from positive data and minimze t,c pairs drawn from negative data\n",
        "* want to maximize sum_(t,c) logP(+|t,c) + sum_(t,c) logP(-|t,c) \n",
        "* focusing on one target word t, loss becomes L(theta) = log(1/1+e^(-t*c)) + sum_i->k log(1/1+e^ni*t)\n",
        "* train with gradient descent, learns W and C embedding matrices, can merge or keep W and throw away C\n",
        "* take paragraph and pair of words that co-exist as positive examples, pair of words that dont co-exist as negative examples, classifier distringuishes and adjusts embeddings, throw away classifier and keep embeddings\n",
        "\n",
        "Properties of embeddings\n",
        "* similarity depends on window size C, context windows change the word embeddings\n",
        "* can solve analogies with word vector embeddings\n",
        "* use vector arithmatic, a:a* as b:b*, b* is hidden vector, b* should be similar to b - a + a*\n",
        "* this can be solved by optimizing argmax_b* in V (cos(b*, b - a + a*))\n",
        "* can capture relational meaning now\n",
        "* if all word-vectors are normalized to unit length then expression is equivalent to argmax_b* in V (cos(b*,b) - cos(b*,a) + cos(b*,a*))\n",
        "* or we can require direction of transformation be maintained, argmax_b* in V (cos(b* - b, a - a*)), say they share the same direction and ignore the distances\n",
        "* diachronic word embeddings can be used for studying language change\n",
        "\n",
        "Embeddings and bias\n",
        "* Paris: France :: Tokyo: x -> x = Japan\n",
        "* man: computer programmer :: woman: x -> x = homemaker\n",
        "* concepts and attributes can be associated, can lead to bias in embeddings\n",
        "* reflect and replicate all sorts of unfair biases\n",
        "* need to debias algorithms for embedding, and use as a tool for studying historic bias\n",
        "* can use hamilton historical embeddings, cosine similarity of embeddings for decade X for occupations to male vs. female names\n",
        "* correlated with actual percentage of gender and occupation\n",
        "* embeddings for competence adjuctives biased towards men\n",
        "* reflect ethnic stereotypes over time, attitudes towards ethnic groups\n",
        "\n",
        "Conclusion\n",
        "* embeddings are vector models of training\n",
        "* more refined than string or index, very good at modeling similarity/analogy\n",
        "* we can use sparse models (tf-idf) or dense models (word2vec, GLoVE)\n",
        "* useful in practice but know they have cultureal stereotypes encoded as well\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "1920fbf5-0fb9-435c-d5cf-9a6e29a3ac5b"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 14 Lecture Notes\", module14_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n\n* Predict next word in sequence given preceding context\n* Can assign conditional probability to every possible next word, get distributions over entire vocabulary, can also assign probabilities to entire sequences by using conditional probabilities with chain rule\n* Evaluate on how well they do on unseen text, good models assign higher probabilities to unseen data\n* Use perplexity to measure quality of a language model, inverse probability that theta assigns to the test set normalized by test set length\n* Can use the chain rule again to expand computation of probability of test set\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n\n* Recurrent neural network is any network that contains cycle within network connections, so values of some unit are directly or indirectly dependent on earlier outputs as input\n* Difficult to reason and train, very effective when applied to language\n* Simple RNNs lead to LSTM networks\n* Input multiplied by weight matrix and passed through non-linear activation function to compute values for hidden unit layer, then used to calculate output \n* Sequences are processed by presenting one item to the network at a time\n* Link augments input to computation at hidden layer with value of hidden layer from preceding point in time\n* Hidden layer from previous step is like a memory unit, informs later decisions, context in previous hidden layer can include information extending to beginning of sequence\n* Can appear more complex than non-recurrent architectures but are not that different, still feed forward but new set of weights tell what to do with previous context\n* Trained with back propagation still\n* Nearly identical to feedforward networks with forward inference\n* Use training set, loss function, back propagation to get gradients to adjust weights\n* Have extra weights from previous hidden layer\n* Back-propagation requires two passes for training of weights, second pass we process sequences in reverse, back propagation through time\n\n\n## SLP Chapter 9.3 RNNs as Language Models\n\n* As language models, RNN process input sequence one word at a time to predict next word from current word and previous hidden state\n* Dot babe limited context problem as n-grams since they can represent information in hidden states\n* Forward inference the same, input sequence is series of word embeddings represented as one-hot vector and output is vector that represents probability distribution over vocabulary\n* Model uses embedding matrix to retrieve embedding for current word and combines with hidden layer to get new hidden layer, generates output layer fed through softmax to get probability\n* Output is set of scores over vocabulary given evidence in hidden layer\n* Probability of entire sequence is product of probabilities of each item in sequence\n* Train to minimize error in predicting true next word in training sequence using cross entropy loss\n* Always give the model the correct history sequence to predict the next word, teacher forcing\n* weights adjusted to minimize CE loss ver training sequence with gradient descent\n* Final layer provides way to score likelihood of each word given evidence present in final hidden layer of network\n* Weight tying is method that uses single set of embeddings at input and softmax layer to reduce redundancy, lower dimensionality\n\n\n## SLP Chapter 6.1 Lexical Semantics\n\n* Representation of word is string of letters or index in vocabulary list\n* Lexical semantics define some of the ways we set up words for our NLP\n* Lemma is the focus word, also called citation form, mouse is lemma for mice e.g.\n* Each lemma can have multiple meanings, call each one a word sense, can be polysemous and have multiple meanings, makes interpretation difficult\n* Word sense disambiguation is process of deducing word from context\n* Synonyms are important to capture, if they have the same propositional meaning\n* Principle of contrast says difference in linguistic form always associated with difference in meaning\n* Words can have lots of similar words that are not similes \n* Words can also be related in ways other than similarity, word association\n* Another relation is if two words belong to the same semantic field, have structured relations between them\n* Semantic frames are set of words that denote perspectives or participants in types of event\n* Words can have different meanings or connotations as well, mean different things with same word, sentiment can reflect positive or negative words\n\n\n## SLP Chapter 6.2 Vector Semantics\n\n* Vector semantics are the standard way to represent word meaning in NLP\n* Define meaning of word by distribution in language use, meaning neighboring words and grammatical environments \n* Represent a word in multidimensional semantic space that is derived from distributions of word neighbors, vectors for representing these words called embeddings, sometimes only applied to word2vec dense vectors\n* Mapping of from one space or structure to another, can project down. To 2d space to see the relationships between words\n* Classifiers can assign sentiment as long as it sees words with similar meanings\n* Tf-idf model has meaning of word defined by simple function of counts of nearby words, gives a sparse representation\n* Word2vec has short dense vectors with useful semantic properties\n* Can also use cosine to compute semantic similarity between too words/sentences\n\n\n## SLP Chapter 6.3 Words and Vectors\n\n* Vector or distributional models are based on co-occurence matrix often, how often words co-occur\n* Term-document matrix, each row represents a word in vocabulary, each column a document from collection of documents\n* Each cell is number of times particular word occurs in particular document\n* First defined as part of vector space model of information retrieval\n* Vector space is collection of vectors characterized by dimension\n* Ordering of numbers in vector space indicate different meaningful dimension where documents vary\n* Vector for a document is a point in V dimension space\n* Two similar documents have similar words, column vectors will be similar as well\n* V rows for number of words, and D columns for each document in collection\n* Information retrieval is process of finding document from documents in some collection that matche a query\n* Represent query as vector, node to compare two vectors for similarity\n* Could use term-term matrix or word-word matrix or term-context matrix, columns are labeled by words instead of documents so V x V, each cell records number of times the row and column word co-occur In some context in training set\n* Context could also be document, most common to use smaller contexts to get neighboring words\n* Size of vocabulary can be tens of thousands of words, most of them are zero so we have sparse vector representation, effective algorithms exist to store them\n\n\n## SLP Chapter 6.4 Cosine for measuring similarity\n\n* Measure similarity between two target words that takes in two vectors, use cosine of the angle between the vectors\n* Cosine based on dot product operator, called inner product\n* Raw dot product favors long vectors, higher is vector is longer with higher values in each direction\n* We can normalize it by diving the dot product by the lengths of each of the two vectors, normalized dot product is the same as the cosine of the angle between the two vectors\n* Can pre-normalize the vector, divide by its length, create a unit vector of length 1\n* Then compute unit vector and take cosine which is equivalent to dot product\n* Cosine ranges from 1 for vectors pointing in same direction, 0 for orthogonal vectors, -1 for pointing in opposite direction\n\n\n## SLP Chapter 6.8 Word2vec\n\n* Word2vec, represent words as short dense embeddings, work better in every NLP task than dense vectors\n* Skip-gram with negative sampling is one of two algorithms in word2vec, static embeddings meaning method learns one fixed embedding for each word in the vocabulary\n* Dynamic contextual embeddings like BERT are separate class\n* Can use running text as implicit supervised training data for a classifier, word that appears near the target is the gold standard, call this self supervision since we don’t need to label anything \n* Simplifies task compared to neural networks, and simplifies architecture\n* Treat word and neighboring context as positive examples, randomly sample other words as negative samples, use logistic regression to train a classifier to distinguish, and use weights as embeddings\n* Given tuple of target word and candidate context word, get probability that it is a real context word, base probability on embedding similarity\n* Word likely to occur near target if embedding vector is similar to target embedding\n* Two vectors are similar if they have a high dot product, sue sigmoid function to turn it into a probability\n* Says all context words are independent so we can multiply their probabilities, need embeddings for each target word and context word in vocabulary \n* Stores two embeddings for each word, one for word as target and one as context\n* Takes input paragraph of text and vocabulary size B, assigns random embedding vector to each word then shifts embeddings to be more like embeddings in training set\n* Noise words can be chosen accord to weighted unigram frequency with alpha as a weight\n* Maximize similarity of target/context word pairs from positive examples and minimize similarity of negative example pairs\n* Minimize loss with stochastic gradient descent\n* Learns target embedding and context embedding, can add them together or throw away C matrix and look at target matrix\n* Fasttext does better with unknown words by using subword models and generalizes better to unknown words\n\n\n## SLP Chapter 6.10 Semantic properties of embeddings\n\n* Size of context window used to collect counts is relevant to both tf-idf vectors and dense word2vec vectors\n* Shorter context dings lead to representations that are more syntactic since coming from nearby words, semantically similar with same parts of speech\n* Long context windows, highest cosine words to target word are topically related but not similar\n* First order co-occurrence if they are typically nearby each other, also called syntagmatic association\n* Two words have second-order co-occurrence if they have similar neighbors, called paradigmatic associated\n* Ability to capture relational meanings also important\n* Parallelogram model for solving simple analogy problems can be constructed \n* Sparse vector models of meaning could solve such analogy problems, better success with word2vec or GLoVe vectors\n* Want to find the argmin distance x of a* - a + b\n* Closest value returned by parallelogram model is Sally not b* but one of the 3 inputs words or morphological variants, so much be explicitly excluded\n* Doesn’t work well for other relations besides frequent words, small instances, and certain relations\n* Can also use embeddings to study how meaning changes over time\n\n\n## SLP Chapter 6.11 Bias and Embeddings\n\n* Embeddings can produce implicit biases and stereotypes latent in text\n* Embedding analogies can exhibit gender stereotypes in word2vec embeddings trained on news text\n* Allocations harm, system allocates resources unfairly to different groups\n* Embeddings also amplify bias, gendered terms become even more gendered in space compared to input text statistics\n* Can also encode implicit associations that are property of human reasoning\n* Representational harm is harm caused by system demeaning or ignoring some social groups\n* Recent research focused on transforming embedding space to remove gender stereotypes, debasing may reduce bias but hard to eliminate since is it in training text\n* Historical embeddings also being used to measure biases in the bast\n* Can document historical gender biases and racial stereotypes\n\n\n## SLP Chapter 6.12 Evaluating Vector Models\n\n* Most important evaluation metric is extrinsic evaluation on tasks, seeing if performance improves over some other model\n* Most common metric is similarity, correlation between algorithms word similarity and ratings assigned by humans\n* Slightly more realistic metrics are intrinsic similarity tasks that include context, they offer richer evaluation scenarios\n* Analogy task is another task used for evaluation\n* All embeddings suffer from variability, may produce different results from the same dataset, individual documents may strongly impact embeddings\n* Train multiple embeddings with bootstrap sampling over documents and average results\n\n\n## Module 14 Lecture Notes\n\nLanguage models\n* estimate the probability of a sentence containing a word sequence w1:n as P(w1:n) + mult_i->n P(wi|wi-k:i-1)\n* need to estimate probability of P(wi+1|wk-1;1) from large paragraph, p_hat(MLE)(wi+1 = m|wi-k:i) = #(wi-k;i+1)/(wi-k:i), p_hat(add - a)(wi+1 = m|wi-k:i) = #(wi-k;i+1) + a/(wi-k:i) + a|V|\n* neural language models have advantages over nn-gram LMs, 1) don't need smomthing, 2) can handle much longer histories, 3) can generalize over contexts of similar words, 4) tend to have higher prediction accuracy\n* can be slower to train than traditional n-gram Lms though\n\nNeural Language models\n* first associate each word in vocabulary with vector-representation and create notion of similarity as a result\n* then express joint probability function of word sequence in terms of word vectors for words in sequewnce\n* then learn the word vectors and the parameters of the function\n* word vectors will be low dimensional dense vectors, probability function expressed as product of conditional probabilites of next word given predvious word using feed forward neural network described earlier\n* input is k-gram of words w1:k, output is probability distribution over next word, k context words treated as word window with each word as an embedded vector\n* input vector x concatenates v(w) for each of k words, v(w) in R^dw, x = [v(w1);v(w2);..v(wk)]\n* fed into a neural network with one ore more hidden layers\n* input layer is one hot vectors with word as 1, then projection layer, hidden layer, output layer\n* training examples are word k-grams from paragraph, identities of first k-1 words used as features, last word is target label, trained using cross-entropy loss\n* better results with better preplexity scores than n-gram LMs, larger N to scale to since parameters are associated with individual words, and they generalize across contexts well, by-product are word embeddings\n\nLanguage modeling\n* want to find a function that returns the joint probability again\n* this is difficult because of the curse of dimensionality (too many parameters to estimate) and we can fail to generalize related words/word sequences that we have observed in n-gram-based models\n* sparse statistics just too high to estimate from the data\n* in LMs we use chain rule to get conditional probability of next word in sequence given all of the previous words, P(w1w2w3..wt) = sum_t->T P(wt|w1..wt-1)\n* probability of the next word only depends on previous n - 1 words, smaller n makes it easier to get a probability from data\n* can construct probability tables to look up probs of words given histories of observed sequences\n* but if we have unseen combination of n words, either back off or smooth/interpolate to stitch together short sequences of observed words\n* try generalizing instead, take a sentence and assign probabilities to similar sentence\n\nNeural probabilistic LM\n* use vector space model where words are vectors with real values, define a function that returns join probability of words in sequence based on previous seqeunce, learn word representations and probability function from data\n* given a training set w1..wt, learn f(w1..wt) = P(wt|w1..wt-1), subject to giving high probability to unseen text/dev set, create probability distribution so we can take product of conditonal probabilities to get joint ones\n* create mapping function C from any word in V onto R^M, store in VxM matrix and initialize with singular value decomposition\n* neural architecture is a function g tha mapes sequence of word vectors onto probability distribution over vocab V, g(C(wt-n) ..C(wt-1)) p P(wt|wt-n ..wt-1)\n* when 50 dimenstional vectors result from training LM prohected to 2 dimensions, see similar words close together\n* ELMo, GPT, BERT, GPT-2 the current state of the art models\n\nWord meaning\n* how to represent meaning of word, in Ngram LMs did so as string of letters or index in vocabulary list\n* want them to encode synonyms (similar meaning), antonyms (opposite meaning), connotations (positive or negative words), semantic roles, support for entailment (e.g. dog is proper subset of animal)\n* leximal semantics used to be trees like wordnet, now use more vector space models\n* most words don't have many synonyms but can use similar words instead for many applications\n* from a lot of context words humans can guess what words mean, so algorithm calls two words similar if their word contexts are similar\n* initially, vector space models were developed with each document as point in space, user query is pseudo document and represented point in same space, retrieved documents that were close to it\n\nTerm-Document matrix\n* each column is a document, each row is a term, value in a cell is how often that word appeared in the document\n* length of document vector is size of vocabulary, document vectors can be sparse, then can measure similarity between two documents by comparing columns\n* word similarity lets you do plagiarism detection\n\nVector comparisons\n* can look at two documents and terms in each, calculate euclidean distance between them, want to use L2 length normalization (||u|| = sqrt(sum_i->n ui^2)) and divde each vector by its L2 length\n* can also do cosine difference, does the L2 normalization as well. 1 - sqrt(sum_i->n ui * vi) / (sqrt(sum_i->n ui^2) * sqrt(sum_i->n vi^2))\n* the cosine angle will tell us the similarity between two terms\n\nTerm-term matrix\n* aka term-context matrix, length of vector is now |V| instead of number of documents, value in a cell indicates how often word appears in context window surrounding other word\n* context windows are w-2, w-1, target_word, w+1, w+2, list how many times this takes place\n* can occur in window of +/- 2 words in same sentence in same document, instead of windows can use more complext contexts like dependency patterns, languages often have long distance dependencies\n* use the syntax to define the word's context\n* raw word frequency not a great measure of association, skewed with more common words like the, of, or, rather have measure of context word as particularly informative about target word\n\nTF-IDF\n* term frequency * inverse document frequency (how often word occured in document * 1 over number of documents it occurred in)\n* can have spare and dense vectors, co-occurrence matrices weighed by Tf-IDF can be long and sparse, would rather have short and dense vectors\n\nDense vectors\n* to get them, train a classifier\n* treat targed word and neighboring context as positive examples, randomly sample other words to get negative samples, use logistic regression to train classifiedr to distinguish and weights as the embeddings\n* short vectors easier to use as features in ML, may generalize better than strong explicit counts\n* may do better at capturing synonyms, work better in practice\n\nWord2Vec\n* learn embeddings as part of word prediction process, train classifier to predict neighboring words, learn dense embeddings for words in paragraphs\n* fast and easy to train, availabe online with pretrained embeddings\n* predict neighboring word in context window of 2c of surrounding words, [wt-2, wt-1, wt+1, wt+2], use negative sampling for training - want prediction of target words to be high and irrelevant words to be low\n* inspired by neural nets, nearest neighbors are pretty good, embeddings can capture relational meanings\n* many possible models, matrix types (term-document, term-context, pattern-pair), reweighting (length norm, TF-IDF, PPMI, probabilities), comparisons (cosine, manhattan, jaccard, KL divergence, JS distance, DICE), dim. reduction (word2vec, GloVe, PCA, LDA, LSA) \n* predict rather than count, instead of looking at how often a word occurs, train classifier on binary prediction task, take learned weights as word embeddings but don't care about task\n* use running text as implicitly supervised trainign data, word near target word, no need for supervision, comes from neural language modeling\n\nWord2Vec Skip-gram with negative sampling (SGNS)\n* 1) treat target word and neighboring context as positive examples\n* 2) randomly sample other words to get negative examples\n* 3) use logistic regression to train classifier for the two\n* 4) use weights as embeddings\n* give a tuple (target, context), return the probability that the context is a real context word P(+|t,c) or P(-|t,c) = 1 - P(+|t,c)\n* words likely to appear close to similar words, to module simularity with dot product, similarity(t,c) = t * c\n* but dot product (and cosine) is not a probability, turn it into a probability using the sigmoid function (1/1+e^-x)\n* p(+|t,c) = 1/(1+e^-(t*c)); P(-|t,c) = 1 - p(+|t,c) = e^(-t*c)/(1+e^-(t*c))\n* assume all context words are independent so we have P(+|t,c1:k) = mult_i->k 1/(1+e^-(t*ci)); log(P(+|t,c1:k)) = sum_i->k log(1/(1+e^-(t*ci))\n* can compute probability of these values, learn the embeddings from Word2vec automatically\n* starts with initial set of embedding vectors, shifts the embedding to be more like embedding of words in nearby texts\n* for each positive example, create k negative ones with noise words and random words that are not term\n* choosing noise words, could pick according to unigram frequency P(w), more common to pick according to Pa(w), where Pa(w) = count(w)^a / sum_w count(w)^a\n* give rare noise words higher probability\n* start with random weights or 0, adjust to make positive pairs more likely and negative pairs less likely\n\nSGNS example\n* start with 300 * V random parameters, words as vectors of length 300\n* maximize similarity of target word, context word pairs from positive data and minimze t,c pairs drawn from negative data\n* want to maximize sum_(t,c) logP(+|t,c) + sum_(t,c) logP(-|t,c) \n* focusing on one target word t, loss becomes L(theta) = log(1/1+e^(-t*c)) + sum_i->k log(1/1+e^ni*t)\n* train with gradient descent, learns W and C embedding matrices, can merge or keep W and throw away C\n* take paragraph and pair of words that co-exist as positive examples, pair of words that dont co-exist as negative examples, classifier distringuishes and adjusts embeddings, throw away classifier and keep embeddings\n\nProperties of embeddings\n* similarity depends on window size C, context windows change the word embeddings\n* can solve analogies with word vector embeddings\n* use vector arithmatic, a:a* as b:b*, b* is hidden vector, b* should be similar to b - a + a*\n* this can be solved by optimizing argmax_b* in V (cos(b*, b - a + a*))\n* can capture relational meaning now\n* if all word-vectors are normalized to unit length then expression is equivalent to argmax_b* in V (cos(b*,b) - cos(b*,a) + cos(b*,a*))\n* or we can require direction of transformation be maintained, argmax_b* in V (cos(b* - b, a - a*)), say they share the same direction and ignore the distances\n* diachronic word embeddings can be used for studying language change\n\nEmbeddings and bias\n* Paris: France :: Tokyo: x -> x = Japan\n* man: computer programmer :: woman: x -> x = homemaker\n* concepts and attributes can be associated, can lead to bias in embeddings\n* reflect and replicate all sorts of unfair biases\n* need to debias algorithms for embedding, and use as a tool for studying historic bias\n* can use hamilton historical embeddings, cosine similarity of embeddings for decade X for occupations to male vs. female names\n* correlated with actual percentage of gender and occupation\n* embeddings for competence adjuctives biased towards men\n* reflect ethnic stereotypes over time, attitudes towards ethnic groups\n\nConclusion\n* embeddings are vector models of training\n* more refined than string or index, very good at modeling similarity/analogy\n* we can use sparse models (tf-idf) or dense models (word2vec, GLoVE)\n* useful in practice but know they have cultureal stereotypes encoded as well\n\n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "T3cm1WV542_a",
        "outputId": "df57f843-306d-4300-e5e4-6821b3b14f39"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\n* Agent makes actions a, has a probability P(s) to each state s\n* Transition model for the uncertain outcomes is P(s' | s,a), probability that action a in state s yields states s'\n* related by the equation $P(RESULT(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n* make decisions based on immedaite outcomes, preferences represented by utility function U(s)\n* utility assigns a number to express the value of a state\n* expected utility EU(a) is the average utility of the outcomes weighed by probability: $EU(a) = \\sum_{s'} P(RESULT(a) = s')U(s')$ \n* maximum expected utility principle says the agent should pick the action that mazimizes the expected utility if it is rational\n* action = argmax(a) EU(a)\n* formalizes what the agent should do but doesnt operate on the advice, lots of other factors to consider\n* if the agent acts to maximize utility function that is correctly representative of the performance, agent will receive the highest possible performance score over all environments averaged\n\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nA >- B - agent prefers A over B\n\nA ~ B - agent is indifferent between A and B\n\nA >-_ B agent prefers A over B or is indifferent\n\n* lottery is a set of outcomes for each action, and each action we assume to be a lottery ticket\n* lottery with possible outcomes S1, S2, ... with probabilityies p1, p2, ... is expressed as L = [p1, S1; p2, S2; ... pn, Sn]\n* each outcome of a lottery can be an atomic state or anotehr lottery\n\nConstraints we expect lotteries to obey\n* Orderability - rational agent must prefer one lottery, the other, or rate them as equally preferrable, cannot avoid a decision\n* A >- B, B >- A, or A ~ B\n\n* Transitivity - if we have lotteries A, B, and C, and the agent prefers A to B and B to C, then it prefers A to C\n* A >- B and B >-C -> A >- C\n\n* Continuity - if a lottery B is between A and C, then there exists a probability p where the agent is indifferent between B guarenteed and lottery A with probability p and C with probability 1 - p\n* A >- B >- C -> exists p [p,A; 1-p, C] ~ B\n\n* Substitutability - agent is indifferent between A and B, then also indifferent for two more complex lotteries that are equal except for B substituted for B\n* A ~ B -> [p, A; 1 - p, C] ~ [p, B; 1 - p, C]\n\n* Monotonicity - agent prefers A to B, must prefer the lottery that has a higher probability for A\n* A >- B -> (p > q <-> [p,A; 1 - p, B] >- [q,A; 1 - q, B])\n\n* Decomposability - compound probabilities can be reduced with the laws of probability, e.g.\n* [p,A; 1 - p, [q,B; 1 - q, C]] ~ [p,A; (1 - p)q,B; (1 - p)(1 - q), C]\n\n* if an agent violates one of these it exhibits irrational behavior\n* Existance of a utility function: if agent behaves all these, then there exists a function U such that U(a) > U(b) iff A is preferred to B, and U(a) = U(b) iff the agent is indifferent\n* expected utility of a lottery is the sum of the probability of each outcome multipled by the utility of the outcome - U([p1,S1; ... ; pn,Sn]) = $\\sum_i piU(Si)$ \n* utility of a compound lottery becomes determined completely\n* agent only needs preference of states, the numbers don't matter, referred to as value function or ordinal utility function\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n\n* agent allowed any preferences it likes so long as it follows the axioms\n* preference elicitation, presents choices to humans and uses the preferences to determine the utility function\n* need to know what the human utility function is to make decisions on our behalf\n* want to create a fixed scale fo preference, normalized utilities scale from 0 to 1\n* using this normalized utility scale, we can determine the utility of a prize S and by asking the agent to pick between S and a standard lottery\n* probability is adjusted until the agent is indifferent, then S is assigned that utility, do this for each prize then have a lookup table of values\n* tradeoffs of life and death made all the time\n* refusal to put monetary value on life can mean its undervalued\n* value of a statistical life determines costs and benefits of regulation/intervention\n* micromort is 1 in a million chance of death, QALY is quality adjusted life years for patients\n* if an agent prefers more money than less, it has a monotonic preference for more money\n* expected monetary value is probabilities multiplied by monetary amounts\n* utility of money generally found to be proportional to the logarithm of the amount\n* positive curve, agents are risk-averse and prefer a sure thing, negative curve, risk-seeking\n* certainty equivalent is the amount someone will except instead of doing the lottery\n* difference between the EMV of the lottery and the certainty equivalent is the insurance premium\n* an agent is risk-neutral if the linear curve is observed\n* often working with estimated expected values EU(a) instead of the true expected utility, assumed to be unbiased when not always the case\n* order statistic is distribution of ranked element of a sample\n* optimizer's curse, tendency for expected utility of the best choice to be too high\n* normative theory describes how rational agents should act, descriptive theory is how actual agents really act\n* certainty effect, people often prefer the sure thing in gains, reduce computational burden, distrust the stated probabilities, regret if it doesnt pan out\n* people also have ambiguity aversion, go with the sure probability\n* framing effect, how decision problems are worded\n* anchoring effect, making relative utiltiy judgements rather than absolute ones, cheaper product compared to more expensive one\n* evolutionary psychology says our decision making processes were not evolved to solve word problems with probabilities\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\n* fully observable environment allows agent to always know where it is\n* transition model describes the outcome of each action in the state, stocastic in our maze so P(s'|s,a) as before, or T(s,a,s')\n* utility function here depends on sequence of states and actions (environmental history), rather than single state\n* agent will receive a reward R(s,a,s') for moving to state s'\n* Markov Decision process (MDP) consists of set of states, actions, transtiion model, reward function\n* involve dynamic programming often, simplifying a problem by breaking it down and solving optimal solutions to subproblems\n* policy solution pi is action recommended for a given state so agent knows what to do, quality measured by expected utility of environment histories generated\n* optimal policy pi* yields highest expected utility\n* reward a big factor in how the agent moves and if/when it chooses to exit\n* with a finite horizon (meaing theres a fixed time after which nothing matters), an optimal action may depend on time left\n* nonstationary policy depends on the time\n* with infinite time (horizon), optimal action only depends on current state, policy is stationary, these are simpler\n* discount factor between 0 and 1, preference of agent for current vs. future rewards. Close to 0, future rewards are insignificant, close to 1 more willing to wait for them. If exactly 1 then purely additive rewards\n* makes infinities go away, rewards not always guarenteed, valued higher\n* utility of infinite sequence now finite\n* if agent contains terminal states and guarenteed to reach it then never need to compare infinite sequences, proper policy if it exits\n* infinite sequences can be compared by average reward obtained per time step\n* expected utility of executing pi starting in s is Upi(s) = E * $\\sum_t->inf y^t R(S_t, pi(S_t), S_{t+1})$\n* optimal policy pi* = argmax of pi U^pi (s)\n* in infinite horizons with discouted utilities, optimal policy independent of starting state\n* utility function allows agent to select actions with highest expected utility: $pi^*(s) = argmax_{a exist A(s)} \\sum_s' P(s'|s,a)[R(s,a,s') + yU(s')]$\n* utility of state is expected reward for next transtion plus discounted utility of next state, so utility of state is: $U(s) = max_{a exist A(s)} \\sum_s' P(s'|s,a)[R(s,a,s') + yU(s')]$, bellman equation\n* action-utility function or Q function, Q(s,a) is expected utility of taking action in given state: U(s) = max_a Q(s,a), optimal policy takes max action that yields highest Q state\n* can develop equation for Q-funtcion, $Q(s,a) = \\sum_s' P(s'|s,a)[R(s,a,s') + ymax_{a'}Q(s',a')]$\n* all policies optimal with a reward function that is always zero\n* represent P(s'|s,a) and R(s,a,s') using 3D tables of size |S|^2|A|, often sparse tables because bounded transitions between states\n* can represent MDPs by extending dynamic bayesian networks to create dynamic decision etwrosk, factored representations with exponential complexity advatnage over atomic representation\n\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\n* if we have n possible states, n Bellman equations, one per state\n* n equations have n unknowns, utilities of each state\n* equations nonlinear because the max operation isn't linear, try iterative approach instead\n* start with arbitrary values for the utilities, then update each utility until we hit equilibrium\n* update assumed to be applied to all states simultaneously, will converge and give unique solutions and the policy\n* value iteration, think of as propagating information through state space to update each state \n* value teration converges due to contraction, function of one argument that produces two output values that are close together by some constant factor, applied to two inputs\n* only has one fixed point, and the value must get closer to the fixed poiint upon every application\n* distance between the utility vectors measured with max norm, measures the length of the vector by value of biggest component\n* Bellman update is contraction by factor of discount on space of utility vectors\n* agent only cares about how well it does if it makes decision using utility function\n* define policy loss as the most an agent can lose by executing a policy instead of the optimal one\n* policy iteration alternates between evaluation and improvement\n* policy evaluation: given policy pi_i, calculate U_i = Upi_i, utility of each step if current policy is executed\n* policy improvement: calculate new MEU policy pi_i + 1 using look ahead based on U_i\n* terminates when policy improvement doesnt update the utilities\n* use q value function, get a simplified version of Bellman equation relating utility of s to neighbor utilities\n* equations now linear since max is removed, so we can solve a n state n linear equation n unknown in O(n^3) time\n* often the most efficient in small state spaces\n* don't have to do exact policy evaluation, can do number of simplified value iteratoin steps to give estimate\n* repeat this many times to get the next utility estimate, modified policy iteration\n* can also update any subset of states and apply policy iteration or simplified value iteration to the subset, asynchronous policy iteration, guarenteed to converge to optimal policy\n* Linear progreamming used to formulate constrained optimization problems\n* can reduce solving MDP to solving formulated linear program\n* want to minimize U(s) for all s\n* MDPs can be solved in polynomial time in number of states, actions, bits to specify model\n* Value and policy iteration both offline algorithms to solve states then agent acts\n* for online, use expectiminimax simplification, evaluation function applied at nonterminal leaves given default value\n* decision taken by backing up the utility values and averaging at the chance nodes\n* limit depth to avoid high branching factor, focus on most likely outcomes\n* real time dynamic programming, explored states create a sub-MDP of MDP, and can be solved to yield decision of current state\n* state space may be too large, or difficult to make heuristic\n\n\n## Module 8 Lecture Notes\n\nAsteroid Field:\n* 4x3 environment with goal states, pick an action at each time step, terminates when goal state reached, stochastic environment\n* pick a sequence of actions, can get there correctly with no errors or with \"correct\" errors\n* Transition model written as T(s,a,s'), or p(s',s,a) - probability of reaching s' given s and action a\n* Markovian transitions, meaning probability of arriving in s' depends only on s, no earlier states\n* utility/reward function can be positive or negative but bounded by some maximum value\n* can receive this reward R(s,a,s') at any time step since the decisions are sequential, additive as well\n\nMarkov Decision Process (MDPs):\n* sequential decision process, fully observable environment, stochastic transition model, additive rewards\n* non-deterministic search problems so we can solve using expectimax search to calculated expected utilities\n* to solve an MDP, need 1) a set of states s in S, 2) a set of actions a in A, 3) a transition function T(s,a,s') or the probabilit that executing action a in s leads to s', 4) a reward function R(s,a,s'), 5) an initial state s0, 6) one or more terminal states (optional)\n* solution is a policy since there is non-determinism, not guaranteed we end up where we want to go, pi(s) = a. Stochastic so may be correct but not happen, optimal policy is pi*\n* same policy executed multiple times may lead to different states and actions and different scores, so need expected utility of all paths generated\n* performance of agent is the sum of the rewards of the transitions it takes, can cause the agent to exit early/late/never\n* Policy is choice of action for each state, utility is sum of the potentially discounted rewards\n\nDiscounting:\n* Want to maximize sum of rewards, prefer now versus later\n* each time we go down a level, multiply discount once, allows algorithms to converge\n* if we assume stationary preferences then we can only define utilities as additive or discounted\n\nInfinite utilities problem:\n* apply finite horizon (similar to depth limited search) to terminate the search after some number of steps to avoid infinite rewards\n* this will yield nonstationary policies since the policy depends on the time left\n* Alternatively, use discounting between 0 and 1 to lower future rewards, smaller y means smaller horizon, shorter term focus\n* or absorbing state, force that for each policy a terminal state will be reached eventually\n\nStar Wars hyperdrive MDP:\n* need to travel far away quickly, can cruise, hyperspace, punch it, faster is more reward\n* each MDP is like an expectimax search tree, has a state s, (s,a) as a q-state, (s,a,s') as a transition\n\nOptimal Qualities:\n* value or utility of a state V*(s) is expected utility starting in s and acting optimally\n* value or utility of a q-state (s,a) Q*(s,a) is expected utility starting out having taken action a from s, from optimal action\n* optimal policy pi*(s) is optimal action from state s\n* want to compute expectimax value of a state, average sum of discounted rewards over expected utility under optimal action\n\nRecursive definition of value (Bellman equations):\n* V*(s) = max_A Q*(s,a)\n* Q*(s,a) = $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$ \n* V*(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$\n* optimal utility via expectimax recurrence has these one step lookahead equations to get optimal values\n\n* can do too much work with expectimax in serach tree, states can be repeated and tree can go on forever\n* instead do depth limited search with increasing depths, depending on gamma depth past a certain point is negligable\n* time limited values, can define Vk(s) to be optimal value if game ends in k more time steps, or what depth k expectimax would yield\n\nValue iteration:\n* start with V0(s) = 0, then do one ply of expectimax from each state, repeat until convergence, yields complexity of O(S^2A)\n* this will converge to unique optimal values, the policy may converge way before the values which is fine\n* Vk+1(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s)]$\n* two cases for convergence: 1) if tree has maximum depth m, then VM holds the untruncated values\n* if discount < 1, will have to converge as k increases since rewards are shrinking\n\nPolicy evaluation:\n* perform expectimax over all actions to get optimal values\n* motivation, if we fix policy, we only have to consider one action per state\n* compute the utility under this fixed policy, not optimal usually\n* perform one step look ahead with this policy: V_pi(s) = $\\sum_s' T(s,pi(s),s')[R(s,pi(s),s') + yV_pi(s')]$\n* turn bellman equations into updates like value iteration\n* V_0 pi(s) = 0, V_k+1 pi(s) = $\\sum_s' T(s,pi(s),s')[R(s,pi(s),s') + yV_pi(s')]$\n* yields O(S^2) efficiency per iteration\n* alternatively could solve with linear systems since there are no maxes\n\n* if we have optimal values V*(s), need to do one step expectimax to get optimal action, pi*(s) = argmax_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV*(s')]$\n* if we have the optimal q-values, very trivial, just pick the best one, pi*(s) = argmax_a Q*(s,a)\n\nValue iteration issues:\n* slow O(S^2A) per iteation, max at each state rarely changes, policy converges long before the values\n\nPolicy iteration approach:\n* 1) policy evaluation - calculate utilities for fixed policy until they converge\n* 2) policy improvement - update policies using one step look with converged utilities as future values\n* repeat until converges, still optimal and can converge much faster\n* Evaluation, find find values V_k+1 pi_i(s) = $\\sum_s' T(s,pi_i(s),s')[R(s,pi_i(s),s') + yVk pi_i(s')]$\n* improvement, get better policy with lookahead, pi_i+1(s) = argmax_A $\\sum_s' T(s,a,s')[R(s,a,s') + yV_pi_i(s')]$\n\n* value and policy iteration compute the same thing\n* value iteration updates values and policy implicitply, don't track the policy and instead take the max over the actions to figure it out\n* policy iteration does several passes to update utility with the same policy, evauate the policy then pick a new one, go until the policy doesn't change\n* both dynamic operations for solving MDPS\n* basically the same, variants of Bellman equations, only difference comes in fixed policy or maximum of actions\n* if you want to compute optimal values, use either, compute values for one policy use policy evaluation, turn values into policy use policy extraction\n\nMaximum expected utility:\n* reational agent should amximize expected utility given knowledge, not using minimax here and averaging instead\n* terminal function scale doesnt matter with worse case, want better states to have higher priorities, insensitivity to monotonic transformations\n* magnitudes are meaningful for average case reasoning\n* utilities have to be functions from outcomes to real numbers, can be simple or more complex, describe preferences\n* agents must have prefernce among prizes, either one to the other or indifferent\n* need constraints on preferences to assign rationality, or agent with intransitive preference can give everything up\n\n5 axioms of rationality:\n* Orderability: A >- B or B >- A, or A ~ B\n* Transitivity: A >- B and B >-C -> A >- C\n* Continuity: A >- B >- C -> exists p [p,A; 1-p, C] ~ B\n* Substitutability: A ~ B -> [p, A; 1 - p, C] ~ [p, B; 1 - p, C]\n* Monotonicity: A >- B -> (p >_ q <-> [p,A; 1 - p, B] >-_ [q,A; 1 - q, B])\n\n* if all of these satisifed, then ther exists utility function that prefers one lottery to the other and gives appropriate utilities\n* choose action that maximizes expected utility\n* normalized utilities between 0 and 1, Micromorts, QALYs, etc.\n* deterministic prizes only, ordinal utility can be determined only, meaning ordering on all the prizes\n* compare prize A to a lottery, when there is indifference we assign that probability to the utility\n* expected monetary value exists but money itself not a utility function\n\n\n## Module 8 Definitions\n* **Utility function** - assigns a single number to express state value\n* **Expected utility** - average utility value of all outcomes weighed by their probability\n* **Maximum expected utility** - agent should pick the action that maximizes their expected utility\n* **Value function** - ordinal utility function, agent only needs preference of states\n* **Markovian** - probability of arriving in new state depends only on the current state\n* **Environment history** - the different states and actions visited by the agent following a policy\n* **Stationary preferences** - if path a > b, path a with a reward > path b with the same reward\n* **Additive utility** - reward over the path is the sum of the reward with no discount\n* **Discounted utility** - reward over the path is the sum of the rewards multiplied by discounts\n* **Expected monetary value** - amount of money expected in terms of probability\n* **Certainty equivalent** - amount of money accepted instead of doing the lottery\n* **Insurance premium** - Difference between EMV and certainty equivalent\n* **Modified policy iteration** - perform small number of simpliefied value iteration with fixed policy\n* **Asynchronous policy iteration** - pick a subset of states and apply policy improvement or simplified value iteration\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n\n* N armed bandit is slot machined with n levels\n* Tradeoff of exploitation of current best action and exploration of previously unknown states and actions to learn information\n* In bandit problems, each arm Mi is a Markov decision process that is an MDP with only one action\n* The overall bandit problem is an MDP with all the same variables as before \n* Important to note that arms are independent, only can work one at a time\n* If we pull a one armed bandit T times, say the stopping time is T\n* Getting index defines the kind of value for M in terms of ability to deliver timely rewards\n* Pull the arm with the highest getting index then update the indices, each decision can be calculated in O(1) time\n* Bernoulli bandit has each arm Mi with a reward of 0 or 1 with a fixed unknown probability, infinitely many states exist in it, but can approximate\n* Arms with higher payoff probabilities are preferred but there is an exploration bonus for those only tried a few times\n* Getting indices for realistic problems is very difficult\n* Upper confidence bound, a heuristic that samples each arm to establish confidence interval for the arm, then choose the arm with then highest upper bound on the confidence interval\n* Just picks the arm with highest UCB value\n* Determines the regression relative to the clairvoyant policy, picks the best arm and averages the reward\n* No algorithm can have regret that grows more slowly than O(logN)\n* Thompson sampling chooses an arm randomly according to the probability that it is optimal given the samples so far, regret of O(logN)\n* No index function exists for a selection problem\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n\n* Agent gets rewards/reinforcements for how it’s doing\n* Same goal as MDPs, maximize the expected sum of the rewards\n* Agent is the MDP instead of being handed one, has to learn transition model and reward function\n* Model based reinforcement learning, agent uses a transition model of environment to help interpret rewards and make decisions on how to act\n* Transition model useful for state estimation, and RL learns utility function\n* Model free reinforcement learning, agent doesn’t know or learn transition model, has to learn direct representation of how to behave\n* Action utility learning is utilized, agent need to learn Q functions and choose actions based on that\n* Learns a policy that maps to the states and has to act accordingly, a reflex agent\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\n* Passive reinforcement learning, agent has fixed policy that determines its actions and it wants to learn the utility function when executing the actions\n* Similar to policy evaluation, but doesn’t know transition model that specifies probability of reaching next state, nor the reward function\n* Usually include discount factor to encourage it to prioritize nearer rewards\n* Direct utility estimation, utility of the state is defined as expected total reward from the state onwards, and each trial is a sample of this quantity for each state visited\n* This approach ignores that utility is determined by reward and expected utility of successor states, so it misses learning and can converge very slowly\n* Adaptive dynamic programming agent learns transitions model that connects states and solves MDPs using dynamic programming\n* Plugging in learned transition model and observed rewards to calculate utilities of states\n* Can also use modified policy iteration, simplified value iteration process to update utility estimates after each change to the model\n* Temporal difference learning, use observed transitions to adjust utilities of the observed states so they align with constraint equations\n* Include a learning rate parameter, update rule uses difference in utilities between successive states\n* Adjust the utility estimates towards ideal equilibrium that holds locally when all estimates are correct\n* Average value of the optimal utility converges to correct quantity even if values continue to move around\n* TD doesn’t need a transition model to perform the updates, environment supplies the connections instead\n* TD adjusts state to agree with observed successors, whereas ADP adjusts state to agree with all successors that might occur weighed by probability\n* Prioritized sweeping heuristic adjusts states whose likely successors have undergone large adjustment in utility estimates first\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\n* Active reinforcement learning agent gets to decide what actions to take instead\n* Needs to learn complete transition model with outcome probabilities for all actions rather than just the policy \n* Sometimes acting greedy will converge to optimal solution, sometimes not\n* Agent still must make a tradeoff between exploration of current best action and exploration of unknown states like in bandit problems\n* Greedy in limit of infinite exploration (GLIE) tries teach action in each state unbounded number of times to avoid having a finite probability that the optimal action is missed\n* Can have it randomly choose an action every so often, or give weights to actions it has not tried often, avoid actions with low believed utilities\n* Have an exploration function then that determines how greed is traded off against curiosity\n* Many actions are irreversible, nothing can restore it to the state you just left from\n* Worst case agent entered an absorbing state where no actions have any effect and no rewards are received \n* Agent in real settings needs to avoid both of these\n* Can end up in a bad state because our model is known or because its incorrect\n* Bayesian reinforcement learning assumes prior probability over hypotheses h about what the true model is, agent can decide to stop learning and get highest expected utility path\n* This becomes an exploration POMDP where belief states are distributions over models\n* Second approach from robust control theory allows set of possible models without assigning probability, then defines optimal robust policy as one that has the best outcome in worst case\n* This can lead to overly conservative behavior\n* Temporal difference Q learning, has to construct active temporal difference learning agent to learn transition model so it can choose an action based on utility function and look ahead\n* Q-learning method avoids need for the model by learning the action utility function instead of utility function, doesn’t need a transition model either for learning or for action selection\n* SARSA is closely related, state action reward state action, update rule is very similar except it updates with Q value of action that is actually taken\n* Q learning is off policy because it learns Q values to whether actions would be worth in this state if you stopped whatever policy you are using and act according to policy that chose best action \n* SARSA is on policy since it learns Q values that ask whether action would be worth it if you stick to the policy, Q learning more flexible as a result\n* Both learn optimal policy but slower than ADP agent\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\n* Generalization in reinforcement learning\n* Look ahead search means behavior can be generated from much simpler utility function approximation learnable from fewer experiences\n* State and sum of rewards are a training example for supervised learning algorithm\n* For RL, makes more sense to use online learning algorithm that updates parameters after each trial\n* Changing the parameters theta_i in response to observed transition between two states changes value of U_theta for every other state in Widrow-Hoff rule for online least squares\n* For passive TD learning, update rule can be shown to converge to closest possible approximation to true function when function approximate is linear in features, when nonlinear anything can happen\n* Need to go beyond linear function approximations because may be no good linear function that comes close to approximating utility function, and may not be able to invent features in new domain\n* Use deep reinforcement learning to try and explore nonlinear function approximations \n* Try to figure out what went wrong in long series of actions, credit assignment\n* Reward shaping, supplying agent with psuedorewards for making progress, can speed up learning quite a bit\n* Hierarchical reinforcement learning, break up long action sequences into smaller pieces, until action sequences are short enough to make learning easy\n* Begins with partial program that outlines hierarchical structure for agents behavior, give it a choose operator to allow agent to do anything from specific set\n* Based on concept of joint state space, each state composite of physical and machine state\n* Choice state is one where program counter for m is at choice point in agent program\n* Can be effective for learning complex behaviors\n* Provides additive decomposition of overall utility function as well\n\n\n## Module 9 Lecture Notes\n\nDouble bandits\n* slot machines, can create MDP when you pull the lever and either win or lose from each state, no discount, both states have same value\n* solving MDPs if offline, all quantities calculated out and you know the details of the MDP, dont need to actually play\n* Online planning suddently different, have to figure out what is going on with the states\n* learning instead of planning, still and MDP but you have to figure it out instead of being able to do so ahead of time\n* Exploration - tried unknown actions to get information\n* Exploitation - eventually use whatever you learn/know already\n* Regret - even if you learn well, still make mistakes\n* Sampling - have to try things many times because of chance\n* Difficulty - learning can be harder than solving MDP computationally\n\nReinforcement learning\n* have an agent that provides action to environment, that returns state s and reward r\n* get feedback in form of rewards, utility of the agent defined by these reward functions, so must maximize the expected rewards\n* all fo the learning is done based on samples of the outcomes\n* still have set of states s in S, actions a in A, model T(s,a,s'), reward function R(s,a,s'), policy pi(s)\n* now we don't know T or R so must try actions to learn what to do and which states to prefer\n* offline is MDPs, online is reinforcement learning\n\nModel-Based learning\n* learn approximate model based on observations, solve for the valuess as if what you learned is correct\n* 1) learn MDP model from experience by counting outcomes s' for each state and action pair, normalize to get estimate of T(s,a,s'), discover each R(s,a,s') when (s,a,s') happens\n* 2) solve the learned MDP, can use value iteration as before\n\nModel Free learning:\nPassive Reinforcement learning\n* now do policy evaluation, given a fixed policy pi(s) and unknown T and R functions, want to learn the state values\n* learner has no choice what actions to take, just execute the policy and learn from what happens\n* actions actually taken, no offline planning involved\n\nDirection Evaluation\n* want to compute values for every state given the current policy\n* do what the policy says, write down the sum of discounted rewards each time you visit a state, and average them all together to get a decent estimate\n* easy to understand, don't need to know T or R, eventually gets the correct averages using transitions\n* but can waste information and state connection s, takes a long time since each state is learned separately\n* in policy evaluation, updates calculaed V for the fixed policy by replacing it with the lookahead step\n* we need T and R to do it though, so not possible here, how can we update the weighted average without knowing the weights\n\nSample Based Policy Evaluation\n* want to calculate the bellman equation of V_k+1 pi_i(s) = $\\sum_s' T(s,pi_i(s),s')[R(s,pi_i(s),s') + yVk pi_i(s')]$, take sames of outcomes s', take action and average\n* sample_1 = R(s, pi(s), s1') + yV_k pi(s1')\n* sample_2 = R(s, pi(s), s2') + yV_k pi(s2')  \n* V_k+1 pi_i(s) = 1/n $\\sum_i sample_i$\n\nTemporal difference learning\n* want to learn from every experience, update V(s) every time we have a transition, so likely s' will contribute more often\n* still a fixed policy with the evaluation, keep a running average fo the successor values\n* samples of V(s): sample = R(s,pi_i(s),s') + yVpi(s')]\n* updates to V(s): Vpi(s) = (1 - a)Vpi(s) + a(sample)\n* same update: Vpi(s) = Vpi(s) + a(sample - Vpi(s))\n* keep an exponential moving average with running interpoliation uupdate: x_n = (1 - a) * x_n-1 + a * x_n\n* makes the recent samples more important and forgets about the distance past values that were less accurate anyway\n* x_n = x_n + (1 - a) * x_n-1 + (1 - a)^2 * x_n-2 + ... / (1 + (1 - a) + (1 - a)^2 + ...)\n* if we decrease the learning rate, can get converging averages\n* mimicks Bellman updates with these running averages, but if we want to turn values into new policy, can't do it because we need Q values for policy\n* so learn Q-values instead of values, makes the actino selection model free as well\n\nActive reinforcement learning\n* full reinforcement learning has optimal policies where you don't know transitions or rewards but can choose actions, learn optimal policy/values\n* tradeoff exploration and explotation\n* Q learning, find the successive depth limited valeus with value iteration: Vk+1(s) = max_A $\\sum_s' T(s,a,s')[R(s,a,s') + yVk(s')]$\n* Q-values more useful so we use them instead: Qk+1(s,a) = $\\sum_s' T(s,a,s')[R(s,a,s') + yMax_a'Qk(s',a')]$\n* start with both terms as 0 which we know is right\n* receive a sample (s,a,s',r), consider old esimate Q(s,a)\n* look at new sample estimate: sample = R(s,a,s') + ymax_a' Q(s',a'), incorporate into running average Q(s,a) = (1 - a)Q(s,a) + a[sample]\n* acting suboptimally, but q learning converges to optimal policy, called off-policy learning\n* to do so, need to explore enough, make learning rate small enough but not too quickly, doesn't matter how you select actions\n\nExploration\n* can pick random actions, epsilon greedy, flip a coin and act randomkly with some probability epsilon, otherwise act on current policy\n* if we do this, can thrash when learning is done, could also lower epsilon over time or look at exploration functions\n* explore areas where we are not sure how bad they are, can eventually stop exploring\n* take a value estimate u and visit count n, return optimistic utility like f(u,n) = u + k/n\n* instead of regular q update Q(s,a) = a R(s,a,s') + y max_a' Q(s',a'), do modified q update: Q(s,a) = a R(s,a,s') + y max_a' f(Q(s',a'), N(s',a'))\n* propagates bonus back to states that lead to unknown ones\n\nRegret - measure of total mistake cost, difference between expected rewards (with suboptimal moves) and optimal expected rewards\n* requres optimal learning to be optimal in minimizing regret, not possible often\n\nApproximate Q-learning\n* basic q-learning keeps a table of all q-values, can't do this often and learn about each state, too many to visit in training/hold in memory\n* generalize instead from small number of training states from experience to new similar situations\n* from evaluation functions, score non-terminals in depth limited search and wanted to return actual minimax function but had to do linear sum of features eval(s) = w1f1(s) + ... wnfn(s)\n* here we can write a q function for any state with a few weights:\n* V(s) = w1f1(s) + ... wnfn(s)\n* Q(s,a) = w1f1(s,a) + ... wnfn(s,a)\n* good since our experience is summer up in a few numbers, but states may share features but be very different in value\n* have our transition function (s,a,s',r), difference = [r + y max_a' Q(s',a')] - Q(s,a)\n* exact Qs are Q(s,a) = Q(s,a) + a * difference, approximate Qs are wi = wi + a * difference * fi(s,a)\n* we adjust the weights of the active features, disprefer bad states if something bad happens, can justify with online least squares\n\n\n\n## Module 9 Definitions\n* **Off-policy learning** - Q learning in active reinforcement learning converges to optimal policy\n* **N-armed bandit** - Slot machine with n levers\n* **One armed bandit** - slot machine with only one arm M that produces rewards and has fixed cost for each pull\n* **Restart mdp** - learn about the MDP through Gittins index and go back to the initial state\n* **Selection problem** - choosing the best option of a bandit environment under certain conditions\n* **Bandit superprocess** - each arm is full MDP rather than a Markov deicion prosess with one action\n* **Opportunity cost** - how much utility is given up by not devoting time to another arm\n* **Dominating policy** - An optimal policy if it exists for ann MDP\n* **Reinforcement learning** - agent interacts with world and periodically gets rewards based on how its doing\n* **Sparse rewards** - agent given no informative reward signal for most of the game\n* **Passive reinforcement learning** - fixed policy and need to learn utility of states\n* **Active reinforcement learning** - agent must also figure out what to do and learn utilities\n* **Exploration** - agent must learn as much about its environment as possible\n* **Trial** - executing the policy in passive reinforcement learning\n* **Pseudoexperience** - simulating the current transition model in TD and ADP\n* **Greedy agent** - greedily takes the action it believes to be optimal at each step in ARL\n* **Evaluation function** - compact measure of desirability for vast state spaces\n* **Function approximation** - process of constructing compact approximation of true utility or Q function\n* **Catastrophic forgetting** - assign zero weights to features when model learns too well\n* **Experience replay** - replay experience perioidcally to avoid catastophic forgetting\n* **Credit assignment** - figuring out where agent went wrong in long string of actions/states\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\n* Agents need to handle uncertainty due to partial observability, nondeterminism, or adversaries\n* Belief state has drawbacks, agent must consider every possible explanation for sensor observations\n* Also must handle every eventuality can grow arbitrarily large and have unlikely contingencies, and sometimes there is no plan guaranteed to achieve the goal, hut still has to act\n* Logical qualification problem, may have a plan but still never arrive due to many reasons, can still be the right thing to do\n* Rational decision depends on relative importances of various goals and likelihood that they will be achieved\n* Trying to use logic to cope with domain like medical diagnosis fails for several reasons, laziness (too much work to list everything), theoretical ignorance, and practical ignorance \n* Agent’s knowledge can provide only degree of belief in relevant sentences\n* Main tool for dealing with these is probability theory, theory of probability provides way of summarizing uncertainty that comes from laziness and ignorance\n* Agent must have preferences among different outcomes of various plans, use utility theory to represent preferences and reason quantitatively\n* Utility of the state is relative to the agent\n* Decision theory = probability theory + utility theory\n* Agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all possible outcomes, principle of maximum expected utility\n* Represent possibilities and probabilities\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n\n* Possible set of all worlds in probability theory is the sample space, worlds are mutually exclusive and exhaustive \n* Probability model associates numerical probability with each possible world\n* Events are sets of actions, sets of worlds correspond to proposition in formal language\n* Unconditional or prior probabilities refer to degree of belief in absence of other information\n* Usually have some information called evidence, in which case look at conditional or posterior probabilities \n* Product rule: P(a and b) = P(a|b)P(b)\n* Variables in probability theory are random variables\n* P can be probability distribution for each random variable, or assignment of probability for each possible value\n* Categorial distribution if there is a finite discrete range\n* Joint probability distributions for two or more variables probability of occurring at the same time\n* Possible world defined to be an assignment of values to all random variables under consideration\n* Could have full joint probability distribution if joint probabilities are known for every variable\n* Inclusion-exclusion principle: P(a or b) = P(a) + P(b) - P(a and b)\n* If agent 1 expresses set of degrees of belief that violate axioms of probability theory then combination of bets by agent 2 guarantees agent 1 loses money every time\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n\n* Probabilistic inference: computation of posterior probabilities given query propositions with observed evidence\n* Full joint distribution used as the knowledge base to answer questions\n* Can isolate one probability by adding all entries in a given row to get marginal probability\n* Marginalization of summing out is summing all possible values of other variables to take them out of the equation\n* Most interested in computing conditional probabilities of some variables given evidence about others\n\n\n## AIMA Chapter 12.4 Independence\n\n* Independence between propositions a and b can be written as \n* P(a|b) = P(a) or P(b|a) = P(b) or P(a and b) = P(a)P(b)\n* Independence assertions usually based on knowledge of the domain\n* If complete set of variables can be divide into independent subsets, then full joint distributions can be factored into separate joint distributions based on them\n* Can reduce size of domain representation and complexity of inference problem, but rare to actually find\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\n* Bayes rule underlies most modern AI systems for probabilistic inference\n* Can compute single term P(b|a) with all other terms P(a|b)P(b)P(a)\n* Often have good probabilistic estimates for some values but not all\n* Diagnostic knowledge in bayes is often more fragile than causal knowledge\n* If dependent on multiple variables, need independence to refine our statements a bit\n* May not be independent, but can be independent given the presence or absence of another variable\n* Creates conditional independence if P(a and b|c) = P(a|c)P(b|c)\n* This leads us to P(X|Y,Z) = P(X|Z) and P(X|Y,Z) = P(Y|Z)\n* Conditional independence assertions can let probabilistic models to scale up and much more commonly available than absolute independence assertions\n* The middle variable then separates the other two since they are a direct cause of both of them\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\n* Bayesian classifiers can work well even when conditions independence is not strictly true\n* Can obtain probability of the cause given some observed events\n* For each cause, multiply the prior probability of the cause by the product of the conditional probabilities of the overspeed effects given the cause, then normalize\n* Linear in number of observed events, doesn’t depend on number of unobserved effects\n* Can be used for text classification, given text decides which predefined set of classes it belongs to\n* Classify them each into a category\n* Model has prior probabilities and conditional probability, estimated as fraction of all previously seen documents that are of category c\n* Check which key words appear in the document then obtain posterior probability distribution over the categories\n* If predicting one class, take the highest posterior probability\n* Assumes words occur independently in documents and determine frequencies by document category\n* Model is overconfident in predictions, but rankings can still be very accurate\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n\n* Wumpus world again\n* Agents sensors only give partial information about the world\n* Pure logical inferences can’t conclude which squares are the most safe, so may have to randomly choose\n* Use the probabilistic agent instead\n* Calculate the probability that each square is a pit\n* As before, sum over entries from the full joint distribution\n* To get efficient solutions, independence and conditional independence relationships can be used to simplify the summations required\n* Axioms of probability constrain the probabilities of logically related propositions, if an agent violates them they may behave irrationally\n* Bayes rules allows unknown probabilities to be computed from known conditional probabilities, often in causal direction\n* Naive babes models assumes conditional independence of all effect variables given a single cause variable, and grows linearly in number of effects\n\n\n## SLP Chapter 3.1 N-Grams\n\n* N-gram language model\n* Assign probability to next possible word, language models assign probabilities to sequences of words\n* N-gram is sqeucen of n words that assigns probability to sentences and sequences of words, refers to either the word sequence itself or the predictive model that assigns it a probability\n* Probability of word given some history, could do so with relative frequency counts \n* With a large enough corpus, could compute counts and estimate the probability, but most of the time isn’t big enough to give good estimations\n* Can decompose the probability of entire sequences using the chain rule of probability and make them dependent on all the words that came before the current word of interest\n* Illustrates the link between computing joint probability of sequence and computing conditional probability of word given previous words\n* Instead of computing probability of word given entire history, can approximate history using just the last few words\n* Bigram looks only at the previous word, making a Markov assumption that the word only depends on the previous word probability\n* Can generalize this to tri-gram and n-gram as well\n* Estimate probabilities using maximum likelihood estimate or MLE, get it by getting counts from corpus and normalizing the counts so they’re between 0 and 1\n* Gets the relative frequency by diving the observed frequency of a particular sequence by the observed frequency of a prefix\n* Resulting parameter maximizes the likelihood of the training set given the model\n* Some bigram statistics encode facts that are strictly syntactic is nature, like parts of speech combinations\n* More common to use trigram models or 4/5-gram models when sufficient training data exists \n* Always compute language model probabilities in log format as log probabilities \n\n\n## SLP Chapter 3.2 Evaluating Language Models\n\n* Evaluate performance of model by embedding it in application and measuring how the application improves, called extrinsic evaluation\n* Only way to know if improvement in component is going to help the task at hand\n* Intrinsic evaluation measure the quality of a model independent of any application, requires a test set\n* Probabilities of the n-gram model come from the training set, test performance on test set and held out set, want to assign higher probabilities to the test set to get better model\n* Training on test set introduces bias that makes probabilities too high and has inaccuracies in perplexity\n* Divide data into training, development, and test set\n* Use perplexity as our metric instead of raw probability, inverse probability of the test set normalize by the number of words\n* The higher the conditional probability of the word sequence, the lower the perplexity, so minimizing it is equivalent to maximizing test set probability\n* Can also look at weighed average branching factor of a language, or number of possible next words that can follow any word\n* In computing perplexities, n-gram model must be constructed without any knowledge of the test set of vocabulary of the test set, or will be artificially low perplexity\n* Perplexities of two language models are only comparable if they use identical vocabularies\n* Intrinsic improvement in perplexity doesn’t guarantee extrinsic improvement in performance of language processing task like speech recognition\n* Should always be confide by an end to end evaluation of a real task\n\n\n## SLP Chapter 3.3 Sampling sentences from a language model\n\n* Sampling from distribution chooses random points according to their likelihood\n* Sampling from language model generates sentences, choosing each sentence according to likelihood as defined by model, so likely to generate sentences the model thinks has high probability over ones with low probability \n* Bigrams done by generating random bigrams that start with s, them choose random bigrams stating with next word and so on until sentence end token is hit\n\n\n## SLP Chapter 3.4 Generalization and Zeros\n\n* Dependent on training corpus, so probabilities often encode specific facts about training corpus, also tend to do better as N increases\n* Can be very similar to trained corpus depending on size and how accurate they become, and size of N\n* Statistical models likely to be useless as predictors if training and test sets are extremely different\n* One solution is to have the training corpus be a similar genre of text to the test set, give it appropriate dialect and variety as well when processing different genres like social media and spoken transcripts versus official documents\n* Matching genres and dialects not sufficient, may be too sparse and missing perfectly good English word sequences from the training set\n* Zeros that are never seen in the training set but in test set mean we are underestimating probability of all sorts of words that may occur, and if probability of any word inn the test set is 0 the entire probability of the test set is 0\n* Closed vocabulary systems are language tasks where we know this won’t happen since we know all the words that can occur, can be reasonable in some domains but not in others\n* Other contexts we have unknown of out of vocabulary words we have never seen before, OOV rate is percentage that occur in test set\n* Open vocabulary system is one where we model potential unknown words in test set by adding a pseudo word\n* Two common ways to train probabilities of unknown word model, one is to turn problem back into closed vocabulary one by choosing fixed vocabulary in advance\n* In this way, we choose a vocabulary that is fixed in advance, convert the training set any word that is not in the set to unknown word token and estimate the probabilities for it like any other word in the set\n* Second alternative is to create a vocabulary implicitly and replace words with pseudo word based on their frequency\n\n\n## SLP Chapter 3.5 Smoothing\n\n* To keep language model from assigning zero probability to unseen events, have to do smoothing or discounting to shave off probability from more seen events and give them to this unseen event\n* Laplace smoothing, add-k smoothing, stupid backoff, and Kneser-Ney smoothing all methods\n* In Laplace, add one to all n-gram counts before normalizing them into probabilities, doesn’t perform well enough to be used in model n-gram models but important conceptually\n* Also called add-one smoothing, just adds one to each count, define an adjusted count to be this added terms and can compare it to the standard MLE count\n* Discounting smoothing, lowering some non-zero counts in order to get probability mass that is assigned to zero counts. Instead of referring to discounted counts, may have a smoothing algorithm in terms of relative discount\n* Add-k smoothing, move a bit less of the probability mass to the unseen events, instead of adding one we add a fractional count k. This requires that we have a method for choosing k, like optimizing on dev set\n* Sometimes using less context is a good thing to generalize more for contexts we have not learned much about\n* In backoff, we use trigram if evidence is sufficient, otherwise the bigram, otherwise unigram, only back off to lower order n-gram if we have zero evidence for a higher order n gram\n* Interpolation, mix the probability estimates from all n-gram estimators, weigh and combine the trigram, bigram, unigram counts\n* Both simple interpolation and conditional interpolation lambdas learned from held out corpus\n* In backoff, continue backing off until we reach a history that has some counts, have to discount the higher order n-grams to save some probability for lower order ones\n* Katz backoff we rely on discounted probability if we have seen n-gram before, otherwise recurveiyl back off to probability for shoter history n gram, good with good-tuning smoothing method \n\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n\n* Can build very large language models from web\n* Efficiency considerations important when building language models that use large sets of n-grams\n* Can be shrunk by pruning or stored as a hash number instead of a string\n* Could also build approximate language models with bloom filter techniques, or use sorted arrays to build probability tables in minimal number of passes through large corpus \n* Very large language models can have much simpler algorithms that are sufficient, called stupid backoff\n* Gives up trim to make language model true probability distribution, no discounting of higher-oder probabilities, just backoff to lower order n-gram weighed by fixed weight if there is a zero count\n\n\n## Module 10 Lecture Notes\n\n* observed variables (evidence): agent knows things about the state of the world through sensors \n* unobserved variables (states): agent needs to figure out other aspects, what/where something is \n* model: agent knows some information about the how known variables ralte to unknown variables\n* probabilistic reasoning creates framework for dealing with all of these\n* probabilities about possible worlds, quantify how likely a possible world is\n* all possible worlds are the sample space, mutually exclusive and exhaustive in probability theory\n* a correct model associates a probability P(w) with each world w\n\nRandom Variables - some aspect of the world where we may have uncertainty, denoted as capital letters\n* probability of any possible world has to be between 0 and 1, and total probability of all possible worlds has to equal 1\n\nProbability distributions\n* unobserved random variables have distributions, or table of probabilities\n* all probabilities must be greater than or equal to 0, sum of all probabilities of variable equal 1\n\nJoint distributions\n* specifies a real number for each assignment or outcome, over a set of random variables\n* probability of two random variables having some value, same principles old\n* the size of the distribution of n variables with domain size d is impractical to calculate except for small distributions\n\nProbabilistic models\n* joint distribution over set of random variables\n* include random variables with domains, assignments called outcomes, joint distributions that assign probability, normalized (sum to 1), hopefully only certain variables directly interact\n* event is a set E of outcomes, sum across however many questions we have or that fit the description\n* e.g. prob that its hot, prob that its hot and sunny, or sunny, etc.\n* events we care about usually are partial assignments\n\nMarginal Distributions\n* sub-tables that eliminate variables, can combine collapsed rows by adding probabilities together\n\nConditional Probabilities:\n* relation between joint and conditional probabilities, taken as the definition\n* p(a|b) = P(a,b) / p(b)\n* use relationships to calculate probability of an event given another event\n* conditional distributions are probability distributions over variables given some other variable values\n\nProbabilistic Inference:\n* compute desired probability from other known probabilities\n* usually interested in conditional probabilities, correspond to agent's beliefs given the evidence\n* these probabilites change with new evidence, beliefs of the agent update\n\nInference by Enumeration\n* generally have evidence variables E with some value e, query variable Q, hidden variable H, want P(Q| e1...ek)\n* 1) select entries consistent with evidence, 2) sum out H to get joint of Query and evidence, 3) Normalize\n* P(Q, e1, .. ek) = sum (h1..hr) P(Q, h1 .. hr, e1 .. ek)\n* normalize by taking Z = sum_z P(Q, e1...ek), multiply 1/Z by P\n\nProduct rule: P(y)P(x|y) = P(x,y)\n\nChain rule\n* can write joint distributions as incremental product of conditional distributions\n* P(x1, x2, x3) = P(x1)P(x2|x1)P(x3|x1,x2)\n* P(x1, x2, ... xn) = mult_i (P(x_i|x1...x_i-1))\n\nBayes Rule\n* P(x,y) = P(x|y)P(y) = P(y|x)(x)\n* we can divide this out to get P(x|y) = (P(y|x) / P(y)) * P(x)\n* can build a conditional from reverse, one often more difficult to find than the other\n\nProbabilistic Language Models\n* want to assign a probability to a sentence\n* autocomplete, machine translation, spell correction, speech recognition, summarization, Q&A, dialog systems\n* complete the probability of a sentence or sequece of words (P(W) = P(w1,w2,..wn), and probability of upcoming word P(w3|w1,w2)\n* model that does either of these is called a language model P(W) or P(wn | w1,..wn-1)\n* P(W) calculated with chain rule of probability\n* P(B|A) = P(A,B)/ P(A), can rewrite as P(A,B) = P(A)P(B|A)\n* P(A,B,C,D) = P(A)P(B|A)P(C|A,B)P(D|A,B,C)\n* joint probability of words in a sentence: P(w1,w2,..wn) = mult_i P(wi|w1w2..wi-1)\n* P(the eagles won) = P(the) * P(Eagles|the) * P(won|the Eagles)\n* Estimate probabilities using maximum likelihood estimate (MLE), doesnt work\n* only depends on the previous k words, not entire context, K is number of context words we take into account\n* unigram (no history), bigram (1 word history), trigram (2 words), 4-gram (3 words)\n\nUnigram model:\n* P(w1w2...wn) = mult_i P(wi)\n* only look at the word itself, no history\nBigram model:\n* condition on previous word: P(wi|w1w2...wi-1) = P(wi|wi-1)\n\nN-gram models\n* can extend this to trigram, 4,5-grams\n* geerally insufficient model of language because of long distance dependenceis, but often can get away with using N-gram models\n\nLanguage Modeling\n* Estimating bigram probabilities\n* maximum likelihood estimate: P(wi|wi-1) = count(wi-1,wi) / count(wi-1)\n* MLE has problems when zeros exist, assign 0 probability to all sentences containing it if it isnt close enough\n* out of vocabulary items (use term <unk> to normalize training data by replace word not in L with it), fixed lexicon L of size V, try to avoid zeros with smoothing\n* practical issues: do everything in log space to avoid underflow: log(p1 * p2 * p3 * p4) = log(p1) + log(p2) + log(p3) + log(p4)\n\n\n## Module 10 Definitions\n* **Sample space** - the possible worlds, mutually exclusive and exhaustive\n* **Marginalization** - summing out, combining rows in distributions by adding probabilities\n* **Belief state** - representation of the set of all possible world states agent might be in\n* **Factored representation** - possible world is represented as set of variable/value pairs\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n\n* Bayesian network expressions, can represent dependencies among variables\n* Can represent any full joint probability distributions often very concisely\n* Each node corresponds to continuous or discrete random variable, directed links or arrows connect pairs of nodes, if there is an arrow from X to Y say that X it the parent of Y\n* No directed cycles and is a directed acyclic graph as a result, and each node has associated probability information that quantifies effect of the parents using number of parameters\n* X has a direct influence on Y, easier to specify than the probabilities that they influence the others\n* Only need local probability information once the map is laid out in form of conditional distribution given the parents\n* Each row in CPT contains conditional probability of each node value for conditioning case, every row must sum to one\n* Node with no parents only has one row, boolean parents have 2^k independently specifiable probabilities\n* Probabilities summarize potentially infinite set of circumstances in the environment\n* Syntax is directed acyclic graph with some local probability information tied to each node\n\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n\n* Semantics, how the syntax corresponds to joint distribution over the variables in the network\n* If a representation of joint distribution, can be used to answer any query by summing relevant joint probability values\n* When one estimates values for conditional distributions, need to be actual conditional probabilities for the variable given its parents\n* First rewrite entries in joint distribution in terms of conditional probability using product rule\n* Repeat the process reducing to join smaller sets of variables to get one big product, the chain rule\n* Determine the set of variables required to model the domain, order them somehow\n* For links, choose minimal set of parents and insert link from parent to child, write down conditional probability in the table\n* Parents should contain all nodes that directly influence them\n* No redundant probability values, so no change of inconsistency so impossible to create a Bayesian network that violates the axioms of probability\n* Can be specified by 2^K * n variables, k is other nodes it is influence by and n is number of Boolean variables, 2^n in joint distribution\n* Can leave out links even if slight dependencies exist, slight accuracy gain not worth complexity\n* Order of the network can affect the number of links and complexity \n* Each variable is conditionally independent of its non-descendants given its parents\n* Network can be viewed as instead defining a set of conditional independence properties and full joint distribution can be derived from the properties\n* Variable is conditionally independent of all other nodes in network given its parents children and Childrens parents - Markov blanket\n* D-separation: consider just ancestral subgraph of X,Y,Z and ancestors\n* Add links between unlinked pairs of nodes that share common child and now have a moral graph\n* Replace all directed links with undirected links\n* If Z blocks all paths between X and Y in the resulting graph then Z d-separates X and Y, X is conditionally independent of Y given Z, otherwise Pringle Bayesian net doesn’t require conditional independence\n* Worst case scenario where parents and child relationship is arbitrary described by canonical distribution to fit some pattern\n* Simplest example provided by deterministic nodes\n* Context specific independence, in conditional distribution if variable is conditionally independent of some of the parents given certain values of others\n* Uncertain relationships can be characterized by noisy logical relationships, O(k) parameters for k parents instead of O(2^K) for full conditional probability table\n* For continuous quantities, do discretization, define continuous variable using family of density functions, or nonparametric representation to define conditional distribution with collection of instances\n* Bayes network includes hidden variables that are neither input nor output by important for structuring network so it is reasonably sparse with manageable number of parameters\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n\n* Need to compute posterior probability for a set of query variables given event and assignment of values to evidence variables\n* Inference by enumeration: query can be answered using a Bayesian net by computing sums of products of conditional probabilities from the network\n* Add terms and multiply the numbers together, worse case have O(2^n) terms in sum, each a product of O(n) probability values so we have O(n2^n) complexity, can be reduced to O(2^n) by nesting computation\n* Enumeration:ask evaluates expressing trees using depth first recursion, similar to backtracking algorithm for solving CSPs, linear in number of variables, time complexity is always O(2^n)\n* Contains repeated subexpressions that we are trying to avoid\n* Do the calculation once and save the expression for later use, like dynamic programming\n* Variable elimination algorithm does this, evaluates expressions right to left or bottom up, then stores intermediate results, summations over each variable that depend on the value\n* Have factors that are matrices indexed by values of argument variables\n* Can sum out variables from product of factors by adding up sub matrices formed by fixing variable to each of values, matrices not multiplied until we need to sum out a variable from the product\n* Time and space complexity of variable elimination dominated by size of largest factor during algorithm runs\n* Find that every variable that is not an ancestor of a query variable or an evidence variable is irrelevant to the query\n* Complexity of Bayesian net depends on structure of network\n* Singly connected trees of polytrees have at most one undirected path between any two nodes in network, time and space complexity is linear in size of network\n* For multiply connected networks, variable elimination can have exponential time/space complexity in worst case even when parents are bounded\n* Includes inference in propositional logic as special case so Bayes net inference is NP-hard\n* Can reduce satisfiability problems to Bayesian net inference problems\n* Similarities between complexity of Bayes net inference and complexity of CPSs, tree width important in both\n* Can also reduced Bayes net inference to satisfiability\n* Using clustering algorithms or join Tre algorithms to solve posterior probabilities for all variable in the network can take it from O(n^2) time to O(n)\n* Join individual nodes to form cluster nodes then replace with meganodes \n\n\n## Module 11 Lecture Notes\n\nProbabilistic models\n* describe how a portion of the world workds, always a simplification and may not account for every variable/actions between them\n* with these models, our agents need to know what to do about unknown variables given evidence\n\n* Two variabels are independent if for all x, y P(x,y) = P(x)P(y), or for all x,y P(x|y) = P(x)\n* Simplifying modeling assumption, usually empirically find they are close to independent at best\n* N independent coin flips will all come out independently, doens't depend on previous trial\n\nConditional independence\n* toothache, cavity, and detection\n* if cavity, probability it is detected doesnt depend on toothache, P(detect|toothache, cavity) = P(detect|cavity)\n* same thing in reverse case: P(detect|toothace, no cavity) = P(detect| no cavity)\n* detect therefore conditionally independt of toothache given cabity\n* P(toothche|detect, cavity) = P(toothache|cavity), P(toothache, detect|cavity) = P(toothache|cavity)P(detect|cavity)\n* using the chain rule here again to derive these probabilities\n* unconditional independence is rare, doesnt allow us to say anything about other variables\n* most basic/robust form of knowledge about unknown environments\n* x conditionally independent of y given z if for all x,y,z P(x,y|z) = P(x|z)P(y|z) or P(x|z,y) = P(x|z)\n\nChain rule traffic rain umbrella\n* P(x1,x2,..xn) = P(x1)P(x2|x1)P(x3|x1,x2)...\n* P(traffic, rain, umbrella) = P(rain)P(traffic|rain)P(umbrella|rain,traffic) - decomposition\n* P(traffic, rain, umbrella) = P(rain)P(traffic|rain)P(umbrella|rain) - conditional independence\n* can express these assumptions with Bayes nets\n\nBayes Nets\n* problems with full joint distribution tables as probabilistic models: unless few variables way to big to represent all entries, hard to learn anything for more than a few variables ach time\n* Bayes nets have a technique for describing complex disjoint distributions using local distributions (models using conditional probabilities)\n* graphical models with local interactions chained together to get global indirect interactions\n* independence assumptions required to define the bayes net, can involve more conditional indepdences, crucial for modeling \n\nGraphical Model\n* Nodes are variables with domains that can be assigned or unassigned\n* Arcs are interactions that have direct influence between variables, encode conditional independence\n* picture that arrows imply direct causation even though generally not the case\n* N coin flips, N nodes with no interactions since they are all independent\n* Traffic: 1) rain and traffic independent, 2) rain causes traffic (more accurate)\n* traffic, rain, low pressure, roof drips, ballgame, cavity: L -> R -> B,T,D, B->T, C alone\n* alarm network, bulgary, alarm, mary calls, john calls, earthquake: B,E -> A -> M,J\n* have a set of nodes, one per variable and a directed acyclic graph\n* also have conditional distribution foe each node, or a collection of distributions over X, one for each parents values combinations\n* get a description of noisy process for causation\n* altogether is a graph and local conditional probabilities\n* implicitly encode join distributions as product of local conditional distributions\n\nProbabilties in Bayes Nets\n* probability bayes net gives to a full assignment, multiply conditionals together P(x1,x2,..xn) = mult(i->n) P(xi|parents(Xi))\n* guarenteed to result in propoer joint distribution, chain rule valid for all distributions P(x1,x2,..xn) = mult(i->n) P(xi|x1..xi-1)\n* assuming conditional independences P(xi|x1,x2..xi-1) = P(xi|parents(Xi))\n* not every bayes net can represent every joint dsitrbution, graph structure enforces conditional independences\n* only distributions with absolutely independent variables can be represented by Bayes net with no arcs\n\nCausality in Bayes Nets\n* when they reflect true causal patterns, often simpler with fewer parents, easier to think about and elicit from experts\n* don't actually need to be causal, sometimes not possible and get correlation arrows instead of causation\n* arrows really referring to conditional independence and may also be causal but not always\n* Bayes nets have joint distribution over N boolean variables of size 2^N\n* N-node net with k parents is O(N * 2^k+1)\n* can calculate these joint probabilties, sae space, get Conditional probability tables easier, answer queries faster\n* conditional independence is property of distribution\n\nD-separation\n* Causal chain, x->y->z, e.g. low pressure -> rain -> traffic\n* P(x,y,z) = P(x)P(y|x)P(z|y)\n* no guarenteed x independent of z, one set where this is not the case shows its not a guarentee, e.g. low pressure causes rain causes traffic, high pressure casues no rain causes no traffic\n* guarenteed x independent of z given y? Yes - P(z|x,y) = P(x,y,z) / P(x,y) = (P(x)P(y|x)P(z|y)) / (P(x)P(y|x)) = P(z|y)\n* evidence along the chain blocks the influence\n\nCommon Cause\n* y-> x,z, P(x,y,z) = P(y)P(x|y)P(z|y)\n* no guarentee x is independent of z, but yes guarenteed x and z indepedent given y\n* P(z|x,y) = P(x,y,z) / P(x,y) = (P(y)P(x|y)P(z|y)) / (P(y)P(x|y)) = P(z|y)\n\nCommon Effect\n* x,y->z, x and y are independent, but not x and y are not independent given z, backwards from all the other cases, now it activates influence between possible causes\n* now we can look at any given bayes net and see if two variables are independent, break down complex examples into simpler ones of these three cases\n\nReachability\n* want to hide evidence nodes and look for paths in the graph\n* identify active/inactve paths\n* x and y are conditionally independent given evidence variabes Z if x and y are d-separated by Z, have to consider all undirected paths from x to y, if no active paths then they are independent\n* path is active if each triple is active, causal chain (a->b->c where b is unobserved), common cause (a <- b -> c where b is unobserved), or common effect (a -> b <- c where b or one descendent is unobserved), if we have one inactive segment then the entire path is blocked\n\n* can now query if two variables are independent, check all undirected paths between the two inputs\n* if one ore more are active than the independence is not guarenteed, but if all paths are inactive then independence is guarenteed\n* can now run d-separation algorithm to build complete list of conditional independences that are true in the form: Xi indep. Xj | {Xk1..Xkn}\n* list determines set of probability distributions that can be represented, now calculate everything\n* can deduce guarenteed independencies of distributions from graphs, joint distribution may have more conditional independence not detectable until you look at specific distributions\n\nInference\n* calculate a useful quantity from the joint probability distributions\n* posterior probability P(Q|E1=e1..Ek=ek), most likely explanation argmax_q P(Q=q|E1=e1..)\n* same steps as before, have evidence variables, query variable, hidden variables\n* 1) select entries that are consistent with the evidence, 2) sum out H to get query and evidence, 3) normalize by Z, want to find P(Q|e1..ek)\n* in bayes nets, easy if we have unlimited time\n* inference by enumeration very slow, have to join up entire joint distribution before sum out hidden variables\n* variable elimination conversely you interleave joining and marginalizing, still Np-hard but much faster than inference by enumeration\n\n\n## Module 11 Definitions\n* **Conditional probability table** - local probability information attached to each node in the bayesian network, only for discrete variables\n* **Conditioning case** - possible combination of values for the parent nodes in bayesian network\n* **Topological order** - numbering nodes in any order consistent with directed graph structure\n* **Deterministic nodes** - Values specified exactly by the value of the parents with no uncertainty\n* **Leak node** - can add to model to cover miscellaneous causes\n* **Discretization** - splitting up possible values into fixed set of intervals\n* **Hybrid bayesian network** - network with both discrete and continuous variables\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\n* Components of agent include direct mapping from conditions on current state to action, means to infer relevant properties of the world, and information about the way world evolves and the results of actions an agent takes\n* Also utility information, action-value information, goal that describes most desirable states, problem generator/critic\n* Assumes little prior knowledge of the agent, transfer learning is when knowledge from one domain transferred to another\n* Induction is going from specific set of observations to general rule, may be incorrect whereas deductive had to be correct\n* Factored representation is vector of attribute values, dealing with classification and regression problems\n* Supervised learning: agent observed input-output pairs and learns a function that maps from input to output\n* Unsupervised learning: agent learns patterns in the input without explicit feedback, most commonly does clustering \n* Reinforcement learning agent learns from series of reinforcements, rewards and punishments\n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n\n* Giving training set of n example inputs, discover function that approximates true function f\n* Hypothesis drawn from hypothesis space, h is a model drawn from model class, or function from function class, ground truth is the true label\n* If we dont have prior knowledge about process that generated data perform exploratory data analysis\n* Look for best fit function for each function close to y\n* Evaluate performance on test set, generalizes well if it accurately predicts outcomes of test set\n* Bias is tendency of predictive hypothesis to deviate from expected value when averaged over different training sets\n* Underfiting when it fails to find pattern in data\n* Variance is amount of change in hypothesis due to fluctuation in training data\n* Overfitting when data pays too much attention to particular data set and performs poorly on unseen data\n* Bias-variance tradeoff between complexity, more complex low bias and low variance generalized better\n* Tradeoff between expressiveness of hypothesis space and computational complexity of finding good hypothesis in space\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n\n* Decision tree is representation of a function that maps vector of attributes values to single output value decision\n* Performs actions until a leaf is reached, boolean classification if it only true or false\n* Can often be very intuitive with interpretable results\n* Want to be consistent with examples and small as possible\n* Can generate error or noise in the data if nondeterministic results\n* Can evaluate performance with learning curve, proportion correct on test set with training set size\n* Importance measured by entropy, fundamental quantity in information theory, measure of uncertainty of a random variable, no uncertainty is 0\n* Information gain from attribute test is expected reduction in entropy\n* Larger hypothesis spaces have more capacity to fit and overfit\n* Can do decision tree pruning to combat overfitting, eliminate irrelevant nodes, can do a significance test to figure out when to cut the tree\n* Early stopping, have decision tree algorithm stop generating nodes when there is no good attribute to split on\n* Split point test, inequality test on value of an attribute can be used to deal with continuous values\n* Information gain ratio tells you which variables to split the data on\n* If trying to predict numerical output value then need regression tree instead of classification tree\n* Decision trees are unstable, adding one new example can change test at the root, random forest model can fix some of the issues\n\n\n## AIMA Chapter 5.5 Stochastic Games\n\n* Stationarity: the assumption that all future examples will be like the past, without it we are doomed\n* Assume each example has the same prior probability distribution and is independent of previous examples\n* Independent and identically distributed\n* Optimal fit is hypothesis that minimizes the error rate, measure this on the testt set of examples\n* Split data into training and test set, can try different hyper parameter classes\n* Need validation set as well to evaluate candidate models and choose the best one\n* Can do k-fold cross validation if we need more data than we have\n* Leave one out cross validation is extreme example of this\n* Model selection chooses good hypothesis class, optimization finds best hypothesis within the space\n* Complexity measured by number of decision tree nodes and by number of neural network parameters\n* Often training set error reaches 0 as complexity increases\n* Model that exactly fits all the training data has interpolated the data, start to overfit once we approach this\n* Want to minimize loss function instead of maximizing utility function in NN\n* Define prior probabilities and get generalization losses for hypothesis h\n* Agent can often only estimate empirical loss on set of examples\n* Learning problem is realizable if hypothesis space contains true function\n* Can be computationally intractable to search all possibilities to get f\n* Penalize hypotheses with regularization, L1 or L2 regularization\n* Can also reduce number of dimensions that model works with through feature selection\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\n* Computational learning theory, any hypothesis that is seriously wrong will almost certainly be discovered after only a few examples because of inaccurate predictions\n* Therefore any hypothesis consistent with sufficiently large set of training examples is unlikely to be seriously wrong, probably approximately correct (PAC)\n* Decision lists outline how PAC learning should be applied to new decision space\n* Branch only in one direction to test PAC \n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\n* Linear functions of continuous valued inputs\n* Coefficients thought of as weights in linear regression, loss function always convex\n* Minimizing loss through gradient descent, estimate the gradient and move in steepest downhill direction\n* Learning rate or step size tells us how quickly to move\n* Loss is quadratic so partial derivative will be linear\n* Batch gradient descent learning rule is over entire dataset at once, an epoch\n* Stochastic gradient descent randomly selects small number at each step, usually with minibatc\n* Can easily extend to multivariable linear regression problems, solve for the data matrices of weight and bias instead since have multiple regressions\n* Usually dont worry about overfitting in univariate linear regression, now need regularization to penalize large weights and avoid overfitting\n* L1 regularization tends to produce sparse models with many weights of 0\n* L2 does not tend to produce zero weights, doesn’t that dimensional axes seriously like l1 does\n* Can do classification as well as regression\n* Decision boundary is line that separates two classes, straight line, linearly separable if this can be done\n* Perceptron learning rule is simple weight update that coverages to solution provided they are linearly separable\n* Correct output, nothing changes\n* If y is 1 but w(x) is 0, wi increased when xi is positive and decreased when xi is negative\n* If y is 0 but w(x) is 1, wi decreased when xi is positive and increased when xi is negative\n* Use sigmoidal output function to make things easier with predictions\n* Process of fitting weights to minimize loss on dataset is logistic regression\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\n* Multinomial naive Bayes classifier, makes simplifying assumption about how features interact\n* Represent text document as bag of words, unordered set and frequency only kept\n* Returns the maximum class based on probability\n* Use Bayes rule to transform this expression into probabilities that have useful properties, can break down conditionals into three other probabilities\n* Naive bayes is a generative model because we can have implicit assumption about how a document is generated, sample a class, then generate words by sampling from conditional probability\n* Compute most probable class given document with the prior probability P(c) of the class and the likelihood of the document P(d|c), c = argmax c P(D|C)P(C)\n* Need some simplifying assumptions to calculate, bag of words assumption says the position doesn’t matter in the document\n* Naive Bayes assumption is second, conditional independence is that probabilities are independent given the class c and can be naively computed as P(f1|c) * P(F2|c) * …\n* Consider word position by walking an index through each word position in the document\n* Done in log space to avoid underflow and increase speed\n* Classifiers that use linear combination of inputs to make classification decision are linear classifiers\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n\n* Use frequencies in data to get estimates of the two terms\n* Maximum likelihood estimate, sum all of the instances divided by all of the words\n* But if we get a word we’ve never seen before can have zero probability and get error\n* Laplace smoothing, add one to pretend we’ve seen every word more than we actually have\n* If we encounter words never before seen, we usually ignore them, can also ignore stop words completely, which are a class of very common words\n\n\n## SLP Chapter 4.3 Worked Example\n\n* Sentiment analysis, + is positive sentiment, - if negative\n* Word doesn’t occur in training set, drop it completely\n* Get a higher probability for the negative sentiment than the positive sentiment, so we predict negative class for the test sentence\n\n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n\n* Whether a word appears or not seems to matter more than frequency, can clip word counts in each document at one, binary multinomial naive Bayes\n* For each document we remove all duplicate words before concatenating into single document\n* Also have to deal with negation when looking at sentiment classifiers\n* May have insufficient labeled training data to train accurate naive Bayes classifiers using all words in training set, so have to derive positive and negative word features from sentiment lexicons\n* Add a feature that is counted whenever a word from the lexicon appears\n* If training data is sparse or not representative of test set, dense lexicon features instead of sparse individual word features may generalize better\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n\n* Naive Bayes doesn’t require our classifier use all words in training data as features\n* Spam detection, deciding if email is spam or appropriate piece of mail\n* Rather than use all words as features, predefine likely sets of words or phrases as features combined with features that really not purely linguistic\n* Language id - determining what language a piece of text is written in, most effective to use character n-grams or byte n-grams isnstead of word n-grams\n* Often trained on multilingual texts\n\n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n\n* Naive bayes classifiers can use any sort of feature\n* We only use individual word features and use all words in the text, then similar to language modeling\n* Naive bayes can be viewed as a set of class-specific unigram language models in which the model for each class instantiates a unigram language model\n* Model assigns probability to each sentence as well, positive model will assign higher probabilities to sentences given a positive sentiment\n* Once we multiply in prior of full naive bayes model a different classification decision may be made\n\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n\n* Gold labels are human defined labels that we are trying to match up to\n* Can build a confusion matrix for visualizing how an algorithm performs with respect to human gold labels using system output and gold labels and label each cell as positive outcomes\n* Can also calculate accuracy, percentage of observations labeled correctly, generally not used for text classification tasks\n* Precision measures percentage of items that system detected that are in fact positive, true positives / true positives + false positives\n* Recall measures the percentage of items actually present that were correctly identified by the system, true positives / true positives + false negatives\n* F-measure incorporates both aspects of precision and recall\n* Precision/recall has to be adjusted slightly if we have more than 2 classes\n* Macro averaging, compute performance for each class then average over all classes \n* Micro averaging, collect decisions for all classes into single confusion matrix then compute precision/recall from a table\n\n\n## SLP Chapter 4.8 Test sets and Cross-validation\n\n* Use development set to tune some parameters, decide what the best model is in general\n* Use of a dev set avoids overfitting the test set, but it may not be large enough to be representative\n* Do cross-validation instead, k disjoint partitions called fold made, each one used as a test set in a run of the models\n* Pick a fold, remaining k-1 folds used in training, then fold used as the test set\n* Average it out, can do 10 times for 10-fold cross validation\n* Ned the whole corpus to be blind, can’t examine what is going on because then would be looking at the test set\n* Common to Crete fixed training set and test set, then do 10-fold cross validation inside the training set\n\n\n## SLP Chapter 4.9 Statistical Significance Testing\n\n* Do statistical significance hypothesis testing to see if one system is better at describing something than another\n* Effect size, bigger value means model A is doing way better than B, smaller says only doing a little bit better\n* H0 is null hypothesis, says value s actually negative or 0 so A is not better than B\n* Calculate the p-value, can reject null hypothesis if below a certain threshold value and statistically significant\n* Two common non-parametric tests in NLP is approximate randomization and bootstrap test\n* Paired tests are those where we compare two sets of observations that are aligned\n* Bootstrapping refers to repeatedly drawing large number of smaller samples with replacement from original larger sample\n* Acts as one-sided empirical p-value\n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n\n* Representational harms caused by a system that demeans a social group and perpetuates negative stereotypes\n* Toxicity detection task of detecting hate speech and other kinds of toxic language, want to reduce societal harm but can backfire\n* False positives can lead to censoring of important issues\n* Ml models replicate and amplify biases they find in their training data, can be cause by labels, resources used, model architecture \n* Can release a model card that details the model to see if any issues are immediately apparent\n\n\n## Module 12 Lecture Notes\n\n* want to acquire model from data/experience, learning the parameters, structure, hidden concepts\n* spam classification, get a large collection of emails labeled by hand, want to predict labels of future emails. Features are attributes that make decision, words, text patterns, non-text details\n* digit recognition, get labeled collection of images, predict new digit images, attributes are pixels and shape patterns\n* have inputs x, predict labels y, spam detection, medical diagnosis, essay grading, fraud detection, etc.\n\nModel-based classification\n* build a model (like a Bayes net) where label and features are random variables, fill in observed features and query for distribution fo labels conditions on features\n* Naive bayes for ditigs assumes all features are independent effects of the labels, one variable for each pixel position, on/off, maps to feature vector\n* general naive bayes learns for y labels, P(Y, F1, ...Fn) = P(Y) mult_i P(F_i|Y), have to specify how features depend on the class\n* number of parameters linear in number of features\n* inference for naive bayes wants to compute posterior distribution over label variable, so we do the following:\n* 1) get joint probability of label and evidence for each of the labels Y, 2) sum them together to get probability of evidence, 3) normalize by taking step 1 / step 2\n* to use naive bayes, need inference method (probabilities P(Y) and P(Fi|Y), calculate P(Y|F1..Fn)), and estimates of local contiional probability tables (P(Y) over labels, P(Fi|Y) for each feature, called parameters of model)\n* Bag of words Naive bayes has word features Wi at position i, predict label conditioned on feature variables, assume they are conditionally independent, but now each Wi is identically distributed\n* in BOW, each position identically distributed, all positions share same contioinal probabilitiy, model is insensitive to word order/reordering\n* spam filtering model e.g. P(Y, W1 ... Wn) = P(Y) mult_i P(Wi|Y) \n\nConcepts/Definitions\n* Data is labeled instances, separated into training set, held out set, test set\n* Features: attributes and values characterizing each x input\n* experimentation cycle: learn the parameters on training set, tune hyperparameters on held out set, compute accuracy of test set (don't look at it ever)\n* evaluation, calculate accuracy as fraction of instances predicted correctly\n* overfitting, fitting the training data very closely but not generalizing well to the test data set. Posterios determined by relative probabilities, if overfit can go to infinite\n\nGeneralization and Overfitting\n* relative frequency parameters overfit training data, don't want to assign unseen events zero probability\n* want to generalize better to the test set, so we smooth or regularize the estimates\n\nParameter estimation\n* need to estimate the distribution of a random variable, difficult to decide so use training data\n* for each outcome x, look at the empirical rate of the value as P(x) = count(x) / total samples, then this is the estimate that maxmizes the likelihood of the data L(x, theta) = mult_i P_theta(xi)\n* relative frequencies give mazimum likelihood estimates, theta_ML = argmax_theta P(X|theta) = argmax_theta mult_i P_theta(Xi)\n* can do Laplace smoothing, pretend you saw every outcome one more time than you did\n* P_lap(x) = c(x) + 1 / sum_x [c(x) + 1] = c(x) + 1 / N + |X|\n* or could pretend you say it k extra times, P_lap(x) = c(x) + k / N + k|X|, k is the strength of the prior\n* for conditionals, smooth each condition separately: P_lap(x) = c(x,y) + k / c(y) + k|X|\n* could also do linear interpolation, get empirical P(x) from the data, make sure estimate P(X|Y) isn't too different from P(x) by P_lin(x|y) = a P_hat(x|y) + (1 - a)P_hat(x)\n* smoothing critical in real classification problems\n\nHeld out data tuning\n* we get two unknowns now, the parameter probabilities P(x,y), P(y) and the hyperparameters (type of smoothing, k, a, etc.)\n* learn paramters from training data, tune hyperparameters on this set, for each value of hyperparameters, train and test on held out set, choose best values and do final run on test data\n\nFeature errors\n* if we get errors, can sometimes need more features as words are not enough\n* add these new information sources as new variables in Naive bayes model\n* naive bayes can incorporate many features, but generally do better with homogenous cases of features\n\nBaselines\n* need a baseline as first step, see how hard task is and what a good accuracy entails\n* a weak baseline gives the most frequent label classifier, give the test set whatever label was most common, accuracy may be high or low, real research uses previous work as baselines\n* confidence of classifier comes from posterior over top label, confidence(x) = max_y P(y,x)\n* says hos sure the classifier is of a prediction, comes with any probabilistic model but no guarentee it is correct\n* weak calibration - higher confidences mean higher accuracy, strong calibration - confidence predicts the accuracy rates\n* bayes lets us do queries with causal probabilities, naive bayes takes all features to be independent, need smoothing and confidences helpful when you can get them\n\nPerceptrons\n* Loosely inspired by human neurons\n* linear classifiers, inputs are feature values, each feature with a weight and the sum if the activation\n* activation_w(x) = sum_i wi * fi(x) = w * f(x), if >= 0 output +1, < 0 output -1\n* in binary case, compare features to weight vector, trying to learn weight vectors from examples\n* have inputs, weights, weighted sum, activation function with theshold, output\n* output is 0 if sum_j wjxj <= theshold, 1 if sum_j wjxj > theshold\n* by varying weights and threshold we get different models of decisions\n* write sum_j wjxj as dot product of input vector and weight vector, w * x\n* move threshold to other side of inequality, define bias as the -1 * theshold b = -theshold\n* output is 0 if wx + b <= 0, 1 if wx + b > 0\n* have complex network of perceptrsont that make subtle decisions\n\nBinary Decision rule\n* in space of feature vectors, each example is a point, weight vector is hyperplane, one side +1, other -1\n* start with weights = 0, for each training instance classify with weights, if correct don't change anything, if wrong adjust the weight vector\n* y = +1 if w * f(x) >= 0, -1 if w * f(x) < 0, adjust w by w = w + y* * f, subtract if y* is -1, separable case very intuitive\n\nMulticlass decition rule\n* now we have weight vector for each class w_y, score activation of a class y is w_y * f(x), highest prediction score wins y = argmax_y w_y * f(X)\n* weights start at 0, for each training instance predict with current weights, if wrong lower score of wrong answer and raise score of right answer\n* w_y = w_y - f(x), w_y* = w_y* + f(x)\n\nPerceptron properties\n* separability: can be true if training set is perfectly separated but not always possible\n* convergence, if training is separable, perceptron eventually converges in binary case\n* maximum number of mistakes in binary case relatest to margin or degree of separability, mistakes < k / d^2\n\nIssues:\n* if data is not separable, weights may thrash, averaging weight vectors over time can help\n* can create mediocre generalization if a barely separating solution found\n* can overtrain if test/held out set accuracy rises then eventually falls, training accuracy keeps increasing\n\nFixing perceptrion:\n* adjust the weight update to lessen effects\n* MIRA* - choose update size thatfixes current mistake but minimzies change to w\n* min_w 0.5 * sum_y ||w_y - w'_y||^2, wy* * f(x) >= wy * f(x) + 1, +1 helps to generalize\n* have some value tao (t) to help fix. if we guess y instead of y* on example x with features f(x), wy = w'y - t(f(x)), wy* = w'y* + t(f(x))\n* minimum correcting update -> min_t ||tf||^2, end up with tao = ((w'y - w'y*) * f + 1) / (2f * f)\n* in practice, bad to make updates that are too large, example may be labeled incorrecrly or you may not have enough features\n* so we cap the maximum value of tao with a constant C, tao* = min(((w'y - w'y*) * f + 1) / (2f * f), C)\n* this will correspond to optimization assuming non-separable data, converges faster than perceptron and better on noisy data\n\nSupport Vector Machines\n* maximize the margin between the support vectors (only ones that matter), all other training examples ignored\n* find separator with max margin, optimize over all examples at once, good according to intuition, theory, pratice\n* min_w 0.5 ||w||^2, for all i, y - wy* * f(xi) >= wy * f(xi) + 1\n* data that is linearly separable is great for linear decision rules, but if its too hard, can map to higher dimensional space\n\n* Naive Bayes builds model training data, gives prediction probabiltiies, has assumptions about feature independence and passes through data once\n* perceptrons/MIRA make less assumptions about data, mistake-driven learning, multiple passes through data, and are often more accurate\n* original feature space can be mapped to higher dimensional feature space where training set is separable\n\nLogical functions:\n* networks of perceptons can compute any logical functions\n* build up these networks with NAND gates\n* XOR cannot be solved with one neural unit, but can be raise dto higher dimensional space to be solved linearly\n\nActivation functions\n* insstead of outputting wx + b as linear function, apply non-linear function to output called activation value\n* activation value is a, output of entire network is y, so we get y = a = f(wx + b)\n* three most commonly used ones are sigmoid function (1/1+e^-z), tanh, and ReLU (x if > 0, 0 if < 0)\n\n\n## Module 12 Definitions\n* **Concept** - definition\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\n* Train a classifier to make binary decision about class of new input observation \n* Logistic regression applies weights and bias term, bias term is also called the intercept, another real number added to the weighed inputs\n* Have the dot product of the logistic regression result z = wx + b\n* Need them between 0 and 1 so we apply the sigmoid function\n* Pushes the outliers between 0 and 1, and it is differentiable which helps in learning\n* P(y = 1) = sig(wx + b), P(y= 0) is 1 minus this\n* Logistic regression applied to all sorts of NLP tasks, any property of the input can be a feature\n* Period disambiguation, whether a period at the end of a sentence is part of a word or end of a sentence\n* Features generally designed by examine training set, helpful to build complex features that are combinations of more primitive ones\n* Feature interactions are designed by hand in logistic regression and Naive Bayes\n* Feature templates are often needed when we have large numbers of features\n* Representation learning, ways to learn features automatically in unsupervised way from input\n* Naive Bayes has overly strong conditional independence assumptions, logistic regression much more robust to correlated features \n* Many correlated features, logistic regression assigns more accurate probability, so works better on larger documents/databases\n* Nice Bayes will still often make correct classification decision and can work well on small datasets and short documents\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n\n* Know the correct labels in each instance\n* Want to learn parameters that make prediction as close as possible to true label\n* Need a metric for how close current label is to the goal, call it the loss of cost function, cross-entropy loss\n* Second is optimization algorithm for updating the weights to minimize loss function, gradient descent (stochastic) in particular\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\n* Need loss function that prefers correct class labels of training examples to be more likely\n* Conditional maximum likelihood estimation, choose parameters web, that maximize log probability of true y labels in training data given observations x, leads to cross entropy loss\n* Want loss to be smaller if models estimate is close to correct, bigger if the model is confused\n* Higher predicted y, the better the classifier, lower the worse the classifier\n* Negative log of probability is a good loss metric since it goes from 0 to infinity, ensures correct answer is maximized, probability of incorrect answer is minimized since two sum to one\n\n\n## SLP Chapter 5.4 Gradient Descent\n\n* Gradient descent, find the optimal weights and minimize the loss function e just defined\n* Loss function is parameterized by the weights, minimize loss averaged over all examples\n* Figures out in which direction the functions slope is rising the most steeply and go the other direction\n* Logistic regression the loss function is convex, so there is only a global minimum and can start from anywhere\n* Finds gradient of loss function at current point and moves in opposite direction\n* Gradient is multi-variable generalization of the slope, just a vector that expresses directional components of sharpest slope\n* Magnitude of amount to move is value of slope weighed by the learning rate\n* Higher learning rate means that we should move more on each step\n* Have a weight for each dimension\n* Derivative of cross entropy function can be used to estimate gradient for a single variable\n* Stochastic gradient descent minimizes loss function by computing gradient after each training example and moving parameters in right direction\n* If learning rate is too high we overshoot minus function, if too low it takes too small of steps and takes a long time\n* Usually have higher learning rate and slowly decrease it, hyper parameter chosen by us\n* Stochastic chooses single random example at each time point\n* In batch training compute gradient over entire dataset \n* Mini-batch trains on group of examples from the entire dataset, batch gradient descent then, computationally efficient \n\n\n## SLP Chapter 5.5 Regularization\n\n* Regularization, used to avoid overfitting, want model to generalize well from training data to unseen test set\n* Regularization term added to objective function, used to penalize large weights, setting where weights match training data perfectly but uses weights with high values will be penalized more than setting that matches data less well with smaller weights\n* L2 regularization is quadratic of weight values, uses square of L2 norm of weight values, same as Euclidean distance, ridge regression\n* L1 regularization is linear function of weight values, Manhattan distance, just sum of weights, lasso regression\n* Both commonly used in language processing, l2 easier to optimizer because of its simple derivative compared to l1 \n* L2 weights prefer weight vectors with many small weights, l1 prefers sparse solutions with larger weights but more weights set to 0\n* L1 viewed as Laplace prior on weights, l2 assuming weights are distributed acorns go Gaussian distribution with mean 0\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n\n* Multinomial logistic regression, if we need more than two classes, also called softmax regression\n* Target y can range over two classes, want to know probability of it belonging to a certain class\n* Uses generalization of sigmoid called softmax that compute probability between 0 and 1 for each class\n* Input to softmax is dot product between weight vector and input vector\n* Need separate weight vectors and biases for each of the K classes\n* Feature can be evidence for or against each individual class now\n* Sometimes make dependence explicitly and represent features themselves as function of input and class\n* Loss function for multinomial logistic regression generalizes loss function from 2 to K classes\n* Takes input and represents it as a one-hot vector, all values 0 except for term corresponding to true class\n* Cross entropy loss is log of output probability corresponding to correct class, call it negative log likelihood loss\n* Gradient is difference between value for true class k and probability the classifier outputs for class k weighted by value of input correspond to element of the weight\n\n\n## SLP Chapter 7.1 Units\n\n* Building block of neural network is single computational unit, takes real valued numbers as input and performs computation on them, produces output\n* Bias term and weighted sum of inputs, express as vector dot product\n* Apply nonlinear transformation f to z, referred to as activation\n* Sigmoid function, tanh, ReLU rectified linear unit all used as activatio functions\n* Sigmoid not used commonly as activation function, tanh almost always better, smoothly differentiable and mapping outlier values towards the mean\n* ReLU takes max of the value and 0, negative values set to zero then, nice properties since its close to linear\n* Gradients that are almost 0 cause error signal to get smaller and smaller until useless, vanishing gradient problem\n\n\n## SLP Chapter 7.2 The XOR problem\n\n* Single neural unit can’t compute XOR function\n* Perceptron is simple neural unit with binary output and does not have nonlinear activation function\n* Cannot linearly classify the output of XOR, decision boundary cannot cover every output, not linearly separable\n* Can be calculated with layered network of units\n* Weights learned automatically using error backpropagation algorithm\n* Hidden layers learn to form useful representations\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n\n* Feedforward network, multilayer network with connected units and no cycles, output of each layer passed to units in next layer with no passes back to previous layer\n* Also called multi-layer perceptrons\n* Input nodes, hidden nodes, output nodes\n* Hidden layer is formed of hidden units, take weighed sum of inputs and apply non-linearity\n* Each layer fully connected meaning takes inputs as outputs of previous layers and link between every pair of units in two adjacent layers\n* Multiply weight matrix by input vector, add bias vector, apply activation function\n* Output layer has a weight matrix, some models don’t include bias in output layer\n* Softmax function normalizing the vector of real values, convert to vector that encodes probability distribution\n* If we dont use nonlinear activation layers then resulting network is exactly equivalent to single layer network\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n\n* Use feedforward networks for NLP\n* Have two layer sentiment classifier, could be scalar feature inputs, output layer could have 2 or 3 nodes\n* Adding hidden layer allows network to represent non-linear interactions between features, may give better sentiment classifier\n* Representing words as word2vec embeddings as input representation\n* Use another algorithm to pretrain the input words before input feeding\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n\n* Language modeling: predicting upcoming words from prior word context, very important NLP task\n* Compared to n-grams, neural language models can handle much longer histories, generalize better over contexts of similar words, more accurate at word prediction\n* But much more complex, slower to train, less interpretable, so small tasks n-gram often still used more often\n* Takes input at time t a representation of some number of previous words and outputs probability distribution over next possible words\n* Then approximates probability of a word given entire prior context by approximating based on N previous words\n* Words in prior context represented as embeddings rather than just word identity as in n-gram models, allows them to generalize better to new data\n* Forward inference, given an input runs forward pass on network to produce probability distribution over possible outputs, next words in this case\n* One hot vector representation used, one dimension for each word in vocabulary and mostly zeros\n* Moving window can see N words into the past, multiply vectors by embedding matrix, concatenated to produce embedding layer, followed by hidden layer then output layer to produce probability distribution\n* Select three embeddings for E, multiply by W, multiply by U, apply softmax\n\n\n## SLP Chapter 7.6 Training Neural Nets\n\n* Want to learn parameters W and b for each layer to get as close as possible to true y\n* Use cross entropy loss for logistic regression, use gradient descent optimization algorithm\n* Need to know gradient of loss function, need to do error backpropagation or backward differentiation\n* Cross entropy loss in neural networks the same as logistic regression\n* Still use one hot vectors, can determine loss as negative log likelihood loss\n* Calculating gradient require partial derivative of the loss function with respect to each parameter\n* Backprop needed to update all weights, not just the ones in the final layer\n* Depends on notion. Of computation graphs, representations of process of computing mathematical expression where broken down into separate operations each as a node\n* Can easily compute the forward pass as before\n* Makes use of the chain rule to calculate backwards differentiation, pass gradients back from final node to all nodes in the graph\n* Takes an upstream gradient and computes local gradient then computes downstream gradient to pass on to earlier node\n* Also take derivatives of the activation functions\n* Optimization in neural networks is non-convex optimization problem, more complex than logistic regression\n* Dropout used as a form of regularization, also tuning hyper parameters is crucial, learning rate, mini batch size, model architecture, need to parallelize onto vector-based GPUs\n\n\n## SLP Chapter 7.7 Training the neural language model\n\n* Training neural language model\n* For some tasks ok to freeze embedding layer with initial word2vec values, hold it constant and only modify W, U, and b, dont update embeddings E\n* Often would still like to refine embeddings while training network, important when taks places strong constraints on what makes a good representation for words\n* Want one single embedding dictionary shared among the words\n* Input as very long text, concatenating all sentences, starting with random weights and iteratively moving through text predicting each word, use cross entropy loss for each word\n* Classes are words in vocabulary, pick the highest predicted one\n* Gradient can then be calculated with backpropagation \n* Results in language omodelling and new set of embeddings E to be used as word representations in other tasks\n\n\n## Module 13 Lecture Notes\n\nLogistic Regression\n* for the classifier we need training samples of M observations, input/output pairs (x,y)\n* also need 1) feature representation of the input as a vector of features [x1,x2,..xn], 2) classification function that computes estimated class y_hat with p(y|x), 3) objective function for learning with minimization of training examples, 4) algorithm for optimizing objective function\n\nSentiment classifier\n* consider input observation x represented as vector of features, output can be 1 (positive) or 0 (negative), and we want to esimate the P(y = 1|x)\n* can use logistic regression to solve this by learning from training set with weights and bias term\n* same as before, z = sum_i wi*xi + b, or z = wx + b\n* apply the sigmoid function y = 1/(1+e^-z), so we get P(y = 1) = sigmoid(wx + b) = 1/(1+e^-(wx + b))\n* we have the probability from this, so if this P(y=1|x) > 0.5 we label it 1, otherwise 0, call 0.5 the decision boundary\n\nLearning in logistic regression\n* learn the parameters (weights and bias) with learning, which requires 1) objective or loss function that tells us distance between system and gold ouput, use cross-entropy loss, and 2) algorithm for optimizing objective function, we will use stochastic gradient descent to minimize the loss function\n* loss function needs to determine how close the classifier output is to the correct output, probabilities are to true 0 and 1\n* we want L(y_hat, y), how much y_hat differs from the true y, could use mean squared error L(y_hat, y) = 0.5(y_hat - y)^2\n\nProbabilistic classification\n* want a loss function that prefers correct class labels of training data\n* choose parameters w and b that maximize the log probabilities of the true labels in training data, loss function is negative log likelihood loss or cross entropy loss\n* for one observation x, want to maximize the probabilitiy of correct label p(y|x), p(y|x) = y_hat^y(1-y_hat)^(1-y)\n* if y = 1, then p(y|x) = y_hat, if y = 0 then p(y|x) = 1 - y_hat\n* change to logs stil mazimizing, log(p(y|x)) = log(y_hat^y(1-y_hat)^(1-y)) = ylog(y_hat) + (1 - y)log(1 - y_hat)\n* tells us what log likelihood is maximized, flip the sign since we want to minimize it\n* so we get cross entropy loss as Lce(y_hat, y) = -log(p(y|x)) = - [ylog(y_hat) + (1 - y)log(1 - y_hat)]\n* plug in definition of y_hat to get Lce(y_hat, y) = - [ylog(sig(wx + b)) + (1 - y)log(1 - sig(wx + b))]\n* want losses to be smaller if model's estimate is close to correct, and loss to be bigger if it gets confused\n* loss on all training examples: log p(training labels) = log mult_i->m p(yi|xi) = sum_i->m log p(yi|xi) = - sum_i->m Lce(y_hat_i|y_i)\n* use gradient descent to find good weight and bias settings by minimizing loss function\n\nGradient descent\n* finds minimum of function by figuring out in which direction in parameter space the function's slope is rising the most and moving in oppostie direction\n* theta_hat = argmin_that 1/m * sum_i->m Lce(yi, xi; theta)\n* for logistic regression, loss function is convex, has just one minimum so no local minima to get stuck on, guarenteed to find the min\n* iteratively find minimum, take steps of learning rate until min reached\n* magnitude of the amount to move in GD is slope weight learned by learning rate\n* higher LR means we move w more on each step\n* w_t+1 = w_t - LR * d/dw f(x;w): new weight = old_weight - Lr * derivative of fn = slope\n* update each dimension wi, by taking partial derivatives with respect to L(f(x;theta), y)\n* final equation for updating theta based on gradient is theta_t+1 = theta_t + LR * L(f(x;theta), y)\n* to update theta, need definition for gradeitn L(f(x;theta), y)\n* for logistric regression, cross entropy loss is Lce(w,b) = -[ylog(sig(wx + b)) + (1 - y)log(1 - sig(wx + b))]\n* derivative of this function for one observation vector and a single weight is [sig(wx + b) - y]xj\n* difference between true y and estimate for x multiplied by corresponding input value xj\n* want to minimze the average loss, so have - 1/m sum_i->m yi log(sig(wxi + b)) + (1 - yi)log(1 - sig(wxi + b))\n* loss for batch or entire dataset is average loss over m examples, gradient fo multiple points is the sum of each individual gradient sum_i->m [sig(wxi + b) - yi]xj_i\n\nStochastic gradient descent algorithm\n* for each training tupe (xi, yi) compute y_hat_i = f(xi, thetha)\n* compute the loss L(y_hat_i, y_u), calculate the gradient as g = L(f(xi;theta, yi)), then move other way, theta = theta - L.R. * g\n* derivative of function for single training example for single weight is [s(sx + b) - y]xj, get the gradient, move the opposite direction\n\nMini-batch training\n* stochastic GD chooses a single random example and updates the weights on the example\n* can also do batch training which computes gradient over the entire dataset which can give a better estimate but takes longer to computer\n* can also do mini-batch training where the model is trained on small batches so all of them can be computed in parallel, then accumulate all the loss\n\n* overfitting can result in poor generaliztion and poor performance on the test set\n* if a feature only occurs in one class in logistic regression then it gets a high weight, can sometimes model noisy factors\n* add regularization to penalize large weights, added to the loss function\n* Lasso regression uses L1 regularization, ridge regression uses L2 regularization\n\nMultinomial logistric regression\n* if we want more than two classes, can extend to be several, for sentiment e.g. can extend to be positive, negative, or neutral\n* want to know the probability of y for each class c in C, p(y = c|x)\n* to get probabilities, have to do a generalization of the sigmoid function called softmax function\n* softmax(zi) = e^zj / sum_i->k e^zj for 1 <= i <= k\n* softmax takes input vector z = [z1,z2..zk] and outputs vector of values normalized to probabilities\n\nPerceptrons\n* developed in 50s/60s inspired by neuron\n* building block of neural networks is single computational unit, takes in real numbers as input and performs computation\n* cell body/dendrites like input, axon like neuron, output like terminal branches of axon\n* neural networks combine input layer, several hidden layers, output layer\n* logistic regression uses weight vecor and bias term like perceptrons\n* z = sum_i = wixi + b, or z = wx + b, transform with sigmoid function\n* have a step function, 1 if wx+ b > 0, 0 is wx + b <= 0\n* sigmoid neurons in place to prevent the weights in the network to suddenly flip output from 0 to 1\n* smoothness of sigmoid means small changes in weights and bias produce small changes in neuron output, can then choose small changes to achieve the desired output\n\nFeed forward neural network\n* simplest kind of NN, multilayer network where all units are fully connectwed with no cycles, outputs from previous layer are passed to the next layer and nothing is bassed back up\n* single hidden unit has parameters w and b, represent parameters for entire hidden layer by combining the weight vector wi and bias bi for each unit i into single W and b terms\n* hidden layer operations can now be done easily for the entire layer\n* 1) multiply weight matrix by input vector x, add bias vector b, apply sigmoid activation function, get h = sig(Wx + b)\n* output layer has weight matrix U, so intermediate output z is z = Uh, then get final output when the softmaxx is applied as y = softmax(z), normalizes output and turns it into probability distribution\n\nBackpropagation\n* derivatives only give correct updates for last layers, need to pass it back to earlier layers as well\n* backward differentiation which depends on computation graphs\n* these graphs represetn computing mathematical expressions by breaking them down into separate operations, each one as a node in a graph\n* calculate everything on the forward pass as usual, then use chain rule to do backwards differentiation, dL/da', dL/db, dL/dc\n* for composite function f(x) = u(v(x)), derivative of f(x) is df/dx = du/dv * dv/dx\n* for f(x) = u(v(w(x))), derivative of f(x) is df/dx = du/dv * dv/dw * dw/dx\n\n\n\n## Module 13 Definitions\n* **Concept** - definition\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n\n* Predict next word in sequence given preceding context\n* Can assign conditional probability to every possible next word, get distributions over entire vocabulary, can also assign probabilities to entire sequences by using conditional probabilities with chain rule\n* Evaluate on how well they do on unseen text, good models assign higher probabilities to unseen data\n* Use perplexity to measure quality of a language model, inverse probability that theta assigns to the test set normalized by test set length\n* Can use the chain rule again to expand computation of probability of test set\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n\n* Recurrent neural network is any network that contains cycle within network connections, so values of some unit are directly or indirectly dependent on earlier outputs as input\n* Difficult to reason and train, very effective when applied to language\n* Simple RNNs lead to LSTM networks\n* Input multiplied by weight matrix and passed through non-linear activation function to compute values for hidden unit layer, then used to calculate output \n* Sequences are processed by presenting one item to the network at a time\n* Link augments input to computation at hidden layer with value of hidden layer from preceding point in time\n* Hidden layer from previous step is like a memory unit, informs later decisions, context in previous hidden layer can include information extending to beginning of sequence\n* Can appear more complex than non-recurrent architectures but are not that different, still feed forward but new set of weights tell what to do with previous context\n* Trained with back propagation still\n* Nearly identical to feedforward networks with forward inference\n* Use training set, loss function, back propagation to get gradients to adjust weights\n* Have extra weights from previous hidden layer\n* Back-propagation requires two passes for training of weights, second pass we process sequences in reverse, back propagation through time\n\n\n## SLP Chapter 9.3 RNNs as Language Models\n\n* As language models, RNN process input sequence one word at a time to predict next word from current word and previous hidden state\n* Dot babe limited context problem as n-grams since they can represent information in hidden states\n* Forward inference the same, input sequence is series of word embeddings represented as one-hot vector and output is vector that represents probability distribution over vocabulary\n* Model uses embedding matrix to retrieve embedding for current word and combines with hidden layer to get new hidden layer, generates output layer fed through softmax to get probability\n* Output is set of scores over vocabulary given evidence in hidden layer\n* Probability of entire sequence is product of probabilities of each item in sequence\n* Train to minimize error in predicting true next word in training sequence using cross entropy loss\n* Always give the model the correct history sequence to predict the next word, teacher forcing\n* weights adjusted to minimize CE loss ver training sequence with gradient descent\n* Final layer provides way to score likelihood of each word given evidence present in final hidden layer of network\n* Weight tying is method that uses single set of embeddings at input and softmax layer to reduce redundancy, lower dimensionality\n\n\n## SLP Chapter 6.1 Lexical Semantics\n\n* Representation of word is string of letters or index in vocabulary list\n* Lexical semantics define some of the ways we set up words for our NLP\n* Lemma is the focus word, also called citation form, mouse is lemma for mice e.g.\n* Each lemma can have multiple meanings, call each one a word sense, can be polysemous and have multiple meanings, makes interpretation difficult\n* Word sense disambiguation is process of deducing word from context\n* Synonyms are important to capture, if they have the same propositional meaning\n* Principle of contrast says difference in linguistic form always associated with difference in meaning\n* Words can have lots of similar words that are not similes \n* Words can also be related in ways other than similarity, word association\n* Another relation is if two words belong to the same semantic field, have structured relations between them\n* Semantic frames are set of words that denote perspectives or participants in types of event\n* Words can have different meanings or connotations as well, mean different things with same word, sentiment can reflect positive or negative words\n\n\n## SLP Chapter 6.2 Vector Semantics\n\n* Vector semantics are the standard way to represent word meaning in NLP\n* Define meaning of word by distribution in language use, meaning neighboring words and grammatical environments \n* Represent a word in multidimensional semantic space that is derived from distributions of word neighbors, vectors for representing these words called embeddings, sometimes only applied to word2vec dense vectors\n* Mapping of from one space or structure to another, can project down. To 2d space to see the relationships between words\n* Classifiers can assign sentiment as long as it sees words with similar meanings\n* Tf-idf model has meaning of word defined by simple function of counts of nearby words, gives a sparse representation\n* Word2vec has short dense vectors with useful semantic properties\n* Can also use cosine to compute semantic similarity between too words/sentences\n\n\n## SLP Chapter 6.3 Words and Vectors\n\n* Vector or distributional models are based on co-occurence matrix often, how often words co-occur\n* Term-document matrix, each row represents a word in vocabulary, each column a document from collection of documents\n* Each cell is number of times particular word occurs in particular document\n* First defined as part of vector space model of information retrieval\n* Vector space is collection of vectors characterized by dimension\n* Ordering of numbers in vector space indicate different meaningful dimension where documents vary\n* Vector for a document is a point in V dimension space\n* Two similar documents have similar words, column vectors will be similar as well\n* V rows for number of words, and D columns for each document in collection\n* Information retrieval is process of finding document from documents in some collection that matche a query\n* Represent query as vector, node to compare two vectors for similarity\n* Could use term-term matrix or word-word matrix or term-context matrix, columns are labeled by words instead of documents so V x V, each cell records number of times the row and column word co-occur In some context in training set\n* Context could also be document, most common to use smaller contexts to get neighboring words\n* Size of vocabulary can be tens of thousands of words, most of them are zero so we have sparse vector representation, effective algorithms exist to store them\n\n\n## SLP Chapter 6.4 Cosine for measuring similarity\n\n* Measure similarity between two target words that takes in two vectors, use cosine of the angle between the vectors\n* Cosine based on dot product operator, called inner product\n* Raw dot product favors long vectors, higher is vector is longer with higher values in each direction\n* We can normalize it by diving the dot product by the lengths of each of the two vectors, normalized dot product is the same as the cosine of the angle between the two vectors\n* Can pre-normalize the vector, divide by its length, create a unit vector of length 1\n* Then compute unit vector and take cosine which is equivalent to dot product\n* Cosine ranges from 1 for vectors pointing in same direction, 0 for orthogonal vectors, -1 for pointing in opposite direction\n\n\n## SLP Chapter 6.8 Word2vec\n\n* Word2vec, represent words as short dense embeddings, work better in every NLP task than dense vectors\n* Skip-gram with negative sampling is one of two algorithms in word2vec, static embeddings meaning method learns one fixed embedding for each word in the vocabulary\n* Dynamic contextual embeddings like BERT are separate class\n* Can use running text as implicit supervised training data for a classifier, word that appears near the target is the gold standard, call this self supervision since we don’t need to label anything \n* Simplifies task compared to neural networks, and simplifies architecture\n* Treat word and neighboring context as positive examples, randomly sample other words as negative samples, use logistic regression to train a classifier to distinguish, and use weights as embeddings\n* Given tuple of target word and candidate context word, get probability that it is a real context word, base probability on embedding similarity\n* Word likely to occur near target if embedding vector is similar to target embedding\n* Two vectors are similar if they have a high dot product, sue sigmoid function to turn it into a probability\n* Says all context words are independent so we can multiply their probabilities, need embeddings for each target word and context word in vocabulary \n* Stores two embeddings for each word, one for word as target and one as context\n* Takes input paragraph of text and vocabulary size B, assigns random embedding vector to each word then shifts embeddings to be more like embeddings in training set\n* Noise words can be chosen accord to weighted unigram frequency with alpha as a weight\n* Maximize similarity of target/context word pairs from positive examples and minimize similarity of negative example pairs\n* Minimize loss with stochastic gradient descent\n* Learns target embedding and context embedding, can add them together or throw away C matrix and look at target matrix\n* Fasttext does better with unknown words by using subword models and generalizes better to unknown words\n\n\n## SLP Chapter 6.10 Semantic properties of embeddings\n\n* Size of context window used to collect counts is relevant to both tf-idf vectors and dense word2vec vectors\n* Shorter context dings lead to representations that are more syntactic since coming from nearby words, semantically similar with same parts of speech\n* Long context windows, highest cosine words to target word are topically related but not similar\n* First order co-occurrence if they are typically nearby each other, also called syntagmatic association\n* Two words have second-order co-occurrence if they have similar neighbors, called paradigmatic associated\n* Ability to capture relational meanings also important\n* Parallelogram model for solving simple analogy problems can be constructed \n* Sparse vector models of meaning could solve such analogy problems, better success with word2vec or GLoVe vectors\n* Want to find the argmin distance x of a* - a + b\n* Closest value returned by parallelogram model is Sally not b* but one of the 3 inputs words or morphological variants, so much be explicitly excluded\n* Doesn’t work well for other relations besides frequent words, small instances, and certain relations\n* Can also use embeddings to study how meaning changes over time\n\n\n## SLP Chapter 6.11 Bias and Embeddings\n\n* Embeddings can produce implicit biases and stereotypes latent in text\n* Embedding analogies can exhibit gender stereotypes in word2vec embeddings trained on news text\n* Allocations harm, system allocates resources unfairly to different groups\n* Embeddings also amplify bias, gendered terms become even more gendered in space compared to input text statistics\n* Can also encode implicit associations that are property of human reasoning\n* Representational harm is harm caused by system demeaning or ignoring some social groups\n* Recent research focused on transforming embedding space to remove gender stereotypes, debasing may reduce bias but hard to eliminate since is it in training text\n* Historical embeddings also being used to measure biases in the bast\n* Can document historical gender biases and racial stereotypes\n\n\n## SLP Chapter 6.12 Evaluating Vector Models\n\n* Most important evaluation metric is extrinsic evaluation on tasks, seeing if performance improves over some other model\n* Most common metric is similarity, correlation between algorithms word similarity and ratings assigned by humans\n* Slightly more realistic metrics are intrinsic similarity tasks that include context, they offer richer evaluation scenarios\n* Analogy task is another task used for evaluation\n* All embeddings suffer from variability, may produce different results from the same dataset, individual documents may strongly impact embeddings\n* Train multiple embeddings with bootstrap sampling over documents and average results\n\n\n## Module 14 Lecture Notes\n\nLanguage models\n* estimate the probability of a sentence containing a word sequence w1:n as P(w1:n) + mult_i->n P(wi|wi-k:i-1)\n* need to estimate probability of P(wi+1|wk-1;1) from large paragraph, p_hat(MLE)(wi+1 = m|wi-k:i) = #(wi-k;i+1)/(wi-k:i), p_hat(add - a)(wi+1 = m|wi-k:i) = #(wi-k;i+1) + a/(wi-k:i) + a|V|\n* neural language models have advantages over nn-gram LMs, 1) don't need smomthing, 2) can handle much longer histories, 3) can generalize over contexts of similar words, 4) tend to have higher prediction accuracy\n* can be slower to train than traditional n-gram Lms though\n\nNeural Language models\n* first associate each word in vocabulary with vector-representation and create notion of similarity as a result\n* then express joint probability function of word sequence in terms of word vectors for words in sequewnce\n* then learn the word vectors and the parameters of the function\n* word vectors will be low dimensional dense vectors, probability function expressed as product of conditional probabilites of next word given predvious word using feed forward neural network described earlier\n* input is k-gram of words w1:k, output is probability distribution over next word, k context words treated as word window with each word as an embedded vector\n* input vector x concatenates v(w) for each of k words, v(w) in R^dw, x = [v(w1);v(w2);..v(wk)]\n* fed into a neural network with one ore more hidden layers\n* input layer is one hot vectors with word as 1, then projection layer, hidden layer, output layer\n* training examples are word k-grams from paragraph, identities of first k-1 words used as features, last word is target label, trained using cross-entropy loss\n* better results with better preplexity scores than n-gram LMs, larger N to scale to since parameters are associated with individual words, and they generalize across contexts well, by-product are word embeddings\n\nLanguage modeling\n* want to find a function that returns the joint probability again\n* this is difficult because of the curse of dimensionality (too many parameters to estimate) and we can fail to generalize related words/word sequences that we have observed in n-gram-based models\n* sparse statistics just too high to estimate from the data\n* in LMs we use chain rule to get conditional probability of next word in sequence given all of the previous words, P(w1w2w3..wt) = sum_t->T P(wt|w1..wt-1)\n* probability of the next word only depends on previous n - 1 words, smaller n makes it easier to get a probability from data\n* can construct probability tables to look up probs of words given histories of observed sequences\n* but if we have unseen combination of n words, either back off or smooth/interpolate to stitch together short sequences of observed words\n* try generalizing instead, take a sentence and assign probabilities to similar sentence\n\nNeural probabilistic LM\n* use vector space model where words are vectors with real values, define a function that returns join probability of words in sequence based on previous seqeunce, learn word representations and probability function from data\n* given a training set w1..wt, learn f(w1..wt) = P(wt|w1..wt-1), subject to giving high probability to unseen text/dev set, create probability distribution so we can take product of conditonal probabilities to get joint ones\n* create mapping function C from any word in V onto R^M, store in VxM matrix and initialize with singular value decomposition\n* neural architecture is a function g tha mapes sequence of word vectors onto probability distribution over vocab V, g(C(wt-n) ..C(wt-1)) p P(wt|wt-n ..wt-1)\n* when 50 dimenstional vectors result from training LM prohected to 2 dimensions, see similar words close together\n* ELMo, GPT, BERT, GPT-2 the current state of the art models\n\nWord meaning\n* how to represent meaning of word, in Ngram LMs did so as string of letters or index in vocabulary list\n* want them to encode synonyms (similar meaning), antonyms (opposite meaning), connotations (positive or negative words), semantic roles, support for entailment (e.g. dog is proper subset of animal)\n* leximal semantics used to be trees like wordnet, now use more vector space models\n* most words don't have many synonyms but can use similar words instead for many applications\n* from a lot of context words humans can guess what words mean, so algorithm calls two words similar if their word contexts are similar\n* initially, vector space models were developed with each document as point in space, user query is pseudo document and represented point in same space, retrieved documents that were close to it\n\nTerm-Document matrix\n* each column is a document, each row is a term, value in a cell is how often that word appeared in the document\n* length of document vector is size of vocabulary, document vectors can be sparse, then can measure similarity between two documents by comparing columns\n* word similarity lets you do plagiarism detection\n\nVector comparisons\n* can look at two documents and terms in each, calculate euclidean distance between them, want to use L2 length normalization (||u|| = sqrt(sum_i->n ui^2)) and divde each vector by its L2 length\n* can also do cosine difference, does the L2 normalization as well. 1 - sqrt(sum_i->n ui * vi) / (sqrt(sum_i->n ui^2) * sqrt(sum_i->n vi^2))\n* the cosine angle will tell us the similarity between two terms\n\nTerm-term matrix\n* aka term-context matrix, length of vector is now |V| instead of number of documents, value in a cell indicates how often word appears in context window surrounding other word\n* context windows are w-2, w-1, target_word, w+1, w+2, list how many times this takes place\n* can occur in window of +/- 2 words in same sentence in same document, instead of windows can use more complext contexts like dependency patterns, languages often have long distance dependencies\n* use the syntax to define the word's context\n* raw word frequency not a great measure of association, skewed with more common words like the, of, or, rather have measure of context word as particularly informative about target word\n\nTF-IDF\n* term frequency * inverse document frequency (how often word occured in document * 1 over number of documents it occurred in)\n* can have spare and dense vectors, co-occurrence matrices weighed by Tf-IDF can be long and sparse, would rather have short and dense vectors\n\nDense vectors\n* to get them, train a classifier\n* treat targed word and neighboring context as positive examples, randomly sample other words to get negative samples, use logistic regression to train classifiedr to distinguish and weights as the embeddings\n* short vectors easier to use as features in ML, may generalize better than strong explicit counts\n* may do better at capturing synonyms, work better in practice\n\nWord2Vec\n* learn embeddings as part of word prediction process, train classifier to predict neighboring words, learn dense embeddings for words in paragraphs\n* fast and easy to train, availabe online with pretrained embeddings\n* predict neighboring word in context window of 2c of surrounding words, [wt-2, wt-1, wt+1, wt+2], use negative sampling for training - want prediction of target words to be high and irrelevant words to be low\n* inspired by neural nets, nearest neighbors are pretty good, embeddings can capture relational meanings\n* many possible models, matrix types (term-document, term-context, pattern-pair), reweighting (length norm, TF-IDF, PPMI, probabilities), comparisons (cosine, manhattan, jaccard, KL divergence, JS distance, DICE), dim. reduction (word2vec, GloVe, PCA, LDA, LSA) \n* predict rather than count, instead of looking at how often a word occurs, train classifier on binary prediction task, take learned weights as word embeddings but don't care about task\n* use running text as implicitly supervised trainign data, word near target word, no need for supervision, comes from neural language modeling\n\nWord2Vec Skip-gram with negative sampling (SGNS)\n* 1) treat target word and neighboring context as positive examples\n* 2) randomly sample other words to get negative examples\n* 3) use logistic regression to train classifier for the two\n* 4) use weights as embeddings\n* give a tuple (target, context), return the probability that the context is a real context word P(+|t,c) or P(-|t,c) = 1 - P(+|t,c)\n* words likely to appear close to similar words, to module simularity with dot product, similarity(t,c) = t * c\n* but dot product (and cosine) is not a probability, turn it into a probability using the sigmoid function (1/1+e^-x)\n* p(+|t,c) = 1/(1+e^-(t*c)); P(-|t,c) = 1 - p(+|t,c) = e^(-t*c)/(1+e^-(t*c))\n* assume all context words are independent so we have P(+|t,c1:k) = mult_i->k 1/(1+e^-(t*ci)); log(P(+|t,c1:k)) = sum_i->k log(1/(1+e^-(t*ci))\n* can compute probability of these values, learn the embeddings from Word2vec automatically\n* starts with initial set of embedding vectors, shifts the embedding to be more like embedding of words in nearby texts\n* for each positive example, create k negative ones with noise words and random words that are not term\n* choosing noise words, could pick according to unigram frequency P(w), more common to pick according to Pa(w), where Pa(w) = count(w)^a / sum_w count(w)^a\n* give rare noise words higher probability\n* start with random weights or 0, adjust to make positive pairs more likely and negative pairs less likely\n\nSGNS example\n* start with 300 * V random parameters, words as vectors of length 300\n* maximize similarity of target word, context word pairs from positive data and minimze t,c pairs drawn from negative data\n* want to maximize sum_(t,c) logP(+|t,c) + sum_(t,c) logP(-|t,c) \n* focusing on one target word t, loss becomes L(theta) = log(1/1+e^(-t*c)) + sum_i->k log(1/1+e^ni*t)\n* train with gradient descent, learns W and C embedding matrices, can merge or keep W and throw away C\n* take paragraph and pair of words that co-exist as positive examples, pair of words that dont co-exist as negative examples, classifier distringuishes and adjusts embeddings, throw away classifier and keep embeddings\n\nProperties of embeddings\n* similarity depends on window size C, context windows change the word embeddings\n* can solve analogies with word vector embeddings\n* use vector arithmatic, a:a* as b:b*, b* is hidden vector, b* should be similar to b - a + a*\n* this can be solved by optimizing argmax_b* in V (cos(b*, b - a + a*))\n* can capture relational meaning now\n* if all word-vectors are normalized to unit length then expression is equivalent to argmax_b* in V (cos(b*,b) - cos(b*,a) + cos(b*,a*))\n* or we can require direction of transformation be maintained, argmax_b* in V (cos(b* - b, a - a*)), say they share the same direction and ignore the distances\n* diachronic word embeddings can be used for studying language change\n\nEmbeddings and bias\n* Paris: France :: Tokyo: x -> x = Japan\n* man: computer programmer :: woman: x -> x = homemaker\n* concepts and attributes can be associated, can lead to bias in embeddings\n* reflect and replicate all sorts of unfair biases\n* need to debias algorithms for embedding, and use as a tool for studying historic bias\n* can use hamilton historical embeddings, cosine similarity of embeddings for decade X for occupations to male vs. female names\n* correlated with actual percentage of gender and occupation\n* embeddings for competence adjuctives biased towards men\n* reflect ethnic stereotypes over time, attitudes towards ethnic groups\n\nConclusion\n* embeddings are vector models of training\n* more refined than string or index, very good at modeling similarity/analogy\n* we can use sparse models (tf-idf) or dense models (word2vec, GLoVE)\n* useful in practice but know they have cultureal stereotypes encoded as well\n\n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}