{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "brNDGX-kvzds"
   },
   "source": [
    "# Template for Notes for CIS 521 Midterm 2\n",
    "\n",
    "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
    "\n",
    "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
    "\n",
    "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aPtM8g7R0Pds"
   },
   "source": [
    "# Instructions\n",
    "\n",
    "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
    "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
    "3. Read the notes below about how you can format your summaries.\n",
    "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
    "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
    "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
    "* Run the last cell of this notebook\n",
    "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
    "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
    "* In the print options, change the layout from Landscape to Portrait. \n",
    "* Clicking the Save, will save a PDF that you can then print.\n",
    "\n",
    "## Formatting Notes\n",
    "\n",
    "1. The max length for each section's summary is 500 words / 3000 characters.\n",
    "2. You can also write a 2000 word summary of the lectures for each module.\n",
    "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
    "2. You can use Markdown formatting for your summaries. \n",
    "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
    "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
    "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
    "```\n",
    "def function():\n",
    "      print('hello world')\n",
    "```\n",
    "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cw5IpDLCZa4W"
   },
   "source": [
    "## Example summary\n",
    "\n",
    "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lzaavW96tcxM",
    "outputId": "d9df20a6-e2d1-433c-de09-0317d83146a0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The example summary is 116 words long, so you could write an additional 384 words.\n"
     ]
    }
   ],
   "source": [
    "chapter_2_1_example = \"\"\"\n",
    "Regular expressions are an algebraic notation for characterizing \\\n",
    "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
    "Only the first matching result will be shown. The search string is case sensitive, \\\n",
    "but it can consist of a single character or a sequence of characters. \\\n",
    "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
    "for characters over a range. The caret specifies the non-existence of a character. \\\n",
    "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
    "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
    "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
    "...\n",
    "\"\"\"\n",
    "\n",
    "length = len(chapter_2_1_example.split(\" \"))\n",
    "print(\"The example summary is {len} words long, so you could write an additional \\\n",
    "{remaining} words.\".format(len=length, remaining=(500-length)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s3Uk9gQTevcj"
   },
   "source": [
    "## Example summary length\n",
    "\n",
    "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8DvZ57A-djo1",
    "outputId": "99e72855-6b47-4a21-f2aa-c55144f9eb14"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The length in words is 485\n",
      "The length in characters is 3029\n"
     ]
    }
   ],
   "source": [
    "# This is an example of a 500 word placeholder\n",
    "placeholder = \"\"\"- policy a solution specify what the agent should do for any state that the agent might reach. A solution of this kind is called a policy. $\\pi(s)$, action recommended by the policy π for state s.\n",
    "- An optimal policy is a policy that yields the highest expected utility. $\\pi^*(s) = argmax U^\\pi(s) = argmax \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
    "- The balance of risk and reward changes depending on the value of $r=R(s,a,s')$ for transitions between nonterminal states. reward can be positive or negative but are bounded by some maximum value.\n",
    "Because the decision process is sequential, we must specify the utility function on a\n",
    "sequence of states and actions. Instead of only giving a reward at the goal states, the agent can receive a reward at\n",
    "each time step,\n",
    "- with a finite horizon, an optimal action in a given state may depend on how much time is left. \n",
    "- With no fixed time limit, on the other hand, there is no reason to behave differently in the  same state at different times.\n",
    "- Policies for the infinite-horizon case are simpler than those for the finite-horizon case\n",
    "- additive discounted rewards: the utility of a history is $U_h([s_0,a_0,s_1,a_1,...]) = R(s_0,a_0,s_1)+\\gamma R(s_1,a_1,s_2)+\\gamma^2 R(s_2,a_2,s_3) + ...$, Sooner rewards probably do have\n",
    "higher utility than later rewards\n",
    "- When γ is close to 0, rewards  in the distant future are viewed as insignificant. When γ is close to 1, an agent is more willing  to wait for long-term rewards.\n",
    "  - With discounted rewards, the utility of an infinite sequence is finite.\n",
    "  - environment contains terminal states and if the agent is guaranteed to get to one  eventually, then we will never need to compare infinite sequences.\n",
    "  - Infinite sequences can be compared in terms of the average reward obtained per time step.\n",
    "- discounted utilities with infinite horizons is that the optimal policy is  independent of the starting state.\n",
    "- the utility of a state is the expected reward for the  next transition plus the discounted utility of the next state, assuming that the agent chooses  the optimal action.\n",
    "- Bellman equation $U(s) = max \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
    "- action utility funtion, Q function $U(s)= max Q(s,a)$, $\\pi^* = argmax Q(s,a)$\n",
    "  - $Q(s,a) = \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma max Q(s',a')]$- reward scales:\n",
    "  - $U'(s) = mU(s)+b$\n",
    "  - $R'(s,a,s') = mR(s,a,s') +b$\n",
    "  - $\\Phi(s)$ be any function of the state s. Then, according  to the shaping theorem, the following transformation leaves the optimal policy unchanged:\n",
    "  - $R'(s,a,s') = R(s,a,s') +\\gamma \\Phi(s')−\\Phi(s).$\n",
    "  - $Q'(s,a)=Q(s,a)−\\Phi(s)= \\sum_{s'}P(s'|s,a)[R'(s,a,s')+\\gamma max Q'(s',a')]$\n",
    "  - $\\Phi(s)$ potential, The term $\\gamma \\Phi(s')−\\Phi(s)$ functions as a gradient of the  potential. \n",
    "  - $\\Phi(s)$  has higher value in states that have higher utility, the addition of $\\gamma \\Phi(s')−\\Phi(s)$ to the reward has the effect of leading the agent “uphill” in utility. \n",
    "  - all policies are optimal  with a reward function that is zero everywhere.\"\"\"\n",
    "\n",
    "print(\"The length in words is\", len(placeholder.split(' ')))\n",
    "print(\"The length in characters is\", len(placeholder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5yG3-YncJfTB"
   },
   "source": [
    "# Permissions to use your summaries\n",
    "\n",
    "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
    "\n",
    "1. Please enter your name.  \n",
    "\n",
    "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
    "\n",
    "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "id": "2RECnJArJew4"
   },
   "outputs": [],
   "source": [
    "name = \"\"\n",
    "permission = True\n",
    "anonymous = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GZZXyrV-xuAp"
   },
   "source": [
    "# Module 8 - Markov Decision Process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "id": "Y2NfE1CKxzS-"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
    "\n",
    "chapter16_1 = \"\"\"- $ P(Result(a)=s')= \\sum_{s}^{} P(s)P(s'|s,a) $, which assumes epsidic\n",
    "- Decision theory, in its simplest form, deals with choosing among actions based on the desirability of their immediate outcomes; that is, the environment is assumed to be episodic\n",
    "- The agent’s preferences are captured by a utility function, U(s), which assigns a single number to express the Utility function  desirability of a state.\n",
    "- $ EU(a) = \\sum_{s'} P(Result(a)=s')U(s') $. expected utility\n",
    "- The principle of maximum expected utility (MEU) says that a rational agent should choose  the action that maximizes the agent’s expected utility: \n",
    "- $ action = argmax EU(a) $, maximum expected utility, a prescription for intelligent behavior\n",
    "- All an  intelligent agent has to do is calculate the various quantities, maximize utility over its actions,  and away it goes. But this does not mean that the AI problem is solved by the definition!\n",
    "  - The MEU principle formalizes the general notion that an intelligent agent should “do the  right thing,” but does not operationalize that advice.\n",
    "  - percerption, learning and knowledge representaion, inference are also needed to compute,\n",
    "  - decision theory is not a panacea that solves the  AI problem—but it does provide the beginnings of a basic mathematical framework that is  general enough to define the AI problem. \n",
    "- If an agent acts so as to maximize a utility function that correctly reflects the performance measure, then the agent will achieve the highest possible performance score (averaged over all the  possible environments)\n",
    "- MEU is not the only rational way\n",
    "- basic preferences:\n",
    "  - $ (A\\succ B) $, agent prefers A over B\n",
    "  - $ (A \\sim B) $, no difference in perference\n",
    "  - $ (A \\succsim B) $, agent prefers A over B or no difference in perference\n",
    "- We can think of the set of outcomes for each action  as a lottery—think of each action as a ticket.\n",
    "- axioms of utility theory  list six constraints that we require any reasonable preference relation to obey: \n",
    "- <strong>Orderability</strong>, Given any two lotteries, a rational agent must either prefer one or else  rate them as equally preferable. That is, the agent cannot avoid deciding.\n",
    "  - Exactly one of $ (A\\succ B) $, $ (B \\succ A)$, or $ (A \\sim B) $ holds\n",
    "- <strong>Transitivity</strong>: Given any three lotteries, if an agent prefers A to B and prefers B to C, then the agent must prefer A to C\n",
    "  - $ (A\\succ B) \\wedge (B \\succ C) \\Rightarrow (A \\succ C) $ holds\n",
    "- <strong>Continuity</strong>: Continuity: If some lottery B is between A and C in preference, then there is some  probability p for which the rational agent will be indifferent between getting B for sure  and the lottery that yields A with probability p and C with probability 1− p.\n",
    "\"\"\"\n",
    "chapter16_2 =  \"\"\" \n",
    "- <strong>Substitutability</strong>: If an agent is indifferent between two lotteries A and B, then the  agent is indifferent between two more complex lotteries that are the same except that  B is substituted for A in one of them. This holds regardless of the probabilities and the  other outcome(s) in the lotteries\n",
    "- <strong>Monotonicity</strong>: Suppose two lotteries have the same two possible outcomes, A and  B. If an agent prefers A to B, then the agent must prefer the lottery that has a higher  probability for A (and vice versa). \n",
    "- <strong>Decomposability</strong>: Compound lotteries can be reduced to simpler ones using the laws of probability. it compresses two consecutive lotteries into a single equivalent lottery.\n",
    "- Rational preference lead to utility:\n",
    "- <strong>Existence of Utility Function</strong>:If an agent’s preferences obey the axioms of utility, then  there exists a function U such that U(A) > U(B) if and only if A is preferred to B, and  U(A) = U(B) if and only if the agent is indifferent between A and B. \n",
    "  - $U(A) > U(B) \\Leftrightarrow A \\succ B $  and  $ U(A) = U(B) \\Leftrightarrow A \\sim B$\n",
    "- <strong>Expected Utility of a Lottery</strong>: The utility of a lottery is the sum of the probability of  each outcome times the utility of that outcome. \n",
    "  - $ U([p_1,S_1;\\dots ; p_n,S_n]) = \\sum_{i} p_i U(S_i)$\n",
    "  - In other words, once the probabilities and utilities of the possible outcome states are specified, the utility of a compound lottery involving those states is completely determined.\n",
    "  - Because the  outcome of a nondeterministic action is a lottery, agent can act rationally, consistently with its preferences—only by choosing an action that maximizes expected  utility\n",
    "  - The preceding theorems establish that (assuming the constraints on rational preferences)  a utility function exists for any rational agent. The theorems do not establish that the utility  function is unique.\n",
    "- As in game-playing, in a deterministic environment an agent needs only a preference ranking on states—the numbers don’t matter. This is called a value function or <strong> ordinal utility function</strong>. \n",
    "- existence of a utility function that describes an agent’s  preference behavior does not necessarily mean that the agent is explicitly maximizing that  utility function in its own deliberations.\"\"\"\n",
    "chapter16_3 =  \"\"\"\n",
    "- system that helps a human make decisions or acts on  his or her behalf, we must first work out what the human’s utility function is. This process, often called preference elicitation, presenting choices to the human and using the Preference elicitation  observed preferences to pin down the underlying utility function. \n",
    "- Normalized utilities: $ u_{bot} = 0, u_{top} = 1 $\n",
    "- standard lottery: $ [p,u_{top};(1-p),u_{bot}] $\n",
    "- expected monetary value (EMV):\n",
    "  - $ EMV = \\sum P(x)*Value(x)$\n",
    "  - Utility is not  directly proportional to monetary value, because the utility for your the momey you own is not the same\n",
    "  - utility of money plateaus at higher, deepening at lower\n",
    "- We will assume, kindly perhaps, that the estimates EU(a) are unbiased, but the real outcome will usually be significantly worse than we estimated,  even though the estimate was unbiased!\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
    "chapter17_1 = \"\"\"\n",
    "- policy a solution specify what the agent should do for any state that the agent might reach. A solution of this kind is called a policy. $\\pi(s)$, action recommended by the policy π for state s.\n",
    "- An optimal policy is a policy that yields the highest expected utility. $\\pi^*(s) = argmax U^\\pi(s) = argmax \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
    "- The balance of risk and reward changes depending on the value of $r=R(s,a,s')$ for transitions between nonterminal states. reward can be positive or negative but are bounded by some maximum value.\n",
    "Because the decision process is sequential, we must specify the utility function on a\n",
    "sequence of states and actions. the agent can receive a reward at\n",
    "each time step,\n",
    "- with a finite horizon, an optimal action in a given state may depend on how much time is left. \n",
    "- With no fixed time limit, on the other hand, there is no reason to behave differently in the  same state at different times.\n",
    "- Policies for the infinite-horizon case are simpler than those for the finite-horizon case\n",
    "- additive discounted rewards: the utility of a history is $U_h([s_0,a_0,s_1,a_1,...]) = R(s_0,a_0,s_1)+\\gamma R(s_1,a_1,s_2)+\\gamma^2 R(s_2,a_2,s_3) + ...$, Sooner rewards probably do have\n",
    "higher utility than later rewards\n",
    "- When γ is close to 0, rewards  in the distant future are viewed as insignificant. When γ is close to 1, an agent is more willing  to wait for long-term rewards.\n",
    "  - With discounted rewards, the utility of an infinite sequence is finite.\n",
    "  - environment contains terminal states and if the agent is guaranteed to get to one  eventually, then we will never need to compare infinite sequences.\n",
    "  - Infinite sequences can be compared in terms of the average reward obtained per time step.\n",
    "- discounted utilities with infinite horizons is that the optimal policy is  independent of the starting state.\n",
    "- the utility of a state is the expected reward for the  next transition plus the discounted utility of the next state, assuming that the agent chooses  the optimal action.\n",
    "- Bellman equation $U(s) = max \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
    "- action utility funtion, Q function $U(s)= max Q(s,a)$, $\\pi^* = argmax Q(s,a)$\n",
    "  - $Q(s,a) = \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma max Q(s',a')]$- reward scales:\n",
    "  - $U'(s) = mU(s)+b$\n",
    "  - $R'(s,a,s') = mR(s,a,s') +b$\n",
    "  - $\\Phi(s)$ be any function of the state s. Then, according  to the shaping theorem, the following transformation leaves the optimal policy unchanged:\n",
    "  - $R'(s,a,s') = R(s,a,s') +\\gamma \\Phi(s')−\\Phi(s).$\n",
    "  - $Q'(s,a)=Q(s,a)−\\Phi(s)= \\sum_{s'}P(s'|s,a)[R'(s,a,s')+\\gamma max Q'(s',a')]$\n",
    "  - $\\Phi(s)$ potential, The term $\\gamma \\Phi(s')−\\Phi(s)$ functions as a gradient of the  potential. \n",
    "  - $\\Phi(s)$  has higher value in states that have higher utility, the addition of $\\gamma \\Phi(s')−\\Phi(s)$ to the reward has the effect of leading the agent “uphill” in utility. \n",
    "  - all policies are optimal  with a reward function that is zero everywhere.\n",
    "\"\"\"\n",
    "chapter17_2 =  \"\"\"\n",
    "- transition model or model describes the outcome of each action in each state $P(s'|s,a)$, Transitions are Markovian:\n",
    "the probability of arriving in\n",
    "s’ only depends on s and\n",
    "not the history of earlier\n",
    "states.\n",
    "- MDPs are non-deterministic search\n",
    "problems. One way of solving them is via\n",
    "expectimax search.\n",
    "- if the game is infinite:\n",
    "  - Finite horizon:\n",
    "    - Terminate episodes after a fixed T steps\n",
    "    - Gives nonstationary policies\n",
    "  - Discounting: Smaller g means smaller “horizon” – shorter term focus\n",
    "  - Absorbing state: guarantee that for every policy, a terminal state will\n",
    "eventually be reached\n",
    "- <strong>value iteration</strong>\n",
    "- is slow, policy converges long before values, max rarely changes\n",
    "- Every iteration updates both the values and (implicitly) the policy\n",
    "- We don’t track the policy, but taking the max over actions implicitly recomputes\n",
    "it\n",
    "- Bellman equation is the basis of the value iteration algorithm for solving MDPs.\n",
    "- $U_{i+1}(s) \\leftarrow max_{a\\in A(s)} \\sum_{s'} P(s'|s,a)[R(s,a,s') +\\gamma U_i(s')]$, complexity of each iteration: $O(s^2a)$\n",
    "- apply the Bellman update infinitely often, we are guaranteed to reach an unique equilibrium \n",
    "- We can think of the value iteration algorithm as propagating  information through the state space by means of local updates. \n",
    "- if $ ||U_{i+1} − U_i|| < \\epsilon (1−\\gamma)/ \\gamma $  then  $||U_{i+1} − U || < \\epsilon$.\n",
    "- if $||U_i-U|| < \\epsilon$ then $||U^{\\pi_i}-U|| < 2\\epsilon $, policy loss\n",
    "- <strong>policy iteration</strong>\n",
    "  - We do several passes that update utilities with fixed policy (each pass is fast\n",
    "because we consider only one action, not all of them)\n",
    "  - After the policy is evaluated, a new policy is chosen (slow like a value iteration\n",
    "pass)\n",
    "  - The new policy will be better (or we’re done)\n",
    "  - Policy evaluation, given $\\pi_i$ calculate $U_i = U^{\\pi i}$ the utility of each state if $\\pi_i$  were to be executed. \n",
    "  - $U_i(s) \\leftarrow \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$\n",
    "  - policy improvement, Calculate a new MEU policy $\\pi_{i+1} $, using one-step look-ahead  based on $U_i$\n",
    "  - $\\pi(s')= argmax_a\\sum_{s'} P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
    "- Policy evaluation\" $U_i(s) = \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$For n states, we have n linear equations with n unknowns, which can be solved exactly in time $O(n^3)$\n",
    "- <strong>linear programming</strong>\n",
    "- basic idea of the formulation is to consider as variables in the LP the utilities U(s) of  each state s, noting that the utilities for an optimal policy are the highest utilities attainable that  are consistent with the Bellman equations.\n",
    "  - s.t. $U(s) \\geq max \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
    "- <strong>online programming</strong>\n",
    "  - actually a simplification of the EXPECTIMINIMAX  algorithm for game trees with chance nodes:\n",
    " \"\"\"\n",
    "\n",
    "# Lecture notes \n",
    "module8_lecture_notes = \"\"\"\n",
    "- Probability theory describes what an agent should believe on the basis of evidence,  utility theory describes what an agent wants, and decision theory puts the two together  to describe what an agent should do. \n",
    "-  We can use decision theory to build a system that makes decisions by considering all  possible actions and choosing the one that leads to the best expected outcome. Such a  system is known as a rational agent.\n",
    "- Utility theory shows that an agent whose preferences between lotteries are consistent  with a set of simple axioms can be described as possessing a utility function; furthermore, the agent selects actions as if maximizing its expected utility.\n",
    "- Multiattribute utility theory deals with utilities that depend on several distinct attributes of states. Stochastic dominance is a particularly useful technique for making  unambiguous decisions, even without precise utility values for attributes. \n",
    "- Sequential decision problems in stochastic environments, also called Markov decision  processes, or MDPs, are defined by a transition model specifying the probabilistic  outcomes of actions and a reward function specifying the reward in each state.\n",
    "- The utility of a state sequence is the sum of all the rewards over the sequence, possibly  discounted over time. The solution of an MDP is a policy that associates a decision  with every state that the agent might reach. An optimal policy maximizes the utility of  the state sequences encountered when it is executed.\n",
    "- The utility of a state is the expected sum of rewards when an optimal policy is executed  from that state. The value iteration algorithm iteratively solves a set of equations  relating the utility of each state to those of its neighbors. \n",
    "- Policy iteration alternates between calculating the utilities of states under the current  policy and improving the current policy with respect to the current utilities. \n",
    "- Theorm: will converge to unique optimal values\n",
    "  - Basic idea: approximations get refined towards optimal values\n",
    "  - Policy may converge long before values do\n",
    "- Converge:\n",
    "  - Case 1: If the tree has maximum depth M, then VM holds\n",
    "the actual untruncated values\n",
    "  - Case 2: If the discount is less than 1\n",
    "- Compute optimal values: use value iteration or policy iteration\n",
    "- Compute values for a particular policy: use policy evaluation\n",
    "- Turn your values into a policy: use policy extraction (one-step\n",
    "lookahead)\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Definitions of concepts from Module 8.\n",
    "# You can define any term in the textbook or lecture, \n",
    "# but you most use your own words for the definitions.\n",
    "# Each definition should be at most 25 words long.\n",
    "\n",
    "module8_definitions = {\n",
    "\"lottery\":\"don't know what is coming\",\n",
    "\"preference elicitation\":\"involves presenting choices to the human and using the observed preferences to pin down the underlying utility function\",\n",
    "\"risk-averse\":\"they prefer a sure thing with a payoff that is less than the expected monetary value of a gamble.\",\n",
    "\"risk-seeking\":\"“desperate” region at large negative wealth, loss more money is fine\",\n",
    "\"normative theory\":\"describe how a rational agent should act.\",\n",
    "\"descriptive theory\":\"how a actual agent (human) would act\",\n",
    "\"certainty effect\":\"people are strongly attracted to gains that are certain\",\n",
    "\"transitions are Markovian\": \"the  probability of reaching s ′ from s depends only on s and not on the history of earlier states. \",\n",
    "\"decision problem is sequential\":\" the utility function will depend  on a sequence of states and actions—an environment history—rather than on a single state. \",\n",
    "\"MDP\":\" a sequential decision problem for a fully observable, stochastic environment with a Markovian transition model and additive rewards\",\n",
    "\"dynamic programming\":\"simplifying a problem by recursively breaking it into smaller pieces and remembering the optimal solutions to the pieces. \",\n",
    "\"finite horizon\": \"means that there is a fixed time N after which nothing matters—the game is over\",\n",
    "\"nonstationary\":\"A policy that depends on the time\",\n",
    "\"stationary\":\"an optimal action depends only on the current state, and  the optimal policy\",\n",
    "\"Proper policy\":\" guaranteed to reach a terminal state is called a proper policy\",\n",
    "\"sparse\":\"with mostly zeros\",\n",
    "\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "elCCgY64spuq"
   },
   "source": [
    "## Preview of your notes for Module 8\n",
    "\n",
    "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "gYYeNcdM_maE",
    "outputId": "e292e222-b14a-4f97-e73f-d66d61a715ca"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 8: Markov Decision Processes\n",
       "## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n",
       "- $ P(Result(a)=s')= \\sum_{s}^{} P(s)P(s'|s,a) $, which assumes epsidic\n",
       "- Decision theory, in its simplest form, deals with choosing among actions based on the desirability of their immediate outcomes; that is, the environment is assumed to be episodic\n",
       "- The agent’s preferences are captured by a utility function, U(s), which assigns a single number to express the Utility function  desirability of a state.\n",
       "- $ EU(a) = \\sum_{s'} P(Result(a)=s')U(s') $. expected utility\n",
       "- The principle of maximum expected utility (MEU) says that a rational agent should choose  the action that maximizes the agent’s expected utility: \n",
       "- $ action = argmax EU(a) $, maximum expected utility, a prescription for intelligent behavior\n",
       "- All an  intelligent agent has to do is calculate the various quantities, maximize utility over its actions,  and away it goes. But this does not mean that the AI problem is solved by the definition!\n",
       "  - The MEU principle formalizes the general notion that an intelligent agent should “do the  right thing,” but does not operationalize that advice.\n",
       "  - percerption, learning and knowledge representaion, inference are also needed to compute,\n",
       "  - decision theory is not a panacea that solves the  AI problem—but it does provide the beginnings of a basic mathematical framework that is  general enough to define the AI problem. \n",
       "- If an agent acts so as to maximize a utility function that correctly reflects the performance measure, then the agent will achieve the highest possible performance score (averaged over all the  possible environments)\n",
       "- MEU is not the only rational way\n",
       "- basic preferences:\n",
       "  - $ (A\\succ B) $, agent prefers A over B\n",
       "  - $ (A \\sim B) $, no difference in perference\n",
       "  - $ (A \\succsim B) $, agent prefers A over B or no difference in perference\n",
       "- We can think of the set of outcomes for each action  as a lottery—think of each action as a ticket.\n",
       "- axioms of utility theory  list six constraints that we require any reasonable preference relation to obey: \n",
       "- <strong>Orderability</strong>, Given any two lotteries, a rational agent must either prefer one or else  rate them as equally preferable. That is, the agent cannot avoid deciding.\n",
       "  - Exactly one of $ (A\\succ B) $, $ (B \\succ A)$, or $ (A \\sim B) $ holds\n",
       "- <strong>Transitivity</strong>: Given any three lotteries, if an agent prefers A to B and prefers B to C, then the agent must prefer A to C\n",
       "  - $ (A\\succ B) \\wedge (B \\succ C) \\Rightarrow (A \\succ C) $ holds\n",
       "- <strong>Continuity</strong>: Continuity: If some lottery B is between A and C in preference, then there is some  probability p for which the rational agent will be indifferent between getting B for sure  and the lottery that yields A with probability p and C with probability 1− p.\n",
       "\n",
       "\n",
       "## AIMA Chapter 16.2 The Basis of Utility Theory\n",
       " \n",
       "- <strong>Substitutability</strong>: If an agent is indifferent between two lotteries A and B, then the  agent is indifferent between two more complex lotteries that are the same except that  B is substituted for A in one of them. This holds regardless of the probabilities and the  other outcome(s) in the lotteries\n",
       "- <strong>Monotonicity</strong>: Suppose two lotteries have the same two possible outcomes, A and  B. If an agent prefers A to B, then the agent must prefer the lottery that has a higher  probability for A (and vice versa). \n",
       "- <strong>Decomposability</strong>: Compound lotteries can be reduced to simpler ones using the laws of probability. it compresses two consecutive lotteries into a single equivalent lottery.\n",
       "- Rational preference lead to utility:\n",
       "- <strong>Existence of Utility Function</strong>:If an agent’s preferences obey the axioms of utility, then  there exists a function U such that U(A) > U(B) if and only if A is preferred to B, and  U(A) = U(B) if and only if the agent is indifferent between A and B. \n",
       "  - $U(A) > U(B) \\Leftrightarrow A \\succ B $  and  $ U(A) = U(B) \\Leftrightarrow A \\sim B$\n",
       "- <strong>Expected Utility of a Lottery</strong>: The utility of a lottery is the sum of the probability of  each outcome times the utility of that outcome. \n",
       "  - $ U([p_1,S_1;\\dots ; p_n,S_n]) = \\sum_{i} p_i U(S_i)$\n",
       "  - In other words, once the probabilities and utilities of the possible outcome states are specified, the utility of a compound lottery involving those states is completely determined.\n",
       "  - Because the  outcome of a nondeterministic action is a lottery, agent can act rationally, consistently with its preferences—only by choosing an action that maximizes expected  utility\n",
       "  - The preceding theorems establish that (assuming the constraints on rational preferences)  a utility function exists for any rational agent. The theorems do not establish that the utility  function is unique.\n",
       "- As in game-playing, in a deterministic environment an agent needs only a preference ranking on states—the numbers don’t matter. This is called a value function or <strong> ordinal utility function</strong>. \n",
       "- existence of a utility function that describes an agent’s  preference behavior does not necessarily mean that the agent is explicitly maximizing that  utility function in its own deliberations.\n",
       "\n",
       "## AIMA Chapter 16.3 Utility Functions\n",
       "\n",
       "- system that helps a human make decisions or acts on  his or her behalf, we must first work out what the human’s utility function is. This process, often called preference elicitation, presenting choices to the human and using the Preference elicitation  observed preferences to pin down the underlying utility function. \n",
       "- Normalized utilities: $ u_{bot} = 0, u_{top} = 1 $\n",
       "- standard lottery: $ [p,u_{top};(1-p),u_{bot}] $\n",
       "- expected monetary value (EMV):\n",
       "  - $ EMV = \\sum P(x)*Value(x)$\n",
       "  - Utility is not  directly proportional to monetary value, because the utility for your the momey you own is not the same\n",
       "  - utility of money plateaus at higher, deepening at lower\n",
       "- We will assume, kindly perhaps, that the estimates EU(a) are unbiased, but the real outcome will usually be significantly worse than we estimated,  even though the estimate was unbiased!\n",
       "\n",
       "\n",
       "## AIMA Chapter 17.1 Sequential Decision Problems\n",
       "\n",
       "- policy a solution specify what the agent should do for any state that the agent might reach. A solution of this kind is called a policy. $\\pi(s)$, action recommended by the policy π for state s.\n",
       "- An optimal policy is a policy that yields the highest expected utility. $\\pi^*(s) = argmax U^\\pi(s) = argmax \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- The balance of risk and reward changes depending on the value of $r=R(s,a,s')$ for transitions between nonterminal states. reward can be positive or negative but are bounded by some maximum value.\n",
       "Because the decision process is sequential, we must specify the utility function on a\n",
       "sequence of states and actions. the agent can receive a reward at\n",
       "each time step,\n",
       "- with a finite horizon, an optimal action in a given state may depend on how much time is left. \n",
       "- With no fixed time limit, on the other hand, there is no reason to behave differently in the  same state at different times.\n",
       "- Policies for the infinite-horizon case are simpler than those for the finite-horizon case\n",
       "- additive discounted rewards: the utility of a history is $U_h([s_0,a_0,s_1,a_1,...]) = R(s_0,a_0,s_1)+\\gamma R(s_1,a_1,s_2)+\\gamma^2 R(s_2,a_2,s_3) + ...$, Sooner rewards probably do have\n",
       "higher utility than later rewards\n",
       "- When γ is close to 0, rewards  in the distant future are viewed as insignificant. When γ is close to 1, an agent is more willing  to wait for long-term rewards.\n",
       "  - With discounted rewards, the utility of an infinite sequence is finite.\n",
       "  - environment contains terminal states and if the agent is guaranteed to get to one  eventually, then we will never need to compare infinite sequences.\n",
       "  - Infinite sequences can be compared in terms of the average reward obtained per time step.\n",
       "- discounted utilities with infinite horizons is that the optimal policy is  independent of the starting state.\n",
       "- the utility of a state is the expected reward for the  next transition plus the discounted utility of the next state, assuming that the agent chooses  the optimal action.\n",
       "- Bellman equation $U(s) = max \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- action utility funtion, Q function $U(s)= max Q(s,a)$, $\\pi^* = argmax Q(s,a)$\n",
       "  - $Q(s,a) = \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma max Q(s',a')]$- reward scales:\n",
       "  - $U'(s) = mU(s)+b$\n",
       "  - $R'(s,a,s') = mR(s,a,s') +b$\n",
       "  - $\\Phi(s)$ be any function of the state s. Then, according  to the shaping theorem, the following transformation leaves the optimal policy unchanged:\n",
       "  - $R'(s,a,s') = R(s,a,s') +\\gamma \\Phi(s')−\\Phi(s).$\n",
       "  - $Q'(s,a)=Q(s,a)−\\Phi(s)= \\sum_{s'}P(s'|s,a)[R'(s,a,s')+\\gamma max Q'(s',a')]$\n",
       "  - $\\Phi(s)$ potential, The term $\\gamma \\Phi(s')−\\Phi(s)$ functions as a gradient of the  potential. \n",
       "  - $\\Phi(s)$  has higher value in states that have higher utility, the addition of $\\gamma \\Phi(s')−\\Phi(s)$ to the reward has the effect of leading the agent “uphill” in utility. \n",
       "  - all policies are optimal  with a reward function that is zero everywhere.\n",
       "\n",
       "\n",
       "## AIMA Chapter 17.2 Algorithms for MDPs\n",
       "\n",
       "- transition model or model describes the outcome of each action in each state $P(s'|s,a)$, Transitions are Markovian:\n",
       "the probability of arriving in\n",
       "s’ only depends on s and\n",
       "not the history of earlier\n",
       "states.\n",
       "- MDPs are non-deterministic search\n",
       "problems. One way of solving them is via\n",
       "expectimax search.\n",
       "- if the game is infinite:\n",
       "  - Finite horizon:\n",
       "    - Terminate episodes after a fixed T steps\n",
       "    - Gives nonstationary policies\n",
       "  - Discounting: Smaller g means smaller “horizon” – shorter term focus\n",
       "  - Absorbing state: guarantee that for every policy, a terminal state will\n",
       "eventually be reached\n",
       "- <strong>value iteration</strong>\n",
       "- is slow, policy converges long before values, max rarely changes\n",
       "- Every iteration updates both the values and (implicitly) the policy\n",
       "- We don’t track the policy, but taking the max over actions implicitly recomputes\n",
       "it\n",
       "- Bellman equation is the basis of the value iteration algorithm for solving MDPs.\n",
       "- $U_{i+1}(s) \\leftarrow max_{a\\in A(s)} \\sum_{s'} P(s'|s,a)[R(s,a,s') +\\gamma U_i(s')]$, complexity of each iteration: $O(s^2a)$\n",
       "- apply the Bellman update infinitely often, we are guaranteed to reach an unique equilibrium \n",
       "- We can think of the value iteration algorithm as propagating  information through the state space by means of local updates. \n",
       "- if $ ||U_{i+1} − U_i|| < \\epsilon (1−\\gamma)/ \\gamma $  then  $||U_{i+1} − U || < \\epsilon$.\n",
       "- if $||U_i-U|| < \\epsilon$ then $||U^{\\pi_i}-U|| < 2\\epsilon $, policy loss\n",
       "- <strong>policy iteration</strong>\n",
       "  - We do several passes that update utilities with fixed policy (each pass is fast\n",
       "because we consider only one action, not all of them)\n",
       "  - After the policy is evaluated, a new policy is chosen (slow like a value iteration\n",
       "pass)\n",
       "  - The new policy will be better (or we’re done)\n",
       "  - Policy evaluation, given $\\pi_i$ calculate $U_i = U^{\\pi i}$ the utility of each state if $\\pi_i$  were to be executed. \n",
       "  - $U_i(s) \\leftarrow \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$\n",
       "  - policy improvement, Calculate a new MEU policy $\\pi_{i+1} $, using one-step look-ahead  based on $U_i$\n",
       "  - $\\pi(s')= argmax_a\\sum_{s'} P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- Policy evaluation\" $U_i(s) = \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$For n states, we have n linear equations with n unknowns, which can be solved exactly in time $O(n^3)$\n",
       "- <strong>linear programming</strong>\n",
       "- basic idea of the formulation is to consider as variables in the LP the utilities U(s) of  each state s, noting that the utilities for an optimal policy are the highest utilities attainable that  are consistent with the Bellman equations.\n",
       "  - s.t. $U(s) \\geq max \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- <strong>online programming</strong>\n",
       "  - actually a simplification of the EXPECTIMINIMAX  algorithm for game trees with chance nodes:\n",
       " \n",
       "\n",
       "## Module 8 Lecture Notes\n",
       "\n",
       "- Probability theory describes what an agent should believe on the basis of evidence,  utility theory describes what an agent wants, and decision theory puts the two together  to describe what an agent should do. \n",
       "-  We can use decision theory to build a system that makes decisions by considering all  possible actions and choosing the one that leads to the best expected outcome. Such a  system is known as a rational agent.\n",
       "- Utility theory shows that an agent whose preferences between lotteries are consistent  with a set of simple axioms can be described as possessing a utility function; furthermore, the agent selects actions as if maximizing its expected utility.\n",
       "- Multiattribute utility theory deals with utilities that depend on several distinct attributes of states. Stochastic dominance is a particularly useful technique for making  unambiguous decisions, even without precise utility values for attributes. \n",
       "- Sequential decision problems in stochastic environments, also called Markov decision  processes, or MDPs, are defined by a transition model specifying the probabilistic  outcomes of actions and a reward function specifying the reward in each state.\n",
       "- The utility of a state sequence is the sum of all the rewards over the sequence, possibly  discounted over time. The solution of an MDP is a policy that associates a decision  with every state that the agent might reach. An optimal policy maximizes the utility of  the state sequences encountered when it is executed.\n",
       "- The utility of a state is the expected sum of rewards when an optimal policy is executed  from that state. The value iteration algorithm iteratively solves a set of equations  relating the utility of each state to those of its neighbors. \n",
       "- Policy iteration alternates between calculating the utilities of states under the current  policy and improving the current policy with respect to the current utilities. \n",
       "- Theorm: will converge to unique optimal values\n",
       "  - Basic idea: approximations get refined towards optimal values\n",
       "  - Policy may converge long before values do\n",
       "- Converge:\n",
       "  - Case 1: If the tree has maximum depth M, then VM holds\n",
       "the actual untruncated values\n",
       "  - Case 2: If the discount is less than 1\n",
       "- Compute optimal values: use value iteration or policy iteration\n",
       "- Compute values for a particular policy: use policy evaluation\n",
       "- Turn your values into a policy: use policy extraction (one-step\n",
       "lookahead)\n",
       "\n",
       "\n",
       "## Module 8 Definitions\n",
       "* **Lottery** - don't know what is coming\n",
       "* **Preference elicitation** - involves presenting choices to the human and using the observed preferences to pin down the underlying utility function\n",
       "* **Risk-averse** - they prefer a sure thing with a payoff that is less than the expected monetary value of a gamble.\n",
       "* **Risk-seeking** - “desperate” region at large negative wealth, loss more money is fine\n",
       "* **Normative theory** - describe how a rational agent should act.\n",
       "* **Descriptive theory** - how a actual agent (human) would act\n",
       "* **Certainty effect** - people are strongly attracted to gains that are certain\n",
       "* **Transitions are markovian** - the  probability of reaching s ′ from s depends only on s and not on the history of earlier states. \n",
       "* **Decision problem is sequential** -  the utility function will depend  on a sequence of states and actions—an environment history—rather than on a single state. \n",
       "* **Mdp** -  a sequential decision problem for a fully observable, stochastic environment with a Markovian transition model and additive rewards\n",
       "* **Dynamic programming** - simplifying a problem by recursively breaking it into smaller pieces and remembering the optimal solutions to the pieces. \n",
       "* **Finite horizon** - means that there is a fixed time N after which nothing matters—the game is over\n",
       "* **Nonstationary** - A policy that depends on the time\n",
       "* **Stationary** - an optimal action depends only on the current state, and  the optimal policy\n",
       "* **Proper policy** -  guaranteed to reach a terminal state is called a proper policy\n",
       "* **Sparse** - with mostly zeros\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "from IPython.display import display, Markdown, Latex, Math\n",
    "\n",
    "module8_sections = [\n",
    "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
    "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
    "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
    "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
    "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
    "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
    "]\n",
    "\n",
    "\n",
    "def length_check(sections):\n",
    "  warnings = \"\"\n",
    "  for section, summary in sections:\n",
    "    max_word_length = 500\n",
    "    max_character_length = 3000\n",
    "    if \"lecture\" in section.lower():\n",
    "      # It's OK to do longer summaries of the lectures\n",
    "      max_word_length = 2000\n",
    "      max_character_length = 12000  \n",
    "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
    "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
    "  return warnings\n",
    "\n",
    "def length_check_definitions(definitions):\n",
    "  warnings = \"\"\n",
    "  for concept, definition in definitions.items():\n",
    "    if(len(definition.split(' ')) > 25):\n",
    "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
    "  return warnings\n",
    "\n",
    "\n",
    "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
    "  markdown = \"\"\n",
    "\n",
    "  # Check summary lengths\n",
    "  warnings = length_check(sections)\n",
    "  if warnings != \"\":\n",
    "    markdown += \"## \" + warnings\n",
    "\n",
    "  # Check definition lengths\n",
    "  warnings = length_check_definitions(definitions)\n",
    "  if warnings != \"\":\n",
    "    markdown += \"## \" + warnings\n",
    "\n",
    "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
    "\n",
    "\n",
    "  for section, summary in sections:\n",
    "    if show_placeholder and summary == \" Your summary here \":\n",
    "      summary = placeholder\n",
    "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
    "  \n",
    "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
    "  for concept, definition in definitions.items():\n",
    "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
    "  return markdown\n",
    "\n",
    "\n",
    "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
    "display(Markdown(module8_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9xRbdqDox2zD"
   },
   "source": [
    "#Module 9 - Reinforcement Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "id": "DpBvu7EMx7sy"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
    "chapter17_3 =  \"\"\"\n",
    "- a one-armed bandit is a slot machine, n arm has n levers\n",
    "- explitation: the current best action to obtain rewards\n",
    "- exploration: previously unknown states and actions to gain information, which can in some cases be converted into a better policy and better long-term rewards.\n",
    "- come back if has time\n",
    "<strong> early start</strong>\n",
    "- Must (learn to) act so as to maximize expected rewards\n",
    "- Reinforcement learning differs from “just solving  an MDP” because the agent is not given the MDP as a problem to solve; the agent is in the  MDP.\n",
    "  - reward function is often (as we saw for chess) very concise and easy to specify\n",
    "  - we don’t have to be experts,  capable of supplying the correct action in any situation, as would be the case if we tried to  apply supervised learning. \n",
    "- sparse rewards, because in the vast majority of states the agent is  given no informative reward signal at all.\n",
    "- Model-based reinforcement learning: agent uses a transition model of the environment to help interpret the reward signals and to make decisions  about how to act. The model may be initially unknown, or it may already be known\n",
    "  - Learn an approximate model based on experiences\n",
    "Solve for values as if the learned model were correct\n",
    "  - Step 1: Learn empirical MDP model:\n",
    "    - Count outcomes s’ for each s, a\n",
    "    - Normalize to give an estimate of trasition\n",
    "    - Discover reward each when we experience (s, a, s’)\n",
    "  - Step 2: Solve the learned MDP\n",
    "    - For example, use value iteration, as before\n",
    "  - the transition model is  also useful for state estimation\n",
    "  - Model-based reinforcement learning  systems often learn a utility function U(s), defined in terms of the  sum of rewards from state s onward.\n",
    "- Model-free reinforcement learning: In these approaches the agent neither knows nor learns a transition model for the environment.\n",
    "  - Action-utility learning: most common form of action-utility learning is Q-learning, where the agent learns a Q-function, or quality-function, Q(s,a), \n",
    "  - Policy search: The agent learns a policy π(s) that maps directly from states to  actions.(reflex agent)\n",
    "- passive reinforcement learning: where the agent’s policy is fixed and the task is to learn the utilities of states\n",
    "- active  reinforcement learning: where the agent must also figure out what to do.\n",
    "\"\"\"\n",
    "\n",
    "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
    "chapter22_1 =  \"\"\"\n",
    "- Simplified task: policy evaluation\n",
    "  - Input: a fixed policy π(s)\n",
    "  - You don’t know the transitions T(s,a,s’)\n",
    "  - You don’t know the rewards R(s,a,s’)\n",
    "  - Goal: learn the state values\n",
    "- In this case:\n",
    "  - No choice about what actions to take\n",
    "  - This is NOT offline planning!\n",
    "- a fully observable environment with a small number of  actions and states, in which an agent already has a fixed policy π(s) that determines its actions.  The agent is trying to learn the utility function U π (s)—the expected total discounted reward  if policy π is executed beginning in state s.\n",
    "- passive learning task is similar to the policy evaluation task, but do not know the trasition model $P(s'|s,a)$ and $R(s,a,s')$\n",
    "- direct utility estimation is that the utility of a state is defined as the expected total reward from that state onward (called the expected reward-to-go), and that each trial  provides a sample of this quantity for each state visited.\n",
    "\"\"\"\n",
    "chapter22_2 =  \"\"\"\n",
    "- Direct evaluation:\n",
    "  - Compute values for each state under π\n",
    "  - Average together observed sample values\n",
    "    - Act according to π\n",
    "    - Every time you visit a state, write down what the\n",
    "    - sum of discounted rewards turned out to be\n",
    "    - Average those samples\n",
    "  - bad:\n",
    "    - It wastes information about state connections\n",
    "    - Each state must be learned separately\n",
    "    - So, it takes a long time to learn\n",
    "- The utility of a state is determined by the reward and the expected utility  of the successor states.\n",
    "  - $U_i(s) = \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$\n",
    "- adaptive dynamic programming (or ADP) agent takes advantage of the constraints among the utilities of states by learning the transition model that connects them and solving the corresponding Markov decision process using dynamic programming.\n",
    "  - we can adopt the approach of modified policy iteration,  using a simplified value iteration process to update the utility estimates after each change to  the learned model.\n",
    "  - Learning the transition model is easy, because the environment is fully observable.\n",
    "  - The algorithm converges fast but can become quite costly to compute for large state spaces. ADP is a model based approach and requires the transition model of the environment.\n",
    "- temperal difference learning:\n",
    "  - learn from every experience!\n",
    "Update V(s) each time we experience a transition (s, a, s’, r)\n",
    "Likely outcomes s’ will contribute updates more often\n",
    "  - Temporal difference learning of values\n",
    "Policy still fixed, still doing evaluation!\n",
    "Move values toward value of whatever successor occurs: running average\n",
    "  - Decreasing learning rate (alpha) can give converging averages\n",
    "  - $U^\\pi(s) \\leftarrow U^\\pi(s)+α[R(s,\\pi(s),s')+\\gamma U^\\pi(s')-U(s)]$\n",
    "  - All temporal-difference methods work by adjusting the utility estimates toward the ideal  equilibrium that holds locally when the utility estimates are correct.\n",
    "  - TD does not need a transition model to perform its updates.\n",
    "  - Forgets about the past (distant past values were wrong anyway\n",
    "- The ADP and TD approaches are closely related. Both try to make local adjustments to  the utility estimates in order to make each state “agree” with its successors.\n",
    "- that TD adjusts a state to agree with its observed successor whereas ADP  adjusts the state to agree with all of the successors that might occur, weighted by their probabilities\n",
    "- TD makes a single adjustment per observed transition, ADP makes  as many as it needs to restore consistency between the utility estimates U and the transition  model P.\n",
    "- Each adjustment made by ADP could be seen, from the TD point of view, as a result of a pseudoexperience generated by simulating the current transition model.\n",
    "- prioritized sweeping  heuristic prefers to make adjustments to states whose likely successors have just undergone a  large adjustment in their own utility estimates. \n",
    "\"\"\"\n",
    "chapter22_3 =  \"\"\"\n",
    "- Q-learning converges to optimal policy -- even\n",
    "if you’re acting suboptimally!\n",
    "- Caveats:\n",
    "  - You have to explore enough\n",
    "  - You have to eventually make the learning rate\n",
    "small enough\n",
    "- exploration function. The function f(u,n) determines how greed (preference Exploration function  for high values of the utility u) is traded off against curiosity\n",
    "  - safe exploration\n",
    "  - Bayes reinforcement learning: assumes a prior probability P(h) over hypotheses h about what the true model is; the posterior probability P(h| e) is obtained  in the usual way by Bayes’ rule given the observations to date.\n",
    "  - $\\pi^* = argmax_\\pi \\sum_h P(h|e)U_h^\\pi$\n",
    "  - robust control theory, allows for a set of possible models H without assigning probabilities to them, and defines an optimal robust policy as  one that gives the best outcome in the worst case over H: \n",
    "  - $\\pi^* - argmax_\\pi min_hU^\\pi_h$\n",
    "- TD Q learning\n",
    "  - $Q(s,a) = \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma max_{a'}Q(s',a')]$\n",
    "  - $Q(s,a) = Q(s,a) + α[R(s,a,s')+\\gamma max_{a'}Q(s',a')-Q(s,a)]$\n",
    "  - a TD Q-learning agent  does not need a transition model, P(s ′ |s,a), either for learning or for action selection.\n",
    "  - the Qlearning agent has no means of looking into the future, so it may have difficulty when rewards  are sparse and long action sequences must be constructed to reach them. \n",
    "  - SARSA\n",
    "  - $Q(s,a) = Q(s,a) + α[R(s,a,s')+\\gamma Q(s',a')-Q(s,a)]$\n",
    "  - If the agent is greedy and always takes the action with the best Q-value, the two algorithms  are identical. When exploration is happening, however, they differ: if the exploration yields  a negative reward, SARSA penalizes the action, while Q-learning does not. \n",
    "  - Q-learning is an off-policy learning algorithm, because it learns Q-values that answer the  question “What would this action be worth in this state, assuming that I stop using whatever  policy I am using now, and start acting according to a policy that chooses the best action \n",
    "  - SARSA is an on-policy algorithm: it learns Q-values that  answer the question “What would this action be worth in this state, assuming I stick with my policy?\n",
    "  - Q-learning is more flexible than SARSA, in the sense that a Q-learning agent can  learn how to behave well when under the control of a wide variety of exploration policies.\n",
    "  - SARSA is appropriate if the overall policy is even partly controlled by other  agents or programs,\n",
    "\"\"\"\n",
    "chapter22_4 =  \"\"\"\n",
    "- How to explror?\n",
    "  - Simplest: random actions (ε-greedy)\n",
    "    - Every time step, flip a coin\n",
    "    - With (small) probability ε, act randomly\n",
    "    - With (large) probability 1-ε, act on current policy\n",
    "  - Problems with random actions?\n",
    "    - You do eventually explore the space, but keep\n",
    "thrashing around once learning is done\n",
    "    - One solution: lower ε over time\n",
    "    - Another solution: exploration functions\n",
    "  - exploration function:\n",
    "    - explore areas whose badness is not\n",
    "(yet) established, eventually stop exploring\n",
    "- regret:\n",
    "  - Even if you learn the optimal policy,\n",
    "you still make mistakes along the way!\n",
    "Regret is a measure of your total\n",
    "mistake cost: the difference between\n",
    "your (expected) rewards, including\n",
    "youthful suboptimality, and optimal\n",
    "(expected) rewards\n",
    "  - Minimizing regret goes beyond\n",
    "learning to be optimal – it requires\n",
    "optimally learning to be optimal\\\n",
    "  - random exploration and\n",
    "exploration functions both end up\n",
    "optimal, but random exploration has\n",
    "higher regret\n",
    "- function approximation for the process of constructing a compact approximation of the true utility function or Q-function. \n",
    "- Basic Q-Learning keeps a table of all q-values\n",
    "In realistic situations, we cannot possibly learn\n",
    "about every single state!\n",
    "  - Learn about some small number of training states from\n",
    "experience\n",
    "  - Generalize that experience to new, similar situations\n",
    "- approximate Q-learning in features\n",
    "  - Q-learning with linear Q-functions:\n",
    "    - Adjust weights of active features\n",
    "    - E.g., if something unexpectedly bad happens, blame the features that were on:\n",
    "disprefer all states with that state’s features\n",
    "\n",
    "\"\"\"\n",
    "chapter22_5 =  \"\"\"\n",
    "- policy search: the  idea is to keep twiddling the policy as long as its performance improves, then stop.\n",
    "- π is a function that  maps states to actions. We are interested primarily in parameterized representations of π that  have far fewer parameters than there are states in the state space\n",
    "- Policy search will then adjust the parameters θ to  improve the policy.\n",
    "- come back if have time\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module9_lecture_notes = \"\"\"\n",
    "- The overall agent design dictates the kind of information that must be learned:  \n",
    "  - A model-based reinforcement learning agent acquires (or is equipped with) a  transition model P(s ′ |s,a) for the environment and learns a utility function U(s).  \n",
    "  - A model-free reinforcement learning agent may learn an action-utility function  Q(s,a) or a policy π(s). \n",
    "- Utilities can be learned using several different approaches:\n",
    "  - Direct utility estimation uses the total observed reward-to-go for a given state as  direct evidence for learning its utility.\n",
    "  - Adaptive dynamic programming (ADP) learns a model and a reward function  from observations and then uses value or policy iteration to obtain the utilities or  an optimal policy. ADP makes optimal use of the local constraints on utilities of  states imposed through the neighborhood structure of the environment. \n",
    "  - Temporal-difference (TD) methods adjust utility estimates to be more consistent  with those of successor states. They can be viewed as simple approximations of the  ADP approach that can learn without requiring a transition model. Using a learned  model to generate pseudoexperiences can, however, result in faster learning.\n",
    "- Action-utility functions, or Q-functions, can be learned by an ADP approach or a TD  approach. With TD, Q-learning requires no model in either the learning or actionselection phase. This simplifies the learning problem but potentially restricts the ability  to learn in complex environments, because the agent cannot simulate the results of  possible courses of action.\n",
    "-  When the learning agent is responsible for selecting actions while it learns, it must  trade off the estimated value of those actions against the potential for learning useful  new information. An exact solution for the exploration problem is infeasible, but some  simple heuristics do a reasonable job. An exploring agent must also take care to avoid  premature death.\n",
    "- In large state spaces, reinforcement learning algorithms must use an approximate functional representation of U(s) or Q(s,a) in order to generalize over states. Deep reinforcement learning—using deep neural networks as function approximators—has  achieved considerable success on hard problems. \n",
    "-  Reward shaping and hierarchical reinforcement learning are helpful for learning  complex behaviors, particularly when rewards are sparse and long action sequences are  required to obtain them. \n",
    "-  Policy-search methods operate directly on a representation of the policy, attempting  to improve it based on observed performance. The variation in the performance in a  stochastic domain is a serious problem; for simulated domains this can be overcome by  fixing the randomness in advance. \n",
    "\"\"\"\n",
    "\n",
    "# Each definition should be at most 25 words long.\n",
    "module9_definitions = {\n",
    "\"Markov reward process or MRP\":\" that is, an MDP with only one Markov reward  process  possible action $a_i$. \",\n",
    "\"Exploration\": \"you have to try unknown actions to get information\",\n",
    "\"Exploitation\": \"eventually, you have to use what you know\",\n",
    "\"Regret\": \"even if you learn intelligently, you make mistakes\",\n",
    "\"Sampling\": \"because of chance, you have to try things repeatedly\",\n",
    "\"Difficulty\": \"learning can be much harder than solving a known MDP\"\n",
    "\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "ydOK__lwWgdp",
    "outputId": "4aa826b4-cd31-4519-a66f-3b65d3f68ba3"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 9: Reinforcement Learning\n",
       "## AIMA Chapter 17.3 Bandit Problems\n",
       "\n",
       "- a one-armed bandit is a slot machine, n arm has n levers\n",
       "- explitation: the current best action to obtain rewards\n",
       "- exploration: previously unknown states and actions to gain information, which can in some cases be converted into a better policy and better long-term rewards.\n",
       "- come back if has time\n",
       "<strong> early start</strong>\n",
       "- Must (learn to) act so as to maximize expected rewards\n",
       "- Reinforcement learning differs from “just solving  an MDP” because the agent is not given the MDP as a problem to solve; the agent is in the  MDP.\n",
       "  - reward function is often (as we saw for chess) very concise and easy to specify\n",
       "  - we don’t have to be experts,  capable of supplying the correct action in any situation, as would be the case if we tried to  apply supervised learning. \n",
       "- sparse rewards, because in the vast majority of states the agent is  given no informative reward signal at all.\n",
       "- Model-based reinforcement learning: agent uses a transition model of the environment to help interpret the reward signals and to make decisions  about how to act. The model may be initially unknown, or it may already be known\n",
       "  - Learn an approximate model based on experiences\n",
       "Solve for values as if the learned model were correct\n",
       "  - Step 1: Learn empirical MDP model:\n",
       "    - Count outcomes s’ for each s, a\n",
       "    - Normalize to give an estimate of trasition\n",
       "    - Discover reward each when we experience (s, a, s’)\n",
       "  - Step 2: Solve the learned MDP\n",
       "    - For example, use value iteration, as before\n",
       "  - the transition model is  also useful for state estimation\n",
       "  - Model-based reinforcement learning  systems often learn a utility function U(s), defined in terms of the  sum of rewards from state s onward.\n",
       "- Model-free reinforcement learning: In these approaches the agent neither knows nor learns a transition model for the environment.\n",
       "  - Action-utility learning: most common form of action-utility learning is Q-learning, where the agent learns a Q-function, or quality-function, Q(s,a), \n",
       "  - Policy search: The agent learns a policy π(s) that maps directly from states to  actions.(reflex agent)\n",
       "- passive reinforcement learning: where the agent’s policy is fixed and the task is to learn the utilities of states\n",
       "- active  reinforcement learning: where the agent must also figure out what to do.\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.1 Learning from Rewards\n",
       "\n",
       "- Simplified task: policy evaluation\n",
       "  - Input: a fixed policy π(s)\n",
       "  - You don’t know the transitions T(s,a,s’)\n",
       "  - You don’t know the rewards R(s,a,s’)\n",
       "  - Goal: learn the state values\n",
       "- In this case:\n",
       "  - No choice about what actions to take\n",
       "  - This is NOT offline planning!\n",
       "- a fully observable environment with a small number of  actions and states, in which an agent already has a fixed policy π(s) that determines its actions.  The agent is trying to learn the utility function U π (s)—the expected total discounted reward  if policy π is executed beginning in state s.\n",
       "- passive learning task is similar to the policy evaluation task, but do not know the trasition model $P(s'|s,a)$ and $R(s,a,s')$\n",
       "- direct utility estimation is that the utility of a state is defined as the expected total reward from that state onward (called the expected reward-to-go), and that each trial  provides a sample of this quantity for each state visited.\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.2 Passive Reinforcement Learning\n",
       "\n",
       "- Direct evaluation:\n",
       "  - Compute values for each state under π\n",
       "  - Average together observed sample values\n",
       "    - Act according to π\n",
       "    - Every time you visit a state, write down what the\n",
       "    - sum of discounted rewards turned out to be\n",
       "    - Average those samples\n",
       "  - bad:\n",
       "    - It wastes information about state connections\n",
       "    - Each state must be learned separately\n",
       "    - So, it takes a long time to learn\n",
       "- The utility of a state is determined by the reward and the expected utility  of the successor states.\n",
       "  - $U_i(s) = \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$\n",
       "- adaptive dynamic programming (or ADP) agent takes advantage of the constraints among the utilities of states by learning the transition model that connects them and solving the corresponding Markov decision process using dynamic programming.\n",
       "  - we can adopt the approach of modified policy iteration,  using a simplified value iteration process to update the utility estimates after each change to  the learned model.\n",
       "  - Learning the transition model is easy, because the environment is fully observable.\n",
       "  - The algorithm converges fast but can become quite costly to compute for large state spaces. ADP is a model based approach and requires the transition model of the environment.\n",
       "- temperal difference learning:\n",
       "  - learn from every experience!\n",
       "Update V(s) each time we experience a transition (s, a, s’, r)\n",
       "Likely outcomes s’ will contribute updates more often\n",
       "  - Temporal difference learning of values\n",
       "Policy still fixed, still doing evaluation!\n",
       "Move values toward value of whatever successor occurs: running average\n",
       "  - Decreasing learning rate (alpha) can give converging averages\n",
       "  - $U^\\pi(s) \\leftarrow U^\\pi(s)+α[R(s,\\pi(s),s')+\\gamma U^\\pi(s')-U(s)]$\n",
       "  - All temporal-difference methods work by adjusting the utility estimates toward the ideal  equilibrium that holds locally when the utility estimates are correct.\n",
       "  - TD does not need a transition model to perform its updates.\n",
       "  - Forgets about the past (distant past values were wrong anyway\n",
       "- The ADP and TD approaches are closely related. Both try to make local adjustments to  the utility estimates in order to make each state “agree” with its successors.\n",
       "- that TD adjusts a state to agree with its observed successor whereas ADP  adjusts the state to agree with all of the successors that might occur, weighted by their probabilities\n",
       "- TD makes a single adjustment per observed transition, ADP makes  as many as it needs to restore consistency between the utility estimates U and the transition  model P.\n",
       "- Each adjustment made by ADP could be seen, from the TD point of view, as a result of a pseudoexperience generated by simulating the current transition model.\n",
       "- prioritized sweeping  heuristic prefers to make adjustments to states whose likely successors have just undergone a  large adjustment in their own utility estimates. \n",
       "\n",
       "\n",
       "## AIMA Chapter 22.3 Active Reinforcement Learning\n",
       "\n",
       "- Q-learning converges to optimal policy -- even\n",
       "if you’re acting suboptimally!\n",
       "- Caveats:\n",
       "  - You have to explore enough\n",
       "  - You have to eventually make the learning rate\n",
       "small enough\n",
       "- exploration function. The function f(u,n) determines how greed (preference Exploration function  for high values of the utility u) is traded off against curiosity\n",
       "  - safe exploration\n",
       "  - Bayes reinforcement learning: assumes a prior probability P(h) over hypotheses h about what the true model is; the posterior probability P(h| e) is obtained  in the usual way by Bayes’ rule given the observations to date.\n",
       "  - $\\pi^* = argmax_\\pi \\sum_h P(h|e)U_h^\\pi$\n",
       "  - robust control theory, allows for a set of possible models H without assigning probabilities to them, and defines an optimal robust policy as  one that gives the best outcome in the worst case over H: \n",
       "  - $\\pi^* - argmax_\\pi min_hU^\\pi_h$\n",
       "- TD Q learning\n",
       "  - $Q(s,a) = \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma max_{a'}Q(s',a')]$\n",
       "  - $Q(s,a) = Q(s,a) + α[R(s,a,s')+\\gamma max_{a'}Q(s',a')-Q(s,a)]$\n",
       "  - a TD Q-learning agent  does not need a transition model, P(s ′ |s,a), either for learning or for action selection.\n",
       "  - the Qlearning agent has no means of looking into the future, so it may have difficulty when rewards  are sparse and long action sequences must be constructed to reach them. \n",
       "  - SARSA\n",
       "  - $Q(s,a) = Q(s,a) + α[R(s,a,s')+\\gamma Q(s',a')-Q(s,a)]$\n",
       "  - If the agent is greedy and always takes the action with the best Q-value, the two algorithms  are identical. When exploration is happening, however, they differ: if the exploration yields  a negative reward, SARSA penalizes the action, while Q-learning does not. \n",
       "  - Q-learning is an off-policy learning algorithm, because it learns Q-values that answer the  question “What would this action be worth in this state, assuming that I stop using whatever  policy I am using now, and start acting according to a policy that chooses the best action \n",
       "  - SARSA is an on-policy algorithm: it learns Q-values that  answer the question “What would this action be worth in this state, assuming I stick with my policy?\n",
       "  - Q-learning is more flexible than SARSA, in the sense that a Q-learning agent can  learn how to behave well when under the control of a wide variety of exploration policies.\n",
       "  - SARSA is appropriate if the overall policy is even partly controlled by other  agents or programs,\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n",
       "\n",
       "- How to explror?\n",
       "  - Simplest: random actions (ε-greedy)\n",
       "    - Every time step, flip a coin\n",
       "    - With (small) probability ε, act randomly\n",
       "    - With (large) probability 1-ε, act on current policy\n",
       "  - Problems with random actions?\n",
       "    - You do eventually explore the space, but keep\n",
       "thrashing around once learning is done\n",
       "    - One solution: lower ε over time\n",
       "    - Another solution: exploration functions\n",
       "  - exploration function:\n",
       "    - explore areas whose badness is not\n",
       "(yet) established, eventually stop exploring\n",
       "- regret:\n",
       "  - Even if you learn the optimal policy,\n",
       "you still make mistakes along the way!\n",
       "Regret is a measure of your total\n",
       "mistake cost: the difference between\n",
       "your (expected) rewards, including\n",
       "youthful suboptimality, and optimal\n",
       "(expected) rewards\n",
       "  - Minimizing regret goes beyond\n",
       "learning to be optimal – it requires\n",
       "optimally learning to be optimal  - random exploration and\n",
       "exploration functions both end up\n",
       "optimal, but random exploration has\n",
       "higher regret\n",
       "- function approximation for the process of constructing a compact approximation of the true utility function or Q-function. \n",
       "- Basic Q-Learning keeps a table of all q-values\n",
       "In realistic situations, we cannot possibly learn\n",
       "about every single state!\n",
       "  - Learn about some small number of training states from\n",
       "experience\n",
       "  - Generalize that experience to new, similar situations\n",
       "- approximate Q-learning in features\n",
       "  - Q-learning with linear Q-functions:\n",
       "    - Adjust weights of active features\n",
       "    - E.g., if something unexpectedly bad happens, blame the features that were on:\n",
       "disprefer all states with that state’s features\n",
       "\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.5 Policy Search\n",
       "\n",
       "- policy search: the  idea is to keep twiddling the policy as long as its performance improves, then stop.\n",
       "- π is a function that  maps states to actions. We are interested primarily in parameterized representations of π that  have far fewer parameters than there are states in the state space\n",
       "- Policy search will then adjust the parameters θ to  improve the policy.\n",
       "- come back if have time\n",
       "\n",
       "\n",
       "## Module 9 Lecture Notes\n",
       "\n",
       "- The overall agent design dictates the kind of information that must be learned:  \n",
       "  - A model-based reinforcement learning agent acquires (or is equipped with) a  transition model P(s ′ |s,a) for the environment and learns a utility function U(s).  \n",
       "  - A model-free reinforcement learning agent may learn an action-utility function  Q(s,a) or a policy π(s). \n",
       "- Utilities can be learned using several different approaches:\n",
       "  - Direct utility estimation uses the total observed reward-to-go for a given state as  direct evidence for learning its utility.\n",
       "  - Adaptive dynamic programming (ADP) learns a model and a reward function  from observations and then uses value or policy iteration to obtain the utilities or  an optimal policy. ADP makes optimal use of the local constraints on utilities of  states imposed through the neighborhood structure of the environment. \n",
       "  - Temporal-difference (TD) methods adjust utility estimates to be more consistent  with those of successor states. They can be viewed as simple approximations of the  ADP approach that can learn without requiring a transition model. Using a learned  model to generate pseudoexperiences can, however, result in faster learning.\n",
       "- Action-utility functions, or Q-functions, can be learned by an ADP approach or a TD  approach. With TD, Q-learning requires no model in either the learning or actionselection phase. This simplifies the learning problem but potentially restricts the ability  to learn in complex environments, because the agent cannot simulate the results of  possible courses of action.\n",
       "-  When the learning agent is responsible for selecting actions while it learns, it must  trade off the estimated value of those actions against the potential for learning useful  new information. An exact solution for the exploration problem is infeasible, but some  simple heuristics do a reasonable job. An exploring agent must also take care to avoid  premature death.\n",
       "- In large state spaces, reinforcement learning algorithms must use an approximate functional representation of U(s) or Q(s,a) in order to generalize over states. Deep reinforcement learning—using deep neural networks as function approximators—has  achieved considerable success on hard problems. \n",
       "-  Reward shaping and hierarchical reinforcement learning are helpful for learning  complex behaviors, particularly when rewards are sparse and long action sequences are  required to obtain them. \n",
       "-  Policy-search methods operate directly on a representation of the policy, attempting  to improve it based on observed performance. The variation in the performance in a  stochastic domain is a serious problem; for simulated domains this can be overcome by  fixing the randomness in advance. \n",
       "\n",
       "\n",
       "## Module 9 Definitions\n",
       "* **Markov reward process or mrp** -  that is, an MDP with only one Markov reward  process  possible action $a_i$. \n",
       "* **Exploration** - you have to try unknown actions to get information\n",
       "* **Exploitation** - eventually, you have to use what you know\n",
       "* **Regret** - even if you learn intelligently, you make mistakes\n",
       "* **Sampling** - because of chance, you have to try things repeatedly\n",
       "* **Difficulty** - learning can be much harder than solving a known MDP\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "\n",
    "module9_sections = [\n",
    "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
    "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
    "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
    "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
    "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
    "  (\"AIMA Chapter 22.5 Policy Search\", chapter22_5),\n",
    "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
    "]\n",
    "\n",
    "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
    "display(Markdown(module9_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "46MQLR-Gx8dZ"
   },
   "source": [
    "#Module 10 - Probabilities and Language Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "id": "BN0b1eVYx-nd"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
    "chapter12_1 = \"\"\" \n",
    "- probabilities are about possible worlds. Instead of strictly\n",
    "ruling out possibilities (where a logical assertion is false), probabilities quantify how\n",
    "likely a particular possible world is.\n",
    "- Uncertainty:  observability, nondeterminism, or adversaries\n",
    "- belief state agent:\n",
    "  - must consider every possible, leads to a large belief-state\n",
    "  - A correct contingent plan that handles every eventuality can grow arbitrarily large and  must consider arbitrarily unlikely contingencies. \n",
    "  - Sometimes there is no plan that is guaranteed to achieve the goal—yet it must act\n",
    "- rational decision: the right thing to do, therefore depends on both the relative importance of various goals and the likelihood that, and degree to which, they will be achieved.\n",
    "- logic to cope with a domain has problems:\n",
    "  - Laziness: It is too much work to list the complete set of antecedents or consequents\n",
    "  - Theoretical ignorance: Medical science has no complete theory for the domain. \n",
    "  - Practical ignorance: Even if we know all the rules, we might be uncertain about a  particular patient because not all the necessary tests have been or can be run. \n",
    "- degree of belief - The agent’s knowledge can at best provide only a degree of belief in the relevant sentences, with degrees of belief is probability theory.\n",
    "- probability provides a way of summarizing the uncertainty that comes from  our laziness and ignorance\n",
    "- agent must first have preferences among the different possible outcomes\n",
    "- decision theory = probability theory + utility theory, odd is not irrational\n",
    "- Maximum expected utility (MEU): an agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all the possible outcomes of the action.\n",
    "- decision-theoretic agent’s belief state represents not just the possibilities for world  states but also their probabilities. belief state and some knowledge of the effects of  actions\"\"\"\n",
    "chapter12_2 = \"\"\"\n",
    "- sample space - the set of all possible world, mutually exclusive and exhaustive\n",
    "- A fully specified probability model associates a numerical probability P(ω) with each Probability model  possible world, add up to 1\n",
    "- event: the unique result of states (dice add up to 11), associate with proposition in this context\n",
    "- unconditional probability = prior probability = prior, degrees of belief in propositions in the absence of any other information.\n",
    "- conditional = posterior, given that the sth is\n",
    "- random variable, variable in probability theory\n",
    "- range, range of possible outcomes\n",
    "- A possible world is defined to be an assignment of values to all of the random variables under consideration. \n",
    "- full joint probability distribution - follows that a probability model is completely determined by the joint distribution for all of the random variables\n",
    "- logically related propositions: If  Agent 1 expresses a set of degrees of belief that violate the axioms of probability theory then  there is a combination of bets by Agent 2 that guarantees that Agent 1 will lose money every  time.<br />\n",
    "\"\"\" + r\"\"\" $P(a|b) = \\frac{P(a,b)}{P(b)}$ <br />\"\"\" + r\"\"\" $P(a,b) = P(a|b)P(b)$ <br />\"\"\"+ r\"\"\" $P(a \\lor b) = P(a)+P(b)-P(a,b)$ <br />\"\"\" + r\"\"\"$P(x_1,x_2,x_3)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2) \\rightarrow P(x_1,x_2,...,x_n)=\\prod_i P(x_i|x_1,...,x_{i-1})$\"\"\"\n",
    "\n",
    "chapter12_3 = \"\"\"\n",
    "- for probabilistic inference—that is, the computation of posterior probabilities for query propositions given observed evidence.\n",
    "- marginal probability: sum of joint probability, summing out \"\"\" + r\"\"\" $P(Y) = \\sum_{z} P(Y,Z=z)$ <br />\"\"\" + \"\"\"\n",
    "- conditioning, add all the conditional probabilities\"\"\"+ r\"\"\" $P(Y) = \\sum_{z} P(Y,Z=z)$ <br />\"\"\" + \"\"\"\n",
    "- for a domain described by n  Boolean variables, it requires an input table of size $O(2^n)$ and takes $O(2^n)$ time to process the table.\n",
    "- The problem is not just memory and computation: the number of examples required will be astronomical. \n",
    "\"\"\"\n",
    "chapter12_4 = \"\"\" \n",
    "- independence :\"\"\" +  r\"\"\" $P(X|Y) = P(X) \\text{ or } P(Y|X) = P(Y) \\text{ or } P(X,Y) = P(X)P(Y)$<br />\"\"\"\n",
    "chapter12_5 = \"\"\"\n",
    "Bayes rule\n",
    "\"\"\" + r\"\"\" $P(Y|X) = \\frac{P(X|Y)P(Y)}{P(X)} \\text{ or } P(Y|X,e) = \\frac{P(X|Y,e)P(Y|e)}{P(X|e)}$<br />\"\"\" + \"\"\"\n",
    "- The conditional probability P(effect| cause) quantifies the relationship in the causal direc- Causal  tion, whereas P(cause | effect) describes the diagnostic direction.\n",
    "- diagnostic knowledge is often more fragile than causal knowledge.\n",
    "- conditional probability:  X is conditionally independent of Y given Z\n",
    "- \n",
    "\"\"\" + r\"\"\"$P(X,Y|Z) = P(X|Z)P(Y|Z)$  or \"\"\"+ r\"\"\"$P(X|Y,Z) = P(X|Z) \\text{ and } P(Y|X,Z) = P(Y|Z)$<br />\"\"\" + \"\"\"\n",
    "- for n symptoms that are all conditionally independent given Cavity, the size  of the representation grows as $O(n)$ instead of $O(2^n)$.\n",
    "- That means that conditional independence assertions can allow probabilistic systems to scale up; moreover, they are much more commonly available than absolute independence assertions. Conceptually, Cavity separates  Toothache and Catch because it is a direct cause of both of them.\n",
    "\"\"\"\n",
    "chapter12_6 = \"\"\"\n",
    "- Naive bayes because it is often  used (as a simplifying assumption) in cases where the “effect” variables are not strictly independent given the cause variable.\n",
    " \"\"\" + r\"\"\"$P(Cause,Effect_1,\\dots,Effect_n) = P(Cause)\\prod_i P(Effect_i|Cause)$ <br />\"\"\" + r\"\"\" $$P(Cause|e) = \\alpha \\sum_{y}P(cause)P(y|Cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$\"\"\"+\\\n",
    "r\"\"\"$$=\\alpha P(cause) \\sum_{y}P(y|Cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$\"\"\"+\\\n",
    "r\"\"\"$$= \\alpha P(cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$ \"\"\" + \"\"\"\n",
    " - text classification, not independent, but works fine, $P(HasWord_i|Category)$\n",
    " \"\"\"\n",
    "chapter12_7 = \"\"\"-\n",
    "- only partial information about  the world, probabilistic agent can do much better than the  logical agent. \n",
    "- seemingly complicated problems can be formulated precisely in probability theory and solved with simple algorithms. To get efficient  solutions, independence and conditional independence relationships can be used to simplify  the summations required.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
    "SLP_chapter3_1 = \"\"\"\n",
    "- $P(w|h) = P(w,h)/P(h)$\n",
    "- but not always work\n",
    "- bigram model, for example, approximates the probability of a word given all the previous words $P(w_n|w_{1:n-1})$ by using only the conditional probability of the preceding word $P(w_n|w_{n-1})$\n",
    "- approximate $P(w_n|w_{1:n-1}) = P(w_n|w_{n-1})$\n",
    "- assumption that the probability of a word depends only on the previous word is Markov assumption. Markov models are the class of probabilistic models that assume we can predict the probability of some future unit without looking too far into the past.\n",
    "- generalize n-gram that $P(w_{1:n}) = \\prod_{k=1}^n P(w_k|w_{k-1})$\n",
    "- $P(w_n|w_{n-1}) = C(w_n-1 w_n)/C(w_{n-1})$\n",
    "- estimates the n-gram probability by dividing the observed frequency of a particular sequence by the observed frequency of a prefix. This ratio is called a relative relative frequency.\n",
    "- Some of the bigram probabilities above encode some facts that we think of as strictly syntactic in nature, like the fact that what comes after eat is usually a noun or an adjective, or that what comes after to is usually a verb\n",
    "- p1* p2* p3* p4 = exp(log p1+log p2+log p3+log p4)\n",
    "- We do everything in log space\n",
    "  - Avoid underflow\n",
    "  - (also adding is faster than multiplying)\n",
    "\"\"\"\n",
    "SLP_chapter3_2 = \"\"\"\n",
    "- Extrinsic evaluation is the only way to know if a particular improvement in a component is really going to help the task at hand. we can compare the performance of two language models by running the speech recognizer twice, once with each language model, and seeing which gives the more accurate transcription\n",
    "- An intrinsic evaluation metric is one that measures the quality of a model independent of any application\n",
    "- whichever model assigns a higher probability to the test set—meaning it more accurately predicts the test set—is a better model\n",
    "- Training on the test set introduces a bias that makes the probabilities all look too high, and causes huge inaccuracies in perplexity\n",
    "- Perplexity:\n",
    "\"\"\" + r\"\"\"$PP(W) = \\sqrt[n]{\\frac{1}{P(w_1,w_2,...,w_N)}} = \\sqrt[N]{\\prod_{i=1}^N\\frac{1}{P(w_i|w_1,...,w_{i-1})}}$\"\"\" + \"\"\"\n",
    "- the higher the conditional probability of the word sequence, the lower the perplexity. Thus, minimizing perplexity is equivalent to maximizing the test set probability according to the language model.\n",
    "- perplexity as the weighted average branching factor of a language. The branching factor of a language is the number of possible next words that can follow any word.\n",
    "- the more information the n-gram gives us about the word sequence, the lower the perplexity\n",
    "- Any kind of knowledge of the test set can cause the perplexity to be artificially low.\n",
    "- An (intrinsic) improvement in perplexity does not guarantee an (extrinsic) improvement in the performance of a language processing task like speech recognition or machine translation. Nonetheless, because perplexity often correlates with such improvements, it is commonly used as a quick check on an algorithm. But a model’s improvement in perplexity should always be confirmed by an end-to-end evaluation of a real task before concluding the evaluation of the model\n",
    "\"\"\"\n",
    "SLP_chapter3_3 = \"\"\" \n",
    "- Sampling from a distribution means to choose random points according to their likelihood. Thus sampling from a language model—which represents a distribution over sentences—means to generate some sentences, choosing each sentence according to its likelihood as defined by the model\n",
    "- list the likelihood from 0 to 1, randomly choose a number and select the word, so avoid sample frequent words\n",
    "\"\"\"\n",
    "SLP_chapter3_4 = \"\"\"\n",
    "- One implication of this is that the probabilities often encode specific facts about a given training corpus. Another implication is that n-grams do a better and better job of modeling the training corpus as we increase the value of N.\n",
    "- The longer the context on which we train the model, the more coherent the sentences. In the unigram sentences, there is no coherent relation between words or any sentence-final punctuation. The bigram sentences have some local word-to-word coherence\n",
    "- One step is to be sure to use a training corpus that has a similar genre to whatever task we are trying to accomplish\n",
    "- get training data in the appropriate dialect or variety, especially when processing social media posts or spoken transcripts\n",
    "- Unknown words\n",
    "- Sometimes we have a language task in which this can’t happen because we know all the words that can occur. In such a closed vocabulary system the test set can only contain words from this lexicon, and there will be no unknown words.\n",
    "- In other cases we have to deal with words we haven’t seen before, which we’ll call unknown OOV words, or out of vocabulary (OOV) words. The percentage of OOV\n",
    "open words that appear in the test set is called the OOV rate. An open vocabulary system  is one in which we model these potential unknown words in the test set by adding a pseudo-word called <UNK>.\n",
    "- First way\n",
    "  -  Choose a vocabulary (word list) that is fixed in advance.\n",
    "  - Convert in the training set any word that is not in this set (any OOV word) to\n",
    "the unknown word token <UNK> in a text normalization step.\n",
    "  - Estimate the probabilities for <UNK> from its counts just like any other regular\n",
    "word in the training set.\n",
    "- second way \n",
    "  - we don’t have a prior vocabulary in advance,\n",
    "is to create such a vocabulary implicitly, replacing words in the training data\n",
    "by <UNK> based on their frequency. For example we can replace by <UNK> all words\n",
    "that occur fewer than n times in the training set, where n is some small number, or\n",
    "equivalently select a vocabulary size V in advance (say 50,000) and choose the top\n",
    "V words by frequency and replace the rest by UNK. In either case we then proceed\n",
    "to train the language model as before, treating <UNK> like a regular word\n",
    "- The exact choice of <UNK> model does have an effect on metrics like perplexity.\n",
    "A language model can achieve low perplexity by choosing a small vocabulary and\n",
    "assigning the unknown word a high probability. For this reason, perplexities should\n",
    "only be compared across language models with the same vocabularies\n",
    "\"\"\"\n",
    "SLP_chapter3_5 = \"\"\"\n",
    "- Laplace\n",
    "  - All the counts that used to be zero will now\n",
    "have a count of 1, the counts of 1 will be 2, and so on.\n",
    "  - Laplace smoothing does not perform well enough to be used in modern n-gram models, but it usefully introduces many of the concepts that we\n",
    "see in other smoothing algorithms, gives a useful baseline, and is also a practical\n",
    "smoothing algorithm for other tasks like text classification\n",
    "  - for unigram: $P(w_i) = c_i/N$\n",
    "  - Laplace: $P_{Laplace} = (c_i+1)/(N+V)$, which V is the number of words in the vocabulaty\n",
    "  - it is convenient to\n",
    "describe how a smoothing algorithm affects the numerator, by defining an adjusted\n",
    "count c*. This adjusted count is easier to compare directly with the MLE counts and\n",
    "can be turned into a probability like an MLE count by normalizing by N. We can now turn c*\n",
    "i into a probability P*\n",
    "i by normalizing by N.\n",
    "  - A related way to view smoothing is as discounting (lowering) some non-zero\n",
    "counts in order to get the probability mass that will be assigned to the zero counts.\n",
    "  - for bigram\n",
    "  - $P^*_{Laplace}(w_n|w_{n-1})=[C(w_{n-1}w_n)+1]/[\\sum_w(C(w_{n-1}w)+1)]$\n",
    "    - $=[C(w_{n-1}w_n)+1]/[C(w_{n-1}+V)]$\n",
    "  - the reconstructed counts.\n",
    "  - $c^*(w_{n-1}w_n) = [C(w_{n-1}w_n)+1]C(w_{n-1})/[C(w_{n-1})+V]$\n",
    "  - The sharp change in counts and probabilities occurs because too much probability\n",
    "mass is moved to all the zeros.\n",
    "- Add-k Smoothing\n",
    "  - One alternative to add-one smoothing is to move a bit less of the probability mass\n",
    "from the seen to the unseen events. Instead of adding 1 to each count, we add a fractional count k (.5? .05? .01?).\n",
    "  - $P^*_{Add-K}(w_n|w_{n-1})==[C(w_{n-1}w_n)+k]/[C(w_{n-1}+kV)]$\n",
    "  - utilize validation set tune hyperparameter\n",
    "- Backoff and Interpolation\n",
    "  - if there is no example of 4-gram, then move to tri gram, and so on, until unigram\n",
    "  - sometimes using less context is a good thing, helping to generalize\n",
    "more for contexts that the model hasn’t learned much about.\n",
    "  - In backoff, we use the trigram if the evidence is\n",
    "sufficient, otherwise we use the bigram, otherwise the unigram. Or we\n",
    "only “back off” to a lower-order n-gram if we have zero evidence for a higher-order n-gram.\n",
    "  - In a backoff n-gram model, if the n-gram we need has zero counts, we approximate\n",
    "it by backing off to the (N-1)-gram. We continue backing off until we reach a\n",
    "history that has some counts.\n",
    "  - In order for a backoff model to give a correct probability distribution, we have\n",
    "discount to discount the higher-order n-grams to save some probability mass for the lower\n",
    "order n-grams. \n",
    "  - in interpolation, we always mix the probability estimates from\n",
    "all the n-gram estimators, weighing and combining the trigram, bigram, and unigram\n",
    "counts.\n",
    "  - interpolation: $\\hat{P}(w_n|w_{n-2}w_{n-1}) = \\lambda_1 P(w_n)+ \\lambda_2 P(w_n|w_{n-1}) + \\lambda_3 P(w_n|w_{n-2}w_{n-1})$ s.t. $\\sum \\lambda =1$\n",
    "  - That is, we fix\n",
    "the n-gram probabilities and then search for the $\\lambda$ values that give us the highest probability of the held-out set, \n",
    "\n",
    " \"\"\"\n",
    "SLP_chapter3_7 = \"\"\"\n",
    "- Efficiency considerations are important when building language models that use\n",
    "such large sets of n-grams. Rather than store each word as a string, it is generally\n",
    "represented in memory as a 64-bit hash number, with the words themselves stored\n",
    "on disk. Probabilities are generally quantized using only 4-8 bits (instead of 8-byte\n",
    "floats), and n-grams are stored in reverse tries.\n",
    "- An n-gram language model can also be shrunk by pruning, for example only\n",
    "storing n-grams with counts greater than some threshold\n",
    "- The algorithm is called\n",
    "- stupid backoff. Stupid backoff gives up the idea of trying to make the language\n",
    "model a true probability distribution. There is no discounting of the higher-order\n",
    "probabilities. If a higher-order n-gram has a zero count, we simply backoff to a\n",
    "lower order n-gram, weighed by a fixed (context-independent) weight\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module10_lecture_notes = \"\"\"\n",
    "- Uncertainty arises because of both laziness and ignorance. It is inescapable in complex,  nondeterministic, or partially observable environments. \n",
    "- Probabilities express the agent’s inability to reach a definite decision regarding the  truth of a sentence. Probabilities summarize the agent’s beliefs relative to the evidence.\n",
    "- Decision theory combines the agent’s beliefs and desires, defining the best action as  the one that maximizes expected utility.  \n",
    "- Basic probability statements include prior or unconditional probabilities and posterior or conditional probabilities over simple and complex propositions. \n",
    "- The axioms of probability constrain the probabilities of logically related propositions.  An agent that violates the axioms must behave irrationally in some cases.  \n",
    "- The full joint probability distribution specifies the probability of each complete assignment of values to random variables. It is usually too large to create or use in its  explicit form, but when it is available it can be used to answer queries simply by adding  up entries for the possible worlds corresponding to the query propositions.  \n",
    "- Absolute independence between subsets of random variables allows the full joint distribution to be factored into smaller joint distributions, greatly reducing its complexity. \n",
    "- Bayes’ rule allows unknown probabilities to be computed from known conditional  probabilities, usually in the causal direction. Applying Bayes’ rule with many pieces of  evidence runs into the same scaling problems as does the full joint distribution.  \n",
    "- Conditional independence brought about by direct causal relationships in the domain  allows the full joint distribution to be factored into smaller, conditional distributions.  The naive Bayes model assumes the conditional independence of all effect variables,  given a single cause variable; its size grows linearly with the number of effects.\n",
    "- A wumpus-world agent can calculate probabilities for unobserved aspects of the world,  thereby improving on the decisions of a purely logical agent. Conditional independence  makes these calculations tractable. \n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Each definition should be at most 25 words long.\n",
    "module10_definitions = {\n",
    "\"belief state\" : \"a representation of the set of all possible world states that it might be in\",\n",
    "\"qualification  problem\":\"conditions can be  deduced for sure, so we can’t infer that the plan succeeds.\",\n",
    "\"ontological commitments\":\"the world is  composed of facts that do or do not hold in any particular case, degree of belief between 0  and 1.\",\n",
    "\"epistemological  commitments\":\"a logical agent believes each sentence to be true or false or has  no opinion\",\n",
    "\"utility theory\":\"represent preference and reason quantitatively with them, utility: usefulness\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "lvtN3p7ET3Tx",
    "outputId": "164100e0-d127-4ffe-a643-7f45bb7e661a"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "## WARNING: Your summary of 'SLP Chapter 3.5 Smoothing' is too long!\n",
       "# Module 10: Probabilies and Language Models\n",
       "## AIMA Chapter 12.1 Acting under Uncertainty\n",
       " \n",
       "- probabilities are about possible worlds. Instead of strictly\n",
       "ruling out possibilities (where a logical assertion is false), probabilities quantify how\n",
       "likely a particular possible world is.\n",
       "- Uncertainty:  observability, nondeterminism, or adversaries\n",
       "- belief state agent:\n",
       "  - must consider every possible, leads to a large belief-state\n",
       "  - A correct contingent plan that handles every eventuality can grow arbitrarily large and  must consider arbitrarily unlikely contingencies. \n",
       "  - Sometimes there is no plan that is guaranteed to achieve the goal—yet it must act\n",
       "- rational decision: the right thing to do, therefore depends on both the relative importance of various goals and the likelihood that, and degree to which, they will be achieved.\n",
       "- logic to cope with a domain has problems:\n",
       "  - Laziness: It is too much work to list the complete set of antecedents or consequents\n",
       "  - Theoretical ignorance: Medical science has no complete theory for the domain. \n",
       "  - Practical ignorance: Even if we know all the rules, we might be uncertain about a  particular patient because not all the necessary tests have been or can be run. \n",
       "- degree of belief - The agent’s knowledge can at best provide only a degree of belief in the relevant sentences, with degrees of belief is probability theory.\n",
       "- probability provides a way of summarizing the uncertainty that comes from  our laziness and ignorance\n",
       "- agent must first have preferences among the different possible outcomes\n",
       "- decision theory = probability theory + utility theory, odd is not irrational\n",
       "- Maximum expected utility (MEU): an agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all the possible outcomes of the action.\n",
       "- decision-theoretic agent’s belief state represents not just the possibilities for world  states but also their probabilities. belief state and some knowledge of the effects of  actions\n",
       "\n",
       "## AIMA Chapter 12.2 Basic Probability Notation\n",
       "\n",
       "- sample space - the set of all possible world, mutually exclusive and exhaustive\n",
       "- A fully specified probability model associates a numerical probability P(ω) with each Probability model  possible world, add up to 1\n",
       "- event: the unique result of states (dice add up to 11), associate with proposition in this context\n",
       "- unconditional probability = prior probability = prior, degrees of belief in propositions in the absence of any other information.\n",
       "- conditional = posterior, given that the sth is\n",
       "- random variable, variable in probability theory\n",
       "- range, range of possible outcomes\n",
       "- A possible world is defined to be an assignment of values to all of the random variables under consideration. \n",
       "- full joint probability distribution - follows that a probability model is completely determined by the joint distribution for all of the random variables\n",
       "- logically related propositions: If  Agent 1 expresses a set of degrees of belief that violate the axioms of probability theory then  there is a combination of bets by Agent 2 that guarantees that Agent 1 will lose money every  time.<br />\n",
       " $P(a|b) = \\frac{P(a,b)}{P(b)}$ <br /> $P(a,b) = P(a|b)P(b)$ <br /> $P(a \\lor b) = P(a)+P(b)-P(a,b)$ <br />$P(x_1,x_2,x_3)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2) \\rightarrow P(x_1,x_2,...,x_n)=\\prod_i P(x_i|x_1,...,x_{i-1})$\n",
       "\n",
       "## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n",
       "\n",
       "- for probabilistic inference—that is, the computation of posterior probabilities for query propositions given observed evidence.\n",
       "- marginal probability: sum of joint probability, summing out  $P(Y) = \\sum_{z} P(Y,Z=z)$ <br />\n",
       "- conditioning, add all the conditional probabilities $P(Y) = \\sum_{z} P(Y,Z=z)$ <br />\n",
       "- for a domain described by n  Boolean variables, it requires an input table of size $O(2^n)$ and takes $O(2^n)$ time to process the table.\n",
       "- The problem is not just memory and computation: the number of examples required will be astronomical. \n",
       "\n",
       "\n",
       "## AIMA Chapter 12.4 Independence\n",
       " \n",
       "- independence : $P(X|Y) = P(X) \\text{ or } P(Y|X) = P(Y) \\text{ or } P(X,Y) = P(X)P(Y)$<br />\n",
       "\n",
       "## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n",
       "\n",
       "Bayes rule\n",
       " $P(Y|X) = \\frac{P(X|Y)P(Y)}{P(X)} \\text{ or } P(Y|X,e) = \\frac{P(X|Y,e)P(Y|e)}{P(X|e)}$<br />\n",
       "- The conditional probability P(effect| cause) quantifies the relationship in the causal direc- Causal  tion, whereas P(cause | effect) describes the diagnostic direction.\n",
       "- diagnostic knowledge is often more fragile than causal knowledge.\n",
       "- conditional probability:  X is conditionally independent of Y given Z\n",
       "- \n",
       "$P(X,Y|Z) = P(X|Z)P(Y|Z)$  or $P(X|Y,Z) = P(X|Z) \\text{ and } P(Y|X,Z) = P(Y|Z)$<br />\n",
       "- for n symptoms that are all conditionally independent given Cavity, the size  of the representation grows as $O(n)$ instead of $O(2^n)$.\n",
       "- That means that conditional independence assertions can allow probabilistic systems to scale up; moreover, they are much more commonly available than absolute independence assertions. Conceptually, Cavity separates  Toothache and Catch because it is a direct cause of both of them.\n",
       "\n",
       "\n",
       "## AIMA Chapter 12.6 Naive Bayes Models\n",
       "\n",
       "- Naive bayes because it is often  used (as a simplifying assumption) in cases where the “effect” variables are not strictly independent given the cause variable.\n",
       " $P(Cause,Effect_1,\\dots,Effect_n) = P(Cause)\\prod_i P(Effect_i|Cause)$ <br /> $$P(Cause|e) = \\alpha \\sum_{y}P(cause)P(y|Cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$$$=\\alpha P(cause) \\sum_{y}P(y|Cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$$$= \\alpha P(cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$ \n",
       " - text classification, not independent, but works fine, $P(HasWord_i|Category)$\n",
       " \n",
       "\n",
       "## AIMA Chapter 12.7 The Wumpus World Revisited\n",
       "-\n",
       "- only partial information about  the world, probabilistic agent can do much better than the  logical agent. \n",
       "- seemingly complicated problems can be formulated precisely in probability theory and solved with simple algorithms. To get efficient  solutions, independence and conditional independence relationships can be used to simplify  the summations required.\n",
       "\n",
       "\n",
       "## SLP Chapter 3.1 N-Grams\n",
       "\n",
       "- $P(w|h) = P(w,h)/P(h)$\n",
       "- but not always work\n",
       "- bigram model, for example, approximates the probability of a word given all the previous words $P(w_n|w_{1:n-1})$ by using only the conditional probability of the preceding word $P(w_n|w_{n-1})$\n",
       "- approximate $P(w_n|w_{1:n-1}) = P(w_n|w_{n-1})$\n",
       "- assumption that the probability of a word depends only on the previous word is Markov assumption. Markov models are the class of probabilistic models that assume we can predict the probability of some future unit without looking too far into the past.\n",
       "- generalize n-gram that $P(w_{1:n}) = \\prod_{k=1}^n P(w_k|w_{k-1})$\n",
       "- $P(w_n|w_{n-1}) = C(w_n-1 w_n)/C(w_{n-1})$\n",
       "- estimates the n-gram probability by dividing the observed frequency of a particular sequence by the observed frequency of a prefix. This ratio is called a relative relative frequency.\n",
       "- Some of the bigram probabilities above encode some facts that we think of as strictly syntactic in nature, like the fact that what comes after eat is usually a noun or an adjective, or that what comes after to is usually a verb\n",
       "- p1* p2* p3* p4 = exp(log p1+log p2+log p3+log p4)\n",
       "- We do everything in log space\n",
       "  - Avoid underflow\n",
       "  - (also adding is faster than multiplying)\n",
       "\n",
       "\n",
       "## SLP Chapter 3.2 Evaluating Language Models\n",
       "\n",
       "- Extrinsic evaluation is the only way to know if a particular improvement in a component is really going to help the task at hand. we can compare the performance of two language models by running the speech recognizer twice, once with each language model, and seeing which gives the more accurate transcription\n",
       "- An intrinsic evaluation metric is one that measures the quality of a model independent of any application\n",
       "- whichever model assigns a higher probability to the test set—meaning it more accurately predicts the test set—is a better model\n",
       "- Training on the test set introduces a bias that makes the probabilities all look too high, and causes huge inaccuracies in perplexity\n",
       "- Perplexity:\n",
       "$PP(W) = \\sqrt[n]{\\frac{1}{P(w_1,w_2,...,w_N)}} = \\sqrt[N]{\\prod_{i=1}^N\\frac{1}{P(w_i|w_1,...,w_{i-1})}}$\n",
       "- the higher the conditional probability of the word sequence, the lower the perplexity. Thus, minimizing perplexity is equivalent to maximizing the test set probability according to the language model.\n",
       "- perplexity as the weighted average branching factor of a language. The branching factor of a language is the number of possible next words that can follow any word.\n",
       "- the more information the n-gram gives us about the word sequence, the lower the perplexity\n",
       "- Any kind of knowledge of the test set can cause the perplexity to be artificially low.\n",
       "- An (intrinsic) improvement in perplexity does not guarantee an (extrinsic) improvement in the performance of a language processing task like speech recognition or machine translation. Nonetheless, because perplexity often correlates with such improvements, it is commonly used as a quick check on an algorithm. But a model’s improvement in perplexity should always be confirmed by an end-to-end evaluation of a real task before concluding the evaluation of the model\n",
       "\n",
       "\n",
       "## SLP Chapter 3.3 Sampling sentences from a language model\n",
       " \n",
       "- Sampling from a distribution means to choose random points according to their likelihood. Thus sampling from a language model—which represents a distribution over sentences—means to generate some sentences, choosing each sentence according to its likelihood as defined by the model\n",
       "- list the likelihood from 0 to 1, randomly choose a number and select the word, so avoid sample frequent words\n",
       "\n",
       "\n",
       "## SLP Chapter 3.4 Generalization and Zeros\n",
       "\n",
       "- One implication of this is that the probabilities often encode specific facts about a given training corpus. Another implication is that n-grams do a better and better job of modeling the training corpus as we increase the value of N.\n",
       "- The longer the context on which we train the model, the more coherent the sentences. In the unigram sentences, there is no coherent relation between words or any sentence-final punctuation. The bigram sentences have some local word-to-word coherence\n",
       "- One step is to be sure to use a training corpus that has a similar genre to whatever task we are trying to accomplish\n",
       "- get training data in the appropriate dialect or variety, especially when processing social media posts or spoken transcripts\n",
       "- Unknown words\n",
       "- Sometimes we have a language task in which this can’t happen because we know all the words that can occur. In such a closed vocabulary system the test set can only contain words from this lexicon, and there will be no unknown words.\n",
       "- In other cases we have to deal with words we haven’t seen before, which we’ll call unknown OOV words, or out of vocabulary (OOV) words. The percentage of OOV\n",
       "open words that appear in the test set is called the OOV rate. An open vocabulary system  is one in which we model these potential unknown words in the test set by adding a pseudo-word called <UNK>.\n",
       "- First way\n",
       "  -  Choose a vocabulary (word list) that is fixed in advance.\n",
       "  - Convert in the training set any word that is not in this set (any OOV word) to\n",
       "the unknown word token <UNK> in a text normalization step.\n",
       "  - Estimate the probabilities for <UNK> from its counts just like any other regular\n",
       "word in the training set.\n",
       "- second way \n",
       "  - we don’t have a prior vocabulary in advance,\n",
       "is to create such a vocabulary implicitly, replacing words in the training data\n",
       "by <UNK> based on their frequency. For example we can replace by <UNK> all words\n",
       "that occur fewer than n times in the training set, where n is some small number, or\n",
       "equivalently select a vocabulary size V in advance (say 50,000) and choose the top\n",
       "V words by frequency and replace the rest by UNK. In either case we then proceed\n",
       "to train the language model as before, treating <UNK> like a regular word\n",
       "- The exact choice of <UNK> model does have an effect on metrics like perplexity.\n",
       "A language model can achieve low perplexity by choosing a small vocabulary and\n",
       "assigning the unknown word a high probability. For this reason, perplexities should\n",
       "only be compared across language models with the same vocabularies\n",
       "\n",
       "\n",
       "## SLP Chapter 3.5 Smoothing\n",
       "\n",
       "- Laplace\n",
       "  - All the counts that used to be zero will now\n",
       "have a count of 1, the counts of 1 will be 2, and so on.\n",
       "  - Laplace smoothing does not perform well enough to be used in modern n-gram models, but it usefully introduces many of the concepts that we\n",
       "see in other smoothing algorithms, gives a useful baseline, and is also a practical\n",
       "smoothing algorithm for other tasks like text classification\n",
       "  - for unigram: $P(w_i) = c_i/N$\n",
       "  - Laplace: $P_{Laplace} = (c_i+1)/(N+V)$, which V is the number of words in the vocabulaty\n",
       "  - it is convenient to\n",
       "describe how a smoothing algorithm affects the numerator, by defining an adjusted\n",
       "count c*. This adjusted count is easier to compare directly with the MLE counts and\n",
       "can be turned into a probability like an MLE count by normalizing by N. We can now turn c*\n",
       "i into a probability P*\n",
       "i by normalizing by N.\n",
       "  - A related way to view smoothing is as discounting (lowering) some non-zero\n",
       "counts in order to get the probability mass that will be assigned to the zero counts.\n",
       "  - for bigram\n",
       "  - $P^*_{Laplace}(w_n|w_{n-1})=[C(w_{n-1}w_n)+1]/[\\sum_w(C(w_{n-1}w)+1)]$\n",
       "    - $=[C(w_{n-1}w_n)+1]/[C(w_{n-1}+V)]$\n",
       "  - the reconstructed counts.\n",
       "  - $c^*(w_{n-1}w_n) = [C(w_{n-1}w_n)+1]C(w_{n-1})/[C(w_{n-1})+V]$\n",
       "  - The sharp change in counts and probabilities occurs because too much probability\n",
       "mass is moved to all the zeros.\n",
       "- Add-k Smoothing\n",
       "  - One alternative to add-one smoothing is to move a bit less of the probability mass\n",
       "from the seen to the unseen events. Instead of adding 1 to each count, we add a fractional count k (.5? .05? .01?).\n",
       "  - $P^*_{Add-K}(w_n|w_{n-1})==[C(w_{n-1}w_n)+k]/[C(w_{n-1}+kV)]$\n",
       "  - utilize validation set tune hyperparameter\n",
       "- Backoff and Interpolation\n",
       "  - if there is no example of 4-gram, then move to tri gram, and so on, until unigram\n",
       "  - sometimes using less context is a good thing, helping to generalize\n",
       "more for contexts that the model hasn’t learned much about.\n",
       "  - In backoff, we use the trigram if the evidence is\n",
       "sufficient, otherwise we use the bigram, otherwise the unigram. Or we\n",
       "only “back off” to a lower-order n-gram if we have zero evidence for a higher-order n-gram.\n",
       "  - In a backoff n-gram model, if the n-gram we need has zero counts, we approximate\n",
       "it by backing off to the (N-1)-gram. We continue backing off until we reach a\n",
       "history that has some counts.\n",
       "  - In order for a backoff model to give a correct probability distribution, we have\n",
       "discount to discount the higher-order n-grams to save some probability mass for the lower\n",
       "order n-grams. \n",
       "  - in interpolation, we always mix the probability estimates from\n",
       "all the n-gram estimators, weighing and combining the trigram, bigram, and unigram\n",
       "counts.\n",
       "  - interpolation: $\\hat{P}(w_n|w_{n-2}w_{n-1}) = \\lambda_1 P(w_n)+ \\lambda_2 P(w_n|w_{n-1}) + \\lambda_3 P(w_n|w_{n-2}w_{n-1})$ s.t. $\\sum \\lambda =1$\n",
       "  - That is, we fix\n",
       "the n-gram probabilities and then search for the $\\lambda$ values that give us the highest probability of the held-out set, \n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n",
       "\n",
       "- Efficiency considerations are important when building language models that use\n",
       "such large sets of n-grams. Rather than store each word as a string, it is generally\n",
       "represented in memory as a 64-bit hash number, with the words themselves stored\n",
       "on disk. Probabilities are generally quantized using only 4-8 bits (instead of 8-byte\n",
       "floats), and n-grams are stored in reverse tries.\n",
       "- An n-gram language model can also be shrunk by pruning, for example only\n",
       "storing n-grams with counts greater than some threshold\n",
       "- The algorithm is called\n",
       "- stupid backoff. Stupid backoff gives up the idea of trying to make the language\n",
       "model a true probability distribution. There is no discounting of the higher-order\n",
       "probabilities. If a higher-order n-gram has a zero count, we simply backoff to a\n",
       "lower order n-gram, weighed by a fixed (context-independent) weight\n",
       "\n",
       "\n",
       "## Module 10 Lecture Notes\n",
       "\n",
       "- Uncertainty arises because of both laziness and ignorance. It is inescapable in complex,  nondeterministic, or partially observable environments. \n",
       "- Probabilities express the agent’s inability to reach a definite decision regarding the  truth of a sentence. Probabilities summarize the agent’s beliefs relative to the evidence.\n",
       "- Decision theory combines the agent’s beliefs and desires, defining the best action as  the one that maximizes expected utility.  \n",
       "- Basic probability statements include prior or unconditional probabilities and posterior or conditional probabilities over simple and complex propositions. \n",
       "- The axioms of probability constrain the probabilities of logically related propositions.  An agent that violates the axioms must behave irrationally in some cases.  \n",
       "- The full joint probability distribution specifies the probability of each complete assignment of values to random variables. It is usually too large to create or use in its  explicit form, but when it is available it can be used to answer queries simply by adding  up entries for the possible worlds corresponding to the query propositions.  \n",
       "- Absolute independence between subsets of random variables allows the full joint distribution to be factored into smaller joint distributions, greatly reducing its complexity. \n",
       "- Bayes’ rule allows unknown probabilities to be computed from known conditional  probabilities, usually in the causal direction. Applying Bayes’ rule with many pieces of  evidence runs into the same scaling problems as does the full joint distribution.  \n",
       "- Conditional independence brought about by direct causal relationships in the domain  allows the full joint distribution to be factored into smaller, conditional distributions.  The naive Bayes model assumes the conditional independence of all effect variables,  given a single cause variable; its size grows linearly with the number of effects.\n",
       "- A wumpus-world agent can calculate probabilities for unobserved aspects of the world,  thereby improving on the decisions of a purely logical agent. Conditional independence  makes these calculations tractable. \n",
       "\n",
       "\n",
       "## Module 10 Definitions\n",
       "* **Belief state** - a representation of the set of all possible world states that it might be in\n",
       "* **Qualification  problem** - conditions can be  deduced for sure, so we can’t infer that the plan succeeds.\n",
       "* **Ontological commitments** - the world is  composed of facts that do or do not hold in any particular case, degree of belief between 0  and 1.\n",
       "* **Epistemological  commitments** - a logical agent believes each sentence to be true or false or has  no opinion\n",
       "* **Utility theory** - represent preference and reason quantitatively with them, utility: usefulness\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module10_sections = [\n",
    "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
    "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
    "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
    "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
    "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
    "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
    "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
    "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
    "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
    "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
    "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
    "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
    "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
    "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
    "]\n",
    "\n",
    "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
    "display(Markdown(module10_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WHREsHSSyA-h"
   },
   "source": [
    "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "id": "tuFC8w7CyCns"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
    "chapter13_1 = \"\"\" \n",
    "- Models describe how (a portion of) the world works\n",
    "- Models are always simplifications\n",
    "  - May not account for every variable\n",
    "  - May not account for all interactions between variables\n",
    "- Two problems with using full joint distribution tables\n",
    "as our probabilistic models:\n",
    "  - Unless there are only a few variables, the joint is WAY too\n",
    "big to represent explicitly\n",
    "  - Hard to learn (estimate) anything empirically about more\n",
    "than a few variables at a time\n",
    "- Bayesian Network: a technique for describing complex joint\n",
    "distributions (models) using simple, local\n",
    "distributions (conditional probabilities)\n",
    "  - We describe how variables locally interact\n",
    "  - Local interactions chain together to give global, indirect\n",
    "interactions\n",
    "  - Each node corresponds to a random variable, which may be discrete or continuous. \n",
    "  - Directed links or arrows connect pairs of nodes. \n",
    "  - Each node Xi has associated probability information θ(Xi |Parents(Xi)) that quantifies the effect of the parents on the node using a finite number of parameters. \n",
    "- Bayes Net Semantics:\n",
    "  - A set of nodes, one per variable X\n",
    "  - A directed, acyclic graph\n",
    "  - A conditional distribution for each node\n",
    "  - Bayes net = Topology (graph) + Local Conditional Probabilities\n",
    "  - Bayes’ nets implicitly encode joint distributions\n",
    "    - As a product of local conditional distributions\n",
    "    - To see what probability a BN gives to a full assignment, multiply all the\n",
    "relevant conditionals together\n",
    "    - Not every BN can represent every joint distribution\n",
    "      - The topology enforces certain conditional independencies\n",
    "- The  intuitive meaning of an arrow is typically that X has a direct influence on Y, which suggests  that causes should be parents of effects.\n",
    "- conditioning case is just a possible combination of values for the parent  nodes—a miniature possible world\n",
    "- each row must sum to 1, because the entries  represent an exhaustive set of cases for the variable.\n",
    "- In general, a table for a Boolean variable with k  Boolean parents contains 2k independently specifiable probabilities. A node with no parents  has only one row, representing the prior probabilities of each possible value of the variable. \n",
    "- some insights:\n",
    "  - When Bayes’ nets reflect the true causal patterns:\n",
    "    - Often simpler (nodes have fewer parents)\n",
    "    - Often easier to think about\n",
    "    - Often easier to elicit from experts\n",
    "  - BNs need not actually be causal\n",
    "  - What do the arrows really mean?： Topology really encodes conditional independence\n",
    "\"\"\"\n",
    "chapter13_2 = \"\"\"\n",
    "- Bayes net, assumptions:\n",
    "  -  $P(x_i|x_1,...,x_{i-1}) = P(x_i|parents(X_i))$\n",
    "\"\"\" + r\"\"\" $P(x_1,...,x_n) = \\prod_{i=1}^n P(x_i|parents(X_i))$<br />\"\"\" + \"\"\"  \n",
    "- product of joint possibility for the final probability\n",
    "- Bayes net construction\n",
    "\"\"\" + r\"\"\" $P(x_1,...,x_n) = \\prod_{i=1}^n P(x_i|x_{i-1},...,x_1)$<br />\"\"\" + \"\"\" \n",
    "- net ordering, any order will work but will have different compactness\n",
    "- Intuitively, the parents of node Xi should contain all those nodes in X1,..., Xi−1 that directly  influence Xi.\n",
    "- Another important property of Bayes nets is that they contain  no redundant probability values. If there is no redundancy, then there is no chance for inconsistency: it is impossible for the knowledge engineer or domain expert to create a Bayesian  network that violates the axioms of probability. \n",
    "- Bayes net can be more compact, in most domains each random variable is directly influenced by at most k others, for some constant k. If we assume  n Boolean variables for simplicity, then the amount of information needed to specify each  conditional probability table will be at most $2^k$ numbers, and the complete network can be  specified by $2^k · n$ numbers.\n",
    "- Incorrect orders result in more links, If we stick to a causal model, we end up  having to specify fewer numbers, and the numbers will often be easier to come up with.\n",
    "- Are X and Y conditionally independent given\n",
    "evidence variables {Z}?\n",
    "  - Yes, if X and Y “d-separated” by Z\n",
    "  - Consider all (undirected) paths from X to Y\n",
    "  - No active paths = independence!\n",
    "- A path is active if each triple is active:\n",
    "  - Causal chain A $\\rightarrow$ B $\\rightarrow$ C where B is unobserved (either direction)\n",
    "  - Common cause A $\\leftarrow$ B $\\rightarrow$ C where B is unobserved\n",
    "  - Common effect (aka v-structure)\n",
    "  - A $\\rightarrow$ B $\\leftarrow$ C where B or one of its descendents is observed\n",
    "\"\"\"\n",
    "chapter13_3 = \"\"\"\n",
    "- probabilistic inference system is to compute the posterior probability  distribution for a set of query variables, given some observed event—usually, some assignment of values to a set of evidence variables.\n",
    "- the joint distribution can be written as products of conditional probabilities from the network. Therefore, a query can  be answered using a Bayes net by computing sums of products of conditional probabilities  from the network. \n",
    "- query In the worst case, where we have to sum out almost all the variables, there will be  $O(2^n)$ terms in the sum, each a product of $O(n)$ probability values. A naive implementation  would therefore have complexity $O(n2^n)$. \n",
    "- Variable elimination:\n",
    "  - every variable that is not an ancestor of a query variable or evidence variable is irrelevant to the query. \n",
    "- Complexity:\n",
    "  - if at most one undirected path between any  two nodes in the network. singly connected networks or polytrees, The time and space complexity of exact inference in polytrees is linear in the size of the network. \n",
    "  - multiply connected This is not surprising when one considers that  because it includes inference in propositional logic as a special case, inference in Bayes nets  is NP-hard.\n",
    "  - in a polytree network, one would need to issue $O(n)$ queries  costing $O(n)$ each, for a total of $O(n^2 )$ time. Using clustering algorithms the time can be reduced to $O(n)$.\n",
    "  - clustering reduce complexity, result in meganode\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module11_lecture_notes = \"\"\" \n",
    "- A Bayesian network is a directed acyclic graph whose nodes correspond to random  variables; each node has a conditional distribution for the node, given its parents. \n",
    "- Bayesian networks provide a concise way to represent conditional independence relationships in the domain. \n",
    "- A Bayesian network specifies a joint probability distribution over its variables. The  probability of any given assignment to all the variables is defined as the product of the  corresponding entries in the local conditional distributions. A Bayesian network is often  exponentially smaller than an explicitly enumerated joint distribution.\n",
    "- Many conditional distributions can be represented compactly by canonical families of  distributions. Hybrid Bayesian networks, which include both discrete and continuous  variables, use a variety of canonical distributions.  \n",
    "- Inference in Bayesian networks means computing the probability distribution of a set  of query variables, given a set of evidence variables. Exact inference algorithms, such  as variable elimination, evaluate sums of products of conditional probabilities as effi-  ciently as possible.  \n",
    "- In polytrees (singly connected networks), exact inference takes time linear in the size  of the network. In the general case, the problem is intractable. \n",
    "- Bayes nets compactly encode joint distributions\n",
    "- Guaranteed independencies of distributions can be\n",
    "deduced from BN graph structure\n",
    "- D-separation gives precise conditional independence\n",
    "guarantees from graph alone\n",
    "- A Bayes’ Net’s joint distribution may have further\n",
    "(conditional) independence that is not detectable until\n",
    "you inspect its specific distribution\n",
    " \"\"\"\n",
    "\n",
    "# Definitions\n",
    "module11_definitions = {\n",
    "\"go back and fill in the meaning\" : \"definition\",    \n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "vqtEs0gHs9sO",
    "outputId": "86772db3-b7b8-49f6-b501-866e932dc251"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n",
       "## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n",
       " \n",
       "- Models describe how (a portion of) the world works\n",
       "- Models are always simplifications\n",
       "  - May not account for every variable\n",
       "  - May not account for all interactions between variables\n",
       "- Two problems with using full joint distribution tables\n",
       "as our probabilistic models:\n",
       "  - Unless there are only a few variables, the joint is WAY too\n",
       "big to represent explicitly\n",
       "  - Hard to learn (estimate) anything empirically about more\n",
       "than a few variables at a time\n",
       "- Bayesian Network: a technique for describing complex joint\n",
       "distributions (models) using simple, local\n",
       "distributions (conditional probabilities)\n",
       "  - We describe how variables locally interact\n",
       "  - Local interactions chain together to give global, indirect\n",
       "interactions\n",
       "  - Each node corresponds to a random variable, which may be discrete or continuous. \n",
       "  - Directed links or arrows connect pairs of nodes. \n",
       "  - Each node Xi has associated probability information θ(Xi |Parents(Xi)) that quantifies the effect of the parents on the node using a finite number of parameters. \n",
       "- Bayes Net Semantics:\n",
       "  - A set of nodes, one per variable X\n",
       "  - A directed, acyclic graph\n",
       "  - A conditional distribution for each node\n",
       "  - Bayes net = Topology (graph) + Local Conditional Probabilities\n",
       "  - Bayes’ nets implicitly encode joint distributions\n",
       "    - As a product of local conditional distributions\n",
       "    - To see what probability a BN gives to a full assignment, multiply all the\n",
       "relevant conditionals together\n",
       "    - Not every BN can represent every joint distribution\n",
       "      - The topology enforces certain conditional independencies\n",
       "- The  intuitive meaning of an arrow is typically that X has a direct influence on Y, which suggests  that causes should be parents of effects.\n",
       "- conditioning case is just a possible combination of values for the parent  nodes—a miniature possible world\n",
       "- each row must sum to 1, because the entries  represent an exhaustive set of cases for the variable.\n",
       "- In general, a table for a Boolean variable with k  Boolean parents contains 2k independently specifiable probabilities. A node with no parents  has only one row, representing the prior probabilities of each possible value of the variable. \n",
       "- some insights:\n",
       "  - When Bayes’ nets reflect the true causal patterns:\n",
       "    - Often simpler (nodes have fewer parents)\n",
       "    - Often easier to think about\n",
       "    - Often easier to elicit from experts\n",
       "  - BNs need not actually be causal\n",
       "  - What do the arrows really mean?： Topology really encodes conditional independence\n",
       "\n",
       "\n",
       "## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n",
       "\n",
       "- Bayes net, assumptions:\n",
       "  -  $P(x_i|x_1,...,x_{i-1}) = P(x_i|parents(X_i))$\n",
       " $P(x_1,...,x_n) = \\prod_{i=1}^n P(x_i|parents(X_i))$<br />  \n",
       "- product of joint possibility for the final probability\n",
       "- Bayes net construction\n",
       " $P(x_1,...,x_n) = \\prod_{i=1}^n P(x_i|x_{i-1},...,x_1)$<br /> \n",
       "- net ordering, any order will work but will have different compactness\n",
       "- Intuitively, the parents of node Xi should contain all those nodes in X1,..., Xi−1 that directly  influence Xi.\n",
       "- Another important property of Bayes nets is that they contain  no redundant probability values. If there is no redundancy, then there is no chance for inconsistency: it is impossible for the knowledge engineer or domain expert to create a Bayesian  network that violates the axioms of probability. \n",
       "- Bayes net can be more compact, in most domains each random variable is directly influenced by at most k others, for some constant k. If we assume  n Boolean variables for simplicity, then the amount of information needed to specify each  conditional probability table will be at most $2^k$ numbers, and the complete network can be  specified by $2^k · n$ numbers.\n",
       "- Incorrect orders result in more links, If we stick to a causal model, we end up  having to specify fewer numbers, and the numbers will often be easier to come up with.\n",
       "- Are X and Y conditionally independent given\n",
       "evidence variables {Z}?\n",
       "  - Yes, if X and Y “d-separated” by Z\n",
       "  - Consider all (undirected) paths from X to Y\n",
       "  - No active paths = independence!\n",
       "- A path is active if each triple is active:\n",
       "  - Causal chain A $\r",
       "ightarrow$ B $\r",
       "ightarrow$ C where B is unobserved (either direction)\n",
       "  - Common cause A $\\leftarrow$ B $\r",
       "ightarrow$ C where B is unobserved\n",
       "  - Common effect (aka v-structure)\n",
       "  - A $\r",
       "ightarrow$ B $\\leftarrow$ C where B or one of its descendents is observed\n",
       "\n",
       "\n",
       "## AIMA 13.3 Exact Inference in Bayesian Networks\n",
       "\n",
       "- probabilistic inference system is to compute the posterior probability  distribution for a set of query variables, given some observed event—usually, some assignment of values to a set of evidence variables.\n",
       "- the joint distribution can be written as products of conditional probabilities from the network. Therefore, a query can  be answered using a Bayes net by computing sums of products of conditional probabilities  from the network. \n",
       "- query In the worst case, where we have to sum out almost all the variables, there will be  $O(2^n)$ terms in the sum, each a product of $O(n)$ probability values. A naive implementation  would therefore have complexity $O(n2^n)$. \n",
       "- Variable elimination:\n",
       "  - every variable that is not an ancestor of a query variable or evidence variable is irrelevant to the query. \n",
       "- Complexity:\n",
       "  - if at most one undirected path between any  two nodes in the network. singly connected networks or polytrees, The time and space complexity of exact inference in polytrees is linear in the size of the network. \n",
       "  - multiply connected This is not surprising when one considers that  because it includes inference in propositional logic as a special case, inference in Bayes nets  is NP-hard.\n",
       "  - in a polytree network, one would need to issue $O(n)$ queries  costing $O(n)$ each, for a total of $O(n^2 )$ time. Using clustering algorithms the time can be reduced to $O(n)$.\n",
       "  - clustering reduce complexity, result in meganode\n",
       "\n",
       "\n",
       "## Module 11 Lecture Notes\n",
       " \n",
       "- A Bayesian network is a directed acyclic graph whose nodes correspond to random  variables; each node has a conditional distribution for the node, given its parents. \n",
       "- Bayesian networks provide a concise way to represent conditional independence relationships in the domain. \n",
       "- A Bayesian network specifies a joint probability distribution over its variables. The  probability of any given assignment to all the variables is defined as the product of the  corresponding entries in the local conditional distributions. A Bayesian network is often  exponentially smaller than an explicitly enumerated joint distribution.\n",
       "- Many conditional distributions can be represented compactly by canonical families of  distributions. Hybrid Bayesian networks, which include both discrete and continuous  variables, use a variety of canonical distributions.  \n",
       "- Inference in Bayesian networks means computing the probability distribution of a set  of query variables, given a set of evidence variables. Exact inference algorithms, such  as variable elimination, evaluate sums of products of conditional probabilities as effi-  ciently as possible.  \n",
       "- In polytrees (singly connected networks), exact inference takes time linear in the size  of the network. In the general case, the problem is intractable. \n",
       "- Bayes nets compactly encode joint distributions\n",
       "- Guaranteed independencies of distributions can be\n",
       "deduced from BN graph structure\n",
       "- D-separation gives precise conditional independence\n",
       "guarantees from graph alone\n",
       "- A Bayes’ Net’s joint distribution may have further\n",
       "(conditional) independence that is not detectable until\n",
       "you inspect its specific distribution\n",
       " \n",
       "\n",
       "## Module 11 Definitions\n",
       "* **Go back and fill in the meaning** - definition\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module11_sections = [\n",
    "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
    "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
    "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
    "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
    "]\n",
    "\n",
    "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
    "                         module11_sections, \n",
    "                         module11_definitions)\n",
    "display(Markdown(module11_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F3FPMzG9yUls"
   },
   "source": [
    "#Module 12 - Naive Bayes and Perceptrons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "id": "iEIOpL7vyWcc"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
    "\n",
    "chapter19_1 = \"\"\"\n",
    "- agent is learning if it improves its performance after making observations about the world. \n",
    "- if agent is computer machine learning: a computer observes some data, builds a model based on the data, and uses the model as both a hypothesis about the world and a piece of software that  can solve problems. \n",
    "  - why learning, 1. the designers cannot anticipate all possible future  situations 2. sometimes the designers have no idea how  to program a solution themselves.\n",
    "- improvements:\n",
    "  - Which component is to be improved. \n",
    "  - What prior knowledge the agent has, which influences the model it builds. \n",
    "  - What data and feedback on that data is available. \n",
    "- components:\n",
    "  - A direct mapping from conditions on the current state to actions. \n",
    "  - A means to infer relevant properties of the world from the percept sequence. \n",
    "  - Information about the way the world evolves and about the results of possible actions  the agent can take. \n",
    "  - Utility information indicating the desirability of world states. \n",
    "  - Action-value information indicating the desirability of actions. \n",
    "  -  Goals that describe the most desirable states. \n",
    "  - A problem generator, critic, and learning element that enable the system\n",
    "- prior knowledge on the part of the agent: it starts from scratch  and learns from the data.\n",
    "- Going from a specific set of observations to a general rule is called induction;\n",
    "- factored representation—a  vector of attribute values.\n",
    "- types of Learning:\n",
    "  - In supervised learning the agent observes input-output pairs and learns a function that maps from input to output.\n",
    "  - In unsupervised learning the agent learns patterns in the input without any explicit Unsupervised  learning  feedback. The most common unsupervised learning task is clustering: detecting potentially useful clusters of input examples.\n",
    "  - In reinforcement learning the agent learns from a series of reinforcements: rewards and punishments.\n",
    "\"\"\"\n",
    "chapter19_2 = \"\"\"\n",
    "- Given a training set of N example input–output pairs, discover a function h that approximates the true function, h is called a hypothesis about the world. It is drawn from a hypothesis space H of possible functions.\n",
    "- h is a model of the data, drawn from a model  class H, or we can say a function drawn from a function class. We call the output yi the ground truth—the true answer we are asking our model to predict.\n",
    "- exploratory data analysis: examining Exploratory data  analysis  the data with statistical tests and visualizations\n",
    "- The true measure of a hypothesis is not how it does on the training set, but rather how  well it handles inputs it has not yet seen. on test set\n",
    "- say that h generalizes well if it accurately predicts the outputs of the test set. \n",
    "- By bias we mean (loosely) the tendency of a predictive hypothesis to deviate from the  expected value when averaged over different training sets.\n",
    "- We say that a hypothesis is underfitting when it fails to find a pattern in the data.\n",
    "- variance we mean the amount of change in the hypothesis due to fluctuation in the training data.\n",
    "- is overfitting the data when it pays too much attention to the particular data set it is trained on, causing it to perform poorly on unseen data. \n",
    "- bias–variance tradeoff: a choice between more complex, low-bias hypotheses that fit the training data well and simpler, low-variance hypotheses that may generalize better.\n",
    "- <strong>perceptron 1</strong>\n",
    "  - Inputs are feature values\n",
    "  - Each feature has a weight\n",
    "  - Sum is the activation\n",
    "  - $activation(x) = \\sum_i w_i\\cdot f_i(x)= w\\cdot f(x)$\n",
    "  - If the activation is:\n",
    "    - Positive, output +1\n",
    "    - Negative, output -1\n",
    "- Binary case: compare features to a weight vector\n",
    "- Learning: figure out the weight vector from examples\n",
    "  - Start with weights = 0\n",
    "  - For each training instance:\n",
    "    - Classify with current weights\n",
    "    - If correct (i.e., y=y*), no change!\n",
    "    - If wrong: adjust the weight vector by adding or subtracting the\n",
    "feature vector. Subtract if y* is -1. $w = w + \\hat y \\cdot f$\n",
    "- Multiclass Decision:\n",
    "  - A weight vector for each class: $w_y$\n",
    "  - Score (activation) of a class y: $w_y\\cdot f(x)$\n",
    "  - Prediction highest score wins: $\\hat y =  argmax_y w_y\\cdot f(x)$\n",
    "- Learning: Multiclass Perceptron\n",
    "  - Start with all weights = 0\n",
    "  - Pick up training examples one by one\n",
    "  - Predict with current weights: $\\hat y =  argmax_y w_y\\cdot f(x)$\n",
    "  - If correct, no change!\n",
    "  - If wrong: lower score of wrong\n",
    "answer, raise score of right answer: $w_y = w_y-f(x)$, $w_{y*} = w_{y*} + f(x)$\n",
    "\"\"\"\n",
    "chapter19_3 = \"\"\"\n",
    "- decision tree is a representation of a function that maps a vector of attribute values to Decision tree  a single output value—a “decision.” A decision tree reaches its decision by performing a  sequence of tests, starting at the root and following the appropriate branch until a leaf is  reached.\n",
    "- no way to represent all kinds of functions. Even just considering Boolean functions with n Boolean attributes, the truth table will have $2^n$ rows, so there are  $2^2^n$ different functions.\n",
    "- using the notion of information gain, which is defined in terms of entropy, which is the fundamental quantity in information theory\n",
    "- Entropy is a measure of the uncertainty of a random variable; the more information, the  less entropy.\n",
    "\"\"\" + r\"\"\"$H(V) = - \\sum_{k}P(v_k)\\log P(v_k)$\"\"\" + \"\"\"\n",
    "- Info gain: \n",
    "\"\"\" + r\"\"\" infoGain = $IG(X) = H(Y)-H(Y|X) =  - \\sum_y P(Y=y)\\log_2 P(Y=y)-  \\sum_x P(X=x)H(Y|X=x) =  - \\sum_y P(Y=y)\\log_2 P(Y=y) + \\sum_{x,y} P(X=x,Y=y) \\log_2 P(Y=y|X=x) $\"\"\" + \"\"\"\n",
    "- Overfitting becomes more likely as the number of attributes  grows, and less likely as we increase the number of training examples. Larger hypothesis  spaces (e.g., decision trees with more nodes or polynomials with high degree) have more capacity both to fit and to overfit; some model classes are more prone to overfitting than  others. \n",
    "- decision tree pruning combats overfitting.  works by eliminating nodes that are not clearly relevant.\n",
    "- The name CART, standing for Classification And Regression Trees, is used to cover both classes. \n",
    "- <strong>perceptron 2</strong>\n",
    "- Properties\n",
    "  - Separability: true if some parameters get the training\n",
    "set perfectly correct\n",
    "  - Convergence: if the training is separable, perceptron\n",
    "will eventually converge (binary case)\n",
    "  - Mistake Bound: the maximum number of mistakes\n",
    "(binary case) related to the margin or degree of\n",
    "separability: $mistake < k/\\delta^2$\n",
    "- Improving the perceptron:\n",
    "  - Noise: if the data isn’t separable,\n",
    "weights might thrash\n",
    "    - Averaging weight vectors over\n",
    "time can help (averaged\n",
    "perceptron)\n",
    "  - Mediocre generalization: finds a\n",
    "“barely” separating solution, not seperated in the center\n",
    "    - SVM\n",
    "  - Overtraining: test / held-out\n",
    "accuracy usually rises, then falls\n",
    "- Fixing perceptron:\n",
    "  - Idea: adjust the weight update to mitigate these effects\n",
    "- MIRA:\n",
    "  - Choose an update size that fixes the current mistake but, minimizes the change to w\n",
    "  - $\\tau = (w'_y-w'_{y*}\\cdot f+1)/2f\\cdot f$, The +1 helps to generalize\n",
    "  - $w_y = w_y-\\tau f(x)$, $w_{y*} = w_{y*} + \\tau f(x)$\n",
    "- SVM:\n",
    "  - Maximizing the margin: good according to intuition, theory, practice\n",
    "  - Only support vectors matter; other training examples are ignorable\n",
    "  - Support vector machines (SVMs) find the separator with max\n",
    "margin\n",
    "\"\"\"\n",
    "chapter19_4 = \"\"\"\n",
    "- will make the assumption that the future examples will be like the past. We call  this the stationarity assumption; without it, all bets are off.\n",
    "- We assume that each example Ej Stationarity  has the same prior probability distribution and is independent of the previous examples: are independent and identically distributed or i.i.d..\n",
    "- k-fold cross-validation. The idea is that each example serves double duty—as training data and validation data—but not at the same time. \n",
    "- is k = n, also  LOOCV known as leave-one-out cross-validation or LOOCV.\n",
    "- We can think of the task of finding a good hypothesis as two subtasks: model selection chooses a good hypothesis space, and optimization (also called training) finds the best hypothesis within that space. \n",
    "- We say that a model that exactly fits all the training data has interpolated the data Model classes typically start to overfit as the capacity approaches the point of interpolation. \n",
    "- to minimize a loss function rather than Loss function  maximize a utility function.\n",
    "- about fitting\n",
    "  - we say that a learning problem is realizable if the hypothesis space H actually  contains the true function\n",
    "  - variance  means that a learning algorithm will in general return different hypotheses for different sets  of examples.\n",
    "  - f may be nondeterministic or noisy—it may return  different values of f(x) for the same x.\n",
    "  - when H is a complicated function in a large hypothesis space,  it can be computationally intractable to systematically search all possibilities; in that case,  a search can explore part of the space and return a reasonably good hypothesis, but can’t  always guarantee the best one. \n",
    "- small-scale learning, where the number of training examples ranged from dozens to the low  thousands.\n",
    "- large-scale learning, with millions of  examples.\n",
    "- regularization: This process of explicitly penalizing complex hypotheses\n",
    "  - choices: the  loss function (L1 or L2), and the complexity measure, which is called a regularization function.\n",
    "  - A  process of feature selection can be performed to discard attributes that appear to be irrelevant.\n",
    "- grid search : try all combinations of Grid search  values and see which performs best on the validation data.\n",
    "- random search:  samples  uniformly from the set of all possible hyperparameter settings\n",
    "- Bayesian optimization treats the task of choosing good hyperparameter values as a  machine learning problem in itself.\n",
    "- If we make the assumption that f can be approximated  by a Gaussian process, then the math of updating our belief about f works out nicely.\n",
    "- <strong>perceptron 3</strong>\n",
    "- Non-Linear Separators\n",
    "  - mapping data to a higher-dimensional space\n",
    "  - the original feature space can always be mapped to some\n",
    "higher-dimensional feature space where the training set is separable:\n",
    "- Comparison:\n",
    "  - Naïve Bayes\n",
    "    - Builds a model training data\n",
    "    - Gives prediction probabilities\n",
    "    - Strong assumptions about feature independence\n",
    "    - One pass through data (counting)\n",
    "  - Perceptrons / MIRA:\n",
    "    - Makes less assumptions about data\n",
    "    - Mistake-driven learning\n",
    "    - Multiple passes through data (prediction)\n",
    "    - Often more accurate\n",
    "\"\"\"\n",
    "chapter19_5 = \"\"\"\n",
    "- any hypothesis that is consistent with a sufficiently large set of training examples is  unlikely to be seriously wrong: that is, it must be probably approximately correct (PAC). \n",
    "- like all theorems, are logical consequences of axioms. When a  theorem (as opposed to, say, a political pundit) states something about the future based on  the past, the axioms have to provide the “juice” to make that connection.\n",
    "- A hypothesis h is called approximately correct if error(h) ≤ ǫ, where ǫ is a small constant. and error is error rate\n",
    "- decision lists. A  decision list consists of a series of tests, each of which is a conjunction of literals. If a  test succeeds when applied to an example description, the decision list specifies the value  to be returned. If the test fails, processing continues with the next test in the list.\n",
    "\"\"\"\n",
    "chapter19_6 = \"\"\"\n",
    "- the class of linear functions of continuous-valued  inputs.\n",
    "- Many forms of learning involve adjusting weights to minimize a loss, so it helps to have a  mental picture of what’s going on in weight space\n",
    "- loss function is convex \n",
    "\"\"\" + r\"\"\"$w = \\frac{\\sum_i x_i y_i}{\\sum_i x_i^2}$\"\"\" + \"\"\"\n",
    "- gradient descent:\n",
    "   - \n",
    "\"\"\" + r\"\"\"$w_i \\leftarrow w_i + \\alpha\\frac{\\partial}{\\partial w_i}Loss(w)$\"\"\" + \"\"\"\n",
    "- batch gradient descent learning rule for univariate linear re- Batch gradient  descent  gression (also called deterministic gradient descent). The loss surface is convex. A  step that covers all the training examples is called an epoch.\n",
    "- A faster variant is called stochastic gradient descent or SGD: it randomly selects a small Stochastic gradient  descent  number of training examples at each step, and updates\n",
    "- multivariable linear regression\n",
    "  - \n",
    "\"\"\" + r\"\"\"$w* = (X^T X)^{-1}X^Ty$\"\"\" + \"\"\"\n",
    "- regularization: Cost(h) = EmpLoss(h) +λComplexity(h). \n",
    "- L1 regularization has an important advantage: it tends to produce a sparse model. That is, it often sets many weights to zero\n",
    "- A decision boundary is a line (or a surface, in higher dimensions) that separates the  two classes. A linear decision boundary is called a linear separator and data that admit such a separator are called linearly separable.\n",
    "- percepton:\n",
    "  - If the output is correct (i.e., y=hw(x)) then the weights are not changed. \n",
    "  - If y is 1 but hw(x) is 0, then wi is increased when the corresponding input xi is positive  and decreased when xi is negative. This makes sense, because we want to make w · x  bigger so that hw(x) outputs a 1. \n",
    "  -  If y is 0 but hw(x) is 1, then wi is decreased when the corresponding input xi is positive  and increased when xi is negative. This makes sense, because we want to make w · x  smaller so that hw(x) outputs a 0. \n",
    "- logistic regression. There is no easy closed-form solution to find the optimal value of w with Logistic regression  this model, but the gradient descent computation is straightforward.\n",
    "- \n",
    "\"\"\" + r\"\"\"$\\frac{\\partial}{\\partial w_i}Loss(w) = -2(y-h_w(x)) \\times g'(w\\cdot x) \\times x_i$ <br />\"\"\" + r\"\"\"$w_i \\leftarrow w_i + \\alpha (y-h_w(x))h_w(x)(1-h_w(x))x_i$<br />\"\"\" + r\"\"\"$h_w(x) = \\frac{1}{1+e^{-w x}}$\"\"\"\n",
    "\n",
    "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
    "slp_chapter4_1 = \"\"\"\n",
    "- General Naive Bayes: $P(Y,F_1,...,F_n) = P(Y)\\prod_i P(F_i|Y)$\n",
    "- We only have to specify how each feature depends on the class\n",
    "Total number of parameters is linear in number of features\n",
    "Model is very simplistic, but often works anyway\n",
    "- multinomial naive Bayes classifier: so called because it is a Bayesian classifier that makes a simplifying (naive) assumption about how the features interact.\n",
    "- bag-of-words, that is, an unordered set of words with their position\n",
    "ignored, keeping only their frequency in the document\n",
    "- at notation ˆ to mean “our\n",
    "estimate of the correct class\n",
    "- $\\hat{c}=argmax_{c\\in C}P(c|d) = argmax_{c\\in C}P(d|c)P(c)/P(d) = argmax_{c\\in C}P(d|c)P(c) $ because $P(d)$ is constant\n",
    "- Naive Bayes is a probabilistic classifier, meaning that for a document d, out of\n",
    "all classes c 2 C the classifier returns the class hat(c) which has the maximum posterior probability given the document.\n",
    "- We call Naive Bayes a generative model because we can read equation stating\n",
    "a kind of implicit assumption about how a document is generated: first a class is\n",
    "sampled from P(c), and then the words are generated by sampling from P(d|c).\n",
    "- we compute the most probable class ˆ c given some\n",
    "document d by choosing the class which has the highest product of two probabilities:\n",
    "the prior probability of prior the class P(c) and the likelihood of the document P(d|c):\n",
    "- we can represent a document d as a set of features\n",
    "- $\\hat{c}= argmax_{c\\in C}P(f_1,f_2,...,f_n|c)P(c) $,\n",
    "- first, we assume\n",
    "that the features f1; f2; ...; fn only encode word identity and not position.\n",
    "- second, naive Bayes assumption: this is the condiassumption\n",
    "tional independence assumption that the probabilities P( fi|c) are independent given\n",
    "the class c and hence can be ‘naively’ multiplied as follows:\n",
    "- $P(f_1,f_2,...,f_n|c) = P(f_1|c)P(f_2|c)...P(f_n|c)$\n",
    "- $c_{NB} = argmax_{c\\in C}P(c)\\prod_{f\\in F}P(f|c)$\n",
    "- To apply the naive Bayes classifier to text, we need to consider word positions, by\n",
    "simply walking an index through every word position in the document:\n",
    "- $c_{NB} = argmax_{c\\in C}P(c)\\prod_{i \\in Positions}P(w_i|c)$\n",
    "- like calculations for language modeling, are done in log\n",
    "space, to avoid underflow and increase speed.\n",
    "- $c_{NB} = argmax_{c\\in C}log P(c)\\sum_{i \\in Positions}log P(w_i|c)$\n",
    "\"\"\"\n",
    "slp_chapter4_2 = \"\"\"\n",
    "-Let Nc be the number of documents in our training data with class c and Ndoc be the total number of documents\n",
    "- $\\hat{P}(c)=N_c/N_{doc}$\n",
    "- $\\hat{P}(w_i|c)=count(w_i,c)/\\sum_{w\\in V} count(w,c)$\n",
    "- Add laplace to avoid zeros\n",
    "- $\\hat{P}(w_i|c)=[count(w_i,c)+1]/\\sum_{w\\in V} [count(w,c)+1]$\n",
    "    - $= [count(w_i,c)+1]/[[\\sum_{w\\in V} count(w,c)]+V]$\n",
    "- processes:\n",
    "  - unknown words is to ignore them—remove them from the test\n",
    "document and not include any probability for them at all.\n",
    "  - ignore another class of words: stop\n",
    "stop words words, very frequent words like the and a\n",
    " \"\"\"\n",
    "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
    "slp_chapter4_4 = \"\"\"\n",
    "- binary NB  uses the except\n",
    "that for each document we remove all duplicate words before concatenating them\n",
    "into the single big document.\n",
    "- addition commonly made when doing text classification for\n",
    "sentiment is to deal with negation. Consider the difference between I really like this\n",
    "movie (positive) and I didn’t like this movie (negative). The negation expressed by\n",
    "didn’t completely alters the inferences we draw from the predicate like. Similarly,\n",
    "negation can modify a negative word to produce a positive review\n",
    "- during text normalization, prepend the prefix NOT to\n",
    "every word after a token of logical negation (n’t, not, no, never) until the next punctuation\n",
    "mark.\n",
    "- NOT like, NOT recommend will thus occur more often\n",
    "in negative document and act as cues for negative sentiment, while words like\n",
    "NOT bored, NOT dismiss will acquire positive associations.\n",
    "- in some situations we might have insufficient labeled training data to\n",
    "train accurate naive Bayes classifiers using all words in the training set to estimate\n",
    "positive and negative sentiment\n",
    "- sentiment lexicons, lists of words that are preannotated with positive or negative sentiment.\n",
    "\"\"\"\n",
    "slp_chapter4_5 = \"\"\"\n",
    "- For other tasks, like language id—determining what language a given piece of text is written in—the most effective naive Bayes features are not words at all,\n",
    "but character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,\n",
    "‘thei’), or, even simpler byte n-grams, where instead of using the multibyte Unicode\n",
    "character representations called codepoints, we just pretend everything is a string of\n",
    "raw bytes.\n",
    "\"\"\"\n",
    "slp_chapter4_6 = \"\"\"\n",
    "- a naive Bayes model can be viewed as a\n",
    "set of class-specific unigram language models, in which the model for each class\n",
    "instantiates a unigram language model.\n",
    "\"\"\"\n",
    "slp_chapter4_7 = \"\"\"\n",
    "- gold labels: human labels\n",
    "- Precision measures the percentage of the items that\n",
    "the system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\n",
    "are positive according to the human gold labels).\n",
    "- $precision = tp/(tp+fp)$\n",
    "- Recall measures the percentage of items actually present in the input that were\n",
    "correctly identified by the system.\n",
    "- $recall = tp/(tp+fn)$\n",
    "- $F_1=2PR/(P+R)$\n",
    "- $F_b = (b^2+1)PR/(b^2P+R)$\n",
    "- Values of b > 1 favor recall, while\n",
    "values of b < 1 favor precision.\n",
    "- Harmonic mean is used because it is a conservative metric; the harmonic mean of\n",
    "two values is closer to the minimum of the two values than the arithmetic mean is.\n",
    "Thus it weighs the lower of the two numbers more heavily.\n",
    "- recall and precision for each row\n",
    "- In macroaveraging, we compute the performance\n",
    "microaveraging for each class, and then average over classes.\n",
    "- In microaveraging, we collect the decisions\n",
    "for all classes into a single confusion matrix, and then compute precision and\n",
    "recall from that table.\n",
    "\"\"\"\n",
    "slp_chapter4_8 = \"\"\"\n",
    "- effect size $\\delta(x) = M(A,x)-M(B,x)$, A is new B is before\n",
    "- That’s because A might just be accidentally better than B on this particular x.\n",
    "We need something more: we want to know if A’s superiority over B is likely to hold\n",
    "again if we checked another test set x0, or under some other set of circumstances.\n",
    "- suppose: $H_0:\\delta(x)\\leq 0$, $H_1: \\delta(x)>0$\n",
    "- P(\\delta(X)\\geq \\delta(x)|H_0 is true)\n",
    "- A very small p-value means that the difference we observed is very unlikely\n",
    "under the null hypothesis, and we can reject the null hypothesis. What counts as very\n",
    "small? It is common to use values like .05 or .01 as the thresholds. A value of .01\n",
    "means that if the p-value (the probability of observing the d we saw assuming H0 is\n",
    "true) is less than .01, we reject the null hypothesis and assume that A is indeed better\n",
    "than statistically B.\n",
    "- bootstrap test:\n",
    "- The intuition\n",
    "of the bootstrap test is that we can create many virtual test sets from an observed\n",
    "test set by repeatedly sampling from it. The method only makes the assumption that\n",
    "the sample is representative of the population.\n",
    "\"\"\"\n",
    "slp_chapter4_9 = \"\"\"\n",
    "- Assuming H0 (A isn’t better than B), we would expect that d(X), estimated\n",
    "over many test sets, would be zero; a much higher value would be surprising,\n",
    "since H0 specifically assumes A isn’t better than B. To measure exactly how surprising\n",
    "is our observed d(x) we would in other circumstances compute the p-value by\n",
    "counting over many test sets how often d(x(i)) exceeds the expected zero value by\n",
    "d(x) or more:\n",
    "- $p-value(x) = 1/b \\sum_{i=1}^b 1 if (\\delta(x^{(i)})\\geq 2\\delta(x))$\n",
    "- this isn’t true for the bootstrapped test\n",
    "sets we created. That’s because we didn’t draw these samples from a distribution\n",
    "with 0 mean; we happened to create them from the original test set x, which happens\n",
    "to be biased (by .20) in favor of A. So to measure how surprising is our observed\n",
    "d(x), we actually compute the p-value by counting over many test sets how often\n",
    "d(x(i)) exceeds the expected value of d(x) by d(x) or more:\n",
    "- Confidence:\n",
    "  - The confidence of a probabilistic classifier:\n",
    "    - Posterior over the top label: $confidence(x) = max_y P(y|x)$\n",
    "    - Represents how sure the classifier is of\n",
    "the classification\n",
    "    - Any probabilistic model will have\n",
    "confidences\n",
    "    - No guarantee confidence is correct\n",
    "  - Calibration\n",
    "    - Weak calibration: higher confidences\n",
    "mean higher accuracy\n",
    "    - Strong calibration: confidence predicts\n",
    "accuracy rate\n",
    "    - What’s the value of calibration?\n",
    "\"\"\"\n",
    "slp_chapter4_10 = \"\"\"\n",
    "- representational harms: harms caused by a system that demeans a social group, for example by perpetuating\n",
    "negative stereotypes about them.\n",
    "- toxicity detection is the task of detecting hate speech, abuse, harassment, or other\n",
    "detection\n",
    "kinds of toxic language\n",
    "- toxicity classifiers can themselves cause harms.\n",
    "- flag under represented minority\n",
    "\"\"\"\n",
    "\n",
    "# Lecture notes \n",
    "module12_lecture_notes = \"\"\"\n",
    "- Naïve Bayes models can incorporate a variety of features, but\n",
    "tend to do best in homogeneous cases\n",
    "\"\"\"\n",
    "\n",
    "# Definitions\n",
    "module12_definitions = {\n",
    "\"concept\" : \"definition\",    \n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "KUc_LgzDvWf-",
    "outputId": "aa656846-b8e2-44bf-f033-c10114d2fc23"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "## WARNING: Your summary of 'AIMA Chapter 19.4 Model Selection and Optimization' is too long!\n",
       "# Module 12: Naive Bayes and Perceptrons\n",
       "## AIMA Chapter 19.1 Forms of Learning\n",
       "\n",
       "- agent is learning if it improves its performance after making observations about the world. \n",
       "- if agent is computer machine learning: a computer observes some data, builds a model based on the data, and uses the model as both a hypothesis about the world and a piece of software that  can solve problems. \n",
       "  - why learning, 1. the designers cannot anticipate all possible future  situations 2. sometimes the designers have no idea how  to program a solution themselves.\n",
       "- improvements:\n",
       "  - Which component is to be improved. \n",
       "  - What prior knowledge the agent has, which influences the model it builds. \n",
       "  - What data and feedback on that data is available. \n",
       "- components:\n",
       "  - A direct mapping from conditions on the current state to actions. \n",
       "  - A means to infer relevant properties of the world from the percept sequence. \n",
       "  - Information about the way the world evolves and about the results of possible actions  the agent can take. \n",
       "  - Utility information indicating the desirability of world states. \n",
       "  - Action-value information indicating the desirability of actions. \n",
       "  -  Goals that describe the most desirable states. \n",
       "  - A problem generator, critic, and learning element that enable the system\n",
       "- prior knowledge on the part of the agent: it starts from scratch  and learns from the data.\n",
       "- Going from a specific set of observations to a general rule is called induction;\n",
       "- factored representation—a  vector of attribute values.\n",
       "- types of Learning:\n",
       "  - In supervised learning the agent observes input-output pairs and learns a function that maps from input to output.\n",
       "  - In unsupervised learning the agent learns patterns in the input without any explicit Unsupervised  learning  feedback. The most common unsupervised learning task is clustering: detecting potentially useful clusters of input examples.\n",
       "  - In reinforcement learning the agent learns from a series of reinforcements: rewards and punishments.\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.2 Supervised Learning\n",
       "\n",
       "- Given a training set of N example input–output pairs, discover a function h that approximates the true function, h is called a hypothesis about the world. It is drawn from a hypothesis space H of possible functions.\n",
       "- h is a model of the data, drawn from a model  class H, or we can say a function drawn from a function class. We call the output yi the ground truth—the true answer we are asking our model to predict.\n",
       "- exploratory data analysis: examining Exploratory data  analysis  the data with statistical tests and visualizations\n",
       "- The true measure of a hypothesis is not how it does on the training set, but rather how  well it handles inputs it has not yet seen. on test set\n",
       "- say that h generalizes well if it accurately predicts the outputs of the test set. \n",
       "- By bias we mean (loosely) the tendency of a predictive hypothesis to deviate from the  expected value when averaged over different training sets.\n",
       "- We say that a hypothesis is underfitting when it fails to find a pattern in the data.\n",
       "- variance we mean the amount of change in the hypothesis due to fluctuation in the training data.\n",
       "- is overfitting the data when it pays too much attention to the particular data set it is trained on, causing it to perform poorly on unseen data. \n",
       "- bias–variance tradeoff: a choice between more complex, low-bias hypotheses that fit the training data well and simpler, low-variance hypotheses that may generalize better.\n",
       "- <strong>perceptron 1</strong>\n",
       "  - Inputs are feature values\n",
       "  - Each feature has a weight\n",
       "  - Sum is the activation\n",
       "  - $activation(x) = \\sum_i w_i\\cdot f_i(x)= w\\cdot f(x)$\n",
       "  - If the activation is:\n",
       "    - Positive, output +1\n",
       "    - Negative, output -1\n",
       "- Binary case: compare features to a weight vector\n",
       "- Learning: figure out the weight vector from examples\n",
       "  - Start with weights = 0\n",
       "  - For each training instance:\n",
       "    - Classify with current weights\n",
       "    - If correct (i.e., y=y*), no change!\n",
       "    - If wrong: adjust the weight vector by adding or subtracting the\n",
       "feature vector. Subtract if y* is -1. $w = w + \\hat y \\cdot f$\n",
       "- Multiclass Decision:\n",
       "  - A weight vector for each class: $w_y$\n",
       "  - Score (activation) of a class y: $w_y\\cdot f(x)$\n",
       "  - Prediction highest score wins: $\\hat y =  argmax_y w_y\\cdot f(x)$\n",
       "- Learning: Multiclass Perceptron\n",
       "  - Start with all weights = 0\n",
       "  - Pick up training examples one by one\n",
       "  - Predict with current weights: $\\hat y =  argmax_y w_y\\cdot f(x)$\n",
       "  - If correct, no change!\n",
       "  - If wrong: lower score of wrong\n",
       "answer, raise score of right answer: $w_y = w_y-f(x)$, $w_{y*} = w_{y*} + f(x)$\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.3 Learning Decision Trees\n",
       "\n",
       "- decision tree is a representation of a function that maps a vector of attribute values to Decision tree  a single output value—a “decision.” A decision tree reaches its decision by performing a  sequence of tests, starting at the root and following the appropriate branch until a leaf is  reached.\n",
       "- no way to represent all kinds of functions. Even just considering Boolean functions with n Boolean attributes, the truth table will have $2^n$ rows, so there are  $2^2^n$ different functions.\n",
       "- using the notion of information gain, which is defined in terms of entropy, which is the fundamental quantity in information theory\n",
       "- Entropy is a measure of the uncertainty of a random variable; the more information, the  less entropy.\n",
       "$H(V) = - \\sum_{k}P(v_k)\\log P(v_k)$\n",
       "- Info gain: \n",
       " infoGain = $IG(X) = H(Y)-H(Y|X) =  - \\sum_y P(Y=y)\\log_2 P(Y=y)-  \\sum_x P(X=x)H(Y|X=x) =  - \\sum_y P(Y=y)\\log_2 P(Y=y) + \\sum_{x,y} P(X=x,Y=y) \\log_2 P(Y=y|X=x) $\n",
       "- Overfitting becomes more likely as the number of attributes  grows, and less likely as we increase the number of training examples. Larger hypothesis  spaces (e.g., decision trees with more nodes or polynomials with high degree) have more capacity both to fit and to overfit; some model classes are more prone to overfitting than  others. \n",
       "- decision tree pruning combats overfitting.  works by eliminating nodes that are not clearly relevant.\n",
       "- The name CART, standing for Classification And Regression Trees, is used to cover both classes. \n",
       "- <strong>perceptron 2</strong>\n",
       "- Properties\n",
       "  - Separability: true if some parameters get the training\n",
       "set perfectly correct\n",
       "  - Convergence: if the training is separable, perceptron\n",
       "will eventually converge (binary case)\n",
       "  - Mistake Bound: the maximum number of mistakes\n",
       "(binary case) related to the margin or degree of\n",
       "separability: $mistake < k/\\delta^2$\n",
       "- Improving the perceptron:\n",
       "  - Noise: if the data isn’t separable,\n",
       "weights might thrash\n",
       "    - Averaging weight vectors over\n",
       "time can help (averaged\n",
       "perceptron)\n",
       "  - Mediocre generalization: finds a\n",
       "“barely” separating solution, not seperated in the center\n",
       "    - SVM\n",
       "  - Overtraining: test / held-out\n",
       "accuracy usually rises, then falls\n",
       "- Fixing perceptron:\n",
       "  - Idea: adjust the weight update to mitigate these effects\n",
       "- MIRA:\n",
       "  - Choose an update size that fixes the current mistake but, minimizes the change to w\n",
       "  - $\tau = (w'_y-w'_{y*}\\cdot f+1)/2f\\cdot f$, The +1 helps to generalize\n",
       "  - $w_y = w_y-\tau f(x)$, $w_{y*} = w_{y*} + \tau f(x)$\n",
       "- SVM:\n",
       "  - Maximizing the margin: good according to intuition, theory, practice\n",
       "  - Only support vectors matter; other training examples are ignorable\n",
       "  - Support vector machines (SVMs) find the separator with max\n",
       "margin\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.4 Model Selection and Optimization\n",
       "\n",
       "- will make the assumption that the future examples will be like the past. We call  this the stationarity assumption; without it, all bets are off.\n",
       "- We assume that each example Ej Stationarity  has the same prior probability distribution and is independent of the previous examples: are independent and identically distributed or i.i.d..\n",
       "- k-fold cross-validation. The idea is that each example serves double duty—as training data and validation data—but not at the same time. \n",
       "- is k = n, also  LOOCV known as leave-one-out cross-validation or LOOCV.\n",
       "- We can think of the task of finding a good hypothesis as two subtasks: model selection chooses a good hypothesis space, and optimization (also called training) finds the best hypothesis within that space. \n",
       "- We say that a model that exactly fits all the training data has interpolated the data Model classes typically start to overfit as the capacity approaches the point of interpolation. \n",
       "- to minimize a loss function rather than Loss function  maximize a utility function.\n",
       "- about fitting\n",
       "  - we say that a learning problem is realizable if the hypothesis space H actually  contains the true function\n",
       "  - variance  means that a learning algorithm will in general return different hypotheses for different sets  of examples.\n",
       "  - f may be nondeterministic or noisy—it may return  different values of f(x) for the same x.\n",
       "  - when H is a complicated function in a large hypothesis space,  it can be computationally intractable to systematically search all possibilities; in that case,  a search can explore part of the space and return a reasonably good hypothesis, but can’t  always guarantee the best one. \n",
       "- small-scale learning, where the number of training examples ranged from dozens to the low  thousands.\n",
       "- large-scale learning, with millions of  examples.\n",
       "- regularization: This process of explicitly penalizing complex hypotheses\n",
       "  - choices: the  loss function (L1 or L2), and the complexity measure, which is called a regularization function.\n",
       "  - A  process of feature selection can be performed to discard attributes that appear to be irrelevant.\n",
       "- grid search : try all combinations of Grid search  values and see which performs best on the validation data.\n",
       "- random search:  samples  uniformly from the set of all possible hyperparameter settings\n",
       "- Bayesian optimization treats the task of choosing good hyperparameter values as a  machine learning problem in itself.\n",
       "- If we make the assumption that f can be approximated  by a Gaussian process, then the math of updating our belief about f works out nicely.\n",
       "- <strong>perceptron 3</strong>\n",
       "- Non-Linear Separators\n",
       "  - mapping data to a higher-dimensional space\n",
       "  - the original feature space can always be mapped to some\n",
       "higher-dimensional feature space where the training set is separable:\n",
       "- Comparison:\n",
       "  - Naïve Bayes\n",
       "    - Builds a model training data\n",
       "    - Gives prediction probabilities\n",
       "    - Strong assumptions about feature independence\n",
       "    - One pass through data (counting)\n",
       "  - Perceptrons / MIRA:\n",
       "    - Makes less assumptions about data\n",
       "    - Mistake-driven learning\n",
       "    - Multiple passes through data (prediction)\n",
       "    - Often more accurate\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.5 The Theory of Learning\n",
       "\n",
       "- any hypothesis that is consistent with a sufficiently large set of training examples is  unlikely to be seriously wrong: that is, it must be probably approximately correct (PAC). \n",
       "- like all theorems, are logical consequences of axioms. When a  theorem (as opposed to, say, a political pundit) states something about the future based on  the past, the axioms have to provide the “juice” to make that connection.\n",
       "- A hypothesis h is called approximately correct if error(h) ≤ ǫ, where ǫ is a small constant. and error is error rate\n",
       "- decision lists. A  decision list consists of a series of tests, each of which is a conjunction of literals. If a  test succeeds when applied to an example description, the decision list specifies the value  to be returned. If the test fails, processing continues with the next test in the list.\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.6 Linear Regression and Classification\n",
       "\n",
       "- the class of linear functions of continuous-valued  inputs.\n",
       "- Many forms of learning involve adjusting weights to minimize a loss, so it helps to have a  mental picture of what’s going on in weight space\n",
       "- loss function is convex \n",
       "$w = \\frac{\\sum_i x_i y_i}{\\sum_i x_i^2}$\n",
       "- gradient descent:\n",
       "   - \n",
       "$w_i \\leftarrow w_i + \\alpha\\frac{\\partial}{\\partial w_i}Loss(w)$\n",
       "- batch gradient descent learning rule for univariate linear re- Batch gradient  descent  gression (also called deterministic gradient descent). The loss surface is convex. A  step that covers all the training examples is called an epoch.\n",
       "- A faster variant is called stochastic gradient descent or SGD: it randomly selects a small Stochastic gradient  descent  number of training examples at each step, and updates\n",
       "- multivariable linear regression\n",
       "  - \n",
       "$w* = (X^T X)^{-1}X^Ty$\n",
       "- regularization: Cost(h) = EmpLoss(h) +λComplexity(h). \n",
       "- L1 regularization has an important advantage: it tends to produce a sparse model. That is, it often sets many weights to zero\n",
       "- A decision boundary is a line (or a surface, in higher dimensions) that separates the  two classes. A linear decision boundary is called a linear separator and data that admit such a separator are called linearly separable.\n",
       "- percepton:\n",
       "  - If the output is correct (i.e., y=hw(x)) then the weights are not changed. \n",
       "  - If y is 1 but hw(x) is 0, then wi is increased when the corresponding input xi is positive  and decreased when xi is negative. This makes sense, because we want to make w · x  bigger so that hw(x) outputs a 1. \n",
       "  -  If y is 0 but hw(x) is 1, then wi is decreased when the corresponding input xi is positive  and increased when xi is negative. This makes sense, because we want to make w · x  smaller so that hw(x) outputs a 0. \n",
       "- logistic regression. There is no easy closed-form solution to find the optimal value of w with Logistic regression  this model, but the gradient descent computation is straightforward.\n",
       "- \n",
       "$\\frac{\\partial}{\\partial w_i}Loss(w) = -2(y-h_w(x)) \\times g'(w\\cdot x) \\times x_i$ <br />$w_i \\leftarrow w_i + \\alpha (y-h_w(x))h_w(x)(1-h_w(x))x_i$<br />$h_w(x) = \\frac{1}{1+e^{-w x}}$\n",
       "\n",
       "## SLP Chapter 4.1 Naive Bayes Classifiers\n",
       "\n",
       "- General Naive Bayes: $P(Y,F_1,...,F_n) = P(Y)\\prod_i P(F_i|Y)$\n",
       "- We only have to specify how each feature depends on the class\n",
       "Total number of parameters is linear in number of features\n",
       "Model is very simplistic, but often works anyway\n",
       "- multinomial naive Bayes classifier: so called because it is a Bayesian classifier that makes a simplifying (naive) assumption about how the features interact.\n",
       "- bag-of-words, that is, an unordered set of words with their position\n",
       "ignored, keeping only their frequency in the document\n",
       "- at notation ˆ to mean “our\n",
       "estimate of the correct class\n",
       "- $\\hat{c}=argmax_{c\\in C}P(c|d) = argmax_{c\\in C}P(d|c)P(c)/P(d) = argmax_{c\\in C}P(d|c)P(c) $ because $P(d)$ is constant\n",
       "- Naive Bayes is a probabilistic classifier, meaning that for a document d, out of\n",
       "all classes c 2 C the classifier returns the class hat(c) which has the maximum posterior probability given the document.\n",
       "- We call Naive Bayes a generative model because we can read equation stating\n",
       "a kind of implicit assumption about how a document is generated: first a class is\n",
       "sampled from P(c), and then the words are generated by sampling from P(d|c).\n",
       "- we compute the most probable class ˆ c given some\n",
       "document d by choosing the class which has the highest product of two probabilities:\n",
       "the prior probability of prior the class P(c) and the likelihood of the document P(d|c):\n",
       "- we can represent a document d as a set of features\n",
       "- $\\hat{c}= argmax_{c\\in C}P(f_1,f_2,...,f_n|c)P(c) $,\n",
       "- first, we assume\n",
       "that the features f1; f2; ...; fn only encode word identity and not position.\n",
       "- second, naive Bayes assumption: this is the condiassumption\n",
       "tional independence assumption that the probabilities P( fi|c) are independent given\n",
       "the class c and hence can be ‘naively’ multiplied as follows:\n",
       "- $P(f_1,f_2,...,f_n|c) = P(f_1|c)P(f_2|c)...P(f_n|c)$\n",
       "- $c_{NB} = argmax_{c\\in C}P(c)\\prod_{f\\in F}P(f|c)$\n",
       "- To apply the naive Bayes classifier to text, we need to consider word positions, by\n",
       "simply walking an index through every word position in the document:\n",
       "- $c_{NB} = argmax_{c\\in C}P(c)\\prod_{i \\in Positions}P(w_i|c)$\n",
       "- like calculations for language modeling, are done in log\n",
       "space, to avoid underflow and increase speed.\n",
       "- $c_{NB} = argmax_{c\\in C}log P(c)\\sum_{i \\in Positions}log P(w_i|c)$\n",
       "\n",
       "\n",
       "## SLP Chapter 4.2 Training the Naive Bayes Classifier\n",
       "\n",
       "-Let Nc be the number of documents in our training data with class c and Ndoc be the total number of documents\n",
       "- $\\hat{P}(c)=N_c/N_{doc}$\n",
       "- $\\hat{P}(w_i|c)=count(w_i,c)/\\sum_{w\\in V} count(w,c)$\n",
       "- Add laplace to avoid zeros\n",
       "- $\\hat{P}(w_i|c)=[count(w_i,c)+1]/\\sum_{w\\in V} [count(w,c)+1]$\n",
       "    - $= [count(w_i,c)+1]/[[\\sum_{w\\in V} count(w,c)]+V]$\n",
       "- processes:\n",
       "  - unknown words is to ignore them—remove them from the test\n",
       "document and not include any probability for them at all.\n",
       "  - ignore another class of words: stop\n",
       "stop words words, very frequent words like the and a\n",
       " \n",
       "\n",
       "## SLP Chapter 4.3 Worked Example\n",
       " Your summary here \n",
       "\n",
       "## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n",
       "\n",
       "- binary NB  uses the except\n",
       "that for each document we remove all duplicate words before concatenating them\n",
       "into the single big document.\n",
       "- addition commonly made when doing text classification for\n",
       "sentiment is to deal with negation. Consider the difference between I really like this\n",
       "movie (positive) and I didn’t like this movie (negative). The negation expressed by\n",
       "didn’t completely alters the inferences we draw from the predicate like. Similarly,\n",
       "negation can modify a negative word to produce a positive review\n",
       "- during text normalization, prepend the prefix NOT to\n",
       "every word after a token of logical negation (n’t, not, no, never) until the next punctuation\n",
       "mark.\n",
       "- NOT like, NOT recommend will thus occur more often\n",
       "in negative document and act as cues for negative sentiment, while words like\n",
       "NOT bored, NOT dismiss will acquire positive associations.\n",
       "- in some situations we might have insufficient labeled training data to\n",
       "train accurate naive Bayes classifiers using all words in the training set to estimate\n",
       "positive and negative sentiment\n",
       "- sentiment lexicons, lists of words that are preannotated with positive or negative sentiment.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n",
       "\n",
       "- For other tasks, like language id—determining what language a given piece of text is written in—the most effective naive Bayes features are not words at all,\n",
       "but character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,\n",
       "‘thei’), or, even simpler byte n-grams, where instead of using the multibyte Unicode\n",
       "character representations called codepoints, we just pretend everything is a string of\n",
       "raw bytes.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.6 Naive Bayes as a Language Model\n",
       "\n",
       "- a naive Bayes model can be viewed as a\n",
       "set of class-specific unigram language models, in which the model for each class\n",
       "instantiates a unigram language model.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n",
       "\n",
       "- gold labels: human labels\n",
       "- Precision measures the percentage of the items that\n",
       "the system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\n",
       "are positive according to the human gold labels).\n",
       "- $precision = tp/(tp+fp)$\n",
       "- Recall measures the percentage of items actually present in the input that were\n",
       "correctly identified by the system.\n",
       "- $recall = tp/(tp+fn)$\n",
       "- $F_1=2PR/(P+R)$\n",
       "- $F_b = (b^2+1)PR/(b^2P+R)$\n",
       "- Values of b > 1 favor recall, while\n",
       "values of b < 1 favor precision.\n",
       "- Harmonic mean is used because it is a conservative metric; the harmonic mean of\n",
       "two values is closer to the minimum of the two values than the arithmetic mean is.\n",
       "Thus it weighs the lower of the two numbers more heavily.\n",
       "- recall and precision for each row\n",
       "- In macroaveraging, we compute the performance\n",
       "microaveraging for each class, and then average over classes.\n",
       "- In microaveraging, we collect the decisions\n",
       "for all classes into a single confusion matrix, and then compute precision and\n",
       "recall from that table.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.8 Test sets and Cross-validation\n",
       "\n",
       "- effect size $\\delta(x) = M(A,x)-M(B,x)$, A is new B is before\n",
       "- That’s because A might just be accidentally better than B on this particular x.\n",
       "We need something more: we want to know if A’s superiority over B is likely to hold\n",
       "again if we checked another test set x0, or under some other set of circumstances.\n",
       "- suppose: $H_0:\\delta(x)\\leq 0$, $H_1: \\delta(x)>0$\n",
       "- P(\\delta(X)\\geq \\delta(x)|H_0 is true)\n",
       "- A very small p-value means that the difference we observed is very unlikely\n",
       "under the null hypothesis, and we can reject the null hypothesis. What counts as very\n",
       "small? It is common to use values like .05 or .01 as the thresholds. A value of .01\n",
       "means that if the p-value (the probability of observing the d we saw assuming H0 is\n",
       "true) is less than .01, we reject the null hypothesis and assume that A is indeed better\n",
       "than statistically B.\n",
       "- bootstrap test:\n",
       "- The intuition\n",
       "of the bootstrap test is that we can create many virtual test sets from an observed\n",
       "test set by repeatedly sampling from it. The method only makes the assumption that\n",
       "the sample is representative of the population.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.9 Statistical Significance Testing\n",
       "\n",
       "- Assuming H0 (A isn’t better than B), we would expect that d(X), estimated\n",
       "over many test sets, would be zero; a much higher value would be surprising,\n",
       "since H0 specifically assumes A isn’t better than B. To measure exactly how surprising\n",
       "is our observed d(x) we would in other circumstances compute the p-value by\n",
       "counting over many test sets how often d(x(i)) exceeds the expected zero value by\n",
       "d(x) or more:\n",
       "- $p-value(x) = 1/b \\sum_{i=1}^b 1 if (\\delta(x^{(i)})\\geq 2\\delta(x))$\n",
       "- this isn’t true for the bootstrapped test\n",
       "sets we created. That’s because we didn’t draw these samples from a distribution\n",
       "with 0 mean; we happened to create them from the original test set x, which happens\n",
       "to be biased (by .20) in favor of A. So to measure how surprising is our observed\n",
       "d(x), we actually compute the p-value by counting over many test sets how often\n",
       "d(x(i)) exceeds the expected value of d(x) by d(x) or more:\n",
       "- Confidence:\n",
       "  - The confidence of a probabilistic classifier:\n",
       "    - Posterior over the top label: $confidence(x) = max_y P(y|x)$\n",
       "    - Represents how sure the classifier is of\n",
       "the classification\n",
       "    - Any probabilistic model will have\n",
       "confidences\n",
       "    - No guarantee confidence is correct\n",
       "  - Calibration\n",
       "    - Weak calibration: higher confidences\n",
       "mean higher accuracy\n",
       "    - Strong calibration: confidence predicts\n",
       "accuracy rate\n",
       "    - What’s the value of calibration?\n",
       "\n",
       "\n",
       "## SLP Chapter 4.10 Avoiding Harms in Classification\n",
       "\n",
       "- representational harms: harms caused by a system that demeans a social group, for example by perpetuating\n",
       "negative stereotypes about them.\n",
       "- toxicity detection is the task of detecting hate speech, abuse, harassment, or other\n",
       "detection\n",
       "kinds of toxic language\n",
       "- toxicity classifiers can themselves cause harms.\n",
       "- flag under represented minority\n",
       "\n",
       "\n",
       "## Module 12 Lecture Notes\n",
       "\n",
       "- Naïve Bayes models can incorporate a variety of features, but\n",
       "tend to do best in homogeneous cases\n",
       "\n",
       "\n",
       "## Module 12 Definitions\n",
       "* **Concept** - definition\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module12_sections = [\n",
    "  (\"AIMA Chapter 19.1 Forms of Learning\", chapter19_1),\n",
    "  (\"AIMA Chapter 19.2 Supervised Learning\", chapter19_2),\n",
    "  (\"AIMA Chapter 19.3 Learning Decision Trees\", chapter19_3),\n",
    "  (\"AIMA Chapter 19.4 Model Selection and Optimization\", chapter19_4),\n",
    "  (\"AIMA Chapter 19.5 The Theory of Learning\", chapter19_5),\n",
    "  (\"AIMA Chapter 19.6 Linear Regression and Classification\", chapter19_6),\n",
    "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
    "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
    "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
    "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
    "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
    "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
    "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
    "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
    "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
    "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
    "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
    "]\n",
    "\n",
    "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
    "                         module12_sections, \n",
    "                         module12_definitions)\n",
    "display(Markdown(module12_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qVSQhTVSyYYO"
   },
   "source": [
    "# Module 13 - Neural Networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "id": "Wm4mfW82yaRH"
   },
   "outputs": [],
   "source": [
    "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
    "\n",
    "slp_chapter5_1 = \"\"\"\n",
    "- Machine learning classifiers require a training corpus of M observations\n",
    "input/output pairs (x(i),y(i)).\n",
    "  - A feature representation of the input. For each input observation\n",
    "x(i), this will be a vector of features [x1,x2,...,xn].\n",
    "  - A classification function that computes the estimated class !𝑦 via\n",
    "p(y|x).\n",
    "  - An objective function for learning, usually involving minimizing\n",
    "error on training examples.\n",
    "  - An algorithm for optimizing the objective function.\n",
    "- Logistic regression solves this task by learning, from a training set, a vector of\n",
    "weights and a bias term.\n",
    "- $z = w \\cdot x + b$\n",
    "- $\\sigma(z) = 1/(1+e^{-z})$\n",
    "- $P(y=1) = \\sigma(w\\cdot x+b)=1/(1+exp(-(w\\cdot x +b)))$\n",
    "- $P(y=0) = 1-\\sigma(w\\cdot x+b)=exp(-(w\\cdot x +b))/(1+exp(-(w\\cdot x +b)))$\n",
    "- $1-\\sigma(x) = \\sigma(-x)$\n",
    "- feature templates, abstract specifications of features. For example a bigram template\n",
    "templates\n",
    "for period disambiguation might create a feature for every pair of words that occurs\n",
    "before a period in the training set.\n",
    "- representation learning: ways to learn features automatically\n",
    "in an unsupervised way from the input\n",
    "- Logistic regression has a number of advantages over naive\n",
    "Bayes. Naive Bayes has overly strong conditional independence assumptions. Consider\n",
    "two features which are strongly correlated; in fact, imagine that we just add the\n",
    "same feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate,\n",
    "multiplying them both in, overestimating the evidence.\n",
    "- By contrast, logistic\n",
    "regression is much more robust to correlated features; if two features f1 and f2 are\n",
    "perfectly correlated, regression will simply assign part of the weight to w1 and part\n",
    "to w2. Thus when there are many correlated features, logistic regression will assign\n",
    "a more accurate probability than naive Bayes.\n",
    "- naive Bayes can work extremely well on small dataset, naive Bayes is\n",
    "easy to implement and very fast to train\n",
    "\"\"\"\n",
    "slp_chapter5_2 = \"\"\"\n",
    "- This requires two components that we foreshadowed in the introduction to the\n",
    "chapter. The first is a metric for how close the current label ( ˆ y) is to the true gold\n",
    "label y.\n",
    "-the distance between the system output and the gold output, and we call this distance the loss function or the cost function\n",
    "- The second thing we need is an optimization algorithm for iteratively updating\n",
    "the weights so as to minimize this loss function.\n",
    "\"\"\"\n",
    "slp_chapter5_3 = \"\"\"\n",
    "- We\n",
    "want the loss to be smaller if the model’s estimate is close to correct,\n",
    "and we want the loss to be bigger if it is confused.\n",
    "- $L(\\hat{y},y)$ = How much $\\hat{y}$ differs froma the true $y$\n",
    "- We do this via a loss function that prefers the correct class labels of the training\n",
    "examples to be more likely. This is called conditional maximum likelihood\n",
    "estimation: we choose the parameters w;b that maximize the log probability of\n",
    "the true y labels in the training data given the observations x.\n",
    "- The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.\n",
    "- $p(y|x)=\\hat{y}^y(1-\\hat{y}^{1-y})$\n",
    "- $log\\,p(y|x) = y\\,log\\,\\hat{y}+(1-y)\\,log(1-\\hat{y})$\n",
    "- cross entropy: $L_{CE}(\\hat{y},y)=-log\\,p(y|x) = -[y\\,log\\,\\hat{y}+(1-y)\\,log(1-\\hat{y})]$\n",
    "- $\\hat{y} = \\sigma(w\\cdot x + b)$\n",
    "- We\n",
    "want the loss to be smaller if the model’s estimate is close to correct, and bigger if\n",
    "the model is confused. minimize the loss function\n",
    "\"\"\"\n",
    "slp_chapter5_4 = \"\"\"\n",
    "- $\\hat{θ} = argmin_θ 1/m \\sum_{i=1}^m L_{CE}(f(x^{(i)};θ),y^{(i)})$\n",
    "- logistic regression, is conveniently convex. A convex function\n",
    "has just one minimum\n",
    "- $w^{t+1}=w^t-\\eta \\, d/dw \\, L(f(x;w),y)$\n",
    "- \n",
    "\"\"\" + r\"\"\"$\\frac{\\partial L_{CE}(\\hat{y},y)}{\\partial w_j}=[\\sigma(w\\cdot x+b)-y]x_j$\"\"\" + \"\"\"\n",
    "- The learning rate h is a hyperparameter that must be adjusted. If it’s too high,\n",
    "the learner will take steps that are too large, overshooting the minimum of the loss\n",
    "function. If it’s too low, the learner will take steps that are too small, and take too\n",
    "long to get to the minimum. It is common to start with a higher learning rate and then\n",
    "slowly decrease it, so that it is a function of the iteration k of training\n",
    "- Stochastic gradient descent is called stochastic because it chooses a single random\n",
    "example at a time, moving the weights so as to improve performance on that single\n",
    "example.\n",
    "- batch training we compute the gradient over the entire dataset.\n",
    "By seeing so many examples, batch training offers a superb estimate of which direction\n",
    "to move the weights, at the cost of spending a lot of time processing every\n",
    "single example in the training set to compute this perfect direction.\n",
    "- mini-batch training: we train on a group of m examples (perhaps\n",
    "512, or 1024) that is less than the whole dataset.\n",
    "- Mini-batch training also has the advantage of computational\n",
    "efficiency. The mini-batches can easily be vectorized, choosing the size of the minibatch\n",
    "based on the computational resources\n",
    "- mini-batch gradient = average of sum of individual gradient\n",
    "\"\"\"\n",
    "slp_chapter5_5 = \"\"\"\n",
    "- To avoid overfitting, a new regularization term R(q) is added to the objective\n",
    "function\n",
    "- $\\hat{θ} = argmax_θ \\sum_{i=1}^m log\\,P(y^i|x^i)-α R(θ)$\n",
    "- L2 regularization is a quadratic function of\n",
    "the weight values, named because it uses the (square of the) L2 norm of the weight\n",
    "values. The L2 norm, jjqjj2, is the same as the Euclidean distance of the vector q\n",
    "from the origin.\n",
    "- $\\hat{θ} = argmax_θ [\\sum_{i=1}^m log\\,P(y^i|x^i)]-α \\sum_{j=1}^n θ_j^2$\n",
    "- L1 regularization is a linear function of the weight values, named after the L1 norm, manhattan distance\n",
    "- But where L2 prefers weight vectors with many small weights, L1 prefers sparse\n",
    "solutions with some larger weights but many more weights set to zero.\n",
    "- Thus L1\n",
    "regularization leads to much sparser weight vectors, that is, far fewer features.\n",
    "- L1 regularization can be viewed as a Laplace\n",
    "prior on the weights. L2 regularization corresponds to assuming that weights are\n",
    "distributed according to a Gaussian distribution with mean m = 0.\n",
    "\"\"\"\n",
    "slp_chapter5_6 = \"\"\"\n",
    "- multinomial logistic regression, also called softmax regression\n",
    "- $softmax(z_i)=exp(z_i)/\\sum_{j=1}^k exp(z_j)$\n",
    "- $L_{CE}(\\hat y,y)=-\\sum_{k=1}^Ky_k\\,log\\,\\hat p(y=k|x)$\n",
    "\"\"\"\n",
    "\n",
    "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
    "\n",
    "slp_chapter7_1 = \"\"\"\n",
    "- sigmoid: $\\sigma(z) = 1/(1+e^{-z})$\n",
    "- tanh:$[e^z-e^{-z}]/[e^z+e^{-z}]$\n",
    "  - the tanh function\n",
    "has the nice properties of being smoothly differentiable and mapping outlier values toward the mean\n",
    "- ReLU: $y=max(z,0)$\n",
    "  - Rectifiers don’t vanish gradient, since the derivative of ReLU for high\n",
    "values of z is 1 rather than very close to 0.\n",
    "\"\"\"\n",
    "slp_chapter7_2 = \"\"\"\n",
    "- perceptron, which is a very simple neural\n",
    "unit that has a binary output and does not have a non-linear activation function\n",
    "- w1x1+w2x2+b = 0 is the equation of a line\n",
    "- XOR is not a linearly separable function\n",
    "\"\"\"\n",
    "slp_chapter7_3 = \"\"\"\n",
    "- The core of the neural network is the hidden layer h formed of hidden units hi,\n",
    "each of which is a neural unit\n",
    "- In the standard architecture, each layer is fully-connected, meaning that each unit in each layer takes as input the outputs\n",
    "from all the units in the previous layer, and there is a link between every pair of units\n",
    "from two adjacent layers.\n",
    "- The advantage of using a single matrix W for the weights of the entire layer is\n",
    "that now the hidden layer computation for a feedforward network can be done very\n",
    "efficiently with simple matrix operations.\n",
    "- So without non-linear activation functions,\n",
    "a multilayer network is just a notational variant of a single layer network with a\n",
    "different set of weights,\n",
    "- we lose all the representational power of multilayer\n",
    "networks\n",
    "- add a dummy node a0 to each layer whose value will\n",
    "always be 1 and a weight associated that is the bias\n",
    "\"\"\"\n",
    "slp_chapter7_4 = \"\"\"\n",
    "-adding this hidden layer to our logistic regression regression\n",
    "classifier allows the network to represent the non-linear interactions between\n",
    "features. This alone might give us a better sentiment classifier.\n",
    "- Instead of using\n",
    "hand-built human-engineered features as the input to our classifier, we draw on deep\n",
    "learning’s ability to learn features from the data by representing words as word2vec\n",
    "- embeddings as our input representation—\n",
    "and more generally the idea of relying on another algorithm to have already learned\n",
    "pretraining an embedding representation for our input words — is called pretraining.\n",
    "\"\"\"\n",
    "slp_chapter7_5 = \"\"\"\n",
    "- language modeling:\n",
    "predicting upcoming words from prior word context. \n",
    "- Compared to n-gram models, neural language models can handle\n",
    "much longer histories, can generalize better over contexts of similar words, and are\n",
    "more accurate at word-prediction\n",
    "- neural net language models\n",
    "are much more complex, slower to train, and less interpretable than n-gram models,\n",
    "so for many (especially smaller) tasks an n-gram language model is still the right\n",
    "tool.\n",
    "- Neural language models represent words in this prior context by their embeddings,\n",
    "rather than just by their word identity as used in n-gram language models. But a neural LM, knowing that “cat” and “dog” have similar embeddings, will\n",
    "be able to generalize from the “cat” context to assign a high enough probability to\n",
    "“fed” even after seeing “dog”.\n",
    "- forward inference or decoding for neural language models\n",
    "- A one-hot vector is\n",
    "a vector that has one element equal to 1—in the dimension corresponding to that\n",
    "word’s index in the vocabulary\n",
    "\"\"\"\n",
    "slp_chapter7_6 = \"\"\"\n",
    "- First, we’ll need a loss function that models the distance between the system\n",
    "output and the gold output, and it’s common to use the loss function used for logistic\n",
    "regression, the cross-entropy loss.\n",
    "- Second, to find the parameters that minimize this loss function, we’ll use the\n",
    "gradient descent optimization algorithm\n",
    "- gradient descent requires knowing the gradient of the loss function, the\n",
    "vector that contains the partial derivative of the loss function with respect to each\n",
    "of the parameters.(error backpropagation or\n",
    "backward differentiation.)\n",
    "- cross entropy loss$L_{CE} = -\\sum_{i=1}^C y_i \\, log\\, \\hat y_i$\n",
    "- Assume this is a hard classification task,\n",
    "meaning that only one class is the correct one, and that there is one output unit in y\n",
    "for each class.\n",
    "- $L_{CE} =- \\, log\\, \\hat y_i$, where y hat could be softmax\n",
    "- language model with softmax that gradient\n",
    "- $gradient = -(1 \\, (\\text{if} y =k) - [exp(w_k\\cdot x+b_k)/\\sum_j exp(w_j\\cdot x+b_j)])$\n",
    "- sigmoid: $d\\,\\sigma(z)/dz = \\sigma(z)(1-\\sigma(z))$\n",
    "- tanh: $d\\,tanh(z)/dz = 1- tanh^2(z)$\n",
    "- ReLU: $d\\, ReLU(z)/dz = 0 \\,\\text{for}\\, z<0; 1 \\,\\text{for}\\, z\\geq 0 $\n",
    "- For logistic regression we can initialize gradient descent with all the weights and\n",
    "biases having the value 0. In neural networks, by contrast, we need to initialize the\n",
    "weights with small random numbers. It’s also helpful to normalize the input values\n",
    "to have 0 mean and unit variance.\n",
    "- One of the most\n",
    "dropout important is dropout: randomly dropping some units and their connections from\n",
    "the network during training avoid overfitting\n",
    "- Tuning of hyperparameters is also important\n",
    "  - learning rate h, the mini-batch size, the model architecture\n",
    "\"\"\"\n",
    "slp_chapter7_7 = \"\"\"\n",
    "- For some tasks, it’s ok to freeze the embedding layer E with initial word2vec values\n",
    "- compute\n",
    "the initial embedding matrix E, and then hold it constant while we only modify\n",
    "W, U, and b\n",
    "- Generally training proceeds by taking as input a very long text, concatenating all\n",
    "the sentences, starting with random weights, and then iteratively moving through the\n",
    "text predicting each word wt .\n",
    "- $L_{CE} = - \\, log\\, \\hat y_i = -log\\, p(w_t|w_{t-1},...,w_{t-n+1})$\n",
    "- Training the parameters to minimize loss will result both in an algorithm for\n",
    "language modeling (a word predictor) but also a new set of embeddings E that can\n",
    "be used as word representations for other tasks.\n",
    "\"\"\"\n",
    "\n",
    "# Lecture notes \n",
    "module13_lecture_notes = \"\"\"\n",
    "- Like with logistic regression, we learn the best parameters for the neural net to make\n",
    "its predictions /𝑦 as close to possible as the gold standard labels in our training data y.\n",
    "We use:\n",
    "  - A cross entropy loss function\n",
    "  - Gradient descent to perform optimization\n",
    "  - Error back propagation to compute the gradient of the loss function\n",
    "- Since Neural Networks combine many perceptron-like neural units, we have many\n",
    "more parameters in the models (weights and biases). This allows us to train systems to\n",
    "perform even more sophisticated tasks.\n",
    "\"\"\"\n",
    "\n",
    "# Definitions\n",
    "module13_definitions = {\n",
    "\"concept\" : \"definition\",    \n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "kKWbCsbfxL3R",
    "outputId": "cb8eb284-307f-4f07-98d7-7f71bc8d9106"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 13: Neural Networks\n",
       "## SLP Chapter 5.1 Classification: the sigmoid\n",
       "\n",
       "- Machine learning classifiers require a training corpus of M observations\n",
       "input/output pairs (x(i),y(i)).\n",
       "  - A feature representation of the input. For each input observation\n",
       "x(i), this will be a vector of features [x1,x2,...,xn].\n",
       "  - A classification function that computes the estimated class !𝑦 via\n",
       "p(y|x).\n",
       "  - An objective function for learning, usually involving minimizing\n",
       "error on training examples.\n",
       "  - An algorithm for optimizing the objective function.\n",
       "- Logistic regression solves this task by learning, from a training set, a vector of\n",
       "weights and a bias term.\n",
       "- $z = w \\cdot x + b$\n",
       "- $\\sigma(z) = 1/(1+e^{-z})$\n",
       "- $P(y=1) = \\sigma(w\\cdot x+b)=1/(1+exp(-(w\\cdot x +b)))$\n",
       "- $P(y=0) = 1-\\sigma(w\\cdot x+b)=exp(-(w\\cdot x +b))/(1+exp(-(w\\cdot x +b)))$\n",
       "- $1-\\sigma(x) = \\sigma(-x)$\n",
       "- feature templates, abstract specifications of features. For example a bigram template\n",
       "templates\n",
       "for period disambiguation might create a feature for every pair of words that occurs\n",
       "before a period in the training set.\n",
       "- representation learning: ways to learn features automatically\n",
       "in an unsupervised way from the input\n",
       "- Logistic regression has a number of advantages over naive\n",
       "Bayes. Naive Bayes has overly strong conditional independence assumptions. Consider\n",
       "two features which are strongly correlated; in fact, imagine that we just add the\n",
       "same feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate,\n",
       "multiplying them both in, overestimating the evidence.\n",
       "- By contrast, logistic\n",
       "regression is much more robust to correlated features; if two features f1 and f2 are\n",
       "perfectly correlated, regression will simply assign part of the weight to w1 and part\n",
       "to w2. Thus when there are many correlated features, logistic regression will assign\n",
       "a more accurate probability than naive Bayes.\n",
       "- naive Bayes can work extremely well on small dataset, naive Bayes is\n",
       "easy to implement and very fast to train\n",
       "\n",
       "\n",
       "## SLP Chapter 5.2 Learning in Logistic Regression\n",
       "\n",
       "- This requires two components that we foreshadowed in the introduction to the\n",
       "chapter. The first is a metric for how close the current label ( ˆ y) is to the true gold\n",
       "label y.\n",
       "-the distance between the system output and the gold output, and we call this distance the loss function or the cost function\n",
       "- The second thing we need is an optimization algorithm for iteratively updating\n",
       "the weights so as to minimize this loss function.\n",
       "\n",
       "\n",
       "## SLP Chapter 5.3 The cross-entropy loss function\n",
       "\n",
       "- We\n",
       "want the loss to be smaller if the model’s estimate is close to correct,\n",
       "and we want the loss to be bigger if it is confused.\n",
       "- $L(\\hat{y},y)$ = How much $\\hat{y}$ differs froma the true $y$\n",
       "- We do this via a loss function that prefers the correct class labels of the training\n",
       "examples to be more likely. This is called conditional maximum likelihood\n",
       "estimation: we choose the parameters w;b that maximize the log probability of\n",
       "the true y labels in the training data given the observations x.\n",
       "- The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.\n",
       "- $p(y|x)=\\hat{y}^y(1-\\hat{y}^{1-y})$\n",
       "- $log\\,p(y|x) = y\\,log\\,\\hat{y}+(1-y)\\,log(1-\\hat{y})$\n",
       "- cross entropy: $L_{CE}(\\hat{y},y)=-log\\,p(y|x) = -[y\\,log\\,\\hat{y}+(1-y)\\,log(1-\\hat{y})]$\n",
       "- $\\hat{y} = \\sigma(w\\cdot x + b)$\n",
       "- We\n",
       "want the loss to be smaller if the model’s estimate is close to correct, and bigger if\n",
       "the model is confused. minimize the loss function\n",
       "\n",
       "\n",
       "## SLP Chapter 5.4 Gradient Descent\n",
       "\n",
       "- $\\hat{θ} = argmin_θ 1/m \\sum_{i=1}^m L_{CE}(f(x^{(i)};θ),y^{(i)})$\n",
       "- logistic regression, is conveniently convex. A convex function\n",
       "has just one minimum\n",
       "- $w^{t+1}=w^t-\\eta \\, d/dw \\, L(f(x;w),y)$\n",
       "- \n",
       "$\\frac{\\partial L_{CE}(\\hat{y},y)}{\\partial w_j}=[\\sigma(w\\cdot x+b)-y]x_j$\n",
       "- The learning rate h is a hyperparameter that must be adjusted. If it’s too high,\n",
       "the learner will take steps that are too large, overshooting the minimum of the loss\n",
       "function. If it’s too low, the learner will take steps that are too small, and take too\n",
       "long to get to the minimum. It is common to start with a higher learning rate and then\n",
       "slowly decrease it, so that it is a function of the iteration k of training\n",
       "- Stochastic gradient descent is called stochastic because it chooses a single random\n",
       "example at a time, moving the weights so as to improve performance on that single\n",
       "example.\n",
       "- batch training we compute the gradient over the entire dataset.\n",
       "By seeing so many examples, batch training offers a superb estimate of which direction\n",
       "to move the weights, at the cost of spending a lot of time processing every\n",
       "single example in the training set to compute this perfect direction.\n",
       "- mini-batch training: we train on a group of m examples (perhaps\n",
       "512, or 1024) that is less than the whole dataset.\n",
       "- Mini-batch training also has the advantage of computational\n",
       "efficiency. The mini-batches can easily be vectorized, choosing the size of the minibatch\n",
       "based on the computational resources\n",
       "- mini-batch gradient = average of sum of individual gradient\n",
       "\n",
       "\n",
       "## SLP Chapter 5.5 Regularization\n",
       "\n",
       "- To avoid overfitting, a new regularization term R(q) is added to the objective\n",
       "function\n",
       "- $\\hat{θ} = argmax_θ \\sum_{i=1}^m log\\,P(y^i|x^i)-α R(θ)$\n",
       "- L2 regularization is a quadratic function of\n",
       "the weight values, named because it uses the (square of the) L2 norm of the weight\n",
       "values. The L2 norm, jjqjj2, is the same as the Euclidean distance of the vector q\n",
       "from the origin.\n",
       "- $\\hat{θ} = argmax_θ [\\sum_{i=1}^m log\\,P(y^i|x^i)]-α \\sum_{j=1}^n θ_j^2$\n",
       "- L1 regularization is a linear function of the weight values, named after the L1 norm, manhattan distance\n",
       "- But where L2 prefers weight vectors with many small weights, L1 prefers sparse\n",
       "solutions with some larger weights but many more weights set to zero.\n",
       "- Thus L1\n",
       "regularization leads to much sparser weight vectors, that is, far fewer features.\n",
       "- L1 regularization can be viewed as a Laplace\n",
       "prior on the weights. L2 regularization corresponds to assuming that weights are\n",
       "distributed according to a Gaussian distribution with mean m = 0.\n",
       "\n",
       "\n",
       "## SLP Chapter 5.6 Multinomial logistic regression\n",
       "\n",
       "- multinomial logistic regression, also called softmax regression\n",
       "- $softmax(z_i)=exp(z_i)/\\sum_{j=1}^k exp(z_j)$\n",
       "- $L_{CE}(\\hat y,y)=-\\sum_{k=1}^Ky_k\\,log\\,\\hat p(y=k|x)$\n",
       "\n",
       "\n",
       "## SLP Chapter 7.1 Units\n",
       "\n",
       "- sigmoid: $\\sigma(z) = 1/(1+e^{-z})$\n",
       "- tanh:$[e^z-e^{-z}]/[e^z+e^{-z}]$\n",
       "  - the tanh function\n",
       "has the nice properties of being smoothly differentiable and mapping outlier values toward the mean\n",
       "- ReLU: $y=max(z,0)$\n",
       "  - Rectifiers don’t vanish gradient, since the derivative of ReLU for high\n",
       "values of z is 1 rather than very close to 0.\n",
       "\n",
       "\n",
       "## SLP Chapter 7.2 The XOR problem\n",
       "\n",
       "- perceptron, which is a very simple neural\n",
       "unit that has a binary output and does not have a non-linear activation function\n",
       "- w1x1+w2x2+b = 0 is the equation of a line\n",
       "- XOR is not a linearly separable function\n",
       "\n",
       "\n",
       "## SLP Chapter 7.3 Feedforward Neural Networks\n",
       "\n",
       "- The core of the neural network is the hidden layer h formed of hidden units hi,\n",
       "each of which is a neural unit\n",
       "- In the standard architecture, each layer is fully-connected, meaning that each unit in each layer takes as input the outputs\n",
       "from all the units in the previous layer, and there is a link between every pair of units\n",
       "from two adjacent layers.\n",
       "- The advantage of using a single matrix W for the weights of the entire layer is\n",
       "that now the hidden layer computation for a feedforward network can be done very\n",
       "efficiently with simple matrix operations.\n",
       "- So without non-linear activation functions,\n",
       "a multilayer network is just a notational variant of a single layer network with a\n",
       "different set of weights,\n",
       "- we lose all the representational power of multilayer\n",
       "networks\n",
       "- add a dummy node a0 to each layer whose value will\n",
       "always be 1 and a weight associated that is the bias\n",
       "\n",
       "\n",
       "## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n",
       "\n",
       "-adding this hidden layer to our logistic regression regression\n",
       "classifier allows the network to represent the non-linear interactions between\n",
       "features. This alone might give us a better sentiment classifier.\n",
       "- Instead of using\n",
       "hand-built human-engineered features as the input to our classifier, we draw on deep\n",
       "learning’s ability to learn features from the data by representing words as word2vec\n",
       "- embeddings as our input representation—\n",
       "and more generally the idea of relying on another algorithm to have already learned\n",
       "pretraining an embedding representation for our input words — is called pretraining.\n",
       "\n",
       "\n",
       "## SLP Chapter 7.5 Feedforward Neural Language Modeling\n",
       "\n",
       "- language modeling:\n",
       "predicting upcoming words from prior word context. \n",
       "- Compared to n-gram models, neural language models can handle\n",
       "much longer histories, can generalize better over contexts of similar words, and are\n",
       "more accurate at word-prediction\n",
       "- neural net language models\n",
       "are much more complex, slower to train, and less interpretable than n-gram models,\n",
       "so for many (especially smaller) tasks an n-gram language model is still the right\n",
       "tool.\n",
       "- Neural language models represent words in this prior context by their embeddings,\n",
       "rather than just by their word identity as used in n-gram language models. But a neural LM, knowing that “cat” and “dog” have similar embeddings, will\n",
       "be able to generalize from the “cat” context to assign a high enough probability to\n",
       "“fed” even after seeing “dog”.\n",
       "- forward inference or decoding for neural language models\n",
       "- A one-hot vector is\n",
       "a vector that has one element equal to 1—in the dimension corresponding to that\n",
       "word’s index in the vocabulary\n",
       "\n",
       "\n",
       "## SLP Chapter 7.6 Training Neural Nets\n",
       "\n",
       "- First, we’ll need a loss function that models the distance between the system\n",
       "output and the gold output, and it’s common to use the loss function used for logistic\n",
       "regression, the cross-entropy loss.\n",
       "- Second, to find the parameters that minimize this loss function, we’ll use the\n",
       "gradient descent optimization algorithm\n",
       "- gradient descent requires knowing the gradient of the loss function, the\n",
       "vector that contains the partial derivative of the loss function with respect to each\n",
       "of the parameters.(error backpropagation or\n",
       "backward differentiation.)\n",
       "- cross entropy loss$L_{CE} = -\\sum_{i=1}^C y_i \\, log\\, \\hat y_i$\n",
       "- Assume this is a hard classification task,\n",
       "meaning that only one class is the correct one, and that there is one output unit in y\n",
       "for each class.\n",
       "- $L_{CE} =- \\, log\\, \\hat y_i$, where y hat could be softmax\n",
       "- language model with softmax that gradient\n",
       "- $gradient = -(1 \\, (\text{if} y =k) - [exp(w_k\\cdot x+b_k)/\\sum_j exp(w_j\\cdot x+b_j)])$\n",
       "- sigmoid: $d\\,\\sigma(z)/dz = \\sigma(z)(1-\\sigma(z))$\n",
       "- tanh: $d\\,tanh(z)/dz = 1- tanh^2(z)$\n",
       "- ReLU: $d\\, ReLU(z)/dz = 0 \\,\text{for}\\, z<0; 1 \\,\text{for}\\, z\\geq 0 $\n",
       "- For logistic regression we can initialize gradient descent with all the weights and\n",
       "biases having the value 0. In neural networks, by contrast, we need to initialize the\n",
       "weights with small random numbers. It’s also helpful to normalize the input values\n",
       "to have 0 mean and unit variance.\n",
       "- One of the most\n",
       "dropout important is dropout: randomly dropping some units and their connections from\n",
       "the network during training avoid overfitting\n",
       "- Tuning of hyperparameters is also important\n",
       "  - learning rate h, the mini-batch size, the model architecture\n",
       "\n",
       "\n",
       "## SLP Chapter 7.7 Training the neural language model\n",
       "\n",
       "- For some tasks, it’s ok to freeze the embedding layer E with initial word2vec values\n",
       "- compute\n",
       "the initial embedding matrix E, and then hold it constant while we only modify\n",
       "W, U, and b\n",
       "- Generally training proceeds by taking as input a very long text, concatenating all\n",
       "the sentences, starting with random weights, and then iteratively moving through the\n",
       "text predicting each word wt .\n",
       "- $L_{CE} = - \\, log\\, \\hat y_i = -log\\, p(w_t|w_{t-1},...,w_{t-n+1})$\n",
       "- Training the parameters to minimize loss will result both in an algorithm for\n",
       "language modeling (a word predictor) but also a new set of embeddings E that can\n",
       "be used as word representations for other tasks.\n",
       "\n",
       "\n",
       "## Module 13 Lecture Notes\n",
       "\n",
       "- Like with logistic regression, we learn the best parameters for the neural net to make\n",
       "its predictions /𝑦 as close to possible as the gold standard labels in our training data y.\n",
       "We use:\n",
       "  - A cross entropy loss function\n",
       "  - Gradient descent to perform optimization\n",
       "  - Error back propagation to compute the gradient of the loss function\n",
       "- Since Neural Networks combine many perceptron-like neural units, we have many\n",
       "more parameters in the models (weights and biases). This allows us to train systems to\n",
       "perform even more sophisticated tasks.\n",
       "\n",
       "\n",
       "## Module 13 Definitions\n",
       "* **Concept** - definition\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module13_sections = [\n",
    "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
    "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
    "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
    "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
    "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
    "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
    "\n",
    "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
    "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
    "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
    "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
    "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
    "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
    "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
    "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
    "]\n",
    "\n",
    "module13_markdown = format_module(13, \"Neural Networks\", \n",
    "                         module13_sections, \n",
    "                         module13_definitions)\n",
    "display(Markdown(module13_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RYESi2kMybL8"
   },
   "source": [
    "# Module 14 - Natural Language Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "id": "qIRJ4YYCzNHL"
   },
   "outputs": [],
   "source": [
    "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
    "\n",
    "slp_chapter9_1 = \"\"\"\n",
    "- <strong>Language Model 1</strong>\n",
    "- <strong>====================</strong>\n",
    "- Goal: Learn a function that returns the joint probability\n",
    "  - There are too many parameters to accurately estimate. This is\n",
    "sometimes called the “curse of dimensionality”\n",
    "  - In n-gram-based models we fail to generalize to related words / word\n",
    "sequences that we have observed. \n",
    "- Curse of dimensionality / sparse statistics：\n",
    "  - Suppose we want a joint distribution over 10 words.\n",
    "  - Suppose we have a vocabulary of size 100,000. This is too high to estimate from data.\n",
    "-  One way: The probability of the next word only depends on the previous n-1\n",
    "words. A small n makes it easier for us to get an estimate of the probability\n",
    "from data.\n",
    "- Neural Language models have several advantages over n-gram LMs:\n",
    "  - They don’t need smoothing\n",
    "  - They can handle much longer histories.\n",
    "  - They can generalize over contexts of similar words.\n",
    "  - Neural LMs tend to have much higher predictive accuracy than n-gram LMs.\n",
    "- Disadvantage: slower to train than traditional n-gram LMs\n",
    "- Associate each word in the vocabulary with a vector-representation, thereby\n",
    "creating a notion of similarity between words.\n",
    "- Express the joint probability function of a word sequence in terms of the word\n",
    "vectors for the words in that sequence.\n",
    "- Simultaneously learn the word vectors and the parameters of the function.\n",
    "The word vectors are low-dimensional (d=30 to d=100) dense vectors, like we’ve seen\n",
    "before.\n",
    "- The probability function is expressed the product of conditional probabilities of the\n",
    "next word given the previous word, using a multi-layer, feed forward neural network.\n",
    "\"\"\"\n",
    "slp_chapter9_2 = \"\"\"\n",
    "-The key\n",
    "difference from a feedforward network lies in the recurrent link shown in the figure\n",
    "with the dashed line. This link augments the input to the computation at the hidden\n",
    "layer with the value of the hidden layer from the preceding point in time.\n",
    "- The hidden layer from the previous time step provides a form of memory, or\n",
    "context, that encodes earlier processing and informs the decisions to be made at\n",
    "later points in time.\n",
    "- appear to be more complex than\n",
    "non-recurrent architectures. But in reality, they’re not all that different.\n",
    "- RNN is nearly identical to what we’ve already seen with feedforward networks. To\n",
    "compute an output yt for an input xt , we need the activation value for the hidden\n",
    "layer ht .\n",
    "- To calculate this, we multiply the input xt with the weight matrix W, and\n",
    "the hidden layer from the previous time step ht􀀀1 with the weight matrix U. We\n",
    "add these values together and pass them through a suitable activation function, g,\n",
    "to arrive at the activation value for the current hidden layer, ht . and proceed with the usual computation to generate the\n",
    "output vector.\n",
    "- The sequential nature of simple\n",
    "recurrent networks can also be seen by unrolling the network in time\n",
    "- First, to compute the loss function for\n",
    "the output at time t we need the hidden layer from time t 􀀀1. Second, the hidden\n",
    "layer at time t influences both the output at time t and the hidden layer at time t +1\n",
    "(and hence the output and loss at t +1).\n",
    "\"\"\"\n",
    "slp_chapter9_3 = \"\"\"\n",
    "- RNN language models (Mikolov et al., 2010) process the input sequence one word at\n",
    "a time, attempting to predict the next word from the current word and the previous\n",
    "hidden state.\n",
    "- RNNs don’t have the limited context problem that n-gram models\n",
    "have, since the hidden state can in principle represent information about all of the\n",
    "preceding words all the way back to the beginning of the sequence.\n",
    "- To train an RNN as a language model, we use a corpus of text as training material,\n",
    "having the model predict the next word at each time step t.\n",
    "- We train the model to\n",
    "minimize the error in predicting the true next word in the training sequence, using\n",
    "cross-entropy as the loss function.\n",
    "- cross-entropy loss measures the\n",
    "difference between a predicted probability distribution and the correct distribution.\n",
    "- Weight tying is a method that\n",
    "dispenses with this redundancy and uses a single set of embeddings at the input and\n",
    "softmax layers. That is, E = V. To do this, we set the dimensionality of the final\n",
    "hidden layer to be the same dh, (or add an additional projection layer to do the\n",
    "same thing), and simply use the same matrix for both layers.\n",
    "- Advantages of NN LMs:\n",
    "  - Better results. They achieve better preplexity scores than SOTA n-gram LMs.\n",
    "  - Larger N. NN LMs can scale to much larger orders of n. This is achievable because\n",
    "parameters are associated only with individual words, and not with n-grams.\n",
    "  - They generalize across contexts. \n",
    "- A by-product of training are word embeddings\n",
    "\"\"\"\n",
    "\n",
    "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
    "\n",
    "slp_chapter6_1 = \"\"\"\n",
    "- summarize some of these desiderata, drawing on results in the\n",
    "lexical linguistic study of word meaning, which is called lexical semantics\n",
    "- Lemmas and Senses\n",
    "  - form mouse is the lemma, also called the citation form. The form mouse would also be the lemma for the word mice;\n",
    "  - The specific forms sung or carpets or sing or duermes are called wordforms\n",
    "  - As the example above shows, each lemma can have multiple meanings; the\n",
    "lemma mouse can refer to the rodent or the cursor control device.\n",
    "  - each\n",
    "of these aspects of the meaning of mouse a word sense\n",
    "- Synonymy\n",
    "  - when one word has a sense whose meaning is\n",
    "identical to a sense of another word, or nearly identical, we say the two senses of\n",
    "those two words are synonyms.\n",
    "  - called the principle of contrast, word H2O is used in scientific contexts and\n",
    "would be inappropriate in a hiking guide\n",
    "- Word Similarity\n",
    "  - The notion of word similarity is very useful in larger semantic tasks. Knowing\n",
    "how similar two words are can help in computing how similar the meaning of two\n",
    "phrases or sentences are\n",
    "- Word relatedness\n",
    "  - The meaning of two words can be related in ways other than similarity. One such class of connections is called word relatedness, or word association in psychology.\n",
    "    - meanings of the words coffee and cup.\n",
    "  - One common kind of relatedness between words is if they belong to the same\n",
    "semantic field semantic field. A semantic field is a set of words which cover a particular semantic\n",
    "domain and bear structured relations with each other\n",
    "  - Semantic fields are also related to topic models, like Latent\n",
    "Dirichlet Allocation, LDA, which apply unsupervised learning on large sets of texts\n",
    "to induce sets of associated words from text.\n",
    "  - useful tools for discovering topical structure in documents\n",
    "- Semantic Frames and Roles\n",
    "  - A semantic frame is a set of words that denote perspectives or\n",
    "participants in a particular type of event.\n",
    "    - e.g. commercial transaction can be encoded with buy, sell, pay\n",
    "- Connotation\n",
    "  - words have affective meanings or connotations\n",
    "  - the aspects of a word’s meaning that are related to a writer or reader’s emotions,\n",
    "sentiment, opinions, or evaluations\n",
    "  - Positive or negative evaluation language is called sentiment,\n",
    "- Support for entailment\n",
    "  - $dog \\subset animal$\n",
    "\"\"\"\n",
    "slp_chapter6_2 = \"\"\"\n",
    "- Vectors semantics is the standard way to represent word meaning in NLP, helping us model many of the aspects of word meaning\n",
    "- Vectors for representing words are called embeddings\n",
    "- classifiers can assign sentiment as long as\n",
    "it sees some words with similar meanings. And as we’ll see, vector semantic models\n",
    "can be learned automatically from text without supervision.\n",
    "- We will see that this method results in very long\n",
    "vectors that are sparse\n",
    "- word2vec model family for constructing\n",
    "short, dense vectors that have useful semantic properties\n",
    " \"\"\"\n",
    "slp_chapter6_3 = \"\"\"\n",
    "- Vector or distributional models of meaning are generally based on a co-occurrence\n",
    "matrix, a way of representing how often words co-occur. We’ll look at two popular\n",
    "matrices: the term-document matrix and the term-term matrix.\n",
    "- term-document matrix, each row represents a word in the vocabulary and each\n",
    "column represents a document from some collection of documents, Each cell in this matrix represents the number of times\n",
    "a particular word, occurs in a particular document\n",
    "- The term-document matrix of was first defined as part of the vector space model of information retrieval\n",
    "- in real term-document matrices, the vectors representing\n",
    "each document would have dimensionality jVj, the vocabulary size.\n",
    "- Term-document matrices were originally defined as a means of finding similar\n",
    "documents for the task of document information retrieval. Two documents that are similar will tend to have similar words\n",
    "- Information retrieval (IR) is the task of finding the document d from the D\n",
    "documents in some collection that best matches a query q.\n",
    "- we saw that similar documents had similar vectors, because similar\n",
    "documents tend to have similar words. This same principle applies to words:\n",
    "similar words have similar vectors because they tend to occur in similar documents.\n",
    "The term-document matrix thus lets us represent the meaning of a word by the documents\n",
    "it tends to occur in.\n",
    "- term-term matrix, also called the word-word matrix or the term-context matrix, in which the columns are labeled by words rather\n",
    "than documents. the number of times the row (target) word and the column (context) word co-occur\n",
    "in some context in some training corpus.\n",
    "- This matrix is thus of dimensionality |V|*|V| and each cell records, the cell represents the number of times the two words appear in the same\n",
    "document. \n",
    "- (PPT)The value in a cell indicates how\n",
    "often a word appears in a\n",
    "context window surrounding\n",
    "another\n",
    "- TF-IDF\n",
    "  - Term frequency * inverse document\n",
    "frequency \n",
    "  - How often a\n",
    "word occurred in\n",
    "a document * 1 over the number\n",
    "of documents that it\n",
    "occurred in\n",
    "- Context windows\n",
    "w-2, w-1 target_word w+1 w+2\n",
    "- Since most of these numbers are zero these are sparse vector representations;\n",
    "there are efficient algorithms for storing and computing with sparse matrices.\n",
    "- Alternates to counts\n",
    "    - Raw word frequency is not a great measure of association between words. It’s very\n",
    "skewed “the” and “of” are very frequent, but maybe not the most discriminative\n",
    "We’d rather have a measure that asks whether a context word is particularly\n",
    "informative about the target word.\n",
    "\"\"\"\n",
    "slp_chapter6_4 = \"\"\"\n",
    "- L2 Normalization: $||u|| = sqrt(\\sum_i^n u_i^2)$, divide each vector by the normalization term, compare each row in term-document matrix\n",
    "- We modify the dot product to normalize for the vector length by dividing the\n",
    "dot product by the lengths of each of the two vectors. This normalized dot product\n",
    "turns out to be the same as the cosine of the angle between the two vectors, following\n",
    "from the definition of the dot product between two vectors a and b:\n",
    "- $cosine(v,w) = (v\\cdot w)/(|v||w|)$, L2-norm\n",
    "- closer the semantic, smaller the angle\n",
    "\"\"\"\n",
    "slp_chapter6_8 = \"\"\"\n",
    "- embeddings, short dense\n",
    "vectors.\n",
    "- embeddings are short, with number\n",
    "of dimensions d ranging from 50-1000, rather than the much larger vocabulary size\n",
    "|V| or number of documents D we’ve seen\n",
    "- vectors are dense: instead of vector entries being sparse,\n",
    "mostly-zero counts or functions of counts, the values will be real-valued numbers\n",
    "that can be negative.\n",
    "- Representing words as 300-dimensional dense vectors requires our classifiers\n",
    "to learn far fewer weights than if we represented words as 50,000-dimensional vectors,\n",
    "and the smaller parameter space possibly helps with generalization and avoiding\n",
    "overfitting.\n",
    "- skip-gram with negative sampling, sometimes called SGNS.\n",
    "- The word2vec methods are fast, efficient to train, and easily available online\n",
    "with code and pretrained embeddings.\n",
    "- Word2vec embeddings are static embeddings, meaning that the method learns one fixed embedding for each word in the vocabulary.\n",
    "- The intuition of word2vec is that instead of counting how often each word w occurs\n",
    "near, say, apricot, we’ll instead train a classifier on a binary prediction task: “Is\n",
    "word w likely to show up near apricot?” We don’t actually care about this prediction\n",
    "task; instead we’ll take the learned classifier weights as the word embeddings.\n",
    "- self-supervision, avoids the need for\n",
    "any sort of hand-labeled supervision signal\n",
    "- assume we’re\n",
    "using a window of +/-2 context words\n",
    "- ... lemon, a [tablespoon of apricot jam, a] pinch ... w = apricot, c1...c4 = tablespoon of jam a\n",
    "- train a classifier such that, given a tuple (w;c) of a target word\n",
    "w paired with a candidate context word c $P(+|w,c)$\n",
    "- $P(-|w|c) = 1-P(+|w,c)$\n",
    "- intuition of the skipgram\n",
    "model is to base this probability on embedding similarity: a word is likely to\n",
    "occur near the target if its embedding vector is similar to the target embedding\n",
    "- compute similarity between these dense embeddings, we rely on the intuition that\n",
    "two vectors are similar if they have a high dot product, similarity = $c\\cdot w$\n",
    "- $P(+|w,c)=\\sigma(c\\cdot w)=1/(1+exp(-c\\cdot w))$\n",
    "- $P(-|w,c)=\\sigma(-c\\cdot w)=1/(1+exp(c\\cdot w))$\n",
    "- $P(+|w,c_{1:L})=\\prod_{i=1}^L\\sigma(c_i\\cdot w)$\n",
    "- $log\\,P(+|w,c_{1:L})=\\sum_{i=1}^L\\log\\,\\sigma(c_i\\cdot w)$\n",
    "- In summary, skip-gram trains a probabilistic classifier that, given a test target word\n",
    "w and its context window of L words c1:L, assigns a probability based on how similar\n",
    "this context window is to the target word. The probability is based on applying the\n",
    "logistic (sigmoid) function to the dot product of the embeddings of the target word\n",
    "with each context word. To compute this probability, we just need embeddings for\n",
    "each target word and context word in the vocabulary.\n",
    "- Skip-gram actually\n",
    "stores two embeddings for each word, one for the word as a target, and one for the word considered as context. Thus the parameters we need to learn are two matrices\n",
    "W and C, each containing an embedding for every one of the jVj words in the\n",
    "vocabulary V\n",
    "\"\"\"\n",
    "slp_chapter6_10 = \"\"\"\n",
    "- <strong>cont.</strong>\n",
    "- Given the set of positive and negative training instances, and an initial set of embeddings,\n",
    "the goal of the learning algorithm is to adjust those embeddings to\n",
    "  - Maximize the similarity of the target word, context word pairs (w;cpos) drawn\n",
    "from the positive examples\n",
    "  - Minimize the similarity of the (w;cneg) pairs from the negative examples.\n",
    "- want to maximize the dot product of the word with the actual context\n",
    "words, and minimize the dot products of the word with the k negative sampled nonneighbor\n",
    "words.\n",
    "- We minimize this loss function using stochastic gradient descent\n",
    "- word2vec maximizes: $\\sum_{(t,c)\\in +}log\\, P(+|t,c)+\\sum_{(t,c)\\in -}log\\, P(-|t,c)$, Maximize the + label for the pairs from the\n",
    "positive training data, and the – label for the\n",
    "pairs sample from the negative data.\n",
    "- <strong>end</strong>\n",
    "- Different types of similarity or association:\n",
    "  - Shorter context windows\n",
    "tend to lead to representations that are a bit more syntactic, since the information is\n",
    "coming from immediately nearby words. When the vectors are computed from short\n",
    "context windows, the most similar words to a target word w tend to be semantically\n",
    "similar words with the same parts of speech. When vectors are computed from long\n",
    "context windows, the highest cosine words to a target word w tend to be words that\n",
    "are topically related but not similar.\n",
    "  - Two words have first-order co-occurrence if they are typically nearby each other.\n",
    "Thus wrote is a first-order associate of book or poem.\n",
    "  - Two words have second-order\n",
    "co-occurrence if they have similar neighbors. Thus wrote is a second-order associate of words like said or remarked.\n",
    "- Analogy/Relational Similarity:\n",
    "  - scholars showed that sparse vector models of meaning could solve such analogy problems but the parallelogram method received more modern attention because of its success\n",
    "with word2vec or GloVe vectors\n",
    "  - problems:\n",
    "    - the closest value returned by the parallelogram\n",
    "algorithm in word2vec or GloVe embedding spaces is usually not in fact\n",
    "b* but one of the 3 input words or their morphological variants\n",
    "    - spaces perform well if the task involves\n",
    "frequent words, small distances, and certain relations the parallelogram method\n",
    "with embeddings doesn’t work as well for other relations\n",
    "- Embeddings can also be a useful tool for studying how meaning changes over time,\n",
    "by computing multiple embedding spaces, each from texts written in a particular\n",
    "time period.\n",
    "\"\"\"\n",
    "slp_chapter6_11 = \"\"\"\n",
    "- allocational harm, when a system allocates resources unfairly to different groups \n",
    "  - algorithm hire programmers would fownweight female\n",
    "- amplifies bias\n",
    "  - gendered terms become more gendered in embedding space\n",
    "- representational harm: a harm caused by a system demeaning or even ignoring some social groups\n",
    "- although these sorts of debiasing may reduce bias in embeddings, they do not eliminate it\n",
    "\"\"\"\n",
    "slp_chapter6_12 = \"\"\"\n",
    "- <strong>Language Model 1</strong>\n",
    "- <strong>====================</strong>\n",
    "- Dense vector:\n",
    "  - Treat the target word and a neighboring context word as positive examples.\n",
    "  - Randomly sample other words in the lexicon to get negative samples.\n",
    "  - Use logistic regression to train a classifier to distinguish those two cases.\n",
    "  - Use the weights as the embeddings.\n",
    "- Sparse vs. dense vectors:\n",
    "  - Short vectors may be easier to use as features in machine learning (fewer\n",
    "weights to tune)\n",
    "  - Dense vectors may generalize better than storing explicit counts\n",
    "  - They may do better at capturing synonymy\n",
    "  - simply works better\n",
    "- Advantages of word2vec:\n",
    "  - Fast, easy to train (much faster than SVD)\n",
    "  - Available online in the word2vec package\n",
    "  Including sets of pretrained embeddings!\n",
    "- word2vec self-supervised:\n",
    "  - we need embeddings for t and c to do it.\n",
    "  - It starts with an initial set of embedding vectors and then iteratively shifts\n",
    "the embedding of each word w to be more like the embeddings of words\n",
    "that occur nearby in texts, and less like the embeddings of words that\n",
    "don’t occur nearby.\n",
    "- learn word2vec (skip-gram) embeddings：\n",
    "  - Start with V random 300-dimensional vectors as initial\n",
    "embeddings\n",
    "  - Use logistic regression\n",
    "    - Take a corpus and take pairs of words that co-occur as positive\n",
    "examples\n",
    "    - Take pairs of words that don't co-occur as negative examples\n",
    "    - Train the classifier to distinguish these by slowly adjusting all the\n",
    "embeddings to improve the classifier performance\n",
    "    - Throw away the classifier code and keep the embeddings.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module14_lecture_notes = \"\"\"\n",
    "- Embeddings = vector models of meaning\n",
    "  - More fine-grained than just a string or index\n",
    "  - Especially good at modeling similarity/analogy\n",
    "    - Just download them and use cosines!!\n",
    "  - Can use sparse models (tf-idf) or dense models (word2vec, GLoVE)\n",
    "- Useful in practice but know they encode cultural stereotypes\n",
    "\"\"\"\n",
    "\n",
    "# Definitions\n",
    "module14_definitions = {\n",
    "\"concept\" : \"definition\",    \n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "lXNcHuKjyNXa",
    "outputId": "99afa239-22bb-4338-993b-c147b8886db8"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 14: Natural Language Processing\n",
       "## SLP Chapter 9.1 Language Models Revisited\n",
       "\n",
       "- <strong>Language Model 1</strong>\n",
       "- <strong>====================</strong>\n",
       "- Goal: Learn a function that returns the joint probability\n",
       "  - There are too many parameters to accurately estimate. This is\n",
       "sometimes called the “curse of dimensionality”\n",
       "  - In n-gram-based models we fail to generalize to related words / word\n",
       "sequences that we have observed. \n",
       "- Curse of dimensionality / sparse statistics：\n",
       "  - Suppose we want a joint distribution over 10 words.\n",
       "  - Suppose we have a vocabulary of size 100,000. This is too high to estimate from data.\n",
       "-  One way: The probability of the next word only depends on the previous n-1\n",
       "words. A small n makes it easier for us to get an estimate of the probability\n",
       "from data.\n",
       "- Neural Language models have several advantages over n-gram LMs:\n",
       "  - They don’t need smoothing\n",
       "  - They can handle much longer histories.\n",
       "  - They can generalize over contexts of similar words.\n",
       "  - Neural LMs tend to have much higher predictive accuracy than n-gram LMs.\n",
       "- Disadvantage: slower to train than traditional n-gram LMs\n",
       "- Associate each word in the vocabulary with a vector-representation, thereby\n",
       "creating a notion of similarity between words.\n",
       "- Express the joint probability function of a word sequence in terms of the word\n",
       "vectors for the words in that sequence.\n",
       "- Simultaneously learn the word vectors and the parameters of the function.\n",
       "The word vectors are low-dimensional (d=30 to d=100) dense vectors, like we’ve seen\n",
       "before.\n",
       "- The probability function is expressed the product of conditional probabilities of the\n",
       "next word given the previous word, using a multi-layer, feed forward neural network.\n",
       "\n",
       "\n",
       "## SLP Chapter 9.2 Recurrent Neural Networks\n",
       "\n",
       "-The key\n",
       "difference from a feedforward network lies in the recurrent link shown in the figure\n",
       "with the dashed line. This link augments the input to the computation at the hidden\n",
       "layer with the value of the hidden layer from the preceding point in time.\n",
       "- The hidden layer from the previous time step provides a form of memory, or\n",
       "context, that encodes earlier processing and informs the decisions to be made at\n",
       "later points in time.\n",
       "- appear to be more complex than\n",
       "non-recurrent architectures. But in reality, they’re not all that different.\n",
       "- RNN is nearly identical to what we’ve already seen with feedforward networks. To\n",
       "compute an output yt for an input xt , we need the activation value for the hidden\n",
       "layer ht .\n",
       "- To calculate this, we multiply the input xt with the weight matrix W, and\n",
       "the hidden layer from the previous time step ht􀀀1 with the weight matrix U. We\n",
       "add these values together and pass them through a suitable activation function, g,\n",
       "to arrive at the activation value for the current hidden layer, ht . and proceed with the usual computation to generate the\n",
       "output vector.\n",
       "- The sequential nature of simple\n",
       "recurrent networks can also be seen by unrolling the network in time\n",
       "- First, to compute the loss function for\n",
       "the output at time t we need the hidden layer from time t 􀀀1. Second, the hidden\n",
       "layer at time t influences both the output at time t and the hidden layer at time t +1\n",
       "(and hence the output and loss at t +1).\n",
       "\n",
       "\n",
       "## SLP Chapter 9.3 RNNs as Language Models\n",
       "\n",
       "- RNN language models (Mikolov et al., 2010) process the input sequence one word at\n",
       "a time, attempting to predict the next word from the current word and the previous\n",
       "hidden state.\n",
       "- RNNs don’t have the limited context problem that n-gram models\n",
       "have, since the hidden state can in principle represent information about all of the\n",
       "preceding words all the way back to the beginning of the sequence.\n",
       "- To train an RNN as a language model, we use a corpus of text as training material,\n",
       "having the model predict the next word at each time step t.\n",
       "- We train the model to\n",
       "minimize the error in predicting the true next word in the training sequence, using\n",
       "cross-entropy as the loss function.\n",
       "- cross-entropy loss measures the\n",
       "difference between a predicted probability distribution and the correct distribution.\n",
       "- Weight tying is a method that\n",
       "dispenses with this redundancy and uses a single set of embeddings at the input and\n",
       "softmax layers. That is, E = V. To do this, we set the dimensionality of the final\n",
       "hidden layer to be the same dh, (or add an additional projection layer to do the\n",
       "same thing), and simply use the same matrix for both layers.\n",
       "- Advantages of NN LMs:\n",
       "  - Better results. They achieve better preplexity scores than SOTA n-gram LMs.\n",
       "  - Larger N. NN LMs can scale to much larger orders of n. This is achievable because\n",
       "parameters are associated only with individual words, and not with n-grams.\n",
       "  - They generalize across contexts. \n",
       "- A by-product of training are word embeddings\n",
       "\n",
       "\n",
       "## SLP Chapter 6.1 Lexical Semantics\n",
       "\n",
       "- summarize some of these desiderata, drawing on results in the\n",
       "lexical linguistic study of word meaning, which is called lexical semantics\n",
       "- Lemmas and Senses\n",
       "  - form mouse is the lemma, also called the citation form. The form mouse would also be the lemma for the word mice;\n",
       "  - The specific forms sung or carpets or sing or duermes are called wordforms\n",
       "  - As the example above shows, each lemma can have multiple meanings; the\n",
       "lemma mouse can refer to the rodent or the cursor control device.\n",
       "  - each\n",
       "of these aspects of the meaning of mouse a word sense\n",
       "- Synonymy\n",
       "  - when one word has a sense whose meaning is\n",
       "identical to a sense of another word, or nearly identical, we say the two senses of\n",
       "those two words are synonyms.\n",
       "  - called the principle of contrast, word H2O is used in scientific contexts and\n",
       "would be inappropriate in a hiking guide\n",
       "- Word Similarity\n",
       "  - The notion of word similarity is very useful in larger semantic tasks. Knowing\n",
       "how similar two words are can help in computing how similar the meaning of two\n",
       "phrases or sentences are\n",
       "- Word relatedness\n",
       "  - The meaning of two words can be related in ways other than similarity. One such class of connections is called word relatedness, or word association in psychology.\n",
       "    - meanings of the words coffee and cup.\n",
       "  - One common kind of relatedness between words is if they belong to the same\n",
       "semantic field semantic field. A semantic field is a set of words which cover a particular semantic\n",
       "domain and bear structured relations with each other\n",
       "  - Semantic fields are also related to topic models, like Latent\n",
       "Dirichlet Allocation, LDA, which apply unsupervised learning on large sets of texts\n",
       "to induce sets of associated words from text.\n",
       "  - useful tools for discovering topical structure in documents\n",
       "- Semantic Frames and Roles\n",
       "  - A semantic frame is a set of words that denote perspectives or\n",
       "participants in a particular type of event.\n",
       "    - e.g. commercial transaction can be encoded with buy, sell, pay\n",
       "- Connotation\n",
       "  - words have affective meanings or connotations\n",
       "  - the aspects of a word’s meaning that are related to a writer or reader’s emotions,\n",
       "sentiment, opinions, or evaluations\n",
       "  - Positive or negative evaluation language is called sentiment,\n",
       "- Support for entailment\n",
       "  - $dog \\subset animal$\n",
       "\n",
       "\n",
       "## SLP Chapter 6.2 Vector Semantics\n",
       "\n",
       "- Vectors semantics is the standard way to represent word meaning in NLP, helping us model many of the aspects of word meaning\n",
       "- Vectors for representing words are called embeddings\n",
       "- classifiers can assign sentiment as long as\n",
       "it sees some words with similar meanings. And as we’ll see, vector semantic models\n",
       "can be learned automatically from text without supervision.\n",
       "- We will see that this method results in very long\n",
       "vectors that are sparse\n",
       "- word2vec model family for constructing\n",
       "short, dense vectors that have useful semantic properties\n",
       " \n",
       "\n",
       "## SLP Chapter 6.3 Words and Vectors\n",
       "\n",
       "- Vector or distributional models of meaning are generally based on a co-occurrence\n",
       "matrix, a way of representing how often words co-occur. We’ll look at two popular\n",
       "matrices: the term-document matrix and the term-term matrix.\n",
       "- term-document matrix, each row represents a word in the vocabulary and each\n",
       "column represents a document from some collection of documents, Each cell in this matrix represents the number of times\n",
       "a particular word, occurs in a particular document\n",
       "- The term-document matrix of was first defined as part of the vector space model of information retrieval\n",
       "- in real term-document matrices, the vectors representing\n",
       "each document would have dimensionality jVj, the vocabulary size.\n",
       "- Term-document matrices were originally defined as a means of finding similar\n",
       "documents for the task of document information retrieval. Two documents that are similar will tend to have similar words\n",
       "- Information retrieval (IR) is the task of finding the document d from the D\n",
       "documents in some collection that best matches a query q.\n",
       "- we saw that similar documents had similar vectors, because similar\n",
       "documents tend to have similar words. This same principle applies to words:\n",
       "similar words have similar vectors because they tend to occur in similar documents.\n",
       "The term-document matrix thus lets us represent the meaning of a word by the documents\n",
       "it tends to occur in.\n",
       "- term-term matrix, also called the word-word matrix or the term-context matrix, in which the columns are labeled by words rather\n",
       "than documents. the number of times the row (target) word and the column (context) word co-occur\n",
       "in some context in some training corpus.\n",
       "- This matrix is thus of dimensionality |V|*|V| and each cell records, the cell represents the number of times the two words appear in the same\n",
       "document. \n",
       "- (PPT)The value in a cell indicates how\n",
       "often a word appears in a\n",
       "context window surrounding\n",
       "another\n",
       "- TF-IDF\n",
       "  - Term frequency * inverse document\n",
       "frequency \n",
       "  - How often a\n",
       "word occurred in\n",
       "a document * 1 over the number\n",
       "of documents that it\n",
       "occurred in\n",
       "- Context windows\n",
       "w-2, w-1 target_word w+1 w+2\n",
       "- Since most of these numbers are zero these are sparse vector representations;\n",
       "there are efficient algorithms for storing and computing with sparse matrices.\n",
       "- Alternates to counts\n",
       "    - Raw word frequency is not a great measure of association between words. It’s very\n",
       "skewed “the” and “of” are very frequent, but maybe not the most discriminative\n",
       "We’d rather have a measure that asks whether a context word is particularly\n",
       "informative about the target word.\n",
       "\n",
       "\n",
       "## SLP Chapter 6.4 Cosine for measuring similarity\n",
       "\n",
       "- L2 Normalization: $||u|| = sqrt(\\sum_i^n u_i^2)$, divide each vector by the normalization term, compare each row in term-document matrix\n",
       "- We modify the dot product to normalize for the vector length by dividing the\n",
       "dot product by the lengths of each of the two vectors. This normalized dot product\n",
       "turns out to be the same as the cosine of the angle between the two vectors, following\n",
       "from the definition of the dot product between two vectors a and b:\n",
       "- $cosine(v,w) = (v\\cdot w)/(|v||w|)$, L2-norm\n",
       "- closer the semantic, smaller the angle\n",
       "\n",
       "\n",
       "## SLP Chapter 6.8 Word2vec\n",
       "\n",
       "- embeddings, short dense\n",
       "vectors.\n",
       "- embeddings are short, with number\n",
       "of dimensions d ranging from 50-1000, rather than the much larger vocabulary size\n",
       "|V| or number of documents D we’ve seen\n",
       "- vectors are dense: instead of vector entries being sparse,\n",
       "mostly-zero counts or functions of counts, the values will be real-valued numbers\n",
       "that can be negative.\n",
       "- Representing words as 300-dimensional dense vectors requires our classifiers\n",
       "to learn far fewer weights than if we represented words as 50,000-dimensional vectors,\n",
       "and the smaller parameter space possibly helps with generalization and avoiding\n",
       "overfitting.\n",
       "- skip-gram with negative sampling, sometimes called SGNS.\n",
       "- The word2vec methods are fast, efficient to train, and easily available online\n",
       "with code and pretrained embeddings.\n",
       "- Word2vec embeddings are static embeddings, meaning that the method learns one fixed embedding for each word in the vocabulary.\n",
       "- The intuition of word2vec is that instead of counting how often each word w occurs\n",
       "near, say, apricot, we’ll instead train a classifier on a binary prediction task: “Is\n",
       "word w likely to show up near apricot?” We don’t actually care about this prediction\n",
       "task; instead we’ll take the learned classifier weights as the word embeddings.\n",
       "- self-supervision, avoids the need for\n",
       "any sort of hand-labeled supervision signal\n",
       "- assume we’re\n",
       "using a window of +/-2 context words\n",
       "- ... lemon, a [tablespoon of apricot jam, a] pinch ... w = apricot, c1...c4 = tablespoon of jam a\n",
       "- train a classifier such that, given a tuple (w;c) of a target word\n",
       "w paired with a candidate context word c $P(+|w,c)$\n",
       "- $P(-|w|c) = 1-P(+|w,c)$\n",
       "- intuition of the skipgram\n",
       "model is to base this probability on embedding similarity: a word is likely to\n",
       "occur near the target if its embedding vector is similar to the target embedding\n",
       "- compute similarity between these dense embeddings, we rely on the intuition that\n",
       "two vectors are similar if they have a high dot product, similarity = $c\\cdot w$\n",
       "- $P(+|w,c)=\\sigma(c\\cdot w)=1/(1+exp(-c\\cdot w))$\n",
       "- $P(-|w,c)=\\sigma(-c\\cdot w)=1/(1+exp(c\\cdot w))$\n",
       "- $P(+|w,c_{1:L})=\\prod_{i=1}^L\\sigma(c_i\\cdot w)$\n",
       "- $log\\,P(+|w,c_{1:L})=\\sum_{i=1}^L\\log\\,\\sigma(c_i\\cdot w)$\n",
       "- In summary, skip-gram trains a probabilistic classifier that, given a test target word\n",
       "w and its context window of L words c1:L, assigns a probability based on how similar\n",
       "this context window is to the target word. The probability is based on applying the\n",
       "logistic (sigmoid) function to the dot product of the embeddings of the target word\n",
       "with each context word. To compute this probability, we just need embeddings for\n",
       "each target word and context word in the vocabulary.\n",
       "- Skip-gram actually\n",
       "stores two embeddings for each word, one for the word as a target, and one for the word considered as context. Thus the parameters we need to learn are two matrices\n",
       "W and C, each containing an embedding for every one of the jVj words in the\n",
       "vocabulary V\n",
       "\n",
       "\n",
       "## SLP Chapter 6.10 Semantic properties of embeddings\n",
       "\n",
       "- <strong>cont.</strong>\n",
       "- Given the set of positive and negative training instances, and an initial set of embeddings,\n",
       "the goal of the learning algorithm is to adjust those embeddings to\n",
       "  - Maximize the similarity of the target word, context word pairs (w;cpos) drawn\n",
       "from the positive examples\n",
       "  - Minimize the similarity of the (w;cneg) pairs from the negative examples.\n",
       "- want to maximize the dot product of the word with the actual context\n",
       "words, and minimize the dot products of the word with the k negative sampled nonneighbor\n",
       "words.\n",
       "- We minimize this loss function using stochastic gradient descent\n",
       "- word2vec maximizes: $\\sum_{(t,c)\\in +}log\\, P(+|t,c)+\\sum_{(t,c)\\in -}log\\, P(-|t,c)$, Maximize the + label for the pairs from the\n",
       "positive training data, and the – label for the\n",
       "pairs sample from the negative data.\n",
       "- <strong>end</strong>\n",
       "- Different types of similarity or association:\n",
       "  - Shorter context windows\n",
       "tend to lead to representations that are a bit more syntactic, since the information is\n",
       "coming from immediately nearby words. When the vectors are computed from short\n",
       "context windows, the most similar words to a target word w tend to be semantically\n",
       "similar words with the same parts of speech. When vectors are computed from long\n",
       "context windows, the highest cosine words to a target word w tend to be words that\n",
       "are topically related but not similar.\n",
       "  - Two words have first-order co-occurrence if they are typically nearby each other.\n",
       "Thus wrote is a first-order associate of book or poem.\n",
       "  - Two words have second-order\n",
       "co-occurrence if they have similar neighbors. Thus wrote is a second-order associate of words like said or remarked.\n",
       "- Analogy/Relational Similarity:\n",
       "  - scholars showed that sparse vector models of meaning could solve such analogy problems but the parallelogram method received more modern attention because of its success\n",
       "with word2vec or GloVe vectors\n",
       "  - problems:\n",
       "    - the closest value returned by the parallelogram\n",
       "algorithm in word2vec or GloVe embedding spaces is usually not in fact\n",
       "b* but one of the 3 input words or their morphological variants\n",
       "    - spaces perform well if the task involves\n",
       "frequent words, small distances, and certain relations the parallelogram method\n",
       "with embeddings doesn’t work as well for other relations\n",
       "- Embeddings can also be a useful tool for studying how meaning changes over time,\n",
       "by computing multiple embedding spaces, each from texts written in a particular\n",
       "time period.\n",
       "\n",
       "\n",
       "## SLP Chapter 6.11 Bias and Embeddings\n",
       "\n",
       "- allocational harm, when a system allocates resources unfairly to different groups \n",
       "  - algorithm hire programmers would fownweight female\n",
       "- amplifies bias\n",
       "  - gendered terms become more gendered in embedding space\n",
       "- representational harm: a harm caused by a system demeaning or even ignoring some social groups\n",
       "- although these sorts of debiasing may reduce bias in embeddings, they do not eliminate it\n",
       "\n",
       "\n",
       "## SLP Chapter 6.12 Evaluating Vector Models\n",
       "\n",
       "- <strong>Language Model 1</strong>\n",
       "- <strong>====================</strong>\n",
       "- Dense vector:\n",
       "  - Treat the target word and a neighboring context word as positive examples.\n",
       "  - Randomly sample other words in the lexicon to get negative samples.\n",
       "  - Use logistic regression to train a classifier to distinguish those two cases.\n",
       "  - Use the weights as the embeddings.\n",
       "- Sparse vs. dense vectors:\n",
       "  - Short vectors may be easier to use as features in machine learning (fewer\n",
       "weights to tune)\n",
       "  - Dense vectors may generalize better than storing explicit counts\n",
       "  - They may do better at capturing synonymy\n",
       "  - simply works better\n",
       "- Advantages of word2vec:\n",
       "  - Fast, easy to train (much faster than SVD)\n",
       "  - Available online in the word2vec package\n",
       "  Including sets of pretrained embeddings!\n",
       "- word2vec self-supervised:\n",
       "  - we need embeddings for t and c to do it.\n",
       "  - It starts with an initial set of embedding vectors and then iteratively shifts\n",
       "the embedding of each word w to be more like the embeddings of words\n",
       "that occur nearby in texts, and less like the embeddings of words that\n",
       "don’t occur nearby.\n",
       "- learn word2vec (skip-gram) embeddings：\n",
       "  - Start with V random 300-dimensional vectors as initial\n",
       "embeddings\n",
       "  - Use logistic regression\n",
       "    - Take a corpus and take pairs of words that co-occur as positive\n",
       "examples\n",
       "    - Take pairs of words that don't co-occur as negative examples\n",
       "    - Train the classifier to distinguish these by slowly adjusting all the\n",
       "embeddings to improve the classifier performance\n",
       "    - Throw away the classifier code and keep the embeddings.\n",
       "\n",
       "\n",
       "## Module 13 Lecture Notes\n",
       "\n",
       "- Like with logistic regression, we learn the best parameters for the neural net to make\n",
       "its predictions /𝑦 as close to possible as the gold standard labels in our training data y.\n",
       "We use:\n",
       "  - A cross entropy loss function\n",
       "  - Gradient descent to perform optimization\n",
       "  - Error back propagation to compute the gradient of the loss function\n",
       "- Since Neural Networks combine many perceptron-like neural units, we have many\n",
       "more parameters in the models (weights and biases). This allows us to train systems to\n",
       "perform even more sophisticated tasks.\n",
       "\n",
       "\n",
       "## Module 14 Definitions\n",
       "* **Concept** - definition\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module14_sections = [\n",
    "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
    "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
    "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
    "\n",
    "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
    "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
    "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
    "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
    "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
    "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
    "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
    "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
    "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
    "]\n",
    "\n",
    "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
    "                         module14_sections, \n",
    "                         module14_definitions)\n",
    "display(Markdown(module14_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vX7lbnJd4sT5"
   },
   "source": [
    "# Your Exam Notes\n",
    "\n",
    "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
    "\n",
    "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
    "* Run the cell below. \n",
    "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
    "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
    "* In the print options, change the layout from Landscape to Portrait. \n",
    "* Clicking the Save, will save a PDF that you can then print.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "T3cm1WV542_a",
    "outputId": "b1ac7409-85d3-496b-b9c0-8047428199bd"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 8: Markov Decision Processes\n",
       "## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n",
       "- $ P(Result(a)=s')= \\sum_{s}^{} P(s)P(s'|s,a) $, which assumes epsidic\n",
       "- Decision theory, in its simplest form, deals with choosing among actions based on the desirability of their immediate outcomes; that is, the environment is assumed to be episodic\n",
       "- The agent’s preferences are captured by a utility function, U(s), which assigns a single number to express the Utility function  desirability of a state.\n",
       "- $ EU(a) = \\sum_{s'} P(Result(a)=s')U(s') $. expected utility\n",
       "- The principle of maximum expected utility (MEU) says that a rational agent should choose  the action that maximizes the agent’s expected utility: \n",
       "- $ action = argmax EU(a) $, maximum expected utility, a prescription for intelligent behavior\n",
       "- All an  intelligent agent has to do is calculate the various quantities, maximize utility over its actions,  and away it goes. But this does not mean that the AI problem is solved by the definition!\n",
       "  - The MEU principle formalizes the general notion that an intelligent agent should “do the  right thing,” but does not operationalize that advice.\n",
       "  - percerption, learning and knowledge representaion, inference are also needed to compute,\n",
       "  - decision theory is not a panacea that solves the  AI problem—but it does provide the beginnings of a basic mathematical framework that is  general enough to define the AI problem. \n",
       "- If an agent acts so as to maximize a utility function that correctly reflects the performance measure, then the agent will achieve the highest possible performance score (averaged over all the  possible environments)\n",
       "- MEU is not the only rational way\n",
       "- basic preferences:\n",
       "  - $ (A\\succ B) $, agent prefers A over B\n",
       "  - $ (A \\sim B) $, no difference in perference\n",
       "  - $ (A \\succsim B) $, agent prefers A over B or no difference in perference\n",
       "- We can think of the set of outcomes for each action  as a lottery—think of each action as a ticket.\n",
       "- axioms of utility theory  list six constraints that we require any reasonable preference relation to obey: \n",
       "- <strong>Orderability</strong>, Given any two lotteries, a rational agent must either prefer one or else  rate them as equally preferable. That is, the agent cannot avoid deciding.\n",
       "  - Exactly one of $ (A\\succ B) $, $ (B \\succ A)$, or $ (A \\sim B) $ holds\n",
       "- <strong>Transitivity</strong>: Given any three lotteries, if an agent prefers A to B and prefers B to C, then the agent must prefer A to C\n",
       "  - $ (A\\succ B) \\wedge (B \\succ C) \\Rightarrow (A \\succ C) $ holds\n",
       "- <strong>Continuity</strong>: Continuity: If some lottery B is between A and C in preference, then there is some  probability p for which the rational agent will be indifferent between getting B for sure  and the lottery that yields A with probability p and C with probability 1− p.\n",
       "\n",
       "\n",
       "## AIMA Chapter 16.2 The Basis of Utility Theory\n",
       " \n",
       "- <strong>Substitutability</strong>: If an agent is indifferent between two lotteries A and B, then the  agent is indifferent between two more complex lotteries that are the same except that  B is substituted for A in one of them. This holds regardless of the probabilities and the  other outcome(s) in the lotteries\n",
       "- <strong>Monotonicity</strong>: Suppose two lotteries have the same two possible outcomes, A and  B. If an agent prefers A to B, then the agent must prefer the lottery that has a higher  probability for A (and vice versa). \n",
       "- <strong>Decomposability</strong>: Compound lotteries can be reduced to simpler ones using the laws of probability. it compresses two consecutive lotteries into a single equivalent lottery.\n",
       "- Rational preference lead to utility:\n",
       "- <strong>Existence of Utility Function</strong>:If an agent’s preferences obey the axioms of utility, then  there exists a function U such that U(A) > U(B) if and only if A is preferred to B, and  U(A) = U(B) if and only if the agent is indifferent between A and B. \n",
       "  - $U(A) > U(B) \\Leftrightarrow A \\succ B $  and  $ U(A) = U(B) \\Leftrightarrow A \\sim B$\n",
       "- <strong>Expected Utility of a Lottery</strong>: The utility of a lottery is the sum of the probability of  each outcome times the utility of that outcome. \n",
       "  - $ U([p_1,S_1;\\dots ; p_n,S_n]) = \\sum_{i} p_i U(S_i)$\n",
       "  - In other words, once the probabilities and utilities of the possible outcome states are specified, the utility of a compound lottery involving those states is completely determined.\n",
       "  - Because the  outcome of a nondeterministic action is a lottery, agent can act rationally, consistently with its preferences—only by choosing an action that maximizes expected  utility\n",
       "  - The preceding theorems establish that (assuming the constraints on rational preferences)  a utility function exists for any rational agent. The theorems do not establish that the utility  function is unique.\n",
       "- As in game-playing, in a deterministic environment an agent needs only a preference ranking on states—the numbers don’t matter. This is called a value function or <strong> ordinal utility function</strong>. \n",
       "- existence of a utility function that describes an agent’s  preference behavior does not necessarily mean that the agent is explicitly maximizing that  utility function in its own deliberations.\n",
       "\n",
       "## AIMA Chapter 16.3 Utility Functions\n",
       "\n",
       "- system that helps a human make decisions or acts on  his or her behalf, we must first work out what the human’s utility function is. This process, often called preference elicitation, presenting choices to the human and using the Preference elicitation  observed preferences to pin down the underlying utility function. \n",
       "- Normalized utilities: $ u_{bot} = 0, u_{top} = 1 $\n",
       "- standard lottery: $ [p,u_{top};(1-p),u_{bot}] $\n",
       "- expected monetary value (EMV):\n",
       "  - $ EMV = \\sum P(x)*Value(x)$\n",
       "  - Utility is not  directly proportional to monetary value, because the utility for your the momey you own is not the same\n",
       "  - utility of money plateaus at higher, deepening at lower\n",
       "- We will assume, kindly perhaps, that the estimates EU(a) are unbiased, but the real outcome will usually be significantly worse than we estimated,  even though the estimate was unbiased!\n",
       "\n",
       "\n",
       "## AIMA Chapter 17.1 Sequential Decision Problems\n",
       "\n",
       "- policy a solution specify what the agent should do for any state that the agent might reach. A solution of this kind is called a policy. $\\pi(s)$, action recommended by the policy π for state s.\n",
       "- An optimal policy is a policy that yields the highest expected utility. $\\pi^*(s) = argmax U^\\pi(s) = argmax \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- The balance of risk and reward changes depending on the value of $r=R(s,a,s')$ for transitions between nonterminal states. reward can be positive or negative but are bounded by some maximum value.\n",
       "Because the decision process is sequential, we must specify the utility function on a\n",
       "sequence of states and actions. the agent can receive a reward at\n",
       "each time step,\n",
       "- with a finite horizon, an optimal action in a given state may depend on how much time is left. \n",
       "- With no fixed time limit, on the other hand, there is no reason to behave differently in the  same state at different times.\n",
       "- Policies for the infinite-horizon case are simpler than those for the finite-horizon case\n",
       "- additive discounted rewards: the utility of a history is $U_h([s_0,a_0,s_1,a_1,...]) = R(s_0,a_0,s_1)+\\gamma R(s_1,a_1,s_2)+\\gamma^2 R(s_2,a_2,s_3) + ...$, Sooner rewards probably do have\n",
       "higher utility than later rewards\n",
       "- When γ is close to 0, rewards  in the distant future are viewed as insignificant. When γ is close to 1, an agent is more willing  to wait for long-term rewards.\n",
       "  - With discounted rewards, the utility of an infinite sequence is finite.\n",
       "  - environment contains terminal states and if the agent is guaranteed to get to one  eventually, then we will never need to compare infinite sequences.\n",
       "  - Infinite sequences can be compared in terms of the average reward obtained per time step.\n",
       "- discounted utilities with infinite horizons is that the optimal policy is  independent of the starting state.\n",
       "- the utility of a state is the expected reward for the  next transition plus the discounted utility of the next state, assuming that the agent chooses  the optimal action.\n",
       "- Bellman equation $U(s) = max \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- action utility funtion, Q function $U(s)= max Q(s,a)$, $\\pi^* = argmax Q(s,a)$\n",
       "  - $Q(s,a) = \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma max Q(s',a')]$- reward scales:\n",
       "  - $U'(s) = mU(s)+b$\n",
       "  - $R'(s,a,s') = mR(s,a,s') +b$\n",
       "  - $\\Phi(s)$ be any function of the state s. Then, according  to the shaping theorem, the following transformation leaves the optimal policy unchanged:\n",
       "  - $R'(s,a,s') = R(s,a,s') +\\gamma \\Phi(s')−\\Phi(s).$\n",
       "  - $Q'(s,a)=Q(s,a)−\\Phi(s)= \\sum_{s'}P(s'|s,a)[R'(s,a,s')+\\gamma max Q'(s',a')]$\n",
       "  - $\\Phi(s)$ potential, The term $\\gamma \\Phi(s')−\\Phi(s)$ functions as a gradient of the  potential. \n",
       "  - $\\Phi(s)$  has higher value in states that have higher utility, the addition of $\\gamma \\Phi(s')−\\Phi(s)$ to the reward has the effect of leading the agent “uphill” in utility. \n",
       "  - all policies are optimal  with a reward function that is zero everywhere.\n",
       "\n",
       "\n",
       "## AIMA Chapter 17.2 Algorithms for MDPs\n",
       "\n",
       "- transition model or model describes the outcome of each action in each state $P(s'|s,a)$, Transitions are Markovian:\n",
       "the probability of arriving in\n",
       "s’ only depends on s and\n",
       "not the history of earlier\n",
       "states.\n",
       "- MDPs are non-deterministic search\n",
       "problems. One way of solving them is via\n",
       "expectimax search.\n",
       "- if the game is infinite:\n",
       "  - Finite horizon:\n",
       "    - Terminate episodes after a fixed T steps\n",
       "    - Gives nonstationary policies\n",
       "  - Discounting: Smaller g means smaller “horizon” – shorter term focus\n",
       "  - Absorbing state: guarantee that for every policy, a terminal state will\n",
       "eventually be reached\n",
       "- <strong>value iteration</strong>\n",
       "- is slow, policy converges long before values, max rarely changes\n",
       "- Every iteration updates both the values and (implicitly) the policy\n",
       "- We don’t track the policy, but taking the max over actions implicitly recomputes\n",
       "it\n",
       "- Bellman equation is the basis of the value iteration algorithm for solving MDPs.\n",
       "- $U_{i+1}(s) \\leftarrow max_{a\\in A(s)} \\sum_{s'} P(s'|s,a)[R(s,a,s') +\\gamma U_i(s')]$, complexity of each iteration: $O(s^2a)$\n",
       "- apply the Bellman update infinitely often, we are guaranteed to reach an unique equilibrium \n",
       "- We can think of the value iteration algorithm as propagating  information through the state space by means of local updates. \n",
       "- if $ ||U_{i+1} − U_i|| < \\epsilon (1−\\gamma)/ \\gamma $  then  $||U_{i+1} − U || < \\epsilon$.\n",
       "- if $||U_i-U|| < \\epsilon$ then $||U^{\\pi_i}-U|| < 2\\epsilon $, policy loss\n",
       "- <strong>policy iteration</strong>\n",
       "  - We do several passes that update utilities with fixed policy (each pass is fast\n",
       "because we consider only one action, not all of them)\n",
       "  - After the policy is evaluated, a new policy is chosen (slow like a value iteration\n",
       "pass)\n",
       "  - The new policy will be better (or we’re done)\n",
       "  - Policy evaluation, given $\\pi_i$ calculate $U_i = U^{\\pi i}$ the utility of each state if $\\pi_i$  were to be executed. \n",
       "  - $U_i(s) \\leftarrow \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$\n",
       "  - policy improvement, Calculate a new MEU policy $\\pi_{i+1} $, using one-step look-ahead  based on $U_i$\n",
       "  - $\\pi(s')= argmax_a\\sum_{s'} P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- Policy evaluation\" $U_i(s) = \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$For n states, we have n linear equations with n unknowns, which can be solved exactly in time $O(n^3)$\n",
       "- <strong>linear programming</strong>\n",
       "- basic idea of the formulation is to consider as variables in the LP the utilities U(s) of  each state s, noting that the utilities for an optimal policy are the highest utilities attainable that  are consistent with the Bellman equations.\n",
       "  - s.t. $U(s) \\geq max \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma U(s')]$\n",
       "- <strong>online programming</strong>\n",
       "  - actually a simplification of the EXPECTIMINIMAX  algorithm for game trees with chance nodes:\n",
       " \n",
       "\n",
       "## Module 8 Lecture Notes\n",
       "\n",
       "- Probability theory describes what an agent should believe on the basis of evidence,  utility theory describes what an agent wants, and decision theory puts the two together  to describe what an agent should do. \n",
       "-  We can use decision theory to build a system that makes decisions by considering all  possible actions and choosing the one that leads to the best expected outcome. Such a  system is known as a rational agent.\n",
       "- Utility theory shows that an agent whose preferences between lotteries are consistent  with a set of simple axioms can be described as possessing a utility function; furthermore, the agent selects actions as if maximizing its expected utility.\n",
       "- Multiattribute utility theory deals with utilities that depend on several distinct attributes of states. Stochastic dominance is a particularly useful technique for making  unambiguous decisions, even without precise utility values for attributes. \n",
       "- Sequential decision problems in stochastic environments, also called Markov decision  processes, or MDPs, are defined by a transition model specifying the probabilistic  outcomes of actions and a reward function specifying the reward in each state.\n",
       "- The utility of a state sequence is the sum of all the rewards over the sequence, possibly  discounted over time. The solution of an MDP is a policy that associates a decision  with every state that the agent might reach. An optimal policy maximizes the utility of  the state sequences encountered when it is executed.\n",
       "- The utility of a state is the expected sum of rewards when an optimal policy is executed  from that state. The value iteration algorithm iteratively solves a set of equations  relating the utility of each state to those of its neighbors. \n",
       "- Policy iteration alternates between calculating the utilities of states under the current  policy and improving the current policy with respect to the current utilities. \n",
       "- Theorm: will converge to unique optimal values\n",
       "  - Basic idea: approximations get refined towards optimal values\n",
       "  - Policy may converge long before values do\n",
       "- Converge:\n",
       "  - Case 1: If the tree has maximum depth M, then VM holds\n",
       "the actual untruncated values\n",
       "  - Case 2: If the discount is less than 1\n",
       "- Compute optimal values: use value iteration or policy iteration\n",
       "- Compute values for a particular policy: use policy evaluation\n",
       "- Turn your values into a policy: use policy extraction (one-step\n",
       "lookahead)\n",
       "\n",
       "\n",
       "## Module 8 Definitions\n",
       "* **Lottery** - don't know what is coming\n",
       "* **Preference elicitation** - involves presenting choices to the human and using the observed preferences to pin down the underlying utility function\n",
       "* **Risk-averse** - they prefer a sure thing with a payoff that is less than the expected monetary value of a gamble.\n",
       "* **Risk-seeking** - “desperate” region at large negative wealth, loss more money is fine\n",
       "* **Normative theory** - describe how a rational agent should act.\n",
       "* **Descriptive theory** - how a actual agent (human) would act\n",
       "* **Certainty effect** - people are strongly attracted to gains that are certain\n",
       "* **Transitions are markovian** - the  probability of reaching s ′ from s depends only on s and not on the history of earlier states. \n",
       "* **Decision problem is sequential** -  the utility function will depend  on a sequence of states and actions—an environment history—rather than on a single state. \n",
       "* **Mdp** -  a sequential decision problem for a fully observable, stochastic environment with a Markovian transition model and additive rewards\n",
       "* **Dynamic programming** - simplifying a problem by recursively breaking it into smaller pieces and remembering the optimal solutions to the pieces. \n",
       "* **Finite horizon** - means that there is a fixed time N after which nothing matters—the game is over\n",
       "* **Nonstationary** - A policy that depends on the time\n",
       "* **Stationary** - an optimal action depends only on the current state, and  the optimal policy\n",
       "* **Proper policy** -  guaranteed to reach a terminal state is called a proper policy\n",
       "* **Sparse** - with mostly zeros\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "# Module 9: Reinforcement Learning\n",
       "## AIMA Chapter 17.3 Bandit Problems\n",
       "\n",
       "- a one-armed bandit is a slot machine, n arm has n levers\n",
       "- explitation: the current best action to obtain rewards\n",
       "- exploration: previously unknown states and actions to gain information, which can in some cases be converted into a better policy and better long-term rewards.\n",
       "- come back if has time\n",
       "<strong> early start</strong>\n",
       "- Must (learn to) act so as to maximize expected rewards\n",
       "- Reinforcement learning differs from “just solving  an MDP” because the agent is not given the MDP as a problem to solve; the agent is in the  MDP.\n",
       "  - reward function is often (as we saw for chess) very concise and easy to specify\n",
       "  - we don’t have to be experts,  capable of supplying the correct action in any situation, as would be the case if we tried to  apply supervised learning. \n",
       "- sparse rewards, because in the vast majority of states the agent is  given no informative reward signal at all.\n",
       "- Model-based reinforcement learning: agent uses a transition model of the environment to help interpret the reward signals and to make decisions  about how to act. The model may be initially unknown, or it may already be known\n",
       "  - Learn an approximate model based on experiences\n",
       "Solve for values as if the learned model were correct\n",
       "  - Step 1: Learn empirical MDP model:\n",
       "    - Count outcomes s’ for each s, a\n",
       "    - Normalize to give an estimate of trasition\n",
       "    - Discover reward each when we experience (s, a, s’)\n",
       "  - Step 2: Solve the learned MDP\n",
       "    - For example, use value iteration, as before\n",
       "  - the transition model is  also useful for state estimation\n",
       "  - Model-based reinforcement learning  systems often learn a utility function U(s), defined in terms of the  sum of rewards from state s onward.\n",
       "- Model-free reinforcement learning: In these approaches the agent neither knows nor learns a transition model for the environment.\n",
       "  - Action-utility learning: most common form of action-utility learning is Q-learning, where the agent learns a Q-function, or quality-function, Q(s,a), \n",
       "  - Policy search: The agent learns a policy π(s) that maps directly from states to  actions.(reflex agent)\n",
       "- passive reinforcement learning: where the agent’s policy is fixed and the task is to learn the utilities of states\n",
       "- active  reinforcement learning: where the agent must also figure out what to do.\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.1 Learning from Rewards\n",
       "\n",
       "- Simplified task: policy evaluation\n",
       "  - Input: a fixed policy π(s)\n",
       "  - You don’t know the transitions T(s,a,s’)\n",
       "  - You don’t know the rewards R(s,a,s’)\n",
       "  - Goal: learn the state values\n",
       "- In this case:\n",
       "  - No choice about what actions to take\n",
       "  - This is NOT offline planning!\n",
       "- a fully observable environment with a small number of  actions and states, in which an agent already has a fixed policy π(s) that determines its actions.  The agent is trying to learn the utility function U π (s)—the expected total discounted reward  if policy π is executed beginning in state s.\n",
       "- passive learning task is similar to the policy evaluation task, but do not know the trasition model $P(s'|s,a)$ and $R(s,a,s')$\n",
       "- direct utility estimation is that the utility of a state is defined as the expected total reward from that state onward (called the expected reward-to-go), and that each trial  provides a sample of this quantity for each state visited.\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.2 Passive Reinforcement Learning\n",
       "\n",
       "- Direct evaluation:\n",
       "  - Compute values for each state under π\n",
       "  - Average together observed sample values\n",
       "    - Act according to π\n",
       "    - Every time you visit a state, write down what the\n",
       "    - sum of discounted rewards turned out to be\n",
       "    - Average those samples\n",
       "  - bad:\n",
       "    - It wastes information about state connections\n",
       "    - Each state must be learned separately\n",
       "    - So, it takes a long time to learn\n",
       "- The utility of a state is determined by the reward and the expected utility  of the successor states.\n",
       "  - $U_i(s) = \\sum_{s'}P(s'|s,\\pi_i(s))[R(s,\\pi_i(s),s')+\\gamma U(s')]$\n",
       "- adaptive dynamic programming (or ADP) agent takes advantage of the constraints among the utilities of states by learning the transition model that connects them and solving the corresponding Markov decision process using dynamic programming.\n",
       "  - we can adopt the approach of modified policy iteration,  using a simplified value iteration process to update the utility estimates after each change to  the learned model.\n",
       "  - Learning the transition model is easy, because the environment is fully observable.\n",
       "  - The algorithm converges fast but can become quite costly to compute for large state spaces. ADP is a model based approach and requires the transition model of the environment.\n",
       "- temperal difference learning:\n",
       "  - learn from every experience!\n",
       "Update V(s) each time we experience a transition (s, a, s’, r)\n",
       "Likely outcomes s’ will contribute updates more often\n",
       "  - Temporal difference learning of values\n",
       "Policy still fixed, still doing evaluation!\n",
       "Move values toward value of whatever successor occurs: running average\n",
       "  - Decreasing learning rate (alpha) can give converging averages\n",
       "  - $U^\\pi(s) \\leftarrow U^\\pi(s)+α[R(s,\\pi(s),s')+\\gamma U^\\pi(s')-U(s)]$\n",
       "  - All temporal-difference methods work by adjusting the utility estimates toward the ideal  equilibrium that holds locally when the utility estimates are correct.\n",
       "  - TD does not need a transition model to perform its updates.\n",
       "  - Forgets about the past (distant past values were wrong anyway\n",
       "- The ADP and TD approaches are closely related. Both try to make local adjustments to  the utility estimates in order to make each state “agree” with its successors.\n",
       "- that TD adjusts a state to agree with its observed successor whereas ADP  adjusts the state to agree with all of the successors that might occur, weighted by their probabilities\n",
       "- TD makes a single adjustment per observed transition, ADP makes  as many as it needs to restore consistency between the utility estimates U and the transition  model P.\n",
       "- Each adjustment made by ADP could be seen, from the TD point of view, as a result of a pseudoexperience generated by simulating the current transition model.\n",
       "- prioritized sweeping  heuristic prefers to make adjustments to states whose likely successors have just undergone a  large adjustment in their own utility estimates. \n",
       "\n",
       "\n",
       "## AIMA Chapter 22.3 Active Reinforcement Learning\n",
       "\n",
       "- Q-learning converges to optimal policy -- even\n",
       "if you’re acting suboptimally!\n",
       "- Caveats:\n",
       "  - You have to explore enough\n",
       "  - You have to eventually make the learning rate\n",
       "small enough\n",
       "- exploration function. The function f(u,n) determines how greed (preference Exploration function  for high values of the utility u) is traded off against curiosity\n",
       "  - safe exploration\n",
       "  - Bayes reinforcement learning: assumes a prior probability P(h) over hypotheses h about what the true model is; the posterior probability P(h| e) is obtained  in the usual way by Bayes’ rule given the observations to date.\n",
       "  - $\\pi^* = argmax_\\pi \\sum_h P(h|e)U_h^\\pi$\n",
       "  - robust control theory, allows for a set of possible models H without assigning probabilities to them, and defines an optimal robust policy as  one that gives the best outcome in the worst case over H: \n",
       "  - $\\pi^* - argmax_\\pi min_hU^\\pi_h$\n",
       "- TD Q learning\n",
       "  - $Q(s,a) = \\sum_{s'}P(s'|s,a)[R(s,a,s')+\\gamma max_{a'}Q(s',a')]$\n",
       "  - $Q(s,a) = Q(s,a) + α[R(s,a,s')+\\gamma max_{a'}Q(s',a')-Q(s,a)]$\n",
       "  - a TD Q-learning agent  does not need a transition model, P(s ′ |s,a), either for learning or for action selection.\n",
       "  - the Qlearning agent has no means of looking into the future, so it may have difficulty when rewards  are sparse and long action sequences must be constructed to reach them. \n",
       "  - SARSA\n",
       "  - $Q(s,a) = Q(s,a) + α[R(s,a,s')+\\gamma Q(s',a')-Q(s,a)]$\n",
       "  - If the agent is greedy and always takes the action with the best Q-value, the two algorithms  are identical. When exploration is happening, however, they differ: if the exploration yields  a negative reward, SARSA penalizes the action, while Q-learning does not. \n",
       "  - Q-learning is an off-policy learning algorithm, because it learns Q-values that answer the  question “What would this action be worth in this state, assuming that I stop using whatever  policy I am using now, and start acting according to a policy that chooses the best action \n",
       "  - SARSA is an on-policy algorithm: it learns Q-values that  answer the question “What would this action be worth in this state, assuming I stick with my policy?\n",
       "  - Q-learning is more flexible than SARSA, in the sense that a Q-learning agent can  learn how to behave well when under the control of a wide variety of exploration policies.\n",
       "  - SARSA is appropriate if the overall policy is even partly controlled by other  agents or programs,\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n",
       "\n",
       "- How to explror?\n",
       "  - Simplest: random actions (ε-greedy)\n",
       "    - Every time step, flip a coin\n",
       "    - With (small) probability ε, act randomly\n",
       "    - With (large) probability 1-ε, act on current policy\n",
       "  - Problems with random actions?\n",
       "    - You do eventually explore the space, but keep\n",
       "thrashing around once learning is done\n",
       "    - One solution: lower ε over time\n",
       "    - Another solution: exploration functions\n",
       "  - exploration function:\n",
       "    - explore areas whose badness is not\n",
       "(yet) established, eventually stop exploring\n",
       "- regret:\n",
       "  - Even if you learn the optimal policy,\n",
       "you still make mistakes along the way!\n",
       "Regret is a measure of your total\n",
       "mistake cost: the difference between\n",
       "your (expected) rewards, including\n",
       "youthful suboptimality, and optimal\n",
       "(expected) rewards\n",
       "  - Minimizing regret goes beyond\n",
       "learning to be optimal – it requires\n",
       "optimally learning to be optimal  - random exploration and\n",
       "exploration functions both end up\n",
       "optimal, but random exploration has\n",
       "higher regret\n",
       "- function approximation for the process of constructing a compact approximation of the true utility function or Q-function. \n",
       "- Basic Q-Learning keeps a table of all q-values\n",
       "In realistic situations, we cannot possibly learn\n",
       "about every single state!\n",
       "  - Learn about some small number of training states from\n",
       "experience\n",
       "  - Generalize that experience to new, similar situations\n",
       "- approximate Q-learning in features\n",
       "  - Q-learning with linear Q-functions:\n",
       "    - Adjust weights of active features\n",
       "    - E.g., if something unexpectedly bad happens, blame the features that were on:\n",
       "disprefer all states with that state’s features\n",
       "\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.5 Policy Search\n",
       "\n",
       "- policy search: the  idea is to keep twiddling the policy as long as its performance improves, then stop.\n",
       "- π is a function that  maps states to actions. We are interested primarily in parameterized representations of π that  have far fewer parameters than there are states in the state space\n",
       "- Policy search will then adjust the parameters θ to  improve the policy.\n",
       "- come back if have time\n",
       "\n",
       "\n",
       "## Module 9 Lecture Notes\n",
       "\n",
       "- The overall agent design dictates the kind of information that must be learned:  \n",
       "  - A model-based reinforcement learning agent acquires (or is equipped with) a  transition model P(s ′ |s,a) for the environment and learns a utility function U(s).  \n",
       "  - A model-free reinforcement learning agent may learn an action-utility function  Q(s,a) or a policy π(s). \n",
       "- Utilities can be learned using several different approaches:\n",
       "  - Direct utility estimation uses the total observed reward-to-go for a given state as  direct evidence for learning its utility.\n",
       "  - Adaptive dynamic programming (ADP) learns a model and a reward function  from observations and then uses value or policy iteration to obtain the utilities or  an optimal policy. ADP makes optimal use of the local constraints on utilities of  states imposed through the neighborhood structure of the environment. \n",
       "  - Temporal-difference (TD) methods adjust utility estimates to be more consistent  with those of successor states. They can be viewed as simple approximations of the  ADP approach that can learn without requiring a transition model. Using a learned  model to generate pseudoexperiences can, however, result in faster learning.\n",
       "- Action-utility functions, or Q-functions, can be learned by an ADP approach or a TD  approach. With TD, Q-learning requires no model in either the learning or actionselection phase. This simplifies the learning problem but potentially restricts the ability  to learn in complex environments, because the agent cannot simulate the results of  possible courses of action.\n",
       "-  When the learning agent is responsible for selecting actions while it learns, it must  trade off the estimated value of those actions against the potential for learning useful  new information. An exact solution for the exploration problem is infeasible, but some  simple heuristics do a reasonable job. An exploring agent must also take care to avoid  premature death.\n",
       "- In large state spaces, reinforcement learning algorithms must use an approximate functional representation of U(s) or Q(s,a) in order to generalize over states. Deep reinforcement learning—using deep neural networks as function approximators—has  achieved considerable success on hard problems. \n",
       "-  Reward shaping and hierarchical reinforcement learning are helpful for learning  complex behaviors, particularly when rewards are sparse and long action sequences are  required to obtain them. \n",
       "-  Policy-search methods operate directly on a representation of the policy, attempting  to improve it based on observed performance. The variation in the performance in a  stochastic domain is a serious problem; for simulated domains this can be overcome by  fixing the randomness in advance. \n",
       "\n",
       "\n",
       "## Module 9 Definitions\n",
       "* **Markov reward process or mrp** -  that is, an MDP with only one Markov reward  process  possible action $a_i$. \n",
       "* **Exploration** - you have to try unknown actions to get information\n",
       "* **Exploitation** - eventually, you have to use what you know\n",
       "* **Regret** - even if you learn intelligently, you make mistakes\n",
       "* **Sampling** - because of chance, you have to try things repeatedly\n",
       "* **Difficulty** - learning can be much harder than solving a known MDP\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "## WARNING: Your summary of 'SLP Chapter 3.5 Smoothing' is too long!\n",
       "# Module 10: Probabilies and Language Models\n",
       "## AIMA Chapter 12.1 Acting under Uncertainty\n",
       " \n",
       "- probabilities are about possible worlds. Instead of strictly\n",
       "ruling out possibilities (where a logical assertion is false), probabilities quantify how\n",
       "likely a particular possible world is.\n",
       "- Uncertainty:  observability, nondeterminism, or adversaries\n",
       "- belief state agent:\n",
       "  - must consider every possible, leads to a large belief-state\n",
       "  - A correct contingent plan that handles every eventuality can grow arbitrarily large and  must consider arbitrarily unlikely contingencies. \n",
       "  - Sometimes there is no plan that is guaranteed to achieve the goal—yet it must act\n",
       "- rational decision: the right thing to do, therefore depends on both the relative importance of various goals and the likelihood that, and degree to which, they will be achieved.\n",
       "- logic to cope with a domain has problems:\n",
       "  - Laziness: It is too much work to list the complete set of antecedents or consequents\n",
       "  - Theoretical ignorance: Medical science has no complete theory for the domain. \n",
       "  - Practical ignorance: Even if we know all the rules, we might be uncertain about a  particular patient because not all the necessary tests have been or can be run. \n",
       "- degree of belief - The agent’s knowledge can at best provide only a degree of belief in the relevant sentences, with degrees of belief is probability theory.\n",
       "- probability provides a way of summarizing the uncertainty that comes from  our laziness and ignorance\n",
       "- agent must first have preferences among the different possible outcomes\n",
       "- decision theory = probability theory + utility theory, odd is not irrational\n",
       "- Maximum expected utility (MEU): an agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all the possible outcomes of the action.\n",
       "- decision-theoretic agent’s belief state represents not just the possibilities for world  states but also their probabilities. belief state and some knowledge of the effects of  actions\n",
       "\n",
       "## AIMA Chapter 12.2 Basic Probability Notation\n",
       "\n",
       "- sample space - the set of all possible world, mutually exclusive and exhaustive\n",
       "- A fully specified probability model associates a numerical probability P(ω) with each Probability model  possible world, add up to 1\n",
       "- event: the unique result of states (dice add up to 11), associate with proposition in this context\n",
       "- unconditional probability = prior probability = prior, degrees of belief in propositions in the absence of any other information.\n",
       "- conditional = posterior, given that the sth is\n",
       "- random variable, variable in probability theory\n",
       "- range, range of possible outcomes\n",
       "- A possible world is defined to be an assignment of values to all of the random variables under consideration. \n",
       "- full joint probability distribution - follows that a probability model is completely determined by the joint distribution for all of the random variables\n",
       "- logically related propositions: If  Agent 1 expresses a set of degrees of belief that violate the axioms of probability theory then  there is a combination of bets by Agent 2 that guarantees that Agent 1 will lose money every  time.<br />\n",
       " $P(a|b) = \\frac{P(a,b)}{P(b)}$ <br /> $P(a,b) = P(a|b)P(b)$ <br /> $P(a \\lor b) = P(a)+P(b)-P(a,b)$ <br />$P(x_1,x_2,x_3)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2) \\rightarrow P(x_1,x_2,...,x_n)=\\prod_i P(x_i|x_1,...,x_{i-1})$\n",
       "\n",
       "## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n",
       "\n",
       "- for probabilistic inference—that is, the computation of posterior probabilities for query propositions given observed evidence.\n",
       "- marginal probability: sum of joint probability, summing out  $P(Y) = \\sum_{z} P(Y,Z=z)$ <br />\n",
       "- conditioning, add all the conditional probabilities $P(Y) = \\sum_{z} P(Y,Z=z)$ <br />\n",
       "- for a domain described by n  Boolean variables, it requires an input table of size $O(2^n)$ and takes $O(2^n)$ time to process the table.\n",
       "- The problem is not just memory and computation: the number of examples required will be astronomical. \n",
       "\n",
       "\n",
       "## AIMA Chapter 12.4 Independence\n",
       " \n",
       "- independence : $P(X|Y) = P(X) \\text{ or } P(Y|X) = P(Y) \\text{ or } P(X,Y) = P(X)P(Y)$<br />\n",
       "\n",
       "## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n",
       "\n",
       "Bayes rule\n",
       " $P(Y|X) = \\frac{P(X|Y)P(Y)}{P(X)} \\text{ or } P(Y|X,e) = \\frac{P(X|Y,e)P(Y|e)}{P(X|e)}$<br />\n",
       "- The conditional probability P(effect| cause) quantifies the relationship in the causal direc- Causal  tion, whereas P(cause | effect) describes the diagnostic direction.\n",
       "- diagnostic knowledge is often more fragile than causal knowledge.\n",
       "- conditional probability:  X is conditionally independent of Y given Z\n",
       "- \n",
       "$P(X,Y|Z) = P(X|Z)P(Y|Z)$  or $P(X|Y,Z) = P(X|Z) \\text{ and } P(Y|X,Z) = P(Y|Z)$<br />\n",
       "- for n symptoms that are all conditionally independent given Cavity, the size  of the representation grows as $O(n)$ instead of $O(2^n)$.\n",
       "- That means that conditional independence assertions can allow probabilistic systems to scale up; moreover, they are much more commonly available than absolute independence assertions. Conceptually, Cavity separates  Toothache and Catch because it is a direct cause of both of them.\n",
       "\n",
       "\n",
       "## AIMA Chapter 12.6 Naive Bayes Models\n",
       "\n",
       "- Naive bayes because it is often  used (as a simplifying assumption) in cases where the “effect” variables are not strictly independent given the cause variable.\n",
       " $P(Cause,Effect_1,\\dots,Effect_n) = P(Cause)\\prod_i P(Effect_i|Cause)$ <br /> $$P(Cause|e) = \\alpha \\sum_{y}P(cause)P(y|Cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$$$=\\alpha P(cause) \\sum_{y}P(y|Cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$$$= \\alpha P(cause)\\left(\\prod_{j}P(e_j|Cause)\\right)$$ \n",
       " - text classification, not independent, but works fine, $P(HasWord_i|Category)$\n",
       " \n",
       "\n",
       "## AIMA Chapter 12.7 The Wumpus World Revisited\n",
       "-\n",
       "- only partial information about  the world, probabilistic agent can do much better than the  logical agent. \n",
       "- seemingly complicated problems can be formulated precisely in probability theory and solved with simple algorithms. To get efficient  solutions, independence and conditional independence relationships can be used to simplify  the summations required.\n",
       "\n",
       "\n",
       "## SLP Chapter 3.1 N-Grams\n",
       "\n",
       "- $P(w|h) = P(w,h)/P(h)$\n",
       "- but not always work\n",
       "- bigram model, for example, approximates the probability of a word given all the previous words $P(w_n|w_{1:n-1})$ by using only the conditional probability of the preceding word $P(w_n|w_{n-1})$\n",
       "- approximate $P(w_n|w_{1:n-1}) = P(w_n|w_{n-1})$\n",
       "- assumption that the probability of a word depends only on the previous word is Markov assumption. Markov models are the class of probabilistic models that assume we can predict the probability of some future unit without looking too far into the past.\n",
       "- generalize n-gram that $P(w_{1:n}) = \\prod_{k=1}^n P(w_k|w_{k-1})$\n",
       "- $P(w_n|w_{n-1}) = C(w_n-1 w_n)/C(w_{n-1})$\n",
       "- estimates the n-gram probability by dividing the observed frequency of a particular sequence by the observed frequency of a prefix. This ratio is called a relative relative frequency.\n",
       "- Some of the bigram probabilities above encode some facts that we think of as strictly syntactic in nature, like the fact that what comes after eat is usually a noun or an adjective, or that what comes after to is usually a verb\n",
       "- p1* p2* p3* p4 = exp(log p1+log p2+log p3+log p4)\n",
       "- We do everything in log space\n",
       "  - Avoid underflow\n",
       "  - (also adding is faster than multiplying)\n",
       "\n",
       "\n",
       "## SLP Chapter 3.2 Evaluating Language Models\n",
       "\n",
       "- Extrinsic evaluation is the only way to know if a particular improvement in a component is really going to help the task at hand. we can compare the performance of two language models by running the speech recognizer twice, once with each language model, and seeing which gives the more accurate transcription\n",
       "- An intrinsic evaluation metric is one that measures the quality of a model independent of any application\n",
       "- whichever model assigns a higher probability to the test set—meaning it more accurately predicts the test set—is a better model\n",
       "- Training on the test set introduces a bias that makes the probabilities all look too high, and causes huge inaccuracies in perplexity\n",
       "- Perplexity:\n",
       "$PP(W) = \\sqrt[n]{\\frac{1}{P(w_1,w_2,...,w_N)}} = \\sqrt[N]{\\prod_{i=1}^N\\frac{1}{P(w_i|w_1,...,w_{i-1})}}$\n",
       "- the higher the conditional probability of the word sequence, the lower the perplexity. Thus, minimizing perplexity is equivalent to maximizing the test set probability according to the language model.\n",
       "- perplexity as the weighted average branching factor of a language. The branching factor of a language is the number of possible next words that can follow any word.\n",
       "- the more information the n-gram gives us about the word sequence, the lower the perplexity\n",
       "- Any kind of knowledge of the test set can cause the perplexity to be artificially low.\n",
       "- An (intrinsic) improvement in perplexity does not guarantee an (extrinsic) improvement in the performance of a language processing task like speech recognition or machine translation. Nonetheless, because perplexity often correlates with such improvements, it is commonly used as a quick check on an algorithm. But a model’s improvement in perplexity should always be confirmed by an end-to-end evaluation of a real task before concluding the evaluation of the model\n",
       "\n",
       "\n",
       "## SLP Chapter 3.3 Sampling sentences from a language model\n",
       " \n",
       "- Sampling from a distribution means to choose random points according to their likelihood. Thus sampling from a language model—which represents a distribution over sentences—means to generate some sentences, choosing each sentence according to its likelihood as defined by the model\n",
       "- list the likelihood from 0 to 1, randomly choose a number and select the word, so avoid sample frequent words\n",
       "\n",
       "\n",
       "## SLP Chapter 3.4 Generalization and Zeros\n",
       "\n",
       "- One implication of this is that the probabilities often encode specific facts about a given training corpus. Another implication is that n-grams do a better and better job of modeling the training corpus as we increase the value of N.\n",
       "- The longer the context on which we train the model, the more coherent the sentences. In the unigram sentences, there is no coherent relation between words or any sentence-final punctuation. The bigram sentences have some local word-to-word coherence\n",
       "- One step is to be sure to use a training corpus that has a similar genre to whatever task we are trying to accomplish\n",
       "- get training data in the appropriate dialect or variety, especially when processing social media posts or spoken transcripts\n",
       "- Unknown words\n",
       "- Sometimes we have a language task in which this can’t happen because we know all the words that can occur. In such a closed vocabulary system the test set can only contain words from this lexicon, and there will be no unknown words.\n",
       "- In other cases we have to deal with words we haven’t seen before, which we’ll call unknown OOV words, or out of vocabulary (OOV) words. The percentage of OOV\n",
       "open words that appear in the test set is called the OOV rate. An open vocabulary system  is one in which we model these potential unknown words in the test set by adding a pseudo-word called <UNK>.\n",
       "- First way\n",
       "  -  Choose a vocabulary (word list) that is fixed in advance.\n",
       "  - Convert in the training set any word that is not in this set (any OOV word) to\n",
       "the unknown word token <UNK> in a text normalization step.\n",
       "  - Estimate the probabilities for <UNK> from its counts just like any other regular\n",
       "word in the training set.\n",
       "- second way \n",
       "  - we don’t have a prior vocabulary in advance,\n",
       "is to create such a vocabulary implicitly, replacing words in the training data\n",
       "by <UNK> based on their frequency. For example we can replace by <UNK> all words\n",
       "that occur fewer than n times in the training set, where n is some small number, or\n",
       "equivalently select a vocabulary size V in advance (say 50,000) and choose the top\n",
       "V words by frequency and replace the rest by UNK. In either case we then proceed\n",
       "to train the language model as before, treating <UNK> like a regular word\n",
       "- The exact choice of <UNK> model does have an effect on metrics like perplexity.\n",
       "A language model can achieve low perplexity by choosing a small vocabulary and\n",
       "assigning the unknown word a high probability. For this reason, perplexities should\n",
       "only be compared across language models with the same vocabularies\n",
       "\n",
       "\n",
       "## SLP Chapter 3.5 Smoothing\n",
       "\n",
       "- Laplace\n",
       "  - All the counts that used to be zero will now\n",
       "have a count of 1, the counts of 1 will be 2, and so on.\n",
       "  - Laplace smoothing does not perform well enough to be used in modern n-gram models, but it usefully introduces many of the concepts that we\n",
       "see in other smoothing algorithms, gives a useful baseline, and is also a practical\n",
       "smoothing algorithm for other tasks like text classification\n",
       "  - for unigram: $P(w_i) = c_i/N$\n",
       "  - Laplace: $P_{Laplace} = (c_i+1)/(N+V)$, which V is the number of words in the vocabulaty\n",
       "  - it is convenient to\n",
       "describe how a smoothing algorithm affects the numerator, by defining an adjusted\n",
       "count c*. This adjusted count is easier to compare directly with the MLE counts and\n",
       "can be turned into a probability like an MLE count by normalizing by N. We can now turn c*\n",
       "i into a probability P*\n",
       "i by normalizing by N.\n",
       "  - A related way to view smoothing is as discounting (lowering) some non-zero\n",
       "counts in order to get the probability mass that will be assigned to the zero counts.\n",
       "  - for bigram\n",
       "  - $P^*_{Laplace}(w_n|w_{n-1})=[C(w_{n-1}w_n)+1]/[\\sum_w(C(w_{n-1}w)+1)]$\n",
       "    - $=[C(w_{n-1}w_n)+1]/[C(w_{n-1}+V)]$\n",
       "  - the reconstructed counts.\n",
       "  - $c^*(w_{n-1}w_n) = [C(w_{n-1}w_n)+1]C(w_{n-1})/[C(w_{n-1})+V]$\n",
       "  - The sharp change in counts and probabilities occurs because too much probability\n",
       "mass is moved to all the zeros.\n",
       "- Add-k Smoothing\n",
       "  - One alternative to add-one smoothing is to move a bit less of the probability mass\n",
       "from the seen to the unseen events. Instead of adding 1 to each count, we add a fractional count k (.5? .05? .01?).\n",
       "  - $P^*_{Add-K}(w_n|w_{n-1})==[C(w_{n-1}w_n)+k]/[C(w_{n-1}+kV)]$\n",
       "  - utilize validation set tune hyperparameter\n",
       "- Backoff and Interpolation\n",
       "  - if there is no example of 4-gram, then move to tri gram, and so on, until unigram\n",
       "  - sometimes using less context is a good thing, helping to generalize\n",
       "more for contexts that the model hasn’t learned much about.\n",
       "  - In backoff, we use the trigram if the evidence is\n",
       "sufficient, otherwise we use the bigram, otherwise the unigram. Or we\n",
       "only “back off” to a lower-order n-gram if we have zero evidence for a higher-order n-gram.\n",
       "  - In a backoff n-gram model, if the n-gram we need has zero counts, we approximate\n",
       "it by backing off to the (N-1)-gram. We continue backing off until we reach a\n",
       "history that has some counts.\n",
       "  - In order for a backoff model to give a correct probability distribution, we have\n",
       "discount to discount the higher-order n-grams to save some probability mass for the lower\n",
       "order n-grams. \n",
       "  - in interpolation, we always mix the probability estimates from\n",
       "all the n-gram estimators, weighing and combining the trigram, bigram, and unigram\n",
       "counts.\n",
       "  - interpolation: $\\hat{P}(w_n|w_{n-2}w_{n-1}) = \\lambda_1 P(w_n)+ \\lambda_2 P(w_n|w_{n-1}) + \\lambda_3 P(w_n|w_{n-2}w_{n-1})$ s.t. $\\sum \\lambda =1$\n",
       "  - That is, we fix\n",
       "the n-gram probabilities and then search for the $\\lambda$ values that give us the highest probability of the held-out set, \n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n",
       "\n",
       "- Efficiency considerations are important when building language models that use\n",
       "such large sets of n-grams. Rather than store each word as a string, it is generally\n",
       "represented in memory as a 64-bit hash number, with the words themselves stored\n",
       "on disk. Probabilities are generally quantized using only 4-8 bits (instead of 8-byte\n",
       "floats), and n-grams are stored in reverse tries.\n",
       "- An n-gram language model can also be shrunk by pruning, for example only\n",
       "storing n-grams with counts greater than some threshold\n",
       "- The algorithm is called\n",
       "- stupid backoff. Stupid backoff gives up the idea of trying to make the language\n",
       "model a true probability distribution. There is no discounting of the higher-order\n",
       "probabilities. If a higher-order n-gram has a zero count, we simply backoff to a\n",
       "lower order n-gram, weighed by a fixed (context-independent) weight\n",
       "\n",
       "\n",
       "## Module 10 Lecture Notes\n",
       "\n",
       "- Uncertainty arises because of both laziness and ignorance. It is inescapable in complex,  nondeterministic, or partially observable environments. \n",
       "- Probabilities express the agent’s inability to reach a definite decision regarding the  truth of a sentence. Probabilities summarize the agent’s beliefs relative to the evidence.\n",
       "- Decision theory combines the agent’s beliefs and desires, defining the best action as  the one that maximizes expected utility.  \n",
       "- Basic probability statements include prior or unconditional probabilities and posterior or conditional probabilities over simple and complex propositions. \n",
       "- The axioms of probability constrain the probabilities of logically related propositions.  An agent that violates the axioms must behave irrationally in some cases.  \n",
       "- The full joint probability distribution specifies the probability of each complete assignment of values to random variables. It is usually too large to create or use in its  explicit form, but when it is available it can be used to answer queries simply by adding  up entries for the possible worlds corresponding to the query propositions.  \n",
       "- Absolute independence between subsets of random variables allows the full joint distribution to be factored into smaller joint distributions, greatly reducing its complexity. \n",
       "- Bayes’ rule allows unknown probabilities to be computed from known conditional  probabilities, usually in the causal direction. Applying Bayes’ rule with many pieces of  evidence runs into the same scaling problems as does the full joint distribution.  \n",
       "- Conditional independence brought about by direct causal relationships in the domain  allows the full joint distribution to be factored into smaller, conditional distributions.  The naive Bayes model assumes the conditional independence of all effect variables,  given a single cause variable; its size grows linearly with the number of effects.\n",
       "- A wumpus-world agent can calculate probabilities for unobserved aspects of the world,  thereby improving on the decisions of a purely logical agent. Conditional independence  makes these calculations tractable. \n",
       "\n",
       "\n",
       "## Module 10 Definitions\n",
       "* **Belief state** - a representation of the set of all possible world states that it might be in\n",
       "* **Qualification  problem** - conditions can be  deduced for sure, so we can’t infer that the plan succeeds.\n",
       "* **Ontological commitments** - the world is  composed of facts that do or do not hold in any particular case, degree of belief between 0  and 1.\n",
       "* **Epistemological  commitments** - a logical agent believes each sentence to be true or false or has  no opinion\n",
       "* **Utility theory** - represent preference and reason quantitatively with them, utility: usefulness\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n",
       "## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n",
       " \n",
       "- Models describe how (a portion of) the world works\n",
       "- Models are always simplifications\n",
       "  - May not account for every variable\n",
       "  - May not account for all interactions between variables\n",
       "- Two problems with using full joint distribution tables\n",
       "as our probabilistic models:\n",
       "  - Unless there are only a few variables, the joint is WAY too\n",
       "big to represent explicitly\n",
       "  - Hard to learn (estimate) anything empirically about more\n",
       "than a few variables at a time\n",
       "- Bayesian Network: a technique for describing complex joint\n",
       "distributions (models) using simple, local\n",
       "distributions (conditional probabilities)\n",
       "  - We describe how variables locally interact\n",
       "  - Local interactions chain together to give global, indirect\n",
       "interactions\n",
       "  - Each node corresponds to a random variable, which may be discrete or continuous. \n",
       "  - Directed links or arrows connect pairs of nodes. \n",
       "  - Each node Xi has associated probability information θ(Xi |Parents(Xi)) that quantifies the effect of the parents on the node using a finite number of parameters. \n",
       "- Bayes Net Semantics:\n",
       "  - A set of nodes, one per variable X\n",
       "  - A directed, acyclic graph\n",
       "  - A conditional distribution for each node\n",
       "  - Bayes net = Topology (graph) + Local Conditional Probabilities\n",
       "  - Bayes’ nets implicitly encode joint distributions\n",
       "    - As a product of local conditional distributions\n",
       "    - To see what probability a BN gives to a full assignment, multiply all the\n",
       "relevant conditionals together\n",
       "    - Not every BN can represent every joint distribution\n",
       "      - The topology enforces certain conditional independencies\n",
       "- The  intuitive meaning of an arrow is typically that X has a direct influence on Y, which suggests  that causes should be parents of effects.\n",
       "- conditioning case is just a possible combination of values for the parent  nodes—a miniature possible world\n",
       "- each row must sum to 1, because the entries  represent an exhaustive set of cases for the variable.\n",
       "- In general, a table for a Boolean variable with k  Boolean parents contains 2k independently specifiable probabilities. A node with no parents  has only one row, representing the prior probabilities of each possible value of the variable. \n",
       "- some insights:\n",
       "  - When Bayes’ nets reflect the true causal patterns:\n",
       "    - Often simpler (nodes have fewer parents)\n",
       "    - Often easier to think about\n",
       "    - Often easier to elicit from experts\n",
       "  - BNs need not actually be causal\n",
       "  - What do the arrows really mean?： Topology really encodes conditional independence\n",
       "\n",
       "\n",
       "## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n",
       "\n",
       "- Bayes net, assumptions:\n",
       "  -  $P(x_i|x_1,...,x_{i-1}) = P(x_i|parents(X_i))$\n",
       " $P(x_1,...,x_n) = \\prod_{i=1}^n P(x_i|parents(X_i))$<br />  \n",
       "- product of joint possibility for the final probability\n",
       "- Bayes net construction\n",
       " $P(x_1,...,x_n) = \\prod_{i=1}^n P(x_i|x_{i-1},...,x_1)$<br /> \n",
       "- net ordering, any order will work but will have different compactness\n",
       "- Intuitively, the parents of node Xi should contain all those nodes in X1,..., Xi−1 that directly  influence Xi.\n",
       "- Another important property of Bayes nets is that they contain  no redundant probability values. If there is no redundancy, then there is no chance for inconsistency: it is impossible for the knowledge engineer or domain expert to create a Bayesian  network that violates the axioms of probability. \n",
       "- Bayes net can be more compact, in most domains each random variable is directly influenced by at most k others, for some constant k. If we assume  n Boolean variables for simplicity, then the amount of information needed to specify each  conditional probability table will be at most $2^k$ numbers, and the complete network can be  specified by $2^k · n$ numbers.\n",
       "- Incorrect orders result in more links, If we stick to a causal model, we end up  having to specify fewer numbers, and the numbers will often be easier to come up with.\n",
       "- Are X and Y conditionally independent given\n",
       "evidence variables {Z}?\n",
       "  - Yes, if X and Y “d-separated” by Z\n",
       "  - Consider all (undirected) paths from X to Y\n",
       "  - No active paths = independence!\n",
       "- A path is active if each triple is active:\n",
       "  - Causal chain A $\r",
       "ightarrow$ B $\r",
       "ightarrow$ C where B is unobserved (either direction)\n",
       "  - Common cause A $\\leftarrow$ B $\r",
       "ightarrow$ C where B is unobserved\n",
       "  - Common effect (aka v-structure)\n",
       "  - A $\r",
       "ightarrow$ B $\\leftarrow$ C where B or one of its descendents is observed\n",
       "\n",
       "\n",
       "## AIMA 13.3 Exact Inference in Bayesian Networks\n",
       "\n",
       "- probabilistic inference system is to compute the posterior probability  distribution for a set of query variables, given some observed event—usually, some assignment of values to a set of evidence variables.\n",
       "- the joint distribution can be written as products of conditional probabilities from the network. Therefore, a query can  be answered using a Bayes net by computing sums of products of conditional probabilities  from the network. \n",
       "- query In the worst case, where we have to sum out almost all the variables, there will be  $O(2^n)$ terms in the sum, each a product of $O(n)$ probability values. A naive implementation  would therefore have complexity $O(n2^n)$. \n",
       "- Variable elimination:\n",
       "  - every variable that is not an ancestor of a query variable or evidence variable is irrelevant to the query. \n",
       "- Complexity:\n",
       "  - if at most one undirected path between any  two nodes in the network. singly connected networks or polytrees, The time and space complexity of exact inference in polytrees is linear in the size of the network. \n",
       "  - multiply connected This is not surprising when one considers that  because it includes inference in propositional logic as a special case, inference in Bayes nets  is NP-hard.\n",
       "  - in a polytree network, one would need to issue $O(n)$ queries  costing $O(n)$ each, for a total of $O(n^2 )$ time. Using clustering algorithms the time can be reduced to $O(n)$.\n",
       "  - clustering reduce complexity, result in meganode\n",
       "\n",
       "\n",
       "## Module 11 Lecture Notes\n",
       " \n",
       "- A Bayesian network is a directed acyclic graph whose nodes correspond to random  variables; each node has a conditional distribution for the node, given its parents. \n",
       "- Bayesian networks provide a concise way to represent conditional independence relationships in the domain. \n",
       "- A Bayesian network specifies a joint probability distribution over its variables. The  probability of any given assignment to all the variables is defined as the product of the  corresponding entries in the local conditional distributions. A Bayesian network is often  exponentially smaller than an explicitly enumerated joint distribution.\n",
       "- Many conditional distributions can be represented compactly by canonical families of  distributions. Hybrid Bayesian networks, which include both discrete and continuous  variables, use a variety of canonical distributions.  \n",
       "- Inference in Bayesian networks means computing the probability distribution of a set  of query variables, given a set of evidence variables. Exact inference algorithms, such  as variable elimination, evaluate sums of products of conditional probabilities as effi-  ciently as possible.  \n",
       "- In polytrees (singly connected networks), exact inference takes time linear in the size  of the network. In the general case, the problem is intractable. \n",
       "- Bayes nets compactly encode joint distributions\n",
       "- Guaranteed independencies of distributions can be\n",
       "deduced from BN graph structure\n",
       "- D-separation gives precise conditional independence\n",
       "guarantees from graph alone\n",
       "- A Bayes’ Net’s joint distribution may have further\n",
       "(conditional) independence that is not detectable until\n",
       "you inspect its specific distribution\n",
       " \n",
       "\n",
       "## Module 11 Definitions\n",
       "* **Go back and fill in the meaning** - definition\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "## WARNING: Your summary of 'AIMA Chapter 19.4 Model Selection and Optimization' is too long!\n",
       "# Module 12: Naive Bayes and Perceptrons\n",
       "## AIMA Chapter 19.1 Forms of Learning\n",
       "\n",
       "- agent is learning if it improves its performance after making observations about the world. \n",
       "- if agent is computer machine learning: a computer observes some data, builds a model based on the data, and uses the model as both a hypothesis about the world and a piece of software that  can solve problems. \n",
       "  - why learning, 1. the designers cannot anticipate all possible future  situations 2. sometimes the designers have no idea how  to program a solution themselves.\n",
       "- improvements:\n",
       "  - Which component is to be improved. \n",
       "  - What prior knowledge the agent has, which influences the model it builds. \n",
       "  - What data and feedback on that data is available. \n",
       "- components:\n",
       "  - A direct mapping from conditions on the current state to actions. \n",
       "  - A means to infer relevant properties of the world from the percept sequence. \n",
       "  - Information about the way the world evolves and about the results of possible actions  the agent can take. \n",
       "  - Utility information indicating the desirability of world states. \n",
       "  - Action-value information indicating the desirability of actions. \n",
       "  -  Goals that describe the most desirable states. \n",
       "  - A problem generator, critic, and learning element that enable the system\n",
       "- prior knowledge on the part of the agent: it starts from scratch  and learns from the data.\n",
       "- Going from a specific set of observations to a general rule is called induction;\n",
       "- factored representation—a  vector of attribute values.\n",
       "- types of Learning:\n",
       "  - In supervised learning the agent observes input-output pairs and learns a function that maps from input to output.\n",
       "  - In unsupervised learning the agent learns patterns in the input without any explicit Unsupervised  learning  feedback. The most common unsupervised learning task is clustering: detecting potentially useful clusters of input examples.\n",
       "  - In reinforcement learning the agent learns from a series of reinforcements: rewards and punishments.\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.2 Supervised Learning\n",
       "\n",
       "- Given a training set of N example input–output pairs, discover a function h that approximates the true function, h is called a hypothesis about the world. It is drawn from a hypothesis space H of possible functions.\n",
       "- h is a model of the data, drawn from a model  class H, or we can say a function drawn from a function class. We call the output yi the ground truth—the true answer we are asking our model to predict.\n",
       "- exploratory data analysis: examining Exploratory data  analysis  the data with statistical tests and visualizations\n",
       "- The true measure of a hypothesis is not how it does on the training set, but rather how  well it handles inputs it has not yet seen. on test set\n",
       "- say that h generalizes well if it accurately predicts the outputs of the test set. \n",
       "- By bias we mean (loosely) the tendency of a predictive hypothesis to deviate from the  expected value when averaged over different training sets.\n",
       "- We say that a hypothesis is underfitting when it fails to find a pattern in the data.\n",
       "- variance we mean the amount of change in the hypothesis due to fluctuation in the training data.\n",
       "- is overfitting the data when it pays too much attention to the particular data set it is trained on, causing it to perform poorly on unseen data. \n",
       "- bias–variance tradeoff: a choice between more complex, low-bias hypotheses that fit the training data well and simpler, low-variance hypotheses that may generalize better.\n",
       "- <strong>perceptron 1</strong>\n",
       "  - Inputs are feature values\n",
       "  - Each feature has a weight\n",
       "  - Sum is the activation\n",
       "  - $activation(x) = \\sum_i w_i\\cdot f_i(x)= w\\cdot f(x)$\n",
       "  - If the activation is:\n",
       "    - Positive, output +1\n",
       "    - Negative, output -1\n",
       "- Binary case: compare features to a weight vector\n",
       "- Learning: figure out the weight vector from examples\n",
       "  - Start with weights = 0\n",
       "  - For each training instance:\n",
       "    - Classify with current weights\n",
       "    - If correct (i.e., y=y*), no change!\n",
       "    - If wrong: adjust the weight vector by adding or subtracting the\n",
       "feature vector. Subtract if y* is -1. $w = w + \\hat y \\cdot f$\n",
       "- Multiclass Decision:\n",
       "  - A weight vector for each class: $w_y$\n",
       "  - Score (activation) of a class y: $w_y\\cdot f(x)$\n",
       "  - Prediction highest score wins: $\\hat y =  argmax_y w_y\\cdot f(x)$\n",
       "- Learning: Multiclass Perceptron\n",
       "  - Start with all weights = 0\n",
       "  - Pick up training examples one by one\n",
       "  - Predict with current weights: $\\hat y =  argmax_y w_y\\cdot f(x)$\n",
       "  - If correct, no change!\n",
       "  - If wrong: lower score of wrong\n",
       "answer, raise score of right answer: $w_y = w_y-f(x)$, $w_{y*} = w_{y*} + f(x)$\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.3 Learning Decision Trees\n",
       "\n",
       "- decision tree is a representation of a function that maps a vector of attribute values to Decision tree  a single output value—a “decision.” A decision tree reaches its decision by performing a  sequence of tests, starting at the root and following the appropriate branch until a leaf is  reached.\n",
       "- no way to represent all kinds of functions. Even just considering Boolean functions with n Boolean attributes, the truth table will have $2^n$ rows, so there are  $2^2^n$ different functions.\n",
       "- using the notion of information gain, which is defined in terms of entropy, which is the fundamental quantity in information theory\n",
       "- Entropy is a measure of the uncertainty of a random variable; the more information, the  less entropy.\n",
       "$H(V) = - \\sum_{k}P(v_k)\\log P(v_k)$\n",
       "- Info gain: \n",
       " infoGain = $IG(X) = H(Y)-H(Y|X) =  - \\sum_y P(Y=y)\\log_2 P(Y=y)-  \\sum_x P(X=x)H(Y|X=x) =  - \\sum_y P(Y=y)\\log_2 P(Y=y) + \\sum_{x,y} P(X=x,Y=y) \\log_2 P(Y=y|X=x) $\n",
       "- Overfitting becomes more likely as the number of attributes  grows, and less likely as we increase the number of training examples. Larger hypothesis  spaces (e.g., decision trees with more nodes or polynomials with high degree) have more capacity both to fit and to overfit; some model classes are more prone to overfitting than  others. \n",
       "- decision tree pruning combats overfitting.  works by eliminating nodes that are not clearly relevant.\n",
       "- The name CART, standing for Classification And Regression Trees, is used to cover both classes. \n",
       "- <strong>perceptron 2</strong>\n",
       "- Properties\n",
       "  - Separability: true if some parameters get the training\n",
       "set perfectly correct\n",
       "  - Convergence: if the training is separable, perceptron\n",
       "will eventually converge (binary case)\n",
       "  - Mistake Bound: the maximum number of mistakes\n",
       "(binary case) related to the margin or degree of\n",
       "separability: $mistake < k/\\delta^2$\n",
       "- Improving the perceptron:\n",
       "  - Noise: if the data isn’t separable,\n",
       "weights might thrash\n",
       "    - Averaging weight vectors over\n",
       "time can help (averaged\n",
       "perceptron)\n",
       "  - Mediocre generalization: finds a\n",
       "“barely” separating solution, not seperated in the center\n",
       "    - SVM\n",
       "  - Overtraining: test / held-out\n",
       "accuracy usually rises, then falls\n",
       "- Fixing perceptron:\n",
       "  - Idea: adjust the weight update to mitigate these effects\n",
       "- MIRA:\n",
       "  - Choose an update size that fixes the current mistake but, minimizes the change to w\n",
       "  - $\tau = (w'_y-w'_{y*}\\cdot f+1)/2f\\cdot f$, The +1 helps to generalize\n",
       "  - $w_y = w_y-\tau f(x)$, $w_{y*} = w_{y*} + \tau f(x)$\n",
       "- SVM:\n",
       "  - Maximizing the margin: good according to intuition, theory, practice\n",
       "  - Only support vectors matter; other training examples are ignorable\n",
       "  - Support vector machines (SVMs) find the separator with max\n",
       "margin\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.4 Model Selection and Optimization\n",
       "\n",
       "- will make the assumption that the future examples will be like the past. We call  this the stationarity assumption; without it, all bets are off.\n",
       "- We assume that each example Ej Stationarity  has the same prior probability distribution and is independent of the previous examples: are independent and identically distributed or i.i.d..\n",
       "- k-fold cross-validation. The idea is that each example serves double duty—as training data and validation data—but not at the same time. \n",
       "- is k = n, also  LOOCV known as leave-one-out cross-validation or LOOCV.\n",
       "- We can think of the task of finding a good hypothesis as two subtasks: model selection chooses a good hypothesis space, and optimization (also called training) finds the best hypothesis within that space. \n",
       "- We say that a model that exactly fits all the training data has interpolated the data Model classes typically start to overfit as the capacity approaches the point of interpolation. \n",
       "- to minimize a loss function rather than Loss function  maximize a utility function.\n",
       "- about fitting\n",
       "  - we say that a learning problem is realizable if the hypothesis space H actually  contains the true function\n",
       "  - variance  means that a learning algorithm will in general return different hypotheses for different sets  of examples.\n",
       "  - f may be nondeterministic or noisy—it may return  different values of f(x) for the same x.\n",
       "  - when H is a complicated function in a large hypothesis space,  it can be computationally intractable to systematically search all possibilities; in that case,  a search can explore part of the space and return a reasonably good hypothesis, but can’t  always guarantee the best one. \n",
       "- small-scale learning, where the number of training examples ranged from dozens to the low  thousands.\n",
       "- large-scale learning, with millions of  examples.\n",
       "- regularization: This process of explicitly penalizing complex hypotheses\n",
       "  - choices: the  loss function (L1 or L2), and the complexity measure, which is called a regularization function.\n",
       "  - A  process of feature selection can be performed to discard attributes that appear to be irrelevant.\n",
       "- grid search : try all combinations of Grid search  values and see which performs best on the validation data.\n",
       "- random search:  samples  uniformly from the set of all possible hyperparameter settings\n",
       "- Bayesian optimization treats the task of choosing good hyperparameter values as a  machine learning problem in itself.\n",
       "- If we make the assumption that f can be approximated  by a Gaussian process, then the math of updating our belief about f works out nicely.\n",
       "- <strong>perceptron 3</strong>\n",
       "- Non-Linear Separators\n",
       "  - mapping data to a higher-dimensional space\n",
       "  - the original feature space can always be mapped to some\n",
       "higher-dimensional feature space where the training set is separable:\n",
       "- Comparison:\n",
       "  - Naïve Bayes\n",
       "    - Builds a model training data\n",
       "    - Gives prediction probabilities\n",
       "    - Strong assumptions about feature independence\n",
       "    - One pass through data (counting)\n",
       "  - Perceptrons / MIRA:\n",
       "    - Makes less assumptions about data\n",
       "    - Mistake-driven learning\n",
       "    - Multiple passes through data (prediction)\n",
       "    - Often more accurate\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.5 The Theory of Learning\n",
       "\n",
       "- any hypothesis that is consistent with a sufficiently large set of training examples is  unlikely to be seriously wrong: that is, it must be probably approximately correct (PAC). \n",
       "- like all theorems, are logical consequences of axioms. When a  theorem (as opposed to, say, a political pundit) states something about the future based on  the past, the axioms have to provide the “juice” to make that connection.\n",
       "- A hypothesis h is called approximately correct if error(h) ≤ ǫ, where ǫ is a small constant. and error is error rate\n",
       "- decision lists. A  decision list consists of a series of tests, each of which is a conjunction of literals. If a  test succeeds when applied to an example description, the decision list specifies the value  to be returned. If the test fails, processing continues with the next test in the list.\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.6 Linear Regression and Classification\n",
       "\n",
       "- the class of linear functions of continuous-valued  inputs.\n",
       "- Many forms of learning involve adjusting weights to minimize a loss, so it helps to have a  mental picture of what’s going on in weight space\n",
       "- loss function is convex \n",
       "$w = \\frac{\\sum_i x_i y_i}{\\sum_i x_i^2}$\n",
       "- gradient descent:\n",
       "   - \n",
       "$w_i \\leftarrow w_i + \\alpha\\frac{\\partial}{\\partial w_i}Loss(w)$\n",
       "- batch gradient descent learning rule for univariate linear re- Batch gradient  descent  gression (also called deterministic gradient descent). The loss surface is convex. A  step that covers all the training examples is called an epoch.\n",
       "- A faster variant is called stochastic gradient descent or SGD: it randomly selects a small Stochastic gradient  descent  number of training examples at each step, and updates\n",
       "- multivariable linear regression\n",
       "  - \n",
       "$w* = (X^T X)^{-1}X^Ty$\n",
       "- regularization: Cost(h) = EmpLoss(h) +λComplexity(h). \n",
       "- L1 regularization has an important advantage: it tends to produce a sparse model. That is, it often sets many weights to zero\n",
       "- A decision boundary is a line (or a surface, in higher dimensions) that separates the  two classes. A linear decision boundary is called a linear separator and data that admit such a separator are called linearly separable.\n",
       "- percepton:\n",
       "  - If the output is correct (i.e., y=hw(x)) then the weights are not changed. \n",
       "  - If y is 1 but hw(x) is 0, then wi is increased when the corresponding input xi is positive  and decreased when xi is negative. This makes sense, because we want to make w · x  bigger so that hw(x) outputs a 1. \n",
       "  -  If y is 0 but hw(x) is 1, then wi is decreased when the corresponding input xi is positive  and increased when xi is negative. This makes sense, because we want to make w · x  smaller so that hw(x) outputs a 0. \n",
       "- logistic regression. There is no easy closed-form solution to find the optimal value of w with Logistic regression  this model, but the gradient descent computation is straightforward.\n",
       "- \n",
       "$\\frac{\\partial}{\\partial w_i}Loss(w) = -2(y-h_w(x)) \\times g'(w\\cdot x) \\times x_i$ <br />$w_i \\leftarrow w_i + \\alpha (y-h_w(x))h_w(x)(1-h_w(x))x_i$<br />$h_w(x) = \\frac{1}{1+e^{-w x}}$\n",
       "\n",
       "## SLP Chapter 4.1 Naive Bayes Classifiers\n",
       "\n",
       "- General Naive Bayes: $P(Y,F_1,...,F_n) = P(Y)\\prod_i P(F_i|Y)$\n",
       "- We only have to specify how each feature depends on the class\n",
       "Total number of parameters is linear in number of features\n",
       "Model is very simplistic, but often works anyway\n",
       "- multinomial naive Bayes classifier: so called because it is a Bayesian classifier that makes a simplifying (naive) assumption about how the features interact.\n",
       "- bag-of-words, that is, an unordered set of words with their position\n",
       "ignored, keeping only their frequency in the document\n",
       "- at notation ˆ to mean “our\n",
       "estimate of the correct class\n",
       "- $\\hat{c}=argmax_{c\\in C}P(c|d) = argmax_{c\\in C}P(d|c)P(c)/P(d) = argmax_{c\\in C}P(d|c)P(c) $ because $P(d)$ is constant\n",
       "- Naive Bayes is a probabilistic classifier, meaning that for a document d, out of\n",
       "all classes c 2 C the classifier returns the class hat(c) which has the maximum posterior probability given the document.\n",
       "- We call Naive Bayes a generative model because we can read equation stating\n",
       "a kind of implicit assumption about how a document is generated: first a class is\n",
       "sampled from P(c), and then the words are generated by sampling from P(d|c).\n",
       "- we compute the most probable class ˆ c given some\n",
       "document d by choosing the class which has the highest product of two probabilities:\n",
       "the prior probability of prior the class P(c) and the likelihood of the document P(d|c):\n",
       "- we can represent a document d as a set of features\n",
       "- $\\hat{c}= argmax_{c\\in C}P(f_1,f_2,...,f_n|c)P(c) $,\n",
       "- first, we assume\n",
       "that the features f1; f2; ...; fn only encode word identity and not position.\n",
       "- second, naive Bayes assumption: this is the condiassumption\n",
       "tional independence assumption that the probabilities P( fi|c) are independent given\n",
       "the class c and hence can be ‘naively’ multiplied as follows:\n",
       "- $P(f_1,f_2,...,f_n|c) = P(f_1|c)P(f_2|c)...P(f_n|c)$\n",
       "- $c_{NB} = argmax_{c\\in C}P(c)\\prod_{f\\in F}P(f|c)$\n",
       "- To apply the naive Bayes classifier to text, we need to consider word positions, by\n",
       "simply walking an index through every word position in the document:\n",
       "- $c_{NB} = argmax_{c\\in C}P(c)\\prod_{i \\in Positions}P(w_i|c)$\n",
       "- like calculations for language modeling, are done in log\n",
       "space, to avoid underflow and increase speed.\n",
       "- $c_{NB} = argmax_{c\\in C}log P(c)\\sum_{i \\in Positions}log P(w_i|c)$\n",
       "\n",
       "\n",
       "## SLP Chapter 4.2 Training the Naive Bayes Classifier\n",
       "\n",
       "-Let Nc be the number of documents in our training data with class c and Ndoc be the total number of documents\n",
       "- $\\hat{P}(c)=N_c/N_{doc}$\n",
       "- $\\hat{P}(w_i|c)=count(w_i,c)/\\sum_{w\\in V} count(w,c)$\n",
       "- Add laplace to avoid zeros\n",
       "- $\\hat{P}(w_i|c)=[count(w_i,c)+1]/\\sum_{w\\in V} [count(w,c)+1]$\n",
       "    - $= [count(w_i,c)+1]/[[\\sum_{w\\in V} count(w,c)]+V]$\n",
       "- processes:\n",
       "  - unknown words is to ignore them—remove them from the test\n",
       "document and not include any probability for them at all.\n",
       "  - ignore another class of words: stop\n",
       "stop words words, very frequent words like the and a\n",
       " \n",
       "\n",
       "## SLP Chapter 4.3 Worked Example\n",
       " Your summary here \n",
       "\n",
       "## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n",
       "\n",
       "- binary NB  uses the except\n",
       "that for each document we remove all duplicate words before concatenating them\n",
       "into the single big document.\n",
       "- addition commonly made when doing text classification for\n",
       "sentiment is to deal with negation. Consider the difference between I really like this\n",
       "movie (positive) and I didn’t like this movie (negative). The negation expressed by\n",
       "didn’t completely alters the inferences we draw from the predicate like. Similarly,\n",
       "negation can modify a negative word to produce a positive review\n",
       "- during text normalization, prepend the prefix NOT to\n",
       "every word after a token of logical negation (n’t, not, no, never) until the next punctuation\n",
       "mark.\n",
       "- NOT like, NOT recommend will thus occur more often\n",
       "in negative document and act as cues for negative sentiment, while words like\n",
       "NOT bored, NOT dismiss will acquire positive associations.\n",
       "- in some situations we might have insufficient labeled training data to\n",
       "train accurate naive Bayes classifiers using all words in the training set to estimate\n",
       "positive and negative sentiment\n",
       "- sentiment lexicons, lists of words that are preannotated with positive or negative sentiment.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n",
       "\n",
       "- For other tasks, like language id—determining what language a given piece of text is written in—the most effective naive Bayes features are not words at all,\n",
       "but character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,\n",
       "‘thei’), or, even simpler byte n-grams, where instead of using the multibyte Unicode\n",
       "character representations called codepoints, we just pretend everything is a string of\n",
       "raw bytes.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.6 Naive Bayes as a Language Model\n",
       "\n",
       "- a naive Bayes model can be viewed as a\n",
       "set of class-specific unigram language models, in which the model for each class\n",
       "instantiates a unigram language model.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n",
       "\n",
       "- gold labels: human labels\n",
       "- Precision measures the percentage of the items that\n",
       "the system detected (i.e., the system labeled as positive) that are in fact positive (i.e.,\n",
       "are positive according to the human gold labels).\n",
       "- $precision = tp/(tp+fp)$\n",
       "- Recall measures the percentage of items actually present in the input that were\n",
       "correctly identified by the system.\n",
       "- $recall = tp/(tp+fn)$\n",
       "- $F_1=2PR/(P+R)$\n",
       "- $F_b = (b^2+1)PR/(b^2P+R)$\n",
       "- Values of b > 1 favor recall, while\n",
       "values of b < 1 favor precision.\n",
       "- Harmonic mean is used because it is a conservative metric; the harmonic mean of\n",
       "two values is closer to the minimum of the two values than the arithmetic mean is.\n",
       "Thus it weighs the lower of the two numbers more heavily.\n",
       "- recall and precision for each row\n",
       "- In macroaveraging, we compute the performance\n",
       "microaveraging for each class, and then average over classes.\n",
       "- In microaveraging, we collect the decisions\n",
       "for all classes into a single confusion matrix, and then compute precision and\n",
       "recall from that table.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.8 Test sets and Cross-validation\n",
       "\n",
       "- effect size $\\delta(x) = M(A,x)-M(B,x)$, A is new B is before\n",
       "- That’s because A might just be accidentally better than B on this particular x.\n",
       "We need something more: we want to know if A’s superiority over B is likely to hold\n",
       "again if we checked another test set x0, or under some other set of circumstances.\n",
       "- suppose: $H_0:\\delta(x)\\leq 0$, $H_1: \\delta(x)>0$\n",
       "- P(\\delta(X)\\geq \\delta(x)|H_0 is true)\n",
       "- A very small p-value means that the difference we observed is very unlikely\n",
       "under the null hypothesis, and we can reject the null hypothesis. What counts as very\n",
       "small? It is common to use values like .05 or .01 as the thresholds. A value of .01\n",
       "means that if the p-value (the probability of observing the d we saw assuming H0 is\n",
       "true) is less than .01, we reject the null hypothesis and assume that A is indeed better\n",
       "than statistically B.\n",
       "- bootstrap test:\n",
       "- The intuition\n",
       "of the bootstrap test is that we can create many virtual test sets from an observed\n",
       "test set by repeatedly sampling from it. The method only makes the assumption that\n",
       "the sample is representative of the population.\n",
       "\n",
       "\n",
       "## SLP Chapter 4.9 Statistical Significance Testing\n",
       "\n",
       "- Assuming H0 (A isn’t better than B), we would expect that d(X), estimated\n",
       "over many test sets, would be zero; a much higher value would be surprising,\n",
       "since H0 specifically assumes A isn’t better than B. To measure exactly how surprising\n",
       "is our observed d(x) we would in other circumstances compute the p-value by\n",
       "counting over many test sets how often d(x(i)) exceeds the expected zero value by\n",
       "d(x) or more:\n",
       "- $p-value(x) = 1/b \\sum_{i=1}^b 1 if (\\delta(x^{(i)})\\geq 2\\delta(x))$\n",
       "- this isn’t true for the bootstrapped test\n",
       "sets we created. That’s because we didn’t draw these samples from a distribution\n",
       "with 0 mean; we happened to create them from the original test set x, which happens\n",
       "to be biased (by .20) in favor of A. So to measure how surprising is our observed\n",
       "d(x), we actually compute the p-value by counting over many test sets how often\n",
       "d(x(i)) exceeds the expected value of d(x) by d(x) or more:\n",
       "- Confidence:\n",
       "  - The confidence of a probabilistic classifier:\n",
       "    - Posterior over the top label: $confidence(x) = max_y P(y|x)$\n",
       "    - Represents how sure the classifier is of\n",
       "the classification\n",
       "    - Any probabilistic model will have\n",
       "confidences\n",
       "    - No guarantee confidence is correct\n",
       "  - Calibration\n",
       "    - Weak calibration: higher confidences\n",
       "mean higher accuracy\n",
       "    - Strong calibration: confidence predicts\n",
       "accuracy rate\n",
       "    - What’s the value of calibration?\n",
       "\n",
       "\n",
       "## SLP Chapter 4.10 Avoiding Harms in Classification\n",
       "\n",
       "- representational harms: harms caused by a system that demeans a social group, for example by perpetuating\n",
       "negative stereotypes about them.\n",
       "- toxicity detection is the task of detecting hate speech, abuse, harassment, or other\n",
       "detection\n",
       "kinds of toxic language\n",
       "- toxicity classifiers can themselves cause harms.\n",
       "- flag under represented minority\n",
       "\n",
       "\n",
       "## Module 12 Lecture Notes\n",
       "\n",
       "- Naïve Bayes models can incorporate a variety of features, but\n",
       "tend to do best in homogeneous cases\n",
       "\n",
       "\n",
       "## Module 12 Definitions\n",
       "* **Concept** - definition\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "# Module 13: Neural Networks\n",
       "## SLP Chapter 5.1 Classification: the sigmoid\n",
       "\n",
       "- Machine learning classifiers require a training corpus of M observations\n",
       "input/output pairs (x(i),y(i)).\n",
       "  - A feature representation of the input. For each input observation\n",
       "x(i), this will be a vector of features [x1,x2,...,xn].\n",
       "  - A classification function that computes the estimated class !𝑦 via\n",
       "p(y|x).\n",
       "  - An objective function for learning, usually involving minimizing\n",
       "error on training examples.\n",
       "  - An algorithm for optimizing the objective function.\n",
       "- Logistic regression solves this task by learning, from a training set, a vector of\n",
       "weights and a bias term.\n",
       "- $z = w \\cdot x + b$\n",
       "- $\\sigma(z) = 1/(1+e^{-z})$\n",
       "- $P(y=1) = \\sigma(w\\cdot x+b)=1/(1+exp(-(w\\cdot x +b)))$\n",
       "- $P(y=0) = 1-\\sigma(w\\cdot x+b)=exp(-(w\\cdot x +b))/(1+exp(-(w\\cdot x +b)))$\n",
       "- $1-\\sigma(x) = \\sigma(-x)$\n",
       "- feature templates, abstract specifications of features. For example a bigram template\n",
       "templates\n",
       "for period disambiguation might create a feature for every pair of words that occurs\n",
       "before a period in the training set.\n",
       "- representation learning: ways to learn features automatically\n",
       "in an unsupervised way from the input\n",
       "- Logistic regression has a number of advantages over naive\n",
       "Bayes. Naive Bayes has overly strong conditional independence assumptions. Consider\n",
       "two features which are strongly correlated; in fact, imagine that we just add the\n",
       "same feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate,\n",
       "multiplying them both in, overestimating the evidence.\n",
       "- By contrast, logistic\n",
       "regression is much more robust to correlated features; if two features f1 and f2 are\n",
       "perfectly correlated, regression will simply assign part of the weight to w1 and part\n",
       "to w2. Thus when there are many correlated features, logistic regression will assign\n",
       "a more accurate probability than naive Bayes.\n",
       "- naive Bayes can work extremely well on small dataset, naive Bayes is\n",
       "easy to implement and very fast to train\n",
       "\n",
       "\n",
       "## SLP Chapter 5.2 Learning in Logistic Regression\n",
       "\n",
       "- This requires two components that we foreshadowed in the introduction to the\n",
       "chapter. The first is a metric for how close the current label ( ˆ y) is to the true gold\n",
       "label y.\n",
       "-the distance between the system output and the gold output, and we call this distance the loss function or the cost function\n",
       "- The second thing we need is an optimization algorithm for iteratively updating\n",
       "the weights so as to minimize this loss function.\n",
       "\n",
       "\n",
       "## SLP Chapter 5.3 The cross-entropy loss function\n",
       "\n",
       "- We\n",
       "want the loss to be smaller if the model’s estimate is close to correct,\n",
       "and we want the loss to be bigger if it is confused.\n",
       "- $L(\\hat{y},y)$ = How much $\\hat{y}$ differs froma the true $y$\n",
       "- We do this via a loss function that prefers the correct class labels of the training\n",
       "examples to be more likely. This is called conditional maximum likelihood\n",
       "estimation: we choose the parameters w;b that maximize the log probability of\n",
       "the true y labels in the training data given the observations x.\n",
       "- The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.\n",
       "- $p(y|x)=\\hat{y}^y(1-\\hat{y}^{1-y})$\n",
       "- $log\\,p(y|x) = y\\,log\\,\\hat{y}+(1-y)\\,log(1-\\hat{y})$\n",
       "- cross entropy: $L_{CE}(\\hat{y},y)=-log\\,p(y|x) = -[y\\,log\\,\\hat{y}+(1-y)\\,log(1-\\hat{y})]$\n",
       "- $\\hat{y} = \\sigma(w\\cdot x + b)$\n",
       "- We\n",
       "want the loss to be smaller if the model’s estimate is close to correct, and bigger if\n",
       "the model is confused. minimize the loss function\n",
       "\n",
       "\n",
       "## SLP Chapter 5.4 Gradient Descent\n",
       "\n",
       "- $\\hat{θ} = argmin_θ 1/m \\sum_{i=1}^m L_{CE}(f(x^{(i)};θ),y^{(i)})$\n",
       "- logistic regression, is conveniently convex. A convex function\n",
       "has just one minimum\n",
       "- $w^{t+1}=w^t-\\eta \\, d/dw \\, L(f(x;w),y)$\n",
       "- \n",
       "$\\frac{\\partial L_{CE}(\\hat{y},y)}{\\partial w_j}=[\\sigma(w\\cdot x+b)-y]x_j$\n",
       "- The learning rate h is a hyperparameter that must be adjusted. If it’s too high,\n",
       "the learner will take steps that are too large, overshooting the minimum of the loss\n",
       "function. If it’s too low, the learner will take steps that are too small, and take too\n",
       "long to get to the minimum. It is common to start with a higher learning rate and then\n",
       "slowly decrease it, so that it is a function of the iteration k of training\n",
       "- Stochastic gradient descent is called stochastic because it chooses a single random\n",
       "example at a time, moving the weights so as to improve performance on that single\n",
       "example.\n",
       "- batch training we compute the gradient over the entire dataset.\n",
       "By seeing so many examples, batch training offers a superb estimate of which direction\n",
       "to move the weights, at the cost of spending a lot of time processing every\n",
       "single example in the training set to compute this perfect direction.\n",
       "- mini-batch training: we train on a group of m examples (perhaps\n",
       "512, or 1024) that is less than the whole dataset.\n",
       "- Mini-batch training also has the advantage of computational\n",
       "efficiency. The mini-batches can easily be vectorized, choosing the size of the minibatch\n",
       "based on the computational resources\n",
       "- mini-batch gradient = average of sum of individual gradient\n",
       "\n",
       "\n",
       "## SLP Chapter 5.5 Regularization\n",
       "\n",
       "- To avoid overfitting, a new regularization term R(q) is added to the objective\n",
       "function\n",
       "- $\\hat{θ} = argmax_θ \\sum_{i=1}^m log\\,P(y^i|x^i)-α R(θ)$\n",
       "- L2 regularization is a quadratic function of\n",
       "the weight values, named because it uses the (square of the) L2 norm of the weight\n",
       "values. The L2 norm, jjqjj2, is the same as the Euclidean distance of the vector q\n",
       "from the origin.\n",
       "- $\\hat{θ} = argmax_θ [\\sum_{i=1}^m log\\,P(y^i|x^i)]-α \\sum_{j=1}^n θ_j^2$\n",
       "- L1 regularization is a linear function of the weight values, named after the L1 norm, manhattan distance\n",
       "- But where L2 prefers weight vectors with many small weights, L1 prefers sparse\n",
       "solutions with some larger weights but many more weights set to zero.\n",
       "- Thus L1\n",
       "regularization leads to much sparser weight vectors, that is, far fewer features.\n",
       "- L1 regularization can be viewed as a Laplace\n",
       "prior on the weights. L2 regularization corresponds to assuming that weights are\n",
       "distributed according to a Gaussian distribution with mean m = 0.\n",
       "\n",
       "\n",
       "## SLP Chapter 5.6 Multinomial logistic regression\n",
       "\n",
       "- multinomial logistic regression, also called softmax regression\n",
       "- $softmax(z_i)=exp(z_i)/\\sum_{j=1}^k exp(z_j)$\n",
       "- $L_{CE}(\\hat y,y)=-\\sum_{k=1}^Ky_k\\,log\\,\\hat p(y=k|x)$\n",
       "\n",
       "\n",
       "## SLP Chapter 7.1 Units\n",
       "\n",
       "- sigmoid: $\\sigma(z) = 1/(1+e^{-z})$\n",
       "- tanh:$[e^z-e^{-z}]/[e^z+e^{-z}]$\n",
       "  - the tanh function\n",
       "has the nice properties of being smoothly differentiable and mapping outlier values toward the mean\n",
       "- ReLU: $y=max(z,0)$\n",
       "  - Rectifiers don’t vanish gradient, since the derivative of ReLU for high\n",
       "values of z is 1 rather than very close to 0.\n",
       "\n",
       "\n",
       "## SLP Chapter 7.2 The XOR problem\n",
       "\n",
       "- perceptron, which is a very simple neural\n",
       "unit that has a binary output and does not have a non-linear activation function\n",
       "- w1x1+w2x2+b = 0 is the equation of a line\n",
       "- XOR is not a linearly separable function\n",
       "\n",
       "\n",
       "## SLP Chapter 7.3 Feedforward Neural Networks\n",
       "\n",
       "- The core of the neural network is the hidden layer h formed of hidden units hi,\n",
       "each of which is a neural unit\n",
       "- In the standard architecture, each layer is fully-connected, meaning that each unit in each layer takes as input the outputs\n",
       "from all the units in the previous layer, and there is a link between every pair of units\n",
       "from two adjacent layers.\n",
       "- The advantage of using a single matrix W for the weights of the entire layer is\n",
       "that now the hidden layer computation for a feedforward network can be done very\n",
       "efficiently with simple matrix operations.\n",
       "- So without non-linear activation functions,\n",
       "a multilayer network is just a notational variant of a single layer network with a\n",
       "different set of weights,\n",
       "- we lose all the representational power of multilayer\n",
       "networks\n",
       "- add a dummy node a0 to each layer whose value will\n",
       "always be 1 and a weight associated that is the bias\n",
       "\n",
       "\n",
       "## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n",
       "\n",
       "-adding this hidden layer to our logistic regression regression\n",
       "classifier allows the network to represent the non-linear interactions between\n",
       "features. This alone might give us a better sentiment classifier.\n",
       "- Instead of using\n",
       "hand-built human-engineered features as the input to our classifier, we draw on deep\n",
       "learning’s ability to learn features from the data by representing words as word2vec\n",
       "- embeddings as our input representation—\n",
       "and more generally the idea of relying on another algorithm to have already learned\n",
       "pretraining an embedding representation for our input words — is called pretraining.\n",
       "\n",
       "\n",
       "## SLP Chapter 7.5 Feedforward Neural Language Modeling\n",
       "\n",
       "- language modeling:\n",
       "predicting upcoming words from prior word context. \n",
       "- Compared to n-gram models, neural language models can handle\n",
       "much longer histories, can generalize better over contexts of similar words, and are\n",
       "more accurate at word-prediction\n",
       "- neural net language models\n",
       "are much more complex, slower to train, and less interpretable than n-gram models,\n",
       "so for many (especially smaller) tasks an n-gram language model is still the right\n",
       "tool.\n",
       "- Neural language models represent words in this prior context by their embeddings,\n",
       "rather than just by their word identity as used in n-gram language models. But a neural LM, knowing that “cat” and “dog” have similar embeddings, will\n",
       "be able to generalize from the “cat” context to assign a high enough probability to\n",
       "“fed” even after seeing “dog”.\n",
       "- forward inference or decoding for neural language models\n",
       "- A one-hot vector is\n",
       "a vector that has one element equal to 1—in the dimension corresponding to that\n",
       "word’s index in the vocabulary\n",
       "\n",
       "\n",
       "## SLP Chapter 7.6 Training Neural Nets\n",
       "\n",
       "- First, we’ll need a loss function that models the distance between the system\n",
       "output and the gold output, and it’s common to use the loss function used for logistic\n",
       "regression, the cross-entropy loss.\n",
       "- Second, to find the parameters that minimize this loss function, we’ll use the\n",
       "gradient descent optimization algorithm\n",
       "- gradient descent requires knowing the gradient of the loss function, the\n",
       "vector that contains the partial derivative of the loss function with respect to each\n",
       "of the parameters.(error backpropagation or\n",
       "backward differentiation.)\n",
       "- cross entropy loss$L_{CE} = -\\sum_{i=1}^C y_i \\, log\\, \\hat y_i$\n",
       "- Assume this is a hard classification task,\n",
       "meaning that only one class is the correct one, and that there is one output unit in y\n",
       "for each class.\n",
       "- $L_{CE} =- \\, log\\, \\hat y_i$, where y hat could be softmax\n",
       "- language model with softmax that gradient\n",
       "- $gradient = -(1 \\, (\text{if} y =k) - [exp(w_k\\cdot x+b_k)/\\sum_j exp(w_j\\cdot x+b_j)])$\n",
       "- sigmoid: $d\\,\\sigma(z)/dz = \\sigma(z)(1-\\sigma(z))$\n",
       "- tanh: $d\\,tanh(z)/dz = 1- tanh^2(z)$\n",
       "- ReLU: $d\\, ReLU(z)/dz = 0 \\,\text{for}\\, z<0; 1 \\,\text{for}\\, z\\geq 0 $\n",
       "- For logistic regression we can initialize gradient descent with all the weights and\n",
       "biases having the value 0. In neural networks, by contrast, we need to initialize the\n",
       "weights with small random numbers. It’s also helpful to normalize the input values\n",
       "to have 0 mean and unit variance.\n",
       "- One of the most\n",
       "dropout important is dropout: randomly dropping some units and their connections from\n",
       "the network during training avoid overfitting\n",
       "- Tuning of hyperparameters is also important\n",
       "  - learning rate h, the mini-batch size, the model architecture\n",
       "\n",
       "\n",
       "## SLP Chapter 7.7 Training the neural language model\n",
       "\n",
       "- For some tasks, it’s ok to freeze the embedding layer E with initial word2vec values\n",
       "- compute\n",
       "the initial embedding matrix E, and then hold it constant while we only modify\n",
       "W, U, and b\n",
       "- Generally training proceeds by taking as input a very long text, concatenating all\n",
       "the sentences, starting with random weights, and then iteratively moving through the\n",
       "text predicting each word wt .\n",
       "- $L_{CE} = - \\, log\\, \\hat y_i = -log\\, p(w_t|w_{t-1},...,w_{t-n+1})$\n",
       "- Training the parameters to minimize loss will result both in an algorithm for\n",
       "language modeling (a word predictor) but also a new set of embeddings E that can\n",
       "be used as word representations for other tasks.\n",
       "\n",
       "\n",
       "## Module 13 Lecture Notes\n",
       "\n",
       "- Like with logistic regression, we learn the best parameters for the neural net to make\n",
       "its predictions /𝑦 as close to possible as the gold standard labels in our training data y.\n",
       "We use:\n",
       "  - A cross entropy loss function\n",
       "  - Gradient descent to perform optimization\n",
       "  - Error back propagation to compute the gradient of the loss function\n",
       "- Since Neural Networks combine many perceptron-like neural units, we have many\n",
       "more parameters in the models (weights and biases). This allows us to train systems to\n",
       "perform even more sophisticated tasks.\n",
       "\n",
       "\n",
       "## Module 13 Definitions\n",
       "* **Concept** - definition\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "# Module 14: Natural Language Processing\n",
       "## SLP Chapter 9.1 Language Models Revisited\n",
       "\n",
       "- <strong>Language Model 1</strong>\n",
       "- <strong>====================</strong>\n",
       "- Goal: Learn a function that returns the joint probability\n",
       "  - There are too many parameters to accurately estimate. This is\n",
       "sometimes called the “curse of dimensionality”\n",
       "  - In n-gram-based models we fail to generalize to related words / word\n",
       "sequences that we have observed. \n",
       "- Curse of dimensionality / sparse statistics：\n",
       "  - Suppose we want a joint distribution over 10 words.\n",
       "  - Suppose we have a vocabulary of size 100,000. This is too high to estimate from data.\n",
       "-  One way: The probability of the next word only depends on the previous n-1\n",
       "words. A small n makes it easier for us to get an estimate of the probability\n",
       "from data.\n",
       "- Neural Language models have several advantages over n-gram LMs:\n",
       "  - They don’t need smoothing\n",
       "  - They can handle much longer histories.\n",
       "  - They can generalize over contexts of similar words.\n",
       "  - Neural LMs tend to have much higher predictive accuracy than n-gram LMs.\n",
       "- Disadvantage: slower to train than traditional n-gram LMs\n",
       "- Associate each word in the vocabulary with a vector-representation, thereby\n",
       "creating a notion of similarity between words.\n",
       "- Express the joint probability function of a word sequence in terms of the word\n",
       "vectors for the words in that sequence.\n",
       "- Simultaneously learn the word vectors and the parameters of the function.\n",
       "The word vectors are low-dimensional (d=30 to d=100) dense vectors, like we’ve seen\n",
       "before.\n",
       "- The probability function is expressed the product of conditional probabilities of the\n",
       "next word given the previous word, using a multi-layer, feed forward neural network.\n",
       "\n",
       "\n",
       "## SLP Chapter 9.2 Recurrent Neural Networks\n",
       "\n",
       "-The key\n",
       "difference from a feedforward network lies in the recurrent link shown in the figure\n",
       "with the dashed line. This link augments the input to the computation at the hidden\n",
       "layer with the value of the hidden layer from the preceding point in time.\n",
       "- The hidden layer from the previous time step provides a form of memory, or\n",
       "context, that encodes earlier processing and informs the decisions to be made at\n",
       "later points in time.\n",
       "- appear to be more complex than\n",
       "non-recurrent architectures. But in reality, they’re not all that different.\n",
       "- RNN is nearly identical to what we’ve already seen with feedforward networks. To\n",
       "compute an output yt for an input xt , we need the activation value for the hidden\n",
       "layer ht .\n",
       "- To calculate this, we multiply the input xt with the weight matrix W, and\n",
       "the hidden layer from the previous time step ht􀀀1 with the weight matrix U. We\n",
       "add these values together and pass them through a suitable activation function, g,\n",
       "to arrive at the activation value for the current hidden layer, ht . and proceed with the usual computation to generate the\n",
       "output vector.\n",
       "- The sequential nature of simple\n",
       "recurrent networks can also be seen by unrolling the network in time\n",
       "- First, to compute the loss function for\n",
       "the output at time t we need the hidden layer from time t 􀀀1. Second, the hidden\n",
       "layer at time t influences both the output at time t and the hidden layer at time t +1\n",
       "(and hence the output and loss at t +1).\n",
       "\n",
       "\n",
       "## SLP Chapter 9.3 RNNs as Language Models\n",
       "\n",
       "- RNN language models (Mikolov et al., 2010) process the input sequence one word at\n",
       "a time, attempting to predict the next word from the current word and the previous\n",
       "hidden state.\n",
       "- RNNs don’t have the limited context problem that n-gram models\n",
       "have, since the hidden state can in principle represent information about all of the\n",
       "preceding words all the way back to the beginning of the sequence.\n",
       "- To train an RNN as a language model, we use a corpus of text as training material,\n",
       "having the model predict the next word at each time step t.\n",
       "- We train the model to\n",
       "minimize the error in predicting the true next word in the training sequence, using\n",
       "cross-entropy as the loss function.\n",
       "- cross-entropy loss measures the\n",
       "difference between a predicted probability distribution and the correct distribution.\n",
       "- Weight tying is a method that\n",
       "dispenses with this redundancy and uses a single set of embeddings at the input and\n",
       "softmax layers. That is, E = V. To do this, we set the dimensionality of the final\n",
       "hidden layer to be the same dh, (or add an additional projection layer to do the\n",
       "same thing), and simply use the same matrix for both layers.\n",
       "- Advantages of NN LMs:\n",
       "  - Better results. They achieve better preplexity scores than SOTA n-gram LMs.\n",
       "  - Larger N. NN LMs can scale to much larger orders of n. This is achievable because\n",
       "parameters are associated only with individual words, and not with n-grams.\n",
       "  - They generalize across contexts. \n",
       "- A by-product of training are word embeddings\n",
       "\n",
       "\n",
       "## SLP Chapter 6.1 Lexical Semantics\n",
       "\n",
       "- summarize some of these desiderata, drawing on results in the\n",
       "lexical linguistic study of word meaning, which is called lexical semantics\n",
       "- Lemmas and Senses\n",
       "  - form mouse is the lemma, also called the citation form. The form mouse would also be the lemma for the word mice;\n",
       "  - The specific forms sung or carpets or sing or duermes are called wordforms\n",
       "  - As the example above shows, each lemma can have multiple meanings; the\n",
       "lemma mouse can refer to the rodent or the cursor control device.\n",
       "  - each\n",
       "of these aspects of the meaning of mouse a word sense\n",
       "- Synonymy\n",
       "  - when one word has a sense whose meaning is\n",
       "identical to a sense of another word, or nearly identical, we say the two senses of\n",
       "those two words are synonyms.\n",
       "  - called the principle of contrast, word H2O is used in scientific contexts and\n",
       "would be inappropriate in a hiking guide\n",
       "- Word Similarity\n",
       "  - The notion of word similarity is very useful in larger semantic tasks. Knowing\n",
       "how similar two words are can help in computing how similar the meaning of two\n",
       "phrases or sentences are\n",
       "- Word relatedness\n",
       "  - The meaning of two words can be related in ways other than similarity. One such class of connections is called word relatedness, or word association in psychology.\n",
       "    - meanings of the words coffee and cup.\n",
       "  - One common kind of relatedness between words is if they belong to the same\n",
       "semantic field semantic field. A semantic field is a set of words which cover a particular semantic\n",
       "domain and bear structured relations with each other\n",
       "  - Semantic fields are also related to topic models, like Latent\n",
       "Dirichlet Allocation, LDA, which apply unsupervised learning on large sets of texts\n",
       "to induce sets of associated words from text.\n",
       "  - useful tools for discovering topical structure in documents\n",
       "- Semantic Frames and Roles\n",
       "  - A semantic frame is a set of words that denote perspectives or\n",
       "participants in a particular type of event.\n",
       "    - e.g. commercial transaction can be encoded with buy, sell, pay\n",
       "- Connotation\n",
       "  - words have affective meanings or connotations\n",
       "  - the aspects of a word’s meaning that are related to a writer or reader’s emotions,\n",
       "sentiment, opinions, or evaluations\n",
       "  - Positive or negative evaluation language is called sentiment,\n",
       "- Support for entailment\n",
       "  - $dog \\subset animal$\n",
       "\n",
       "\n",
       "## SLP Chapter 6.2 Vector Semantics\n",
       "\n",
       "- Vectors semantics is the standard way to represent word meaning in NLP, helping us model many of the aspects of word meaning\n",
       "- Vectors for representing words are called embeddings\n",
       "- classifiers can assign sentiment as long as\n",
       "it sees some words with similar meanings. And as we’ll see, vector semantic models\n",
       "can be learned automatically from text without supervision.\n",
       "- We will see that this method results in very long\n",
       "vectors that are sparse\n",
       "- word2vec model family for constructing\n",
       "short, dense vectors that have useful semantic properties\n",
       " \n",
       "\n",
       "## SLP Chapter 6.3 Words and Vectors\n",
       "\n",
       "- Vector or distributional models of meaning are generally based on a co-occurrence\n",
       "matrix, a way of representing how often words co-occur. We’ll look at two popular\n",
       "matrices: the term-document matrix and the term-term matrix.\n",
       "- term-document matrix, each row represents a word in the vocabulary and each\n",
       "column represents a document from some collection of documents, Each cell in this matrix represents the number of times\n",
       "a particular word, occurs in a particular document\n",
       "- The term-document matrix of was first defined as part of the vector space model of information retrieval\n",
       "- in real term-document matrices, the vectors representing\n",
       "each document would have dimensionality jVj, the vocabulary size.\n",
       "- Term-document matrices were originally defined as a means of finding similar\n",
       "documents for the task of document information retrieval. Two documents that are similar will tend to have similar words\n",
       "- Information retrieval (IR) is the task of finding the document d from the D\n",
       "documents in some collection that best matches a query q.\n",
       "- we saw that similar documents had similar vectors, because similar\n",
       "documents tend to have similar words. This same principle applies to words:\n",
       "similar words have similar vectors because they tend to occur in similar documents.\n",
       "The term-document matrix thus lets us represent the meaning of a word by the documents\n",
       "it tends to occur in.\n",
       "- term-term matrix, also called the word-word matrix or the term-context matrix, in which the columns are labeled by words rather\n",
       "than documents. the number of times the row (target) word and the column (context) word co-occur\n",
       "in some context in some training corpus.\n",
       "- This matrix is thus of dimensionality |V|*|V| and each cell records, the cell represents the number of times the two words appear in the same\n",
       "document. \n",
       "- (PPT)The value in a cell indicates how\n",
       "often a word appears in a\n",
       "context window surrounding\n",
       "another\n",
       "- TF-IDF\n",
       "  - Term frequency * inverse document\n",
       "frequency \n",
       "  - How often a\n",
       "word occurred in\n",
       "a document * 1 over the number\n",
       "of documents that it\n",
       "occurred in\n",
       "- Context windows\n",
       "w-2, w-1 target_word w+1 w+2\n",
       "- Since most of these numbers are zero these are sparse vector representations;\n",
       "there are efficient algorithms for storing and computing with sparse matrices.\n",
       "- Alternates to counts\n",
       "    - Raw word frequency is not a great measure of association between words. It’s very\n",
       "skewed “the” and “of” are very frequent, but maybe not the most discriminative\n",
       "We’d rather have a measure that asks whether a context word is particularly\n",
       "informative about the target word.\n",
       "\n",
       "\n",
       "## SLP Chapter 6.4 Cosine for measuring similarity\n",
       "\n",
       "- L2 Normalization: $||u|| = sqrt(\\sum_i^n u_i^2)$, divide each vector by the normalization term, compare each row in term-document matrix\n",
       "- We modify the dot product to normalize for the vector length by dividing the\n",
       "dot product by the lengths of each of the two vectors. This normalized dot product\n",
       "turns out to be the same as the cosine of the angle between the two vectors, following\n",
       "from the definition of the dot product between two vectors a and b:\n",
       "- $cosine(v,w) = (v\\cdot w)/(|v||w|)$, L2-norm\n",
       "- closer the semantic, smaller the angle\n",
       "\n",
       "\n",
       "## SLP Chapter 6.8 Word2vec\n",
       "\n",
       "- embeddings, short dense\n",
       "vectors.\n",
       "- embeddings are short, with number\n",
       "of dimensions d ranging from 50-1000, rather than the much larger vocabulary size\n",
       "|V| or number of documents D we’ve seen\n",
       "- vectors are dense: instead of vector entries being sparse,\n",
       "mostly-zero counts or functions of counts, the values will be real-valued numbers\n",
       "that can be negative.\n",
       "- Representing words as 300-dimensional dense vectors requires our classifiers\n",
       "to learn far fewer weights than if we represented words as 50,000-dimensional vectors,\n",
       "and the smaller parameter space possibly helps with generalization and avoiding\n",
       "overfitting.\n",
       "- skip-gram with negative sampling, sometimes called SGNS.\n",
       "- The word2vec methods are fast, efficient to train, and easily available online\n",
       "with code and pretrained embeddings.\n",
       "- Word2vec embeddings are static embeddings, meaning that the method learns one fixed embedding for each word in the vocabulary.\n",
       "- The intuition of word2vec is that instead of counting how often each word w occurs\n",
       "near, say, apricot, we’ll instead train a classifier on a binary prediction task: “Is\n",
       "word w likely to show up near apricot?” We don’t actually care about this prediction\n",
       "task; instead we’ll take the learned classifier weights as the word embeddings.\n",
       "- self-supervision, avoids the need for\n",
       "any sort of hand-labeled supervision signal\n",
       "- assume we’re\n",
       "using a window of +/-2 context words\n",
       "- ... lemon, a [tablespoon of apricot jam, a] pinch ... w = apricot, c1...c4 = tablespoon of jam a\n",
       "- train a classifier such that, given a tuple (w;c) of a target word\n",
       "w paired with a candidate context word c $P(+|w,c)$\n",
       "- $P(-|w|c) = 1-P(+|w,c)$\n",
       "- intuition of the skipgram\n",
       "model is to base this probability on embedding similarity: a word is likely to\n",
       "occur near the target if its embedding vector is similar to the target embedding\n",
       "- compute similarity between these dense embeddings, we rely on the intuition that\n",
       "two vectors are similar if they have a high dot product, similarity = $c\\cdot w$\n",
       "- $P(+|w,c)=\\sigma(c\\cdot w)=1/(1+exp(-c\\cdot w))$\n",
       "- $P(-|w,c)=\\sigma(-c\\cdot w)=1/(1+exp(c\\cdot w))$\n",
       "- $P(+|w,c_{1:L})=\\prod_{i=1}^L\\sigma(c_i\\cdot w)$\n",
       "- $log\\,P(+|w,c_{1:L})=\\sum_{i=1}^L\\log\\,\\sigma(c_i\\cdot w)$\n",
       "- In summary, skip-gram trains a probabilistic classifier that, given a test target word\n",
       "w and its context window of L words c1:L, assigns a probability based on how similar\n",
       "this context window is to the target word. The probability is based on applying the\n",
       "logistic (sigmoid) function to the dot product of the embeddings of the target word\n",
       "with each context word. To compute this probability, we just need embeddings for\n",
       "each target word and context word in the vocabulary.\n",
       "- Skip-gram actually\n",
       "stores two embeddings for each word, one for the word as a target, and one for the word considered as context. Thus the parameters we need to learn are two matrices\n",
       "W and C, each containing an embedding for every one of the jVj words in the\n",
       "vocabulary V\n",
       "\n",
       "\n",
       "## SLP Chapter 6.10 Semantic properties of embeddings\n",
       "\n",
       "- <strong>cont.</strong>\n",
       "- Given the set of positive and negative training instances, and an initial set of embeddings,\n",
       "the goal of the learning algorithm is to adjust those embeddings to\n",
       "  - Maximize the similarity of the target word, context word pairs (w;cpos) drawn\n",
       "from the positive examples\n",
       "  - Minimize the similarity of the (w;cneg) pairs from the negative examples.\n",
       "- want to maximize the dot product of the word with the actual context\n",
       "words, and minimize the dot products of the word with the k negative sampled nonneighbor\n",
       "words.\n",
       "- We minimize this loss function using stochastic gradient descent\n",
       "- word2vec maximizes: $\\sum_{(t,c)\\in +}log\\, P(+|t,c)+\\sum_{(t,c)\\in -}log\\, P(-|t,c)$, Maximize the + label for the pairs from the\n",
       "positive training data, and the – label for the\n",
       "pairs sample from the negative data.\n",
       "- <strong>end</strong>\n",
       "- Different types of similarity or association:\n",
       "  - Shorter context windows\n",
       "tend to lead to representations that are a bit more syntactic, since the information is\n",
       "coming from immediately nearby words. When the vectors are computed from short\n",
       "context windows, the most similar words to a target word w tend to be semantically\n",
       "similar words with the same parts of speech. When vectors are computed from long\n",
       "context windows, the highest cosine words to a target word w tend to be words that\n",
       "are topically related but not similar.\n",
       "  - Two words have first-order co-occurrence if they are typically nearby each other.\n",
       "Thus wrote is a first-order associate of book or poem.\n",
       "  - Two words have second-order\n",
       "co-occurrence if they have similar neighbors. Thus wrote is a second-order associate of words like said or remarked.\n",
       "- Analogy/Relational Similarity:\n",
       "  - scholars showed that sparse vector models of meaning could solve such analogy problems but the parallelogram method received more modern attention because of its success\n",
       "with word2vec or GloVe vectors\n",
       "  - problems:\n",
       "    - the closest value returned by the parallelogram\n",
       "algorithm in word2vec or GloVe embedding spaces is usually not in fact\n",
       "b* but one of the 3 input words or their morphological variants\n",
       "    - spaces perform well if the task involves\n",
       "frequent words, small distances, and certain relations the parallelogram method\n",
       "with embeddings doesn’t work as well for other relations\n",
       "- Embeddings can also be a useful tool for studying how meaning changes over time,\n",
       "by computing multiple embedding spaces, each from texts written in a particular\n",
       "time period.\n",
       "\n",
       "\n",
       "## SLP Chapter 6.11 Bias and Embeddings\n",
       "\n",
       "- allocational harm, when a system allocates resources unfairly to different groups \n",
       "  - algorithm hire programmers would fownweight female\n",
       "- amplifies bias\n",
       "  - gendered terms become more gendered in embedding space\n",
       "- representational harm: a harm caused by a system demeaning or even ignoring some social groups\n",
       "- although these sorts of debiasing may reduce bias in embeddings, they do not eliminate it\n",
       "\n",
       "\n",
       "## SLP Chapter 6.12 Evaluating Vector Models\n",
       "\n",
       "- <strong>Language Model 1</strong>\n",
       "- <strong>====================</strong>\n",
       "- Dense vector:\n",
       "  - Treat the target word and a neighboring context word as positive examples.\n",
       "  - Randomly sample other words in the lexicon to get negative samples.\n",
       "  - Use logistic regression to train a classifier to distinguish those two cases.\n",
       "  - Use the weights as the embeddings.\n",
       "- Sparse vs. dense vectors:\n",
       "  - Short vectors may be easier to use as features in machine learning (fewer\n",
       "weights to tune)\n",
       "  - Dense vectors may generalize better than storing explicit counts\n",
       "  - They may do better at capturing synonymy\n",
       "  - simply works better\n",
       "- Advantages of word2vec:\n",
       "  - Fast, easy to train (much faster than SVD)\n",
       "  - Available online in the word2vec package\n",
       "  Including sets of pretrained embeddings!\n",
       "- word2vec self-supervised:\n",
       "  - we need embeddings for t and c to do it.\n",
       "  - It starts with an initial set of embedding vectors and then iteratively shifts\n",
       "the embedding of each word w to be more like the embeddings of words\n",
       "that occur nearby in texts, and less like the embeddings of words that\n",
       "don’t occur nearby.\n",
       "- learn word2vec (skip-gram) embeddings：\n",
       "  - Start with V random 300-dimensional vectors as initial\n",
       "embeddings\n",
       "  - Use logistic regression\n",
       "    - Take a corpus and take pairs of words that co-occur as positive\n",
       "examples\n",
       "    - Take pairs of words that don't co-occur as negative examples\n",
       "    - Train the classifier to distinguish these by slowly adjusting all the\n",
       "embeddings to improve the classifier performance\n",
       "    - Throw away the classifier code and keep the embeddings.\n",
       "\n",
       "\n",
       "## Module 13 Lecture Notes\n",
       "\n",
       "- Like with logistic regression, we learn the best parameters for the neural net to make\n",
       "its predictions /𝑦 as close to possible as the gold standard labels in our training data y.\n",
       "We use:\n",
       "  - A cross entropy loss function\n",
       "  - Gradient descent to perform optimization\n",
       "  - Error back propagation to compute the gradient of the loss function\n",
       "- Since Neural Networks combine many perceptron-like neural units, we have many\n",
       "more parameters in the models (weights and biases). This allows us to train systems to\n",
       "perform even more sophisticated tasks.\n",
       "\n",
       "\n",
       "## Module 14 Definitions\n",
       "* **Concept** - definition\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# This will generate the full set of notes that you can use for your exam.\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "#\n",
    "full_markdown = \"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\".join([module8_markdown, \n",
    "                             module9_markdown, \n",
    "                             module10_markdown, \n",
    "                             module11_markdown, \n",
    "                             module12_markdown,\n",
    "                             module13_markdown,\n",
    "                             module14_markdown,]\n",
    "                       )\n",
    "display(Markdown(full_markdown))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
