{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "brNDGX-kvzds"
   },
   "source": [
    "# Template for Notes for CIS 521 Midterm 2\n",
    "\n",
    "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
    "\n",
    "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
    "\n",
    "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aPtM8g7R0Pds"
   },
   "source": [
    "# Instructions\n",
    "\n",
    "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
    "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
    "3. Read the notes below about how you can format your summaries.\n",
    "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
    "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
    "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
    "* Run the last cell of this notebook\n",
    "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
    "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
    "* In the print options, change the layout from Landscape to Portrait. \n",
    "* Clicking the Save, will save a PDF that you can then print.\n",
    "\n",
    "## Formatting Notes\n",
    "\n",
    "1. The max length for each section's summary is 500 words / 3000 characters.\n",
    "2. You can also write a 2000 word summary of the lectures for each module.\n",
    "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
    "2. You can use Markdown formatting for your summaries. \n",
    "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
    "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
    "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
    "```\n",
    "def function():\n",
    "      print('hello world')\n",
    "```\n",
    "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cw5IpDLCZa4W"
   },
   "source": [
    "## Example summary\n",
    "\n",
    "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "id": "lzaavW96tcxM",
    "outputId": "fdc823f4-2163-4578-fcf0-204d1624317c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The example summary is 116 words long, so you could write an additional 384 words.\n"
     ]
    }
   ],
   "source": [
    "chapter_2_1_example = \"\"\"\n",
    "Regular expressions are an algebraic notation for characterizing \\\n",
    "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
    "Only the first matching result will be shown. The search string is case sensitive, \\\n",
    "but it can consist of a single character or a sequence of characters. \\\n",
    "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
    "for characters over a range. The caret specifies the non-existence of a character. \\\n",
    "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
    "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
    "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
    "...\n",
    "\"\"\"\n",
    "\n",
    "length = len(chapter_2_1_example.split(\" \"))\n",
    "print(\"The example summary is {len} words long, so you could write an additional \\\n",
    "{remaining} words.\".format(len=length, remaining=(500-length)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s3Uk9gQTevcj"
   },
   "source": [
    "## Example summary length\n",
    "\n",
    "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "id": "8DvZ57A-djo1",
    "outputId": "0f7d21a7-4238-44ab-dff6-c0d7a258f38b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The length in words is 500\n",
      "The length in characters is 3030\n"
     ]
    }
   ],
   "source": [
    "# This is an example of a 500 word placeholder\n",
    "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
    " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
    " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
    " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
    " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
    " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
    " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
    " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
    " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
    " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
    " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
    " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
    " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
    " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
    " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
    " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
    " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
    " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
    " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
    " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
    " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
    " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
    " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
    " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
    " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
    " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
    " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
    " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
    " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
    " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
    " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
    " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
    " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
    " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
    " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
    " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
    " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
    " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
    "\n",
    "print(\"The length in words is\", len(placeholder.split(' ')))\n",
    "print(\"The length in characters is\", len(placeholder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5yG3-YncJfTB"
   },
   "source": [
    "# Permissions to use your summaries\n",
    "\n",
    "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
    "\n",
    "1. Please enter your name.  \n",
    "\n",
    "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
    "\n",
    "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "2RECnJArJew4"
   },
   "outputs": [],
   "source": [
    "name = \"\"\n",
    "permission = True\n",
    "anonymous = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GZZXyrV-xuAp"
   },
   "source": [
    "# Module 8 - Markov Decision Process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "Y2NfE1CKxzS-"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
    "\n",
    "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
    "\n",
    "chapter16_1 = \"\"\" $P(RESULT(a) = s') = \\sum_s P(s) P(s' | s, a)$ \\n\n",
    "\n",
    "$Expected Utility = \\sum_s' P(RESULT(a) = s') U(s')$ \\n\n",
    "\n",
    "MEU principle formalizes \"doing the right thing\" but does not operationalize: \\\n",
    "still need to estimate probability of states thruogh learning, and compute P(RESULT(a)=s') \\\n",
    "using a causal model.  \"\"\"\n",
    "\n",
    "chapter16_2 = \"\"\" Constraints/axioms on rational preference relations: (1) orderability \\\n",
    "either two lotteries are indifferent or agent prefers one over other, (2) transitivity \\\n",
    ", (3) continuity: if B is between A and C in preference, there is some probability p \\\n",
    "s.t. agent will be indifferent between B for sure and A / C with probability \\\n",
    "(4) substitutability: If indifferent between A and B, then indifferent between two lotteries \\\n",
    "with only difference being A vs B, (5) Monotonicity: If A $\\succ$ B then agent prefers \\\n",
    "lottery with higher probability for A, (6) decomposability: Compound lotteries can \\\n",
    "be reduced to simpler ones\\n\n",
    "\n",
    "von Neumann & Morgenstern: Existence of Utility function. If agent's preference \\\n",
    "obeys axiom, then there is a function s.t. U(A) > U(B) iff A $\\succ$ B and U(A) = U(B) \\\n",
    "if $A \\sim B$. \"\"\"\n",
    "\n",
    "chapter16_3 = \"\"\" Utility of money is not directly proportional to monetary value.\\\n",
    "Value of first million is very high, while marginal value decreases. s shaped curve. \\\n",
    "A billionnaire thus more likely to take a risky but beneficial move than a stable one \\n\n",
    "\n",
    "Grayson study: utility of money almost exactly proportional to log of the amount. \\\n",
    "We typically have concave utility function for positive wealth (facing downwards),\\n\n",
    "\n",
    "People in the slope-decreasing part of utility curve will assign higher utility to \\\n",
    "a guaranteed Expected Monetary Value than a lottery. Risk-avrese agents. Risk-seeking agents \\\n",
    "show up in the negative part of the wealth utility curve. If the changes in wealth are \\\n",
    "small, curve will be nearly linear, and agent will be risk-neutral \\n\n",
    "\n",
    "Most rational action a = $argmax_a EU(a)$. Most of the time we can only estimate EU. \\\n",
    "Unbiased estimate: expected value of error is 0. Since we select action with highest \\\n",
    "EU estimate, we will have an optimistic bias/ optimizer's curse. Bias higher when we \\\n",
    "are choosing among more choices. If variance in utility estimate is high, what appears \\\n",
    "to be the best choice may not even be the best chioce: so make sure sample size is large \\\n",
    "enough for trials. We can avoid it by using a Bayesian model using a probability model \\\n",
    "of the error $P(EU_{est} | EU)$, updating EU based on our prior knowledge of it.\\\n",
    "\n",
    "Human preferences are predictably irrataional: there may be no utility function consistent with \\\n",
    "our choices. Possible reasons: (1) certainty effect (reduce computational burden, distrust stated probs\\\n",
    "accoutning for emotion) (2) Ellsberg paradox: people have ambiguity aversion, prefer \\\n",
    "known probability. (3) framing effect, (4) anchoring effect. \"\"\"\n",
    "\n",
    "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
    "chapter17_1 = \"\"\" In sequential decision problems agent's utility is the result of \\\n",
    "a sequence of decisions. Markov decision process = sequential decision problem for fully \\\n",
    "observable, stochastic environment with a fully observable transition model and additive \\\n",
    "rewards. Solving MDPs usually require dynamic programming. \\n\n",
    "\n",
    "If a game has finite time horizon, policy may change based on the time left. \\n\n",
    "\n",
    "We use additive discount rewards to define utility for two reasons: (1) empirical evidence \\\n",
    "shows that humans value near-term rewards. (2) economic reasoning suggests early reward \\\n",
    "is momre valuable, (3) uncertainty about true rewards, (4) assumption of stationary \\\n",
    "preference, (5) avoid problems with infinite horizons. \\n\n",
    "\n",
    "On point (5), three ways to avoid infinite utility. (1): use discounted rewards such that \\\n",
    "utility is bounded by $(R_{max}) / (1 - gamma)$. (2): design proper policies that \\\n",
    "guarantee agents will reach terminal states, save us from discouting (3): compare \\\n",
    "sequences using average rather than sum of reward. \\n\n",
    "\n",
    "When we use discounted utilities with infinite horizons, the optimal policy is \\\n",
    "independent of the starting state. Thus, the utility of each state is the expected \\\n",
    "sum of discounted rewards if the agent carrys out an optimal policy, or U(s) \\n\n",
    "\n",
    "Utility of one state = expected reward for next transition + discounted utility \\\n",
    "of the next state. This is the Bellman equation. \\n\n",
    "\n",
    "An affine transformation of reward will leave the optimal policy unchanged. Moreoever \\\n",
    "the shaping theorem suggests that any transformation in this form will also produce no \\\n",
    "change to optimal policy: \\n\n",
    "\n",
    "\\t $R'(s, a, s') = R(s, a, s') + \\gamma \\phi (s') - \\phi (s)$ where $\\phi (s)$ is \\\n",
    "any function of state s. \\n\n",
    "\n",
    "Function $\\phi (s)$ is called a potential. $ \\gamma \\phi (s') - \\phi (s)$ is the \\\n",
    "gradient of the potential. If $\\phi (s)$ is proportional to higher utility values, then adding \\\n",
    "$ \\gamma \\phi (s') - \\phi (s)$ will lead agent \\\"uphill\\\" in utility. \\n\n",
    "\n",
    "Ways to represent MDPs: \\n\n",
    "1) use 3-D tables of size $|S|^2 |A|$. Sparse tables with many 0's only have size \\\n",
    "$O(|S||A|)$. \\n\n",
    "2) Dynamic decision networks (DDNs) with decision, reward, and utility nodes. Use factored \\\n",
    "represetnation, and have complexity exponential times better than atomic representation. \n",
    "\"\"\"\n",
    "chapter17_2 =  \"\"\" Problem with solving Bellman equations: the MAX \\\n",
    "operator is not linear, so can't use linear system solver. Instead we use \\\n",
    "an iterative approach. \\n\n",
    "\n",
    "Showing that the values would converge: \\n\n",
    "Contraction: A function that, when apply to two values, return two values \\\n",
    "that are closer together by at least some constant factor. E.g. divide by four \\n\n",
    "\n",
    "A contraction function has only one fixed point. The value gets closer to the fixed \\\n",
    "point each time contraction is applied. \\n\n",
    "\n",
    "We can show that Bellman update is a contraction by a factor of $\\gamma$ on utility \\\n",
    "vectors. Define \\\"length\\\" of a vector as the abs value of its biggest component. \\\n",
    "Then difference between two vectors = max difference between two corresponding elements. \\\n",
    "Then we can show $||BU_i - BU'_i||$ $\\leq$ $\\gamma$ $||U_i - U'_i||$. \\n\n",
    "\n",
    "On each iteration, the error decreases by a factor $\\geq \\gamma$. So value iteration \\\n",
    "converges in exponential rate. The maximum number of iterations = \\\n",
    "ceiling of $log (2R_{max} / \\epsilon (1 - \\gamma) / log (1 / \\gamma))$. \\\n",
    "Implies fast convergence if $\\gamma$ is small / horizon is short, but N does not \\\n",
    "depend too much on $\\epsilon / R_{max}$. \\n\n",
    "\n",
    "It can be shown that in value iteration, policy often becomes optimal long before value \\\n",
    "converges. Policy loss is connected to error in value estimate by: \\n\n",
    "\n",
    "if $||U_i - U|| < \\epsilon $ then $||U^{\\pi_i} - U|| < 2 \\epsilon$. \\n\n",
    "\n",
    "Policy iteration is faster than solving Bellman equations: O(n^3) time since we don't \\\n",
    "have max operator, so equations are linear. To further save time, use modified \\\n",
    "policy iteration where do a small number of value iterations. \\n\n",
    "\n",
    "Another alternative is asynchronous policy iteration: pick subset of states \\\n",
    "then apply either policy improvement or simplified value iteration. \\n \n",
    "\n",
    "We can also convert MDPs into linear programming problems, by treating utilities as \\\n",
    "variables and seeking to minimize utilities under constraints. (bellman equations). \\\n",
    "Makes MDPs solvable in polynomial time in the number of states and actions. However, \\\n",
    "number of states may be quite large, so LP may be less efficient than DP. \\n\n",
    "\n",
    "We can also solve MDPs using Expectimax trees, limiting depths by using the $\\epsilon$ horizon \\\n",
    "which is to cut the tree at a depth where sum of reward is lower than $\\epsilon$. We \\\n",
    "can also limit branching factor by sampling values instead of getting all of them. \\\n",
    "This approach is Real-Time Dynamic Programming (RTDP), similar to LRTA\\* algorithm. \\\n",
    "Useful in solving moderate sized problems,  but limited for large ones. \\n\n",
    "\n",
    "Enhancements to RTDP: (1) apply reinforcement learning to generate better heuristic. \\\n",
    "(2) Look further ahead in MDP with Monte Carlo approach. \n",
    "\"\"\"\n",
    "\n",
    "# Lecture notes \n",
    "module8_lecture_notes =\"\"\" A MDP is a sequential, fully observable decision problem \\\n",
    "with additive rewards and stochastic transition model (i.e. nondeterministic). Need \\\n",
    "to be solved by expectimax search. \\n\n",
    "\n",
    "MDP is defined by: (1) set of states, (2) set of actions, (3) transition function, \\\n",
    "specifying the probability (aka the model), (4) reward function, (5) initial state, and \\\n",
    "(6) (optional) terminal state. \\n \n",
    "\n",
    "In MDPs rewards are (1) either positive or negative, \\\n",
    "but bounded. (2) specified on a sequence of actions since MDP is sequential, (3) \\\n",
    "can be received both at goal state or at each transition, (4) additive. \\n\n",
    "\n",
    "A solution to a MDP is a policy: a mapping from state to action. vs solution to search \\\n",
    "problem is a plan. $\\pi$ denotes a policy and $\\pi *$ denotes the optimal policy. Each \\\n",
    "policy is associated with an expected utility. Results of same policy may vary \\\n",
    "according to the different environment history. \\n\n",
    "\n",
    "Two reasons for discounting future rewards: (1) fit empirical observation and \\\n",
    "(2) help algorithm converges. \\n\n",
    "\n",
    "Theorem: assuming stationary preferences (i.e. if agent prefers seq A to seq B, \\\n",
    "then agent prefers action r + seq A to action r + sequence B), we can only define \\\n",
    "utility as additive (simple sum) or discounted (sum weighted by discount). \\n\n",
    "\n",
    "Three solutions for infinite games: (1) finite horizon (stop after T steps), which \\\n",
    "will cause nonstationary policies, (2) discounting (wiht smaller $\\gamma$ meaning \\\n",
    "smaller time horizon), and (3) absorbing state that every policy is guaranteed \\\n",
    "to reach. \\n\n",
    "\n",
    "Value of state s = expected utility of starting in the state then acting optimally. \\\n",
    "$V^*(s) = max_a Q^*(s,a)$\\n\n",
    "\n",
    "Value of Q state (s, a) = expected utility of taking a from s then acting optimally. \\\n",
    "$Q^*(s,a) = \\sum_{s'} T(s,a,s') [R(s,a,s') + $ $\\gamma$ $V^*(s')]$ \\n\n",
    "\n",
    "We can use time limited values to reduce amount of work with expectimax. \\\n",
    "$V_k(s)$ = optimal value of s if game ends in k more time steps (i.e. value \\\n",
    "from k-depth expectimax on s). \\n\n",
    "\n",
    "Value iteration starts from $V_0 (s) = 0$, then update using formula: \\\n",
    "$V_{k+1} (s) = max_a \\sum_{s'} T(s,a,s') [R(s,a,s') + $ $\\gamma$ $V_k(s')]$ \\\n",
    "Repeated until convergence (theorem guarantees that convergence will be \\\n",
    "reached). Each iteration is $O(S^2 A)$ complexity. \\n\n",
    "\n",
    "Proof that values will converge: \\n\n",
    "Case 1: if tree has max depth M then $V_M$ will hold actual values\\n\n",
    "Case 2: if discount < 1, then the difference between $V_k$ and $V_{k+1}$ will be \\\n",
    "at most $\\gamma$ $^k$ $max|R|$ different. As k increases the values will converge. \\n\n",
    "\n",
    "Alternative to value iteration is policy iteration. Define $V^{\\pi}(s)$ as the expected \\\n",
    "total award of starting in s and following $\\pi$. Update with formula: \\\n",
    "$V^{\\pi}_{k+1} (s) = \\sum_{s'} T(s, \\pi (s), s')] [R(s, \\pi (s),s') +$ $\\gamma$ $V_k^{\\pi}(s')]$ Each update is $O(s^2)$ in time complexity. \\n\n",
    "\n",
    "We can also solve for V's under fixed policy using solvers for linear system, since \\\n",
    "there is no max operations involved. \\n\n",
    "\n",
    "It is easier to select actions from q-values than from state-values. \\n\n",
    "\n",
    "Three problems with value iteration: (1) slow runtime, (2) \"max\" at each state \\\n",
    "does not often change, and (3) policy can converge must quicker than values. \\n\n",
    "\n",
    "Policy iteration steps: \\n\n",
    "1) Policy evaluation: calculate utilities for fixed policy until values converge \\n\n",
    "$V^{\\pi}_{k+1} (s) = \\sum_{s'} T(s, \\pi (s), s')] [R(s, \\pi (s),s') + $ $\\gamma$ $V_k^{\\pi}(s')]$\\n\n",
    "\n",
    "2) Policy improvement: update policy with one-step lookahead, using resulting converged \\\n",
    "utilities as future values. \\n\n",
    "$\\pi_{i+1}(s) = argmax_a \\sum_{s'} T(s,a,s')[R(s,a,s') + $ $\\gamma V^{\\pi_i}(s')$]\\n \n",
    "3) Go back to step 1 until policy does not change \\n\n",
    "\n",
    "Value iterations do not track the policy, only implicitly recompute it during the max \\\n",
    "operations. \n",
    "\n",
    "Theorem from Ramsey, von Neumann & Morgenstern: \\n\n",
    "If preferences satisfy orderability, transitivity, continuity, substitutability, and \\\n",
    "monotonicity, then there is a utility function corresponding to it preserving the \\\n",
    "preferences. \n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Definitions of concepts from Module 8.\n",
    "# You can define any term in the textbook or lecture, \n",
    "# but you most use your own words for the definitions.\n",
    "# Each definition should be at most 25 words long.\n",
    "\n",
    "module8_definitions = {\n",
    "\"Markovian transitions\" : \"Probability of arriving in s' only depends on the \\\n",
    "previous state\",\n",
    "\"MEU Principle\" : \"agent should choose action that maximizes expected utility.\",\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "elCCgY64spuq"
   },
   "source": [
    "## Preview of your notes for Module 8\n",
    "\n",
    "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "gYYeNcdM_maE",
    "outputId": "66a46f66-4943-4c59-8646-03294ee099f6"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 8: Markov Decision Processes\n",
       "## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n",
       " $P(RESULT(a) = s') = \\sum_s P(s) P(s' | s, a)$ \n",
       "\n",
       "\n",
       "$Expected Utility = \\sum_s' P(RESULT(a) = s') U(s')$ \n",
       "\n",
       "\n",
       "MEU principle formalizes \"doing the right thing\" but does not operationalize: still need to estimate probability of states thruogh learning, and compute P(RESULT(a)=s') using a causal model.  \n",
       "\n",
       "## AIMA Chapter 16.2 The Basis of Utility Theory\n",
       " Constraints/axioms on rational preference relations: (1) orderability either two lotteries are indifferent or agent prefers one over other, (2) transitivity , (3) continuity: if B is between A and C in preference, there is some probability p s.t. agent will be indifferent between B for sure and A / C with probability (4) substitutability: If indifferent between A and B, then indifferent between two lotteries with only difference being A vs B, (5) Monotonicity: If A $\\succ$ B then agent prefers lottery with higher probability for A, (6) decomposability: Compound lotteries can be reduced to simpler ones\n",
       "\n",
       "\n",
       "von Neumann & Morgenstern: Existence of Utility function. If agent's preference obeys axiom, then there is a function s.t. U(A) > U(B) iff A $\\succ$ B and U(A) = U(B) if $A \\sim B$. \n",
       "\n",
       "## AIMA Chapter 16.3 Utility Functions\n",
       " Utility of money is not directly proportional to monetary value.Value of first million is very high, while marginal value decreases. s shaped curve. A billionnaire thus more likely to take a risky but beneficial move than a stable one \n",
       "\n",
       "\n",
       "Grayson study: utility of money almost exactly proportional to log of the amount. We typically have concave utility function for positive wealth (facing downwards),\n",
       "\n",
       "\n",
       "People in the slope-decreasing part of utility curve will assign higher utility to a guaranteed Expected Monetary Value than a lottery. Risk-avrese agents. Risk-seeking agents show up in the negative part of the wealth utility curve. If the changes in wealth are small, curve will be nearly linear, and agent will be risk-neutral \n",
       "\n",
       "\n",
       "Most rational action a = $argmax_a EU(a)$. Most of the time we can only estimate EU. Unbiased estimate: expected value of error is 0. Since we select action with highest EU estimate, we will have an optimistic bias/ optimizer's curse. Bias higher when we are choosing among more choices. If variance in utility estimate is high, what appears to be the best choice may not even be the best chioce: so make sure sample size is large enough for trials. We can avoid it by using a Bayesian model using a probability model of the error $P(EU_{est} | EU)$, updating EU based on our prior knowledge of it.\n",
       "Human preferences are predictably irrataional: there may be no utility function consistent with our choices. Possible reasons: (1) certainty effect (reduce computational burden, distrust stated probsaccoutning for emotion) (2) Ellsberg paradox: people have ambiguity aversion, prefer known probability. (3) framing effect, (4) anchoring effect. \n",
       "\n",
       "## AIMA Chapter 17.1 Sequential Decision Problems\n",
       " In sequential decision problems agent's utility is the result of a sequence of decisions. Markov decision process = sequential decision problem for fully observable, stochastic environment with a fully observable transition model and additive rewards. Solving MDPs usually require dynamic programming. \n",
       "\n",
       "\n",
       "If a game has finite time horizon, policy may change based on the time left. \n",
       "\n",
       "\n",
       "We use additive discount rewards to define utility for two reasons: (1) empirical evidence shows that humans value near-term rewards. (2) economic reasoning suggests early reward is momre valuable, (3) uncertainty about true rewards, (4) assumption of stationary preference, (5) avoid problems with infinite horizons. \n",
       "\n",
       "\n",
       "On point (5), three ways to avoid infinite utility. (1): use discounted rewards such that utility is bounded by $(R_{max}) / (1 - gamma)$. (2): design proper policies that guarantee agents will reach terminal states, save us from discouting (3): compare sequences using average rather than sum of reward. \n",
       "\n",
       "\n",
       "When we use discounted utilities with infinite horizons, the optimal policy is independent of the starting state. Thus, the utility of each state is the expected sum of discounted rewards if the agent carrys out an optimal policy, or U(s) \n",
       "\n",
       "\n",
       "Utility of one state = expected reward for next transition + discounted utility of the next state. This is the Bellman equation. \n",
       "\n",
       "\n",
       "An affine transformation of reward will leave the optimal policy unchanged. Moreoever the shaping theorem suggests that any transformation in this form will also produce no change to optimal policy: \n",
       "\n",
       "\n",
       "\t $R'(s, a, s') = R(s, a, s') + \\gamma \\phi (s') - \\phi (s)$ where $\\phi (s)$ is any function of state s. \n",
       "\n",
       "\n",
       "Function $\\phi (s)$ is called a potential. $ \\gamma \\phi (s') - \\phi (s)$ is the gradient of the potential. If $\\phi (s)$ is proportional to higher utility values, then adding $ \\gamma \\phi (s') - \\phi (s)$ will lead agent \"uphill\" in utility. \n",
       "\n",
       "\n",
       "Ways to represent MDPs: \n",
       "\n",
       "1) use 3-D tables of size $|S|^2 |A|$. Sparse tables with many 0's only have size $O(|S||A|)$. \n",
       "\n",
       "2) Dynamic decision networks (DDNs) with decision, reward, and utility nodes. Use factored represetnation, and have complexity exponential times better than atomic representation. \n",
       "\n",
       "\n",
       "## AIMA Chapter 17.2 Algorithms for MDPs\n",
       " Problem with solving Bellman equations: the MAX operator is not linear, so can't use linear system solver. Instead we use an iterative approach. \n",
       "\n",
       "\n",
       "Showing that the values would converge: \n",
       "\n",
       "Contraction: A function that, when apply to two values, return two values that are closer together by at least some constant factor. E.g. divide by four \n",
       "\n",
       "\n",
       "A contraction function has only one fixed point. The value gets closer to the fixed point each time contraction is applied. \n",
       "\n",
       "\n",
       "We can show that Bellman update is a contraction by a factor of $\\gamma$ on utility vectors. Define \"length\" of a vector as the abs value of its biggest component. Then difference between two vectors = max difference between two corresponding elements. Then we can show $||BU_i - BU'_i||$ $\\leq$ $\\gamma$ $||U_i - U'_i||$. \n",
       "\n",
       "\n",
       "On each iteration, the error decreases by a factor $\\geq \\gamma$. So value iteration converges in exponential rate. The maximum number of iterations = ceiling of $log (2R_{max} / \\epsilon (1 - \\gamma) / log (1 / \\gamma))$. Implies fast convergence if $\\gamma$ is small / horizon is short, but N does not depend too much on $\\epsilon / R_{max}$. \n",
       "\n",
       "\n",
       "It can be shown that in value iteration, policy often becomes optimal long before value converges. Policy loss is connected to error in value estimate by: \n",
       "\n",
       "\n",
       "if $||U_i - U|| < \\epsilon $ then $||U^{\\pi_i} - U|| < 2 \\epsilon$. \n",
       "\n",
       "\n",
       "Policy iteration is faster than solving Bellman equations: O(n^3) time since we don't have max operator, so equations are linear. To further save time, use modified policy iteration where do a small number of value iterations. \n",
       "\n",
       "\n",
       "Another alternative is asynchronous policy iteration: pick subset of states then apply either policy improvement or simplified value iteration. \n",
       " \n",
       "\n",
       "We can also convert MDPs into linear programming problems, by treating utilities as variables and seeking to minimize utilities under constraints. (bellman equations). Makes MDPs solvable in polynomial time in the number of states and actions. However, number of states may be quite large, so LP may be less efficient than DP. \n",
       "\n",
       "\n",
       "We can also solve MDPs using Expectimax trees, limiting depths by using the $\\epsilon$ horizon which is to cut the tree at a depth where sum of reward is lower than $\\epsilon$. We can also limit branching factor by sampling values instead of getting all of them. This approach is Real-Time Dynamic Programming (RTDP), similar to LRTA\\* algorithm. Useful in solving moderate sized problems,  but limited for large ones. \n",
       "\n",
       "\n",
       "Enhancements to RTDP: (1) apply reinforcement learning to generate better heuristic. (2) Look further ahead in MDP with Monte Carlo approach. \n",
       "\n",
       "\n",
       "## Module 8 Lecture Notes\n",
       " A MDP is a sequential, fully observable decision problem with additive rewards and stochastic transition model (i.e. nondeterministic). Need to be solved by expectimax search. \n",
       "\n",
       "\n",
       "MDP is defined by: (1) set of states, (2) set of actions, (3) transition function, specifying the probability (aka the model), (4) reward function, (5) initial state, and (6) (optional) terminal state. \n",
       " \n",
       "\n",
       "In MDPs rewards are (1) either positive or negative, but bounded. (2) specified on a sequence of actions since MDP is sequential, (3) can be received both at goal state or at each transition, (4) additive. \n",
       "\n",
       "\n",
       "A solution to a MDP is a policy: a mapping from state to action. vs solution to search problem is a plan. $\\pi$ denotes a policy and $\\pi *$ denotes the optimal policy. Each policy is associated with an expected utility. Results of same policy may vary according to the different environment history. \n",
       "\n",
       "\n",
       "Two reasons for discounting future rewards: (1) fit empirical observation and (2) help algorithm converges. \n",
       "\n",
       "\n",
       "Theorem: assuming stationary preferences (i.e. if agent prefers seq A to seq B, then agent prefers action r + seq A to action r + sequence B), we can only define utility as additive (simple sum) or discounted (sum weighted by discount). \n",
       "\n",
       "\n",
       "Three solutions for infinite games: (1) finite horizon (stop after T steps), which will cause nonstationary policies, (2) discounting (wiht smaller $\\gamma$ meaning smaller time horizon), and (3) absorbing state that every policy is guaranteed to reach. \n",
       "\n",
       "\n",
       "Value of state s = expected utility of starting in the state then acting optimally. $V^*(s) = max_a Q^*(s,a)$\n",
       "\n",
       "\n",
       "Value of Q state (s, a) = expected utility of taking a from s then acting optimally. $Q^*(s,a) = \\sum_{s'} T(s,a,s') [R(s,a,s') + $ $\\gamma$ $V^*(s')]$ \n",
       "\n",
       "\n",
       "We can use time limited values to reduce amount of work with expectimax. $V_k(s)$ = optimal value of s if game ends in k more time steps (i.e. value from k-depth expectimax on s). \n",
       "\n",
       "\n",
       "Value iteration starts from $V_0 (s) = 0$, then update using formula: $V_{k+1} (s) = max_a \\sum_{s'} T(s,a,s') [R(s,a,s') + $ $\\gamma$ $V_k(s')]$ Repeated until convergence (theorem guarantees that convergence will be reached). Each iteration is $O(S^2 A)$ complexity. \n",
       "\n",
       "\n",
       "Proof that values will converge: \n",
       "\n",
       "Case 1: if tree has max depth M then $V_M$ will hold actual values\n",
       "\n",
       "Case 2: if discount < 1, then the difference between $V_k$ and $V_{k+1}$ will be at most $\\gamma$ $^k$ $max|R|$ different. As k increases the values will converge. \n",
       "\n",
       "\n",
       "Alternative to value iteration is policy iteration. Define $V^{\\pi}(s)$ as the expected total award of starting in s and following $\\pi$. Update with formula: $V^{\\pi}_{k+1} (s) = \\sum_{s'} T(s, \\pi (s), s')] [R(s, \\pi (s),s') +$ $\\gamma$ $V_k^{\\pi}(s')]$ Each update is $O(s^2)$ in time complexity. \n",
       "\n",
       "\n",
       "We can also solve for V's under fixed policy using solvers for linear system, since there is no max operations involved. \n",
       "\n",
       "\n",
       "It is easier to select actions from q-values than from state-values. \n",
       "\n",
       "\n",
       "Three problems with value iteration: (1) slow runtime, (2) \"max\" at each state does not often change, and (3) policy can converge must quicker than values. \n",
       "\n",
       "\n",
       "Policy iteration steps: \n",
       "\n",
       "1) Policy evaluation: calculate utilities for fixed policy until values converge \n",
       "\n",
       "$V^{\\pi}_{k+1} (s) = \\sum_{s'} T(s, \\pi (s), s')] [R(s, \\pi (s),s') + $ $\\gamma$ $V_k^{\\pi}(s')]$\n",
       "\n",
       "\n",
       "2) Policy improvement: update policy with one-step lookahead, using resulting converged utilities as future values. \n",
       "\n",
       "$\\pi_{i+1}(s) = argmax_a \\sum_{s'} T(s,a,s')[R(s,a,s') + $ $\\gamma V^{\\pi_i}(s')$]\n",
       " \n",
       "3) Go back to step 1 until policy does not change \n",
       "\n",
       "\n",
       "Value iterations do not track the policy, only implicitly recompute it during the max operations. \n",
       "\n",
       "Theorem from Ramsey, von Neumann & Morgenstern: \n",
       "\n",
       "If preferences satisfy orderability, transitivity, continuity, substitutability, and monotonicity, then there is a utility function corresponding to it preserving the preferences. \n",
       "\n",
       "\n",
       "## Module 8 Definitions\n",
       "* **Markovian transitions** - Probability of arriving in s' only depends on the previous state\n",
       "* **Meu principle** - agent should choose action that maximizes expected utility.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "from IPython.display import display, Markdown, Latex\n",
    "\n",
    "module8_sections = [\n",
    "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
    "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
    "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
    "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
    "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
    "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
    "]\n",
    "\n",
    "\n",
    "def length_check(sections):\n",
    "  warnings = \"\"\n",
    "  for section, summary in sections:\n",
    "    max_word_length = 500\n",
    "    max_character_length = 3000\n",
    "    if \"lecture\" in section.lower():\n",
    "      # It's OK to do longer summaries of the lectures\n",
    "      max_word_length = 2000\n",
    "      max_character_length = 12000  \n",
    "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
    "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
    "  return warnings\n",
    "\n",
    "def length_check_definitions(definitions):\n",
    "  warnings = \"\"\n",
    "  for concept, definition in definitions.items():\n",
    "    if(len(definition.split(' ')) > 25):\n",
    "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
    "  return warnings\n",
    "\n",
    "\n",
    "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
    "  markdown = \"\"\n",
    "\n",
    "  # Check summary lengths\n",
    "  warnings = length_check(sections)\n",
    "  if warnings != \"\":\n",
    "    markdown += \"## \" + warnings\n",
    "\n",
    "  # Check definition lengths\n",
    "  warnings = length_check_definitions(definitions)\n",
    "  if warnings != \"\":\n",
    "    markdown += \"## \" + warnings\n",
    "\n",
    "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
    "\n",
    "\n",
    "  for section, summary in sections:\n",
    "    if show_placeholder and summary == \" Your summary here \":\n",
    "      summary = placeholder\n",
    "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
    "  \n",
    "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
    "  for concept, definition in definitions.items():\n",
    "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
    "  return markdown\n",
    "\n",
    "\n",
    "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
    "display(Markdown(module8_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9xRbdqDox2zD"
   },
   "source": [
    "#Module 9 - Reinforcement Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "DpBvu7EMx7sy"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
    "chapter17_3 =  \"\"\" Bandit problem is a formal models for many important problems, e.g. \\\n",
    "deciding when to apply certain treatments / investments. \\n\n",
    "\n",
    "Definition of Bandit Problems: \\n\n",
    "(1) Each arm is a Markov Reward Process (MRP): MDP with only one possible action. \\n\n",
    "(2) Overall bandit problem as an MDP with state space = Cartesian product of all states. \\n\n",
    "(The arms are independent, and agent can only pull on at a time) \\n\n",
    "\n",
    "Gittins index of a lever is defined as: \\n\n",
    "$\\lambda$ = $max_{T > 0}$ $ (E(\\sum^{T-1}_{t=0} \\gamma ^t R_t)) / (E(\\sum^{T-1}_{t=0} \\gamma ^ t))$ \\n\n",
    "\n",
    "Gittins index captures the maximum obtainable utility per unit of discounted time. \\\n",
    "Optimal policy for any bandit problem is to pull arm with higest GIndex then update the \\\n",
    "indices. Since index of arm only depends on info about that arm, optimal decision of first \\\n",
    "iteration can be calculated in O(n) time, and subsequent decision in O(1) time. \\n\n",
    "\n",
    "At the point where optimal policy in indifferent between an arm and a fixed arm, \\\n",
    "the value of M is same as choosing an infinit sequence with reward = Gittens index. \\n\n",
    "\n",
    "Restart MDP: replace the choice to enter an infinite sequence with lambda reward with \\\n",
    "choice to go back and restart M from initial state. Gittens index for M = (1 - $\\lambda$) \\\n",
    "times value of optimal policy for restart MDP. When we solve the MDP we can get Gittens index. \\n\n",
    "\n",
    "Bernoulli bandit problem has each arm with reward 0 or 1. Cannot solve for Gittins index \\\n",
    "directly using restart MDP since state space is infinite, but we can approximate by truncating \\\n",
    "the MDP with a discount factor and a limit on numbers of trial. Result is preference \\\n",
    "for higher payoffs + an exploration bonus. \\n\n",
    "\n",
    "Methods to generate almost optimal policies: (1) use Upper Confidence Bound (UCB) \\\n",
    "heuristic, using sampels to establish confidence interval for the payoff of each arm. \\\n",
    "Approximate index value for arm = \\n\n",
    "\n",
    "$UCB(M_i) = \\mu_i + g(N) / \\sqrt(N_i)$ \\n\n",
    "\n",
    "(2) Thompson sampling, which chooses an arm with probability related to the probability \\\n",
    "that the arm is optimal. Regret grows as O(log N) for both (1) and (2). \\n\n",
    "\n",
    "Non-indexable problems: \\n\n",
    "\n",
    "(1) Selection problem: choosing best options when there is fixed cost for each. No index function \\\n",
    "unlike bandit problems. Selection solvers will explore more than bandit solvers. \\n\n",
    "\n",
    "(2) multitasking, or bandit superprocesses, or BSPs: Generalized bandit problems where \\\n",
    "each bandit is a full MDP in their own right. In BSPs globally optimal problem may require \\\n",
    "acting suboptimally locally. We often need to be greedier in each MDP. We can solve BSPs \\\n",
    "by modeling interactions between each MDP using opportunity cost. We can also convert a bandit \\\n",
    "arm into a MRP by coming up with an optimal dominating policy, which gives ceilign to the value \\\n",
    "of acting in the arm. Lower bound can be given by computing Gittens index. Then we \\\n",
    "can use search and recomputing bounds to solve BSPs.\n",
    "\"\"\"\n",
    "\n",
    "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
    "chapter22_1 = \"\"\" As opposed to teaching agent the optimal moves, it is easier to \\\n",
    "give them rewards because (1) rewards are concise and easy to specify, and (2) \\\n",
    "we don't need to have the expertise. A problem is sparse rewards - when many states \\\n",
    "do not give reward at all. \\n\n",
    "\n",
    "Two kinds of reinforcement learning appraoches:\\n\n",
    "\n",
    "1) Model-based reinforcement learning: agent uses a transition model, learn a utility function \\\n",
    "U(s) in terms of sums of rewards from state onward.\\n\n",
    "\n",
    "2) Model-free reinforcement learning: two forms. a) action utility learning. most common form is Q-learning - agent learns a \\\n",
    "Q-function with sum of rewards when taking certain action in certain state. b) Policy \\\n",
    "search: agent learns a policy, behaves like reflex agent. \\n\n",
    "\n",
    "Passive reinforcement learning fixes the policy. Active RL requires agents \\\n",
    "to alos decide what to do/ i.e. exploration. \n",
    "\"\"\"\n",
    "\n",
    "chapter22_2 = \"\"\" Given fixed policy, agent tries to learn the expected discounted reward \\\n",
    "of beginning from state s. \\n\n",
    "\n",
    "Direct utility estimatation: agent conducts trials, treat each resulting reward \\\n",
    "as a sample. Problem is that it ignores the connection between states (i.e. given by \\\n",
    "Bellma equation). Therefore it converges very slowly. \\n\n",
    "\n",
    "Adaptive dynamic programming (model-based, passive, online): agent includes learned transition model into Bellman \\\n",
    "equation to calculate utilities. Since Bellman is linear, linear algebra packages can \\\n",
    "be used to solve them. \\n \n",
    "\n",
    "\\tdef passive_adp: \\n\n",
    "\\t(input: current state s' and reward r)\n",
    "\\t(persistent: $\\pi$, mdp with model P, rewards R, actions A, discount $\\gamma$, table of utilities \\\n",
    "U, $N_{s'|s,a}$ table of outcome counts, s, and a (initially null)) \\n\n",
    "  \\t if s' is new then U[s'] = 0\\n\n",
    "  \\t if s is not null then:\\n\n",
    "  \\t\\t increment $N_{s'|s,a}$[s,a][s']\\n\n",
    "  \\t\\t R[s,a,s'] = r\\n\n",
    "  \\t\\t add a to A[s]\\n\n",
    "  \\t\\t P($\\dot$|s,a) = NORMALIZE($N_{s'|s,a}$[s,a])\\n\n",
    "  \\t\\t U = policy_evaluation($\\pi$, U, mdp)\\n\n",
    "  \\t\\t s,a = s', $\\pi$s' \\n\n",
    "  \\t\\t return a \\n\n",
    "\n",
    "We can also use modified policy iteration with simplified value iteration for updating \\\n",
    "utility estimates after each change to the learned model. \\n\n",
    "\n",
    "Passive ADP provides a standard by which other RL algorithms are measured. \\n\n",
    "It is impracticle for large state spaces. May take some time to \"discover\" rarely \\\n",
    "visited states \\n\n",
    "\n",
    "Temporal difference learning (model-free, passive, offline): update utility of a state every time it transitions \\\n",
    "to a new state, with the function: \\n\n",
    "\n",
    "$U^{\\pi}(s) = U^{\\pi}(s) +$ $a$ $[R(s, \\pi (s), s') +$ $\\gamma$ $U^{\\pi}(s') - U^{\\pi}(s)]$ \\n\n",
    "\n",
    "$a$ stands for the learning rate. The term multiplied with alpha is an error signal. \\\n",
    "If $a$ becomes a function that decreases with the number of state visits, then $U^{\\pi}(s)$ will \\\n",
    "converge to the correct value. \\n\n",
    "\n",
    "TD learning does not converge as fast, but is simpler and require fewer computations, since \\\n",
    "it does not require a transition model. TD adjusts utility based on observed successors, \\\n",
    "while ADP adjusts utility based on all successors. TD makes one adjustment every observed transition, \\\n",
    "while ADP makes as many as it needs. \\n\n",
    "\n",
    "We can exstend TD appraoch to generate psuedoexperiences - to make the utility estimates \\\n",
    "closer to ADP, at the cost of higher computation time. \\n\n",
    "\n",
    "We can also make ADP more efficient in the other way. Prioritized sweeping heuristic \\\n",
    "prioritizes adjusting states whose possible successors have got a large adjustment in utility \\\n",
    "estimates. Speeds up approximate ADP and allows it to handle larger state spaces than ADP. \\n\n",
    "\"\"\"\n",
    "\n",
    "chapter22_3 = \"\"\" Greedy agents suffer from the problem of no exploration. To improve, \\\n",
    "we use GILE, or \"Greedy in the limit of infinite exploration\". GILE scheme \\\n",
    "tries each action in eact states certain times to avoid missing optimal actions. \\n\n",
    "\n",
    "GILE scheme 1: random action at each point. Problem is slow conversion. \\n\n",
    "\n",
    "GILE scheme 2: give weight to unexplored actions. Include exploration function f(u, n) \\\n",
    "which increases with u and decreases with n. E.g. we can define f(u, n) to return \\\n",
    "max reward if an action is not yet explored for k times, and return u otherwise. \\n\n",
    "\n",
    "We should use $U^+$ (optimistic estimate of utility/ expected reward-to-go) for updates \\\n",
    "instead of U, since it propagates benefits of exploration. \\n\n",
    "\n",
    "Need to handle the problem of entering absorbing states and exercising irreversible actions. \\\n",
    "We should therefore prefer policy that works well across all models, even if the policy \\\n",
    "is not optimal for the model with max probability. Three appraoches:\\n\n",
    "\n",
    "1) Bayesian Reinforcement Learning: assumes a prior probaility, than update with each observation \\\n",
    "optimal policy is one with highest expected utility. Requires priors to be able to \\\n",
    "tell danger. Problem can be transformed into an exploration POMDP where belief states \\\n",
    "are distribution over models, and can be solved before agent starts moving. \\n\n",
    "\n",
    "2) Robust control theory: defines optimal policy as one that performs best in worst \\\n",
    "case. Essentially maximin rule - may result in conservative policy \\n\n",
    "\n",
    "3) Use human expertise: have human teachers do a demo, or write down constraints. \\n\n",
    "\n",
    "Temporal difference Q-learning: model-free learning with agents learning Q(s,a) rather \\\n",
    "than U(s). It does not need a transition model. Update function is: \\n\n",
    "\n",
    "$Q(s,a) = Q(s,a) + alpha [R(s,a,s') +$ $/gamma$  $max_{a'} Q(s',a') - Q(s,a)$\\n\n",
    "\n",
    "Q-learning agent cannot look into the future, so if rewards are sparse it would find \\\n",
    "difficulty. It is off-policy, since it assumes stopping the current policy \\\n",
    "and adopting the best policy according to estimates. \\n\n",
    "\n",
    "SARSA (state,action,reward,state,action): similar to Q-learning, except that SARSA waits \\\n",
    "until an action is taken, then update based on the taken action. If exploration gives negative \\\n",
    "reward, SARSA will punish the action but Q-learning will not. It is on-policy, since \\\n",
    "it assumes continuing with current policy. SARSA is more appropriate when we do not \\\n",
    "have full control over what the agent does. \\n\n",
    "\"\"\"\n",
    "\n",
    "chapter22_4 = \"\"\" In large gamespaces, convergence of utilities will be slow. We can \\\n",
    "approximate utility/Q-function instead, using weighted linear combination of features. \\n\n",
    "\n",
    "E.g. Approximating direct utility estimation: parametrize the utility, then find best \\\n",
    "fit parameters (we use online learning, so we update parameters after each trial). \\\n",
    "Define error has half of the squared difference between predicted and actual \\\n",
    "utility. Use the Widrow-Hoff rule / delta rule for online least-squares. \\ \n",
    "If true utility function is linear than learning will be easy; if not, we can have \\\n",
    "features be nonlinear functions of state variables, can continue to use linear function \\\n",
    "approximators. \\n\n",
    "\n",
    "Approximate TD learning, similar idea: vary the parameters and optimize. Converge \\\n",
    "in the case of TD learning and when function approximator is linear in features. \\\n",
    "When using active learning and non-linear functions, there may be problems with infinite \\\n",
    "parameters. Also problem with castastrophic forgetting - learning too well that \\\n",
    "it forgot previous bad experience. Potential solution is experience relay - remind \\\n",
    "self of bad experience at regular intervals. \\n\n",
    "\n",
    "Deep RL goes beyond linear function approximators for 2 reasons: 1) there may not be \\\n",
    "good linear functions, 2) we may not be able to come up with features. More complex \\\n",
    "function but same gradients as before. Deep RL can play games well but may not handle \\\n",
    "varied environments. \\n\n",
    "\n",
    "Credit assignment problem: agent needs to figure out where it does wrong. Solution is reward \\\n",
    "shaping: use additional pseudorewards. Need to ensure that optimal policy is not changed with \\\n",
    "their addition: we can use potential function $+ \\gamma \\phi (s') - \\phi (s)$ to reflect the aspect we want to \\\n",
    "keep track of. \\n\n",
    "\n",
    "Hierarchical RL: break up long sequences of action into small pieces. Builds partial \\\n",
    "programs that extend normal programming languages and provide general know-how. \\n\n",
    "\n",
    "HRL based on joint state space: (physical state, machine state) pairs. Define choice \\\n",
    "state as state at which program is at a choice point. Transform HRL into a Markovian decision \\\n",
    "problem, with state = choice states, actions = choices, reward = rewards for physical \\\n",
    "transitions. For transition model, if c chooses physical action, then use physical model \\\n",
    "if computational action taken, then modifies computational states. Solving MDP gets \\\n",
    "use the optimal policy consistent with partial program. \\n\n",
    "\n",
    "Two additional advantages of HRL: 1) more opportunity for reward shaping, 2) provides \\\n",
    "additive decomposition of overall utility function. \n",
    "\"\"\"\n",
    "\n",
    "chapter22_5 = \"\"\" Another alternative RL approach is policy search. Represents policies using \\\n",
    "parameterization, with fewer parameters than number of staqtes. Then adjust parmeters to improve the \\\n",
    "policy. \\n\n",
    "\n",
    "A problem is that policy is sometimes a discontinous function. To address this, we use a \\\n",
    "stochastic policy representation that specifies probability of selecting certain action in a stet. \\\n",
    "Example is softmax function. \\n\n",
    "\n",
    "To improve policy in deterministic environment, we optimize $p(\\theta)$ by following the \\\n",
    "policy gradient vector $\\delta_{\\theta} p(\\theta)$, or by executing the policy and observing the rewards \\\n",
    ", following empirical gradient by hill climbing. Eventually it will converge to local optimum. \\n\n",
    "\n",
    "When environment is non-deterministic, rewards may fluctuate a lot, whcih require either \\\n",
    "a lot of samples, approximate true gradient by summing up terms involving the gradient \\\n",
    "of action-selection probability in each trial (Ron Williams appraoch), or use correlated sampling \\\n",
    "(i.e. the same set of samples for two policies). \n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module9_lecture_notes = \"\"\" \n",
    "In MDPs agents are given reward function and transition function (we know probability \\\n",
    "distribution of the states). In reinforcement learning \\\n",
    "agents must learn these. Tradeoffs between exploration and exploitation \\n\n",
    "\n",
    "Double bandits (come from slot machines): actions are play blue vs play red \\\n",
    "states are win and lose. No discount, 100 time steps, and both states have the same \\\n",
    "value. If probabilities are specified, \\\n",
    "offline planning can solve MDP - we don't actually need to play. Without the probabilties, \\\n",
    "we need to learn rather than plan. Learning can be much harder than solving a known \\\n",
    "MDP. \\n\n",
    "\n",
    "In RL, agents choose an action, receive reward from environment. Agents' utility \\\n",
    "is defined by reward function. Agent must learn to maximize expected rewards. \\\n",
    "All learning will be based on observed samples of outcomes. \\ We still assuming MDP \\\n",
    "with set of states, set of actions, model T, and reward function R, with T and R \\\n",
    "hidden initially. \\n\n",
    "\n",
    "Two different kinds of learning: model-based learning and model-free learning \\n\n",
    "\n",
    "Model based learning: learn an approximate model of R and T, then use MDP algo. \\\n",
    "Count outcomes for each action, normalize to estimate transition function, then discover \\\n",
    "rewards and come up with maximum likelihood estimates. E.g. fix input policy, \\\n",
    "start at different points, observe episodes. \\n\n",
    "\n",
    "Passive RL: Every time we take an action, we compare what is expected vs what happens. Does not keep track \\\n",
    "of T and R. Given fixed policy, we want to learn state values. Execute policy and learn \\\n",
    "from experience. Goal is to compute values for every state under the action $\\pi$. \\\n",
    "Direct evaluation - average the observed sample values. \\n\n",
    "\n",
    "In each passive RL training episode, we run through the policy to the end (from each state) \\\n",
    "and compute the value of state accordingly. \\n\n",
    "\n",
    "Benefits of direct evaluation: \\n\n",
    "1. easy to understand\\n\n",
    "2. does not need to understand R and L\\n\n",
    "3. Eventually compute the correct average values\\n\n",
    "\n",
    "Disadvantages: \\n\n",
    "1. Waste information about state connections (learn value of state wihtout \\\n",
    "looking at other states), \\n\n",
    "2. each state learned separately \\n\n",
    "3. So, takes a long time to learn\\n\n",
    "\n",
    "We can improve using policy evaulation, which use connections between states. \\\n",
    "However, normal policy evaluation uses weighted average, which requires knowing the \\\n",
    "weights. Instead, we can use sample-based policy evaluation: for each (state, action) \\\n",
    "pair, collect sample resulting stages, sum up sample values and divide by num of samples. \\\n",
    "However, that requires \"Crystal ball\" simulation of results. \\n\n",
    "\n",
    "Temporal difference learning: update V(s) every time we experience a transition that \\\n",
    "starts from s. Likely outcomes will lead to updates more often. Every time we have sample, \\\n",
    "we update V(s) as: $V^{\\pi}(s) = (1 - alpha) V^{\\pi}(s) + alpha * sample$. \\\n",
    "Alpha is a value we set which says how much our observations impact vs old V. \\n\n",
    "\n",
    "We end up calculating an exponential moving average, which weights current observation \\\n",
    "more heavily than new observations. Forgets about the past (but the past is not as important). \\\n",
    "$x_n = (x_n + (1 - alpha) x_{n-1} + (1 - alpha)^2 x_{n-2} + ... ) / (1 + (1 - alpha) + (1 - alpha) ^ 2 + ...)$ \\n\n",
    "\n",
    "TD value learning is model-free, mimicking Bellman updates with moving averages. \\n\n",
    "\n",
    "Problems with TD value learning: we do not have Q-values, so we can't turn values into a new \\\n",
    "policy. \\n\n",
    "\n",
    "Q-value iteration: given Qk, calculate k+1 q-values for all q-states. Formula: \\n\n",
    "\\t $Q_{k+1}(s,a) = \\sum_{s'} T(s, a, s') [R(s, a, s') + \\gamma max_{a'} Q_k (s', a')]$ \\n\n",
    "\n",
    "Acting RL/ Q-learning: sample based Q value iteration. When we receive our sample, \\\n",
    "we compare to our previous esimate, then update. \\n\n",
    "\n",
    "$Q(s,a) = (1 - a) Q(s,a) + a * [sample]$ \\n\n",
    "\n",
    "Q-learning converges to optimal policy - example of off-policy learning. Caveats: \\\n",
    "we need to explore enoguh, we will eventually have to make learning rate small enough \\\n",
    "but not decrease it too quickly. \\n\n",
    "\n",
    "Choosing when to explore: \\n\n",
    "1. Simplest way: random actions ($\\epsilon$-greedy). WIth small probability = $\\epsilon$ we act randomly \\\n",
    "else act on current policy. Problem: value of random exploration decreases as we go. \\\n",
    "Solution 1: lower $\\epsilon$ over time. Solution 2: explicitly write a function \\n\n",
    "2. Exploration functinos: explore areas whose value has not yet established. Function \\\n",
    "$f(u,n) = u + k / n$, u = value estimate, n = visit count, k: constant bonus, \\\n",
    "returns an optimistic utility. \\n\n",
    "\n",
    "Modified Q-update: $Q(s,a) = R(s,a,s') + \\gamma max_{a'} f(Q(s',a'), N(s',a'))$. vs just \\\n",
    "using old Q value in regular Q update. \\n\n",
    "\n",
    "We can quantify effectiveness of exploration strategy using regret. Regret measures \\\n",
    "total mistake cost: difference between expected rewards and optimal rewards. Minimize regret = \\\n",
    "requires us to reach optimal policy as quickly as we can. Random exploration has higher regret. \\n\n",
    "\n",
    "Approximate Q-learning: Some problems have many states, so we need to generalize states. \\\n",
    "Basic q-learning keeps track of all q values, approx learning learn about small number of \\\n",
    "states, then generalize learning to new states. Similar to evaluating chess states using \\\n",
    "features. Use a linear value function that assigns weights to each feature. Pro: experienced summed \\\n",
    "up in a few numbers. Disadvantage: features may not adequately predict values. \\n\n",
    "\n",
    "E.g. $Q(s,a) = w_1 f_1(s,a) + w_2 f_2(s,a) + .... $\\n\n",
    "\n",
    "\\t Transition = (s, a, r, s')\\n\n",
    "\\t Difference/error term = $[r + \\gamma max_{a'} Q(s', a')] - Q(s,a)$\\n\n",
    "\\t Q(s,a) = Q(s,a) + $alpha$ [difference]  (Exact Q's) \\n\n",
    "\\t\\t $w_i = w_i + alpha$[difference]$f_i (s,a)$  (Approximate Q's)\\n\n",
    "\\t\\t (we update each weight instead of the entire Q state) \\n\n",
    "\n",
    "We adjust the weights of active features as we go. If something bad happens, blame them \\\n",
    "feature that were on.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Each definition should be at most 25 words long.\n",
    "module9_definitions = {\n",
    "\"Prioritized sweeping heuristic\" : \"Adjustment to ADP which prioritizes states whose \\\n",
    "likely successors have just undergone large adjustment in utilities\",\n",
    "\"Adaptive dynamic programming\" : \"Estimates transition model from counts, then \\\n",
    "apply policy evaluation \",\n",
    "\"Temporal difference learning\" : \"Learning that uses difference in utilities between \\\n",
    "successive states\",\n",
    "\"Greedy in the limit of infinite exploration (GLIE)\" : \"Agent must try each action \\\n",
    "in each state an unbounded number of times\",\n",
    "\"Robust control theory\" : \"Defines optimal policy as one that gives best outcome \\\n",
    "in the worst case\",\n",
    "\"exploration\" : \"trying previously unknown states to gain information\",\n",
    "\"exploitation\" : \"exercising currently known best action to obtain rewards\",\n",
    "\"Markovian transitions\" : \"transitions depends only on the current state but not previous states.\",\n",
    "\"Environment history\" : \"a sequence of states/actions on which the utility function is dependent\",\n",
    "\"policy\" : \"instruction that tells agents what kind of action they should carry out \\\n",
    "under each state\",\n",
    "\"optimal policy\" : \"Policy that yields the highest expected utility\", \n",
    "\"Finite horizon\" : \"a fixed time point after which the game is done\", \n",
    "\"nonstationary policy\" : \"a policy that is dependent on time\", \n",
    "\"Infinite horizon\" : \"games with an infinite state sequence (may still contain \\\n",
    "terminal state(s)\",\n",
    "\"additive discounted rewards\" : \"utility function taht defines utility as sum of rewards \\\n",
    "weighted by a discount factor\",\n",
    "\"stationaray preferences\" : \"if we prefer one future to another starting tomorrow, \\\n",
    "we would prefer it even if it starts today\",\n",
    "\"proper policy\" : \"a policy where an agent is guaranteed to reach a terminal state\", \n",
    "\"Bellman equation\" : \"Utility of one state = expected reward for next transition + discounted utility \\\n",
    "of the next state. \",\n",
    "\"n-armed bandit\" : \"slot machine with n levers\", \n",
    "\"absorbing state\" : \"state where no actions from agent has any effect and no rewards \\\n",
    "will be received\",\n",
    "\"reward to-go\" : \"expected reward starting from a certain state\",\n",
    "\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "ydOK__lwWgdp",
    "outputId": "c66daaaf-dec9-4499-fa7a-ef0ad66499da"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 9: Reinforcement Learning\n",
       "## AIMA Chapter 17.3 Bandit Problems\n",
       " Bandit problem is a formal models for many important problems, e.g. deciding when to apply certain treatments / investments. \n",
       "\n",
       "\n",
       "Definition of Bandit Problems: \n",
       "\n",
       "(1) Each arm is a Markov Reward Process (MRP): MDP with only one possible action. \n",
       "\n",
       "(2) Overall bandit problem as an MDP with state space = Cartesian product of all states. \n",
       "\n",
       "(The arms are independent, and agent can only pull on at a time) \n",
       "\n",
       "\n",
       "Gittins index of a lever is defined as: \n",
       "\n",
       "$\\lambda$ = $max_{T > 0}$ $ (E(\\sum^{T-1}_{t=0} \\gamma ^t R_t)) / (E(\\sum^{T-1}_{t=0} \\gamma ^ t))$ \n",
       "\n",
       "\n",
       "Gittins index captures the maximum obtainable utility per unit of discounted time. Optimal policy for any bandit problem is to pull arm with higest GIndex then update the indices. Since index of arm only depends on info about that arm, optimal decision of first iteration can be calculated in O(n) time, and subsequent decision in O(1) time. \n",
       "\n",
       "\n",
       "At the point where optimal policy in indifferent between an arm and a fixed arm, the value of M is same as choosing an infinit sequence with reward = Gittens index. \n",
       "\n",
       "\n",
       "Restart MDP: replace the choice to enter an infinite sequence with lambda reward with choice to go back and restart M from initial state. Gittens index for M = (1 - $\\lambda$) times value of optimal policy for restart MDP. When we solve the MDP we can get Gittens index. \n",
       "\n",
       "\n",
       "Bernoulli bandit problem has each arm with reward 0 or 1. Cannot solve for Gittins index directly using restart MDP since state space is infinite, but we can approximate by truncating the MDP with a discount factor and a limit on numbers of trial. Result is preference for higher payoffs + an exploration bonus. \n",
       "\n",
       "\n",
       "Methods to generate almost optimal policies: (1) use Upper Confidence Bound (UCB) heuristic, using sampels to establish confidence interval for the payoff of each arm. Approximate index value for arm = \n",
       "\n",
       "\n",
       "$UCB(M_i) = \\mu_i + g(N) / \\sqrt(N_i)$ \n",
       "\n",
       "\n",
       "(2) Thompson sampling, which chooses an arm with probability related to the probability that the arm is optimal. Regret grows as O(log N) for both (1) and (2). \n",
       "\n",
       "\n",
       "Non-indexable problems: \n",
       "\n",
       "\n",
       "(1) Selection problem: choosing best options when there is fixed cost for each. No index function unlike bandit problems. Selection solvers will explore more than bandit solvers. \n",
       "\n",
       "\n",
       "(2) multitasking, or bandit superprocesses, or BSPs: Generalized bandit problems where each bandit is a full MDP in their own right. In BSPs globally optimal problem may require acting suboptimally locally. We often need to be greedier in each MDP. We can solve BSPs by modeling interactions between each MDP using opportunity cost. We can also convert a bandit arm into a MRP by coming up with an optimal dominating policy, which gives ceilign to the value of acting in the arm. Lower bound can be given by computing Gittens index. Then we can use search and recomputing bounds to solve BSPs.\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.1 Learning from Rewards\n",
       " As opposed to teaching agent the optimal moves, it is easier to give them rewards because (1) rewards are concise and easy to specify, and (2) we don't need to have the expertise. A problem is sparse rewards - when many states do not give reward at all. \n",
       "\n",
       "\n",
       "Two kinds of reinforcement learning appraoches:\n",
       "\n",
       "\n",
       "1) Model-based reinforcement learning: agent uses a transition model, learn a utility function U(s) in terms of sums of rewards from state onward.\n",
       "\n",
       "\n",
       "2) Model-free reinforcement learning: two forms. a) action utility learning. most common form is Q-learning - agent learns a Q-function with sum of rewards when taking certain action in certain state. b) Policy search: agent learns a policy, behaves like reflex agent. \n",
       "\n",
       "\n",
       "Passive reinforcement learning fixes the policy. Active RL requires agents to alos decide what to do/ i.e. exploration. \n",
       "\n",
       "\n",
       "## AIMA Chapter 22.2 Passive Reinforcement Learning\n",
       " Given fixed policy, agent tries to learn the expected discounted reward of beginning from state s. \n",
       "\n",
       "\n",
       "Direct utility estimatation: agent conducts trials, treat each resulting reward as a sample. Problem is that it ignores the connection between states (i.e. given by Bellma equation). Therefore it converges very slowly. \n",
       "\n",
       "\n",
       "Adaptive dynamic programming (model-based, passive, online): agent includes learned transition model into Bellman equation to calculate utilities. Since Bellman is linear, linear algebra packages can be used to solve them. \n",
       " \n",
       "\n",
       "\tdef passive_adp: \n",
       "\n",
       "\t(input: current state s' and reward r)\n",
       "\t(persistent: $\\pi$, mdp with model P, rewards R, actions A, discount $\\gamma$, table of utilities U, $N_{s'|s,a}$ table of outcome counts, s, and a (initially null)) \n",
       "\n",
       "  \t if s' is new then U[s'] = 0\n",
       "\n",
       "  \t if s is not null then:\n",
       "\n",
       "  \t\t increment $N_{s'|s,a}$[s,a][s']\n",
       "\n",
       "  \t\t R[s,a,s'] = r\n",
       "\n",
       "  \t\t add a to A[s]\n",
       "\n",
       "  \t\t P($\\dot$|s,a) = NORMALIZE($N_{s'|s,a}$[s,a])\n",
       "\n",
       "  \t\t U = policy_evaluation($\\pi$, U, mdp)\n",
       "\n",
       "  \t\t s,a = s', $\\pi$s' \n",
       "\n",
       "  \t\t return a \n",
       "\n",
       "\n",
       "We can also use modified policy iteration with simplified value iteration for updating utility estimates after each change to the learned model. \n",
       "\n",
       "\n",
       "Passive ADP provides a standard by which other RL algorithms are measured. \n",
       "\n",
       "It is impracticle for large state spaces. May take some time to \"discover\" rarely visited states \n",
       "\n",
       "\n",
       "Temporal difference learning (model-free, passive, offline): update utility of a state every time it transitions to a new state, with the function: \n",
       "\n",
       "\n",
       "$U^{\\pi}(s) = U^{\\pi}(s) +$ $a$ $[R(s, \\pi (s), s') +$ $\\gamma$ $U^{\\pi}(s') - U^{\\pi}(s)]$ \n",
       "\n",
       "\n",
       "$a$ stands for the learning rate. The term multiplied with alpha is an error signal. If $a$ becomes a function that decreases with the number of state visits, then $U^{\\pi}(s)$ will converge to the correct value. \n",
       "\n",
       "\n",
       "TD learning does not converge as fast, but is simpler and require fewer computations, since it does not require a transition model. TD adjusts utility based on observed successors, while ADP adjusts utility based on all successors. TD makes one adjustment every observed transition, while ADP makes as many as it needs. \n",
       "\n",
       "\n",
       "We can exstend TD appraoch to generate psuedoexperiences - to make the utility estimates closer to ADP, at the cost of higher computation time. \n",
       "\n",
       "\n",
       "We can also make ADP more efficient in the other way. Prioritized sweeping heuristic prioritizes adjusting states whose possible successors have got a large adjustment in utility estimates. Speeds up approximate ADP and allows it to handle larger state spaces than ADP. \n",
       "\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.3 Active Reinforcement Learning\n",
       " Greedy agents suffer from the problem of no exploration. To improve, we use GILE, or \"Greedy in the limit of infinite exploration\". GILE scheme tries each action in eact states certain times to avoid missing optimal actions. \n",
       "\n",
       "\n",
       "GILE scheme 1: random action at each point. Problem is slow conversion. \n",
       "\n",
       "\n",
       "GILE scheme 2: give weight to unexplored actions. Include exploration function f(u, n) which increases with u and decreases with n. E.g. we can define f(u, n) to return max reward if an action is not yet explored for k times, and return u otherwise. \n",
       "\n",
       "\n",
       "We should use $U^+$ (optimistic estimate of utility/ expected reward-to-go) for updates instead of U, since it propagates benefits of exploration. \n",
       "\n",
       "\n",
       "Need to handle the problem of entering absorbing states and exercising irreversible actions. We should therefore prefer policy that works well across all models, even if the policy is not optimal for the model with max probability. Three appraoches:\n",
       "\n",
       "\n",
       "1) Bayesian Reinforcement Learning: assumes a prior probaility, than update with each observation optimal policy is one with highest expected utility. Requires priors to be able to tell danger. Problem can be transformed into an exploration POMDP where belief states are distribution over models, and can be solved before agent starts moving. \n",
       "\n",
       "\n",
       "2) Robust control theory: defines optimal policy as one that performs best in worst case. Essentially maximin rule - may result in conservative policy \n",
       "\n",
       "\n",
       "3) Use human expertise: have human teachers do a demo, or write down constraints. \n",
       "\n",
       "\n",
       "Temporal difference Q-learning: model-free learning with agents learning Q(s,a) rather than U(s). It does not need a transition model. Update function is: \n",
       "\n",
       "\n",
       "$Q(s,a) = Q(s,a) + alpha [R(s,a,s') +$ $/gamma$  $max_{a'} Q(s',a') - Q(s,a)$\n",
       "\n",
       "\n",
       "Q-learning agent cannot look into the future, so if rewards are sparse it would find difficulty. It is off-policy, since it assumes stopping the current policy and adopting the best policy according to estimates. \n",
       "\n",
       "\n",
       "SARSA (state,action,reward,state,action): similar to Q-learning, except that SARSA waits until an action is taken, then update based on the taken action. If exploration gives negative reward, SARSA will punish the action but Q-learning will not. It is on-policy, since it assumes continuing with current policy. SARSA is more appropriate when we do not have full control over what the agent does. \n",
       "\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n",
       " In large gamespaces, convergence of utilities will be slow. We can approximate utility/Q-function instead, using weighted linear combination of features. \n",
       "\n",
       "\n",
       "E.g. Approximating direct utility estimation: parametrize the utility, then find best fit parameters (we use online learning, so we update parameters after each trial). Define error has half of the squared difference between predicted and actual utility. Use the Widrow-Hoff rule / delta rule for online least-squares. \\ \n",
       "If true utility function is linear than learning will be easy; if not, we can have features be nonlinear functions of state variables, can continue to use linear function approximators. \n",
       "\n",
       "\n",
       "Approximate TD learning, similar idea: vary the parameters and optimize. Converge in the case of TD learning and when function approximator is linear in features. When using active learning and non-linear functions, there may be problems with infinite parameters. Also problem with castastrophic forgetting - learning too well that it forgot previous bad experience. Potential solution is experience relay - remind self of bad experience at regular intervals. \n",
       "\n",
       "\n",
       "Deep RL goes beyond linear function approximators for 2 reasons: 1) there may not be good linear functions, 2) we may not be able to come up with features. More complex function but same gradients as before. Deep RL can play games well but may not handle varied environments. \n",
       "\n",
       "\n",
       "Credit assignment problem: agent needs to figure out where it does wrong. Solution is reward shaping: use additional pseudorewards. Need to ensure that optimal policy is not changed with their addition: we can use potential function $+ \\gamma \\phi (s') - \\phi (s)$ to reflect the aspect we want to keep track of. \n",
       "\n",
       "\n",
       "Hierarchical RL: break up long sequences of action into small pieces. Builds partial programs that extend normal programming languages and provide general know-how. \n",
       "\n",
       "\n",
       "HRL based on joint state space: (physical state, machine state) pairs. Define choice state as state at which program is at a choice point. Transform HRL into a Markovian decision problem, with state = choice states, actions = choices, reward = rewards for physical transitions. For transition model, if c chooses physical action, then use physical model if computational action taken, then modifies computational states. Solving MDP gets use the optimal policy consistent with partial program. \n",
       "\n",
       "\n",
       "Two additional advantages of HRL: 1) more opportunity for reward shaping, 2) provides additive decomposition of overall utility function. \n",
       "\n",
       "\n",
       "## Module 9 Lecture Notes\n",
       " \n",
       "In MDPs agents are given reward function and transition function (we know probability distribution of the states). In reinforcement learning agents must learn these. Tradeoffs between exploration and exploitation \n",
       "\n",
       "\n",
       "Double bandits (come from slot machines): actions are play blue vs play red states are win and lose. No discount, 100 time steps, and both states have the same value. If probabilities are specified, offline planning can solve MDP - we don't actually need to play. Without the probabilties, we need to learn rather than plan. Learning can be much harder than solving a known MDP. \n",
       "\n",
       "\n",
       "In RL, agents choose an action, receive reward from environment. Agents' utility is defined by reward function. Agent must learn to maximize expected rewards. All learning will be based on observed samples of outcomes. \\ We still assuming MDP with set of states, set of actions, model T, and reward function R, with T and R hidden initially. \n",
       "\n",
       "\n",
       "Two different kinds of learning: model-based learning and model-free learning \n",
       "\n",
       "\n",
       "Model based learning: learn an approximate model of R and T, then use MDP algo. Count outcomes for each action, normalize to estimate transition function, then discover rewards and come up with maximum likelihood estimates. E.g. fix input policy, start at different points, observe episodes. \n",
       "\n",
       "\n",
       "Passive RL: Every time we take an action, we compare what is expected vs what happens. Does not keep track of T and R. Given fixed policy, we want to learn state values. Execute policy and learn from experience. Goal is to compute values for every state under the action $\\pi$. Direct evaluation - average the observed sample values. \n",
       "\n",
       "\n",
       "In each passive RL training episode, we run through the policy to the end (from each state) and compute the value of state accordingly. \n",
       "\n",
       "\n",
       "Benefits of direct evaluation: \n",
       "\n",
       "1. easy to understand\n",
       "\n",
       "2. does not need to understand R and L\n",
       "\n",
       "3. Eventually compute the correct average values\n",
       "\n",
       "\n",
       "Disadvantages: \n",
       "\n",
       "1. Waste information about state connections (learn value of state wihtout looking at other states), \n",
       "\n",
       "2. each state learned separately \n",
       "\n",
       "3. So, takes a long time to learn\n",
       "\n",
       "\n",
       "We can improve using policy evaulation, which use connections between states. However, normal policy evaluation uses weighted average, which requires knowing the weights. Instead, we can use sample-based policy evaluation: for each (state, action) pair, collect sample resulting stages, sum up sample values and divide by num of samples. However, that requires \"Crystal ball\" simulation of results. \n",
       "\n",
       "\n",
       "Temporal difference learning: update V(s) every time we experience a transition that starts from s. Likely outcomes will lead to updates more often. Every time we have sample, we update V(s) as: $V^{\\pi}(s) = (1 - alpha) V^{\\pi}(s) + alpha * sample$. Alpha is a value we set which says how much our observations impact vs old V. \n",
       "\n",
       "\n",
       "We end up calculating an exponential moving average, which weights current observation more heavily than new observations. Forgets about the past (but the past is not as important). $x_n = (x_n + (1 - alpha) x_{n-1} + (1 - alpha)^2 x_{n-2} + ... ) / (1 + (1 - alpha) + (1 - alpha) ^ 2 + ...)$ \n",
       "\n",
       "\n",
       "TD value learning is model-free, mimicking Bellman updates with moving averages. \n",
       "\n",
       "\n",
       "Problems with TD value learning: we do not have Q-values, so we can't turn values into a new policy. \n",
       "\n",
       "\n",
       "Q-value iteration: given Qk, calculate k+1 q-values for all q-states. Formula: \n",
       "\n",
       "\t $Q_{k+1}(s,a) = \\sum_{s'} T(s, a, s') [R(s, a, s') + \\gamma max_{a'} Q_k (s', a')]$ \n",
       "\n",
       "\n",
       "Acting RL/ Q-learning: sample based Q value iteration. When we receive our sample, we compare to our previous esimate, then update. \n",
       "\n",
       "\n",
       "$Q(s,a) = (1 - a) Q(s,a) + a * [sample]$ \n",
       "\n",
       "\n",
       "Q-learning converges to optimal policy - example of off-policy learning. Caveats: we need to explore enoguh, we will eventually have to make learning rate small enough but not decrease it too quickly. \n",
       "\n",
       "\n",
       "Choosing when to explore: \n",
       "\n",
       "1. Simplest way: random actions ($\\epsilon$-greedy). WIth small probability = $\\epsilon$ we act randomly else act on current policy. Problem: value of random exploration decreases as we go. Solution 1: lower $\\epsilon$ over time. Solution 2: explicitly write a function \n",
       "\n",
       "2. Exploration functinos: explore areas whose value has not yet established. Function $f(u,n) = u + k / n$, u = value estimate, n = visit count, k: constant bonus, returns an optimistic utility. \n",
       "\n",
       "\n",
       "Modified Q-update: $Q(s,a) = R(s,a,s') + \\gamma max_{a'} f(Q(s',a'), N(s',a'))$. vs just using old Q value in regular Q update. \n",
       "\n",
       "\n",
       "We can quantify effectiveness of exploration strategy using regret. Regret measures total mistake cost: difference between expected rewards and optimal rewards. Minimize regret = requires us to reach optimal policy as quickly as we can. Random exploration has higher regret. \n",
       "\n",
       "\n",
       "Approximate Q-learning: Some problems have many states, so we need to generalize states. Basic q-learning keeps track of all q values, approx learning learn about small number of states, then generalize learning to new states. Similar to evaluating chess states using features. Use a linear value function that assigns weights to each feature. Pro: experienced summed up in a few numbers. Disadvantage: features may not adequately predict values. \n",
       "\n",
       "\n",
       "E.g. $Q(s,a) = w_1 f_1(s,a) + w_2 f_2(s,a) + .... $\n",
       "\n",
       "\n",
       "\t Transition = (s, a, r, s')\n",
       "\n",
       "\t Difference/error term = $[r + \\gamma max_{a'} Q(s', a')] - Q(s,a)$\n",
       "\n",
       "\t Q(s,a) = Q(s,a) + $alpha$ [difference]  (Exact Q's) \n",
       "\n",
       "\t\t $w_i = w_i + alpha$[difference]$f_i (s,a)$  (Approximate Q's)\n",
       "\n",
       "\t\t (we update each weight instead of the entire Q state) \n",
       "\n",
       "\n",
       "We adjust the weights of active features as we go. If something bad happens, blame them feature that were on.\n",
       "\n",
       "\n",
       "## Module 9 Definitions\n",
       "* **Prioritized sweeping heuristic** - Adjustment to ADP which prioritizes states whose likely successors have just undergone large adjustment in utilities\n",
       "* **Adaptive dynamic programming** - Estimates transition model from counts, then apply policy evaluation \n",
       "* **Temporal difference learning** - Learning that uses difference in utilities between successive states\n",
       "* **Greedy in the limit of infinite exploration (glie)** - Agent must try each action in each state an unbounded number of times\n",
       "* **Robust control theory** - Defines optimal policy as one that gives best outcome in the worst case\n",
       "* **Exploration** - trying previously unknown states to gain information\n",
       "* **Exploitation** - exercising currently known best action to obtain rewards\n",
       "* **Markovian transitions** - transitions depends only on the current state but not previous states.\n",
       "* **Environment history** - a sequence of states/actions on which the utility function is dependent\n",
       "* **Policy** - instruction that tells agents what kind of action they should carry out under each state\n",
       "* **Optimal policy** - Policy that yields the highest expected utility\n",
       "* **Finite horizon** - a fixed time point after which the game is done\n",
       "* **Nonstationary policy** - a policy that is dependent on time\n",
       "* **Infinite horizon** - games with an infinite state sequence (may still contain terminal state(s)\n",
       "* **Additive discounted rewards** - utility function taht defines utility as sum of rewards weighted by a discount factor\n",
       "* **Stationaray preferences** - if we prefer one future to another starting tomorrow, we would prefer it even if it starts today\n",
       "* **Proper policy** - a policy where an agent is guaranteed to reach a terminal state\n",
       "* **Bellman equation** - Utility of one state = expected reward for next transition + discounted utility of the next state. \n",
       "* **N-armed bandit** - slot machine with n levers\n",
       "* **Absorbing state** - state where no actions from agent has any effect and no rewards will be received\n",
       "* **Reward to-go** - expected reward starting from a certain state\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "\n",
    "module9_sections = [\n",
    "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
    "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
    "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
    "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
    "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
    "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
    "]\n",
    "\n",
    "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
    "display(Markdown(module9_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "46MQLR-Gx8dZ"
   },
   "source": [
    "#Module 10 - Probabilities and Language Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "BN0b1eVYx-nd"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
    "chapter12_1 = \"\"\" For real world problems, we often cannot generate a contingency \\\n",
    "plan for every possible state. To do so will result in (1) too many contingenicies \\\n",
    "considered, (2) belief-state with many unlikely possibilities, and (3) no way \\\n",
    "to compare value of plans when results are not guaranteed. Approach also suffer from \\\n",
    "qualification problem (we do not know what to exclude) \\n\n",
    "\n",
    "Rational decision will depend on both importance of a goal and the probability at which \\\n",
    "it will be achieved. \\n\n",
    "\n",
    "Problems with using logical rules to model the real world: 1) laziness (too much \\\n",
    "work to list all possibilities), 2) theoretical ignorance (do not have complete theory \\\n",
    "for the domain), 3) practical ignorance (have not tested knowledge under domain) \\n\n",
    "\n",
    "To deal with degrees of belief we use probability theory. Logic and probability theory \\\n",
    "share the same ontological commitment (that the world is made of facts that are true \\\n",
    "or false in particular case), but differnet epistemological commitment. \\n\n",
    "\n",
    "Decision theory = probability theory + utility theory. Fundamental idea is that a rational \\\n",
    "action chooses actions with teh highest expected utility, averaged over all possible \\\n",
    "outcomes of the action. (MEU principle)\n",
    "\"\"\"\n",
    "\n",
    "chapter12_2 = \"\"\" Set of all possible world = sample space. Possible worlds are \\\n",
    "mutually exclusive and collectively exhaustive. Probability model is mapping between \\\n",
    "possible world and probabilities. Two basic axioms: prob between 0 and 1, and all add \\\n",
    "up to 1. \\n\n",
    "\n",
    "Set of possible worlds is called outcomes (vs in logic, set of worlds = a proposition). \\n\n",
    "\n",
    "Unconditional = prior probabilities = degrees of belief without extra info. \\n\n",
    "\n",
    "Conditional = posterior probabilties = degrees of belief given evidence. \\n\n",
    "\n",
    "We use (in this chapter) a factored representation to describe sets of possible \\\n",
    "worlds. \\n\n",
    "\n",
    "Random variable = mapping from possible worlds to a range (i.e. set of possible values \\\n",
    "that it can take on). Bernoulli distribution has range {0, 1}. \\n\n",
    "\n",
    "We can use bold P(RV), a vector of values, to represent probability of all values \\\n",
    "of a Random variable.  This will define a probability distribution for the RV. This \\\n",
    "will work for categorical distribution, where range is finite and discrete. \\n\n",
    "\n",
    "For continuous variables , we define probability density function. Probabiltiy densities \\\n",
    "are not probabilites: they are measured with a unit. PDFs will show us how likely it is \\\n",
    "fall a value to fall close to a the value we enter. \\n\n",
    "\n",
    "Joint probability distribution: table specifying distribution of multiple variables. \\ \n",
    "Full joint probability distribution: joint distribution of all random variables. \\n\n",
    "\n",
    "From the two basic axioms we can derive: \\n\n",
    "1) Probability of negation is 1 - probability \\n\n",
    "2) Inclusion-exclusion principle: P(a or b) = P(a) + P(b) - P(a and b)\\n\n",
    "\n",
    "Kolmogorov's axioms = inclusion-exclusion and two basic axioms \\n\n",
    "\n",
    "Debates about whether our degrees of belief really need to obey axioms. \\\n",
    "De Finett's theorem: if agent's belief violates axioms of probabilty then \\\n",
    "they can be taken advantage of by another agent. \\n\n",
    "\n",
    "Objection: what if one refuses to bet, rejoinder: one unavoidably participate \\\n",
    "in games like those. \n",
    "\"\"\"\n",
    "\n",
    "chapter12_3 = \"\"\" Probabilistic inference is the process of computing posterior \\\n",
    "probabilities for query propositions, given observed evidence \\n\n",
    "\n",
    "Marginalization/ summing out is the process of taking out variables by summing up \\\n",
    "all their possible values. \\n\n",
    "\n",
    "Conditioning rule: $P(y) = \\sum_z P(y|z) P(z)$ \\n\n",
    "\n",
    "Combining these rules yield the general inference function: \\n\n",
    "$P(X|e) = a P(X,e) = a \\sum_y P(X,e,y)$ where E = list of evidence variables, e = \\\n",
    "observed values, Y = remaining unobserved variables. \\n\n",
    "\n",
    "Using this approach with full joint distribution, we can answer prob queries for \\\n",
    "discrete variables. However, it takes input table of size $O(2^n)$ and time of $O(2^n)$ .\n",
    "\"\"\"\n",
    "\n",
    "chapter12_4 = \"\"\"\n",
    "If a and b are independent then: \\n\n",
    "P(a|b) = P(a) and P(b|a) = P(b) and P(a and b) = P(a)P(b)\\n\n",
    "\n",
    "Independence can be used to deconstruct a joint distribution. E.g. distribution of \\\n",
    "coinflip can be represented as product of n single-variable distribution instead of a \\\n",
    "table with $2^n$ entries. \n",
    "\"\"\"\n",
    "\n",
    "chapter12_5 = \"\"\" Bayes rule: $P(b|a) = P(a|b)P(b) / P(a)$\\n\n",
    "\n",
    "Alternatively, P(cause|effect) = P(effect|cause)P(cause)/P(effect)\\n\n",
    "\n",
    "P(effect|cause): quantifies relationship in causal direction. \\\n",
    "P(cause|effect) describes diagnostic direction. Diagnostic knowledge \\\n",
    "is often more fragile than causal knowledge. \\n\n",
    "\n",
    "Conditional independence: P(X,Y|Z) = P(X|Z)P(Y|Z)\\n\n",
    "\n",
    "We can also understand conditional independence as some common cause \\\n",
    "seperating two possible effects. It can be used to reduce representation \\\n",
    "from O(2^n) to O(n) and hence help prob systems scale up.\n",
    " \"\"\"\n",
    "\n",
    "chapter12_6 = \"\"\" Naive bayes model uses equation: \\\n",
    "P(Cause, effect1, effect2,...) = P(cause) product of (P(effect|cause)).\\n\n",
    "\n",
    "Run time of calculaiton is linear in number of observed effects \\\n",
    "and does not depend on unobserved ones. \\n\n",
    "\n",
    "Naive Bayes assumes that words occur independently in documents.  \"\"\"\n",
    "\n",
    "chapter12_7 = \"\"\" We can use naive bayes and conditional independence \\\n",
    "to calculate probabilities for unobserved aspects of Wumpus world.  \"\"\"\n",
    "\n",
    "\n",
    "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
    "SLP_chapter3_1 = \"\"\"Chain rule: $P(X_1, X_2, .... X_n) = \\\n",
    "P(X_1)P(X_2|X_1)P(X_3|X_{1:2})...P(X_n|X_{1:n-1})$.\\n\n",
    "\n",
    "One way to come up with MLE is using relative frequency: Counts of n-gram in a corpus, \\\n",
    "divided/normalized by sum of n-grams that share the same first word(s). \\n\n",
    "\n",
    "We can use log probabilities to prevent numerical underflow. p1p2p3p4 = exp(log p1\\\n",
    "+ log p2 + log p3 + log p4).\n",
    "  \"\"\"\n",
    "SLP_chapter3_2 = \"\"\" Extrinsic evaluation: apply LM in an application and see \\\n",
    "how much it improves. Intrinsic evaluation: evaluates without application. \\n\n",
    "\n",
    "For intrinsic evaluation we need a test set, a training set, and a holdout set. \\\n",
    "We want to avoid training on the test set, which results in inaccurate perplexity. \\\n",
    "We may also want to introduce a development set / deveset which is a fresh \\\n",
    "test set that is never seen. We want a test size with a size that gives us \\\n",
    "enough statistical power to measure a stat. significant difference between 2 models. \\\n",
    "An example is 80% training, 10% dev and 10% test. \\n\n",
    "\n",
    "Minimizing perplexity = maximizing test set probability given the language \\\n",
    "model. Perplexity is also same as weighted average branching factor of a language. \\\n",
    "With branching factor of language = num of words that can follow any word. \\\n",
    "If a word is very frequent perplexity will be lower. \\n\n",
    "\n",
    "Using n-grams with higher n's give us more info and hence lower perplexity. \\\n",
    "If the n-gram model is constructed with knowledge of test set (which should be \\\n",
    "forbidden), the perplexity will be artificially low. \n",
    " \"\"\"\n",
    "SLP_chapter3_3 = \"\"\" We want to ensure that training and test set is in similar \\\n",
    "genres, and that the training data has appropriate dialect and variety. \\n\n",
    "\n",
    "Problem of sparsity: some n-grams may have never appeared in training set (zeros). \\\n",
    "Two resulting problems: we may underestimate many probabilities, and we will estimate \\\n",
    "the prob of test set as 0. \\n\n",
    "\n",
    "In more extreme case a word can never been seen (unknown words/ out of vocabulary (OOV) \\\n",
    "words). % of OOV words = OOV rate. \\n\n",
    "\n",
    "Two ways to handle unknown word <UNK>: (1) turn problem into closed vocab. \\\n",
    "First choose a fixed vocab, then convert every OOV word to <UNK>, then estimate prob \\\n",
    "of <UNK> as a regular word. (2) Create an implicit vocab, by replacing words in training \\\n",
    "date by <UNK> based on their frequency. E.g. replace any word that appears fewer \\\n",
    "than k times by <UNK>, or choose top 50000 words and rest as <UNK>. \\n\n",
    " \"\"\"\n",
    "SLP_chapter3_4 = \"\"\" Smoothing/ discounting = taking away probability mass from \\\n",
    "high freq events and adding to unknown events. \\n\n",
    "\n",
    "Laplace smoothing: adding one to all bigram counts. $P_{Laplace}(w_i) = \\\n",
    "(c_i + 1) / (N + V)$ Adjusted count $c* = (c_i + 1)$ $N / (N+V)$ can be \\\n",
    "turned into probability by normalizing by N. \\n\n",
    "\n",
    "We can also see smoothing as discounting. Laplace smoothing = applying a discount \\\n",
    "$d_c = (c*)/c$.\\n\n",
    "\n",
    "Downside of Lapalce smoothign: sharp changes in probabilities, since a lot of mass \\\n",
    "is moved to the zeros. \\n\n",
    "\n",
    "Alternative: add-k smoothing, add a fractional count k. We can choose \\\n",
    "k by optimizing on a devset. \\n\n",
    "\n",
    "Backoff: using n-gram if evidence is sufficient, else use n-1-gram, and n-2 \\\n",
    "and so on. We discount higher order n-grams to save some probability \\\n",
    "mass for lower order n-grams. Backoff with discount is also called \\\n",
    "Katz backoff. Katz backoff can be combined with a smoothing method \\\n",
    "called Good-Turing to become Good Turing backoff. \\n\n",
    "\n",
    "Interpolation: mixing probability estimates from all \\\n",
    "n-gram estimators. Assign a $\\lambda$ weight to each n-gram, and select \\\n",
    "lambda by learning from held-out corpus (additiona corpus to learn \\\n",
    "hyperparameters). We can also use EM algorithm to find $\\lambda$. \\n\n",
    "\"\"\"\n",
    "SLP_chapter3_5 = \"\"\" Absolute discounting: subtract a fixed discount d \\\n",
    "from all counts. High counts will not be affected much while small counts \\\n",
    "will be modified more. $P_{AbsoluteDiscounting}(w_i | w_{i-1})$ = $(C(w_{i-1}w_i - d))/(\\sum_v C(w_{i-1}v)) + \\lambda (w_{i-1}) P(w_i)$ \\n\n",
    " \n",
    "Kneser-Ney smoothing: instead of getting unigram model, we seek the probability \\\n",
    "of a word appearing as a novel continuation. $P_{continuation} (w) = |{v: C(vw) > 0}|$ \\\n",
    "$ \\sum_{w'} |{v: C(vw') > 0}|$. Frequent word appearing in only a few contexts \\\n",
    "will have low continuation probability. \\n\n",
    "\n",
    "P_{KN} (w_i | w_{i-1}) = max(C(w_{i-1}w_i) - d, 0) / C(w_{i-1}) + \\lambda (w_{i-1}) P_{continuation} \\n \\n\n",
    "\n",
    "Modified Kneser-Ney: use three different discounts for n-grams with counts of 1, 2, and \\\n",
    "3+ \"\"\"\n",
    "\n",
    "SLP_chapter3_7 = \"\"\" Given random variable X, entropy is: \\n\n",
    "\n",
    "H(X) = - \\sum_{x \\in range(X)} p(x) log_2 p(x) \\n\n",
    "\n",
    "Entropy calculated using log base 2 is measured in bits. \\n\n",
    "\n",
    "Entropy = lower bound on number of bits that is needed to encode a decision \\\n",
    "given an optimal encoding scheme. \\n\n",
    "\n",
    "Entropy for sequence (of words) = replace p(x) with $p(W^n_1)$. \\n\n",
    "\n",
    "Entropy rate/ per-word entropy = entropy of sequence / number of words. \\n\n",
    "\n",
    "To calculate entropy for a language that is regular, we can calculate \\\n",
    "the entropy of a single sequence that is long enough. By Shannon-McMillan-Breiman theorem. \\n\n",
    "\n",
    "A stochastic process is stationary if probability it assigns to a \\\n",
    "sequence are invariant with respect to shifts in the time index. Natural \\\n",
    "language is not stationary, since there can be long-distance depedency. \\\n",
    "Markov models are stationary, and are therefore only approx of natural language \\n\n",
    "\n",
    "Cross-entropy: defined on m, a model of probability p that we do not know. \\\n",
    "H(p, m) = $lim_{n -> \\inf} - 1 / n \\sum_{W \\in L} p(w1, ..., wn) log m (w1, ..., wn)$\\n\n",
    "For stationary process it can be calculated as H(p, m) = $lim_{n -> \\inf} - 1 / n  log m (w1, ..., wn)$.\\n\n",
    "\n",
    "Cross entropy H(p,m) is upper bound on entropy H(p) for any model m. The difference \\\n",
    "between H(p,m) and H(p) is the measure of the accuracy of a model. \\n\n",
    "\n",
    "Perplexity of a model P on sequence W can be defined as the exponent of cross-entropy. \\\n",
    "Perplexity(W) = $2 ^ {H(W)}$\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module10_lecture_notes = \"\"\" Agents often know (observed variables) evidence about the state of the world \\\n",
    "but need to reason about unobserved variables (states) of the world. A model is how the \\\n",
    "known variables related to the unknown. Probabilistic reasoning is the underlying process. \\n\n",
    "\n",
    "Probability is about how likely a possible world is. Possible worlds = sample space, and \\\n",
    "they are mutually exclusive and collectively exhaustive. Fully specified probability model \\\n",
    "associates probability P(w) with each possible world w. \\n\n",
    "\n",
    "Random variable is an aspect of the world that may be uncertain. Denoted with capital letters. \\\n",
    "\n",
    "Two axioms:\\n\n",
    "1) Probability of any possible world is between 0 and 1\\n\n",
    "2) Total probability of all possible worlds is 1\\n\n",
    "\n",
    "Random variables have distributions. \\\n",
    "Distribution = table of probabilities of values, indexed by outcomes. \\\n",
    "If all domain entries are unique we can use shorthand: P(hot) = P(T=hot) \\n\n",
    "\n",
    "Joint distribution specifies probability for each assignment to a set of RVs. \\\n",
    "Size of distribution with n variables and domain size d = $d^n$. \\n\n",
    "\n",
    "Probabilistic model: joint distribution over a set of R.V.s. Assignments are called outcomes, \\\n",
    "probabilties normalized (sum to 1.0), and ideally only certain variables directly interact. \\\n",
    "(i.e. many will be independent). \\n\n",
    "\n",
    "Event = set E of outcomes. P(E) = P(x1) + P(x2) + ... for all x in E. \\n\n",
    "\n",
    "In many cases the events we care about are partial assignments. We can use \\\n",
    "marginal distributinons - subtables with variables eliminated. Process of marginalization \\\n",
    "= combining eliminated rows by adding. \\n\n",
    "\n",
    "Definition of conditional probability: P(a|b) = P(a,b) / P(b). A given B. Defined in terms of \\\n",
    "joint probability. \\n\n",
    "\n",
    "Probabilistic inference is to use known probablities to compute desired probabilities. \\\n",
    "Update beliefs based on evidence. \\n\n",
    "\n",
    "Inference by enumeration: We have (1) evidence variables, (2) Query variable, and (3) hidden variables. \\\n",
    "Should sum up to all variables. We want P(Q | e1e2....ek). Steps: (1) select entries consistent with evidence, (2) sum \\\n",
    "H to get P(Q, e1e2...) = sum over h1h2h3... of P(Q and h1h2h3 and e1e2e3...), (3) \\\n",
    "normalize by dividing with $\\sum_q P(Q,e1e2...ek)$. \\n\n",
    "\n",
    "To construct joint probability table, we use product rule \\\n",
    "P(y)P(x|y) = P(x and y). Generalize to chain rule: P(x1, x2, x3) \\\n",
    "= P(x1) P(x2|x1) P(x3 | x1,x2). \\n\n",
    "\n",
    "Two ways to factor joint distribution over two vars: \\n\n",
    "P(x,y) = P(x|y)P(y) = P(y|x)P(x)\\n\n",
    "\n",
    "Bayes rule: $P(x|y) = \\frac{P(y|x)}{P(y)} P(x)$ Allows building one conditional from the reverse. \\n\n",
    "\n",
    "Probabilistic language model: take each word position as an RV \\\n",
    "estimate probability distribution to the next RV. Goal is to compute prob of \\\n",
    "a sentence/ a sequence of words, or compute probability of upcoming word. A \\\n",
    "model that computes either P(W) or $P(w_n | w_1, w_2...)$ is a language model or \\\n",
    "a grammar. \\n\n",
    "\n",
    "$P(w1,w2,...wn) = \\prod_i P(wi | w1,w2,w_{i-1})$. \\n\n",
    "\n",
    "Maximum likelihood estimate: count and divide. Problem is that some sentences \\\n",
    "would never appear. We will not get enough data. \\n\n",
    "\n",
    "Markov assumption: we can use the current state alone to predict the next state. \\\n",
    "In language models, this mean the next word will only depend on the last k words. \\n\n",
    "\n",
    "unigram = no history used, bigram = 1 word, trigram = 2 words, 4-gram = 3 words as history. \\n\n",
    "\n",
    "Markov built LM by counting up 20k letter in Onegin in 1913. 1948 Shannon approximates \\\n",
    "English by n-grams. 1956 Chomsky paper suggests that finite state Markov Model \\\n",
    "is not a good model for human languages. Fred Jelinek in 1980 uses \\\n",
    "n-gram for ASR, and in 1993 develops statistical machine translation. \\n\n",
    "\n",
    "Formula for machine translation: $argmax_e p(e|f) = p(e) p(f|e)$\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "\n",
    "# Each definition should be at most 25 words long.\n",
    "module10_definitions = {\n",
    "    \"Product rule\" : \"P(a and b) = P(a|b) P(b)\",\n",
    "    \"normalization constant\" : \"a constant that, when applied, ensures a \\\n",
    "    distribution will add up to 1\",\n",
    "    \"Language models\" : \"Models that assign probabilities to sequences \\\n",
    "    of words\",\n",
    "    \"Markov assumption\" : \"Prob of a word depends only on the previous word\", \n",
    "    \"Maximum likelihood estimation (MLE)\" : \"Estimated paremeters of a probability \\\n",
    "    distribution, s.t. probability of observed data is the highest\",\n",
    "    \"Perplexity\" : \"Inverse probabilty of test set, normalized by number of words\",\n",
    "    \"Closed vocabulary system\" : \"test set can only contain words from a lexicon\",\n",
    "    \"Open vocab system\" : \"system in which we model unknown words as <UNK>\",\n",
    "\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "lvtN3p7ET3Tx",
    "outputId": "e0334dbc-4606-405a-ad99-579b9a5b1084"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 10: Probabilies and Language Models\n",
       "## AIMA Chapter 12.1 Acting under Uncertainty\n",
       " For real world problems, we often cannot generate a contingency plan for every possible state. To do so will result in (1) too many contingenicies considered, (2) belief-state with many unlikely possibilities, and (3) no way to compare value of plans when results are not guaranteed. Approach also suffer from qualification problem (we do not know what to exclude) \n",
       "\n",
       "\n",
       "Rational decision will depend on both importance of a goal and the probability at which it will be achieved. \n",
       "\n",
       "\n",
       "Problems with using logical rules to model the real world: 1) laziness (too much work to list all possibilities), 2) theoretical ignorance (do not have complete theory for the domain), 3) practical ignorance (have not tested knowledge under domain) \n",
       "\n",
       "\n",
       "To deal with degrees of belief we use probability theory. Logic and probability theory share the same ontological commitment (that the world is made of facts that are true or false in particular case), but differnet epistemological commitment. \n",
       "\n",
       "\n",
       "Decision theory = probability theory + utility theory. Fundamental idea is that a rational action chooses actions with teh highest expected utility, averaged over all possible outcomes of the action. (MEU principle)\n",
       "\n",
       "\n",
       "## AIMA Chapter 12.2 Basic Probability Notation\n",
       " Set of all possible world = sample space. Possible worlds are mutually exclusive and collectively exhaustive. Probability model is mapping between possible world and probabilities. Two basic axioms: prob between 0 and 1, and all add up to 1. \n",
       "\n",
       "\n",
       "Set of possible worlds is called outcomes (vs in logic, set of worlds = a proposition). \n",
       "\n",
       "\n",
       "Unconditional = prior probabilities = degrees of belief without extra info. \n",
       "\n",
       "\n",
       "Conditional = posterior probabilties = degrees of belief given evidence. \n",
       "\n",
       "\n",
       "We use (in this chapter) a factored representation to describe sets of possible worlds. \n",
       "\n",
       "\n",
       "Random variable = mapping from possible worlds to a range (i.e. set of possible values that it can take on). Bernoulli distribution has range {0, 1}. \n",
       "\n",
       "\n",
       "We can use bold P(RV), a vector of values, to represent probability of all values of a Random variable.  This will define a probability distribution for the RV. This will work for categorical distribution, where range is finite and discrete. \n",
       "\n",
       "\n",
       "For continuous variables , we define probability density function. Probabiltiy densities are not probabilites: they are measured with a unit. PDFs will show us how likely it is fall a value to fall close to a the value we enter. \n",
       "\n",
       "\n",
       "Joint probability distribution: table specifying distribution of multiple variables. \\ \n",
       "Full joint probability distribution: joint distribution of all random variables. \n",
       "\n",
       "\n",
       "From the two basic axioms we can derive: \n",
       "\n",
       "1) Probability of negation is 1 - probability \n",
       "\n",
       "2) Inclusion-exclusion principle: P(a or b) = P(a) + P(b) - P(a and b)\n",
       "\n",
       "\n",
       "Kolmogorov's axioms = inclusion-exclusion and two basic axioms \n",
       "\n",
       "\n",
       "Debates about whether our degrees of belief really need to obey axioms. De Finett's theorem: if agent's belief violates axioms of probabilty then they can be taken advantage of by another agent. \n",
       "\n",
       "\n",
       "Objection: what if one refuses to bet, rejoinder: one unavoidably participate in games like those. \n",
       "\n",
       "\n",
       "## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n",
       " Probabilistic inference is the process of computing posterior probabilities for query propositions, given observed evidence \n",
       "\n",
       "\n",
       "Marginalization/ summing out is the process of taking out variables by summing up all their possible values. \n",
       "\n",
       "\n",
       "Conditioning rule: $P(y) = \\sum_z P(y|z) P(z)$ \n",
       "\n",
       "\n",
       "Combining these rules yield the general inference function: \n",
       "\n",
       "$P(X|e) = a P(X,e) = a \\sum_y P(X,e,y)$ where E = list of evidence variables, e = observed values, Y = remaining unobserved variables. \n",
       "\n",
       "\n",
       "Using this approach with full joint distribution, we can answer prob queries for discrete variables. However, it takes input table of size $O(2^n)$ and time of $O(2^n)$ .\n",
       "\n",
       "\n",
       "## AIMA Chapter 12.4 Independence\n",
       "\n",
       "If a and b are independent then: \n",
       "\n",
       "P(a|b) = P(a) and P(b|a) = P(b) and P(a and b) = P(a)P(b)\n",
       "\n",
       "\n",
       "Independence can be used to deconstruct a joint distribution. E.g. distribution of coinflip can be represented as product of n single-variable distribution instead of a table with $2^n$ entries. \n",
       "\n",
       "\n",
       "## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n",
       " Bayes rule: $P(b|a) = P(a|b)P(b) / P(a)$\n",
       "\n",
       "\n",
       "Alternatively, P(cause|effect) = P(effect|cause)P(cause)/P(effect)\n",
       "\n",
       "\n",
       "P(effect|cause): quantifies relationship in causal direction. P(cause|effect) describes diagnostic direction. Diagnostic knowledge is often more fragile than causal knowledge. \n",
       "\n",
       "\n",
       "Conditional independence: P(X,Y|Z) = P(X|Z)P(Y|Z)\n",
       "\n",
       "\n",
       "We can also understand conditional independence as some common cause seperating two possible effects. It can be used to reduce representation from O(2^n) to O(n) and hence help prob systems scale up.\n",
       " \n",
       "\n",
       "## AIMA Chapter 12.6 Naive Bayes Models\n",
       " Naive bayes model uses equation: P(Cause, effect1, effect2,...) = P(cause) product of (P(effect|cause)).\n",
       "\n",
       "\n",
       "Run time of calculaiton is linear in number of observed effects and does not depend on unobserved ones. \n",
       "\n",
       "\n",
       "Naive Bayes assumes that words occur independently in documents.  \n",
       "\n",
       "## AIMA Chapter 12.7 The Wumpus World Revisited\n",
       " We can use naive bayes and conditional independence to calculate probabilities for unobserved aspects of Wumpus world.  \n",
       "\n",
       "## SLP Chapter 3.1 N-Grams\n",
       "Chain rule: $P(X_1, X_2, .... X_n) = P(X_1)P(X_2|X_1)P(X_3|X_{1:2})...P(X_n|X_{1:n-1})$.\n",
       "\n",
       "\n",
       "One way to come up with MLE is using relative frequency: Counts of n-gram in a corpus, divided/normalized by sum of n-grams that share the same first word(s). \n",
       "\n",
       "\n",
       "We can use log probabilities to prevent numerical underflow. p1p2p3p4 = exp(log p1+ log p2 + log p3 + log p4).\n",
       "  \n",
       "\n",
       "## SLP Chapter 3.2 Evaluating Language Models\n",
       " Extrinsic evaluation: apply LM in an application and see how much it improves. Intrinsic evaluation: evaluates without application. \n",
       "\n",
       "\n",
       "For intrinsic evaluation we need a test set, a training set, and a holdout set. We want to avoid training on the test set, which results in inaccurate perplexity. We may also want to introduce a development set / deveset which is a fresh test set that is never seen. We want a test size with a size that gives us enough statistical power to measure a stat. significant difference between 2 models. An example is 80% training, 10% dev and 10% test. \n",
       "\n",
       "\n",
       "Minimizing perplexity = maximizing test set probability given the language model. Perplexity is also same as weighted average branching factor of a language. With branching factor of language = num of words that can follow any word. If a word is very frequent perplexity will be lower. \n",
       "\n",
       "\n",
       "Using n-grams with higher n's give us more info and hence lower perplexity. If the n-gram model is constructed with knowledge of test set (which should be forbidden), the perplexity will be artificially low. \n",
       " \n",
       "\n",
       "## SLP Chapter 3.3 Sampling sentences from a language model\n",
       " We want to ensure that training and test set is in similar genres, and that the training data has appropriate dialect and variety. \n",
       "\n",
       "\n",
       "Problem of sparsity: some n-grams may have never appeared in training set (zeros). Two resulting problems: we may underestimate many probabilities, and we will estimate the prob of test set as 0. \n",
       "\n",
       "\n",
       "In more extreme case a word can never been seen (unknown words/ out of vocabulary (OOV) words). % of OOV words = OOV rate. \n",
       "\n",
       "\n",
       "Two ways to handle unknown word <UNK>: (1) turn problem into closed vocab. First choose a fixed vocab, then convert every OOV word to <UNK>, then estimate prob of <UNK> as a regular word. (2) Create an implicit vocab, by replacing words in training date by <UNK> based on their frequency. E.g. replace any word that appears fewer than k times by <UNK>, or choose top 50000 words and rest as <UNK>. \n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 3.4 Generalization and Zeros\n",
       " Smoothing/ discounting = taking away probability mass from high freq events and adding to unknown events. \n",
       "\n",
       "\n",
       "Laplace smoothing: adding one to all bigram counts. $P_{Laplace}(w_i) = (c_i + 1) / (N + V)$ Adjusted count $c* = (c_i + 1)$ $N / (N+V)$ can be turned into probability by normalizing by N. \n",
       "\n",
       "\n",
       "We can also see smoothing as discounting. Laplace smoothing = applying a discount $d_c = (c*)/c$.\n",
       "\n",
       "\n",
       "Downside of Lapalce smoothign: sharp changes in probabilities, since a lot of mass is moved to the zeros. \n",
       "\n",
       "\n",
       "Alternative: add-k smoothing, add a fractional count k. We can choose k by optimizing on a devset. \n",
       "\n",
       "\n",
       "Backoff: using n-gram if evidence is sufficient, else use n-1-gram, and n-2 and so on. We discount higher order n-grams to save some probability mass for lower order n-grams. Backoff with discount is also called Katz backoff. Katz backoff can be combined with a smoothing method called Good-Turing to become Good Turing backoff. \n",
       "\n",
       "\n",
       "Interpolation: mixing probability estimates from all n-gram estimators. Assign a $\\lambda$ weight to each n-gram, and select lambda by learning from held-out corpus (additiona corpus to learn hyperparameters). We can also use EM algorithm to find $\\lambda$. \n",
       "\n",
       "\n",
       "\n",
       "## SLP Chapter 3.5 Smoothing\n",
       " Absolute discounting: subtract a fixed discount d from all counts. High counts will not be affected much while small counts will be modified more. $P_{AbsoluteDiscounting}(w_i | w_{i-1})$ = $(C(w_{i-1}w_i - d))/(\\sum_v C(w_{i-1}v)) + \\lambda (w_{i-1}) P(w_i)$ \n",
       "\n",
       " \n",
       "Kneser-Ney smoothing: instead of getting unigram model, we seek the probability of a word appearing as a novel continuation. $P_{continuation} (w) = |{v: C(vw) > 0}|$ $ \\sum_{w'} |{v: C(vw') > 0}|$. Frequent word appearing in only a few contexts will have low continuation probability. \n",
       "\n",
       "\n",
       "P_{KN} (w_i | w_{i-1}) = max(C(w_{i-1}w_i) - d, 0) / C(w_{i-1}) + \\lambda (w_{i-1}) P_{continuation} \n",
       " \n",
       "\n",
       "\n",
       "Modified Kneser-Ney: use three different discounts for n-grams with counts of 1, 2, and 3+ \n",
       "\n",
       "## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n",
       " Given random variable X, entropy is: \n",
       "\n",
       "\n",
       "H(X) = - \\sum_{x \\in range(X)} p(x) log_2 p(x) \n",
       "\n",
       "\n",
       "Entropy calculated using log base 2 is measured in bits. \n",
       "\n",
       "\n",
       "Entropy = lower bound on number of bits that is needed to encode a decision given an optimal encoding scheme. \n",
       "\n",
       "\n",
       "Entropy for sequence (of words) = replace p(x) with $p(W^n_1)$. \n",
       "\n",
       "\n",
       "Entropy rate/ per-word entropy = entropy of sequence / number of words. \n",
       "\n",
       "\n",
       "To calculate entropy for a language that is regular, we can calculate the entropy of a single sequence that is long enough. By Shannon-McMillan-Breiman theorem. \n",
       "\n",
       "\n",
       "A stochastic process is stationary if probability it assigns to a sequence are invariant with respect to shifts in the time index. Natural language is not stationary, since there can be long-distance depedency. Markov models are stationary, and are therefore only approx of natural language \n",
       "\n",
       "\n",
       "Cross-entropy: defined on m, a model of probability p that we do not know. H(p, m) = $lim_{n -> \\inf} - 1 / n \\sum_{W \\in L} p(w1, ..., wn) log m (w1, ..., wn)$\n",
       "\n",
       "For stationary process it can be calculated as H(p, m) = $lim_{n -> \\inf} - 1 / n  log m (w1, ..., wn)$.\n",
       "\n",
       "\n",
       "Cross entropy H(p,m) is upper bound on entropy H(p) for any model m. The difference between H(p,m) and H(p) is the measure of the accuracy of a model. \n",
       "\n",
       "\n",
       "Perplexity of a model P on sequence W can be defined as the exponent of cross-entropy. Perplexity(W) = $2 ^ {H(W)}$\n",
       "\n",
       "\n",
       "## Module 10 Lecture Notes\n",
       " Agents often know (observed variables) evidence about the state of the world but need to reason about unobserved variables (states) of the world. A model is how the known variables related to the unknown. Probabilistic reasoning is the underlying process. \n",
       "\n",
       "\n",
       "Probability is about how likely a possible world is. Possible worlds = sample space, and they are mutually exclusive and collectively exhaustive. Fully specified probability model associates probability P(w) with each possible world w. \n",
       "\n",
       "\n",
       "Random variable is an aspect of the world that may be uncertain. Denoted with capital letters. \n",
       "Two axioms:\n",
       "\n",
       "1) Probability of any possible world is between 0 and 1\n",
       "\n",
       "2) Total probability of all possible worlds is 1\n",
       "\n",
       "\n",
       "Random variables have distributions. Distribution = table of probabilities of values, indexed by outcomes. If all domain entries are unique we can use shorthand: P(hot) = P(T=hot) \n",
       "\n",
       "\n",
       "Joint distribution specifies probability for each assignment to a set of RVs. Size of distribution with n variables and domain size d = $d^n$. \n",
       "\n",
       "\n",
       "Probabilistic model: joint distribution over a set of R.V.s. Assignments are called outcomes, probabilties normalized (sum to 1.0), and ideally only certain variables directly interact. (i.e. many will be independent). \n",
       "\n",
       "\n",
       "Event = set E of outcomes. P(E) = P(x1) + P(x2) + ... for all x in E. \n",
       "\n",
       "\n",
       "In many cases the events we care about are partial assignments. We can use marginal distributinons - subtables with variables eliminated. Process of marginalization = combining eliminated rows by adding. \n",
       "\n",
       "\n",
       "Definition of conditional probability: P(a|b) = P(a,b) / P(b). A given B. Defined in terms of joint probability. \n",
       "\n",
       "\n",
       "Probabilistic inference is to use known probablities to compute desired probabilities. Update beliefs based on evidence. \n",
       "\n",
       "\n",
       "Inference by enumeration: We have (1) evidence variables, (2) Query variable, and (3) hidden variables. Should sum up to all variables. We want P(Q | e1e2....ek). Steps: (1) select entries consistent with evidence, (2) sum H to get P(Q, e1e2...) = sum over h1h2h3... of P(Q and h1h2h3 and e1e2e3...), (3) normalize by dividing with $\\sum_q P(Q,e1e2...ek)$. \n",
       "\n",
       "\n",
       "To construct joint probability table, we use product rule P(y)P(x|y) = P(x and y). Generalize to chain rule: P(x1, x2, x3) = P(x1) P(x2|x1) P(x3 | x1,x2). \n",
       "\n",
       "\n",
       "Two ways to factor joint distribution over two vars: \n",
       "\n",
       "P(x,y) = P(x|y)P(y) = P(y|x)P(x)\n",
       "\n",
       "\n",
       "Bayes rule: $P(x|y) = \f",
       "rac{P(y|x)}{P(y)} P(x)$ Allows building one conditional from the reverse. \n",
       "\n",
       "\n",
       "Probabilistic language model: take each word position as an RV estimate probability distribution to the next RV. Goal is to compute prob of a sentence/ a sequence of words, or compute probability of upcoming word. A model that computes either P(W) or $P(w_n | w_1, w_2...)$ is a language model or a grammar. \n",
       "\n",
       "\n",
       "$P(w1,w2,...wn) = \\prod_i P(wi | w1,w2,w_{i-1})$. \n",
       "\n",
       "\n",
       "Maximum likelihood estimate: count and divide. Problem is that some sentences would never appear. We will not get enough data. \n",
       "\n",
       "\n",
       "Markov assumption: we can use the current state alone to predict the next state. In language models, this mean the next word will only depend on the last k words. \n",
       "\n",
       "\n",
       "unigram = no history used, bigram = 1 word, trigram = 2 words, 4-gram = 3 words as history. \n",
       "\n",
       "\n",
       "Markov built LM by counting up 20k letter in Onegin in 1913. 1948 Shannon approximates English by n-grams. 1956 Chomsky paper suggests that finite state Markov Model is not a good model for human languages. Fred Jelinek in 1980 uses n-gram for ASR, and in 1993 develops statistical machine translation. \n",
       "\n",
       "\n",
       "Formula for machine translation: $argmax_e p(e|f) = p(e) p(f|e)$\n",
       "\n",
       "\n",
       "## Module 10 Definitions\n",
       "* **Product rule** - P(a and b) = P(a|b) P(b)\n",
       "* **Normalization constant** - a constant that, when applied, ensures a     distribution will add up to 1\n",
       "* **Language models** - Models that assign probabilities to sequences     of words\n",
       "* **Markov assumption** - Prob of a word depends only on the previous word\n",
       "* **Maximum likelihood estimation (mle)** - Estimated paremeters of a probability     distribution, s.t. probability of observed data is the highest\n",
       "* **Perplexity** - Inverse probabilty of test set, normalized by number of words\n",
       "* **Closed vocabulary system** - test set can only contain words from a lexicon\n",
       "* **Open vocab system** - system in which we model unknown words as <UNK>\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module10_sections = [\n",
    "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
    "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
    "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
    "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
    "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
    "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
    "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
    "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
    "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
    "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
    "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
    "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
    "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
    "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
    "]\n",
    "\n",
    "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
    "display(Markdown(module10_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WHREsHSSyA-h"
   },
   "source": [
    "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "tuFC8w7CyCns"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
    "chapter13_1 = \"\"\" Bayesian network: each node is a random variable. No cycles, \\\n",
    "so a DAG. Each node has probability info that quantifies effect of parents \\\n",
    "on the node using a number of parameters. Intuitive meaning of an arrow \\\n",
    "is that X has a direct influence on Y. \\n\n",
    "\n",
    "Local probability information in each node is represented in a conditional \\\n",
    "probability table. Each row contains cond prob for a conditioning case. \\n\n",
    "\n",
    "Constructing Bayesian networks: (1) determine set of variables required \\\n",
    "to model the domain. (2) order them, with the network being more compact if causes \\\n",
    "precede effects in the order, (3) choose minimal set of parents for X_i such that \\\n",
    "P(X_i | X_(i-1)...X1) = P(X_i | Parents(X_i)), (4) insert a link from each parent \\\n",
    "to X_i, and (5) write down conditional probability table. \\n\n",
    "\n",
    "The above construction method guarantees acyclicity, and no redundant prob values. \\\n",
    "So there cannot be inconsistency or violation of axioms of probability. \\n\n",
    " \"\"\"\n",
    "\n",
    "\n",
    "chapter13_2 = \"\"\" P(x1,...,xn) = product(P(x_i | parents(X_i))) defines \\\n",
    "the semantics of Bayesian networks. \\n\n",
    "\n",
    "Bayesian networks are locally structured, or sparse systems. \\\n",
    "Each subcomponent interact directly only with a bounded \\\n",
    "number of other components, resulting in higher compactness. \\n\n",
    "\n",
    "We can often suppose most RVs are influenced by at most k others \\\n",
    "for some k. THe conditional prob table will have at most 2^k numbers \\\n",
    "so complete network can be specified with 2^k n numbers vs 2^n \\n\n",
    "\n",
    "Node ordering can affect how compact a NB is. WIth a causal model \\\n",
    "we will have to specify fewer numbers and numbers are easier to \\\n",
    "come up with. \\n\n",
    "\n",
    "We can prove the property that in a NB each variable is cond \\\n",
    "independent of its non-descendents, given its parents. \\n\n",
    "\n",
    "Also the property that a var is cond indp of all other notdes in the \\\n",
    "network, given its parents, children, and children's parents \\\n",
    "(i.e. its Markov blanket). \\n\n",
    "\n",
    "Markov blanket property enables inference algos that use \\\n",
    "completely local and distributed stochastic sampling processes. \\n\n",
    "\n",
    "To examine whether a set Z d-separates X and Y (X is cond indp \\of Y given Z), we:\\n\n",
    "\n",
    "1) consider ancestral subgraph with just XYZ and ancestors. \\n\n",
    "\n",
    "2) Add links between unliked pairs that share common child (create \\\n",
    "a moral graph)\\n\n",
    "\n",
    "3) replace directed by undirected links.\\n\n",
    "\n",
    "4) If Z blocks all paths between X and Y then Z d-separates \\\n",
    "X and Y.\\n\n",
    "\n",
    "We can use canonical distribution to generalize the relatipnship \\\n",
    "between parent and children nodes. E.g. 1) deterministic nodes, 2) \\\n",
    "context specific indepdenecne (CSI), with two vars being cond indp provided \\\n",
    "certain values of others. \\n\n",
    "\n",
    "We can use noisy logical relationships (e.g. noisy OR) to model \\\n",
    "uncertain relationships. The model assumes that all possible causes \\\n",
    "are listed (we can add a leak node to fill up miscell items), and that \\\n",
    "inhibition of each parent is indp from inhibition from others. Noisy \\\n",
    "relationships depending on k parents can be described using \\\n",
    "O(k) parameters instead of the full O(2^k). \\n\n",
    "\n",
    "To model continuous vars we can: 1) use discretization, 2) define \\\n",
    "using standard prob density functions (e.g. Gaussian), 3) \\\n",
    "use non-paremetric representation, 4) use hybrid Bayesian Network with \\\n",
    "both discrete and continuous values. \\n\n",
    "\n",
    "Example of distributions: linear-Gaussian conditional distribution, \\\n",
    "with mean varying linearly with value of parent and fixed s.d. Networking with \\\n",
    "only vars with linear-Gaussian distributions have joint distirbution that \\\n",
    "is a multivariate Gaussian distrbution. \\n\n",
    "\n",
    "When discrete vars are added as paretns of cont vars, network defines \\\n",
    "a conditional Gaussian distribution.\\n\n",
    "\n",
    "Probit and expit(inverse logit) model can model relationship \\\n",
    "with soft thresholds.\n",
    "  \"\"\"\n",
    "\n",
    "\n",
    "chapter13_3 = \"\"\" Inference by enumeration: O(n 2^n) runtime with \\\n",
    "naive implementation, and O(2^n) if we take advantage of the nested \\\n",
    "structure of computation. We calculate P(X|e) = alpha * $\\sum_y P(X, e, y)$ \\\n",
    "Evaluation trees are calcualted in depth-first, left to right recurison, like \\\n",
    "backtracking and DPLL algorithm \\n\n",
    "\n",
    "Variable elimination reduces repeated calculation. We evaluate expressions \\\n",
    "in right-to-left order, storing intermediate results. We can annotate each \\\n",
    "part of expression with the corresponding factor, a matrix indexed by values \\\n",
    "of its argument varible.s Then we can sum the factors up and get the pointwise \\\n",
    "product between them. Pointwise product yields a new factor whose varaibles are the \\\n",
    "union of vars of original factors, and elements are the product of corresponding \\\n",
    "elments. \\n\n",
    "\n",
    "\n",
    "Different orderings chosen for the algorithm will generate differnet algo. Time and space \\\n",
    "requirements of var elimination is dominated by the largest factor constructed. \\n\n",
    "\n",
    "Complexity of exact inference depends on structure of network. If there is at most \\\n",
    "one undirected path between two nodes, then the network if singly connected or \\\n",
    "called polytrees. The time and space complexity of exact inference is linear in size of \\\n",
    "network, with size being number of CPT entries. \\n\n",
    "\n",
    "For multiply connected networks, var elimination can have exponential runtime in the \\\n",
    "worst case. Inference in Bayes nets is NP-hard (we can reduce \\\n",
    "satisfiability problems to BN inference problems). \\n\n",
    "\n",
    "We can also reduce Bayes net inference to satisifability and use develoepd solvers. \\\n",
    "e.g. weighted model counting. \\n\n",
    "\n",
    "Answering multiple queries (e.g. for all vars) can be done in O(n) time with \\\n",
    "clustering algorithms (or join tree algos). \\n\n",
    "\n",
    "We can join indiv nodes to form clusters such that resulting network is a \\\n",
    "polytree. Nodes are replaced by meganodes. We then use a special purpose inference algo \\\n",
    "with a technique like constraint propagation to compute posterior probabilties \\\n",
    "in time lienar to size of clustered network. \n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module11_lecture_notes = \"\"\" X and Y are conditionally independent given Z if \\\n",
    "P(x,y|z) = P(x|z)P(y|z), or equivalently iff P(x|z,y) = P(x|z) \\n\n",
    "\n",
    "Proof of equivalence: p(x|y,z) = p(x,y,z)/p(y,z) = p(z)p(x,y|z)/p(y,z) = \\\n",
    "p(z)p(x|z)p(y|z)/p(z)p(y|z) = P(x|z). \n",
    "\n",
    "C-independence is a property of a distribution. E.g. Alarm indp Fire given Smoke \\n\n",
    "\n",
    "Unconditional independent is very rare, and does not help reasoning on probability. \\\n",
    "Conditional independence consistutes the most basic and robust form of knowledge \\\n",
    "about unknown environments. \\n\n",
    "\n",
    "In Bayes net, nodes are variables with domains. Can be assigned/observed or \\\n",
    "unassigned (unobserved). Arcs are interactions between variables, showing \\\n",
    "direct influence/ conditional independence (but an arrow does not \\\n",
    "necessarily mean causal influence). \\n\n",
    "\n",
    "Bayes net are direct, acyclic graph. Each node X with parents A1, A2, ... has \\\n",
    "a distribution for each combination of parent values. Same as topology + local conditional \\\n",
    "probabilities. \\n\n",
    "\n",
    "Joint distributions are implicitly encoded in Baynes nets, as products of local \\\n",
    "conditional distributions. Joint probability can be obtained by multiplying all \\\n",
    "conditional probabilities, assuming conditional independence.\\n\n",
    "\n",
    "Baynes net does not need to be causal: arrows can represent correlation rather \\\n",
    "than causation. The topology in Baynes net should be taken to represent conditional \\\n",
    "independence (P(xi|x1,...,x(i-1))) = P(xi|parents(Xi)). \\n\n",
    "\n",
    "An N-node net with up to k-parents per node has O(N 2^(k+1)) size. Space \\\n",
    "savings compared to having full joint probability model. \\n\n",
    "\n",
    "In Bayes net, we assume $P(x_i | x_1..x_{i-1}) = P(x_i | parents(X_i))$. \\\n",
    "i.e. xi only depends on its defined parents. \\n\n",
    "\n",
    "There are often additional assumptions that can be read off the graph. \\\n",
    "e.g. d-separation. \\n\n",
    "\n",
    "Suppose we have causal chain where x -> y -> z. We can prove that X is indp of \\\n",
    "Z given y by showing p(z|x,y) = p(z|y). i.e. evidence along the chain blocks causal \\\n",
    "influence. \\n\n",
    "\n",
    "Suppose we have common cause Y which influences X and Z. X is conditionally \\\n",
    "independent of Z given y by showing p(z|x,y) = p(z|y). i.e. seeing the cause \\\n",
    "blocks the influence between effects \\n\n",
    "\n",
    "Suppose we have X and Y causing a common effect Z (v-strucutre). X and Y are \\\n",
    "independent. X and Y are not independent given Z (seeing effect activates \\\n",
    "influence between possible causes).  \\n\n",
    "\n",
    "General question: are two variables independent in a given Bayes Net? \\\n",
    "We reason using the three cases above. Shade evidence nodes in the graph and examine \\\n",
    "resulting graph. \\n\n",
    "\n",
    "Wrong approach: any two connected nodes not blocked by a shaded node is c. indp. \\\n",
    "Problem is with v-structures. \\n\n",
    "\n",
    "Correct approach: define active triples: \\n\n",
    "(U = unobserved, O = observed)\\n\n",
    "1) U -> U -> U/O \\n\n",
    "2) U <- U -> U \\n\n",
    "3) U -> O <- U \\n\n",
    "4) U -> (U->O) <- U \\n\n",
    "vs inactive triples\\n\n",
    "1) U -> O -> U\\n\n",
    "2) U <- O -> U\\n\n",
    "3) U -> U <- U \\n\n",
    "\n",
    "Any path is active if each triple is active. \\n\n",
    "\n",
    "If there are no active paths between X and Y then they are conditionally indp. \\\n",
    "i.e. d-separated. \\n\n",
    "\n",
    "D separation algorithm: want to know if i, j are cond indp given evidence. \\\n",
    "Approach: check all undirected paths between i and j. If one or more paths \\\n",
    "are active, than indp not guaranteed. Otherwise guaranteed.\\n\n",
    "\n",
    "Note that a path is active only if all triples along the path are active. \\n\n",
    "\n",
    "Inference is the process of getting useful numbers from joint probabilty distribution \\\n",
    "e.g. posterior probability, most likely explanation. Inference by enumeration: \\\n",
    "sum hidden variables, calculate P(Q|e1,e2...). \\n\n",
    "\n",
    "We can do inference by enumeration in bayes net but it will \\\n",
    "be expensive. Exponential time with respect to non observed variables. \\\n",
    "Reason is that we need to join the whole joint distribution \\\n",
    "before summing out hidden variables. To prevent that we can do \\\n",
    "variable elimination (np hard but faster), interleave joining and \\\n",
    "marginalizing.  \n",
    "\"\"\"\n",
    "\n",
    "# Definitions\n",
    "module11_definitions = {\n",
    "\"Hidden variables\" : \"Vars in Bayes net that are neither input nor output\"\n",
    ",    \n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "vqtEs0gHs9sO",
    "outputId": "55098d61-7877-4454-c09d-d31c258ba429"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n",
       "## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n",
       " Bayesian network: each node is a random variable. No cycles, so a DAG. Each node has probability info that quantifies effect of parents on the node using a number of parameters. Intuitive meaning of an arrow is that X has a direct influence on Y. \n",
       "\n",
       "\n",
       "Local probability information in each node is represented in a conditional probability table. Each row contains cond prob for a conditioning case. \n",
       "\n",
       "\n",
       "Constructing Bayesian networks: (1) determine set of variables required to model the domain. (2) order them, with the network being more compact if causes precede effects in the order, (3) choose minimal set of parents for X_i such that P(X_i | X_(i-1)...X1) = P(X_i | Parents(X_i)), (4) insert a link from each parent to X_i, and (5) write down conditional probability table. \n",
       "\n",
       "\n",
       "The above construction method guarantees acyclicity, and no redundant prob values. So there cannot be inconsistency or violation of axioms of probability. \n",
       "\n",
       " \n",
       "\n",
       "## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n",
       " P(x1,...,xn) = product(P(x_i | parents(X_i))) defines the semantics of Bayesian networks. \n",
       "\n",
       "\n",
       "Bayesian networks are locally structured, or sparse systems. Each subcomponent interact directly only with a bounded number of other components, resulting in higher compactness. \n",
       "\n",
       "\n",
       "We can often suppose most RVs are influenced by at most k others for some k. THe conditional prob table will have at most 2^k numbers so complete network can be specified with 2^k n numbers vs 2^n \n",
       "\n",
       "\n",
       "Node ordering can affect how compact a NB is. WIth a causal model we will have to specify fewer numbers and numbers are easier to come up with. \n",
       "\n",
       "\n",
       "We can prove the property that in a NB each variable is cond independent of its non-descendents, given its parents. \n",
       "\n",
       "\n",
       "Also the property that a var is cond indp of all other notdes in the network, given its parents, children, and children's parents (i.e. its Markov blanket). \n",
       "\n",
       "\n",
       "Markov blanket property enables inference algos that use completely local and distributed stochastic sampling processes. \n",
       "\n",
       "\n",
       "To examine whether a set Z d-separates X and Y (X is cond indp \\of Y given Z), we:\n",
       "\n",
       "\n",
       "1) consider ancestral subgraph with just XYZ and ancestors. \n",
       "\n",
       "\n",
       "2) Add links between unliked pairs that share common child (create a moral graph)\n",
       "\n",
       "\n",
       "3) replace directed by undirected links.\n",
       "\n",
       "\n",
       "4) If Z blocks all paths between X and Y then Z d-separates X and Y.\n",
       "\n",
       "\n",
       "We can use canonical distribution to generalize the relatipnship between parent and children nodes. E.g. 1) deterministic nodes, 2) context specific indepdenecne (CSI), with two vars being cond indp provided certain values of others. \n",
       "\n",
       "\n",
       "We can use noisy logical relationships (e.g. noisy OR) to model uncertain relationships. The model assumes that all possible causes are listed (we can add a leak node to fill up miscell items), and that inhibition of each parent is indp from inhibition from others. Noisy relationships depending on k parents can be described using O(k) parameters instead of the full O(2^k). \n",
       "\n",
       "\n",
       "To model continuous vars we can: 1) use discretization, 2) define using standard prob density functions (e.g. Gaussian), 3) use non-paremetric representation, 4) use hybrid Bayesian Network with both discrete and continuous values. \n",
       "\n",
       "\n",
       "Example of distributions: linear-Gaussian conditional distribution, with mean varying linearly with value of parent and fixed s.d. Networking with only vars with linear-Gaussian distributions have joint distirbution that is a multivariate Gaussian distrbution. \n",
       "\n",
       "\n",
       "When discrete vars are added as paretns of cont vars, network defines a conditional Gaussian distribution.\n",
       "\n",
       "\n",
       "Probit and expit(inverse logit) model can model relationship with soft thresholds.\n",
       "  \n",
       "\n",
       "## AIMA 13.3 Exact Inference in Bayesian Networks\n",
       " Inference by enumeration: O(n 2^n) runtime with naive implementation, and O(2^n) if we take advantage of the nested structure of computation. We calculate P(X|e) = alpha * $\\sum_y P(X, e, y)$ Evaluation trees are calcualted in depth-first, left to right recurison, like backtracking and DPLL algorithm \n",
       "\n",
       "\n",
       "Variable elimination reduces repeated calculation. We evaluate expressions in right-to-left order, storing intermediate results. We can annotate each part of expression with the corresponding factor, a matrix indexed by values of its argument varible.s Then we can sum the factors up and get the pointwise product between them. Pointwise product yields a new factor whose varaibles are the union of vars of original factors, and elements are the product of corresponding elments. \n",
       "\n",
       "\n",
       "\n",
       "Different orderings chosen for the algorithm will generate differnet algo. Time and space requirements of var elimination is dominated by the largest factor constructed. \n",
       "\n",
       "\n",
       "Complexity of exact inference depends on structure of network. If there is at most one undirected path between two nodes, then the network if singly connected or called polytrees. The time and space complexity of exact inference is linear in size of network, with size being number of CPT entries. \n",
       "\n",
       "\n",
       "For multiply connected networks, var elimination can have exponential runtime in the worst case. Inference in Bayes nets is NP-hard (we can reduce satisfiability problems to BN inference problems). \n",
       "\n",
       "\n",
       "We can also reduce Bayes net inference to satisifability and use develoepd solvers. e.g. weighted model counting. \n",
       "\n",
       "\n",
       "Answering multiple queries (e.g. for all vars) can be done in O(n) time with clustering algorithms (or join tree algos). \n",
       "\n",
       "\n",
       "We can join indiv nodes to form clusters such that resulting network is a polytree. Nodes are replaced by meganodes. We then use a special purpose inference algo with a technique like constraint propagation to compute posterior probabilties in time lienar to size of clustered network. \n",
       "\n",
       "\n",
       "\n",
       "## Module 11 Lecture Notes\n",
       " X and Y are conditionally independent given Z if P(x,y|z) = P(x|z)P(y|z), or equivalently iff P(x|z,y) = P(x|z) \n",
       "\n",
       "\n",
       "Proof of equivalence: p(x|y,z) = p(x,y,z)/p(y,z) = p(z)p(x,y|z)/p(y,z) = p(z)p(x|z)p(y|z)/p(z)p(y|z) = P(x|z). \n",
       "\n",
       "C-independence is a property of a distribution. E.g. Alarm indp Fire given Smoke \n",
       "\n",
       "\n",
       "Unconditional independent is very rare, and does not help reasoning on probability. Conditional independence consistutes the most basic and robust form of knowledge about unknown environments. \n",
       "\n",
       "\n",
       "In Bayes net, nodes are variables with domains. Can be assigned/observed or unassigned (unobserved). Arcs are interactions between variables, showing direct influence/ conditional independence (but an arrow does not necessarily mean causal influence). \n",
       "\n",
       "\n",
       "Bayes net are direct, acyclic graph. Each node X with parents A1, A2, ... has a distribution for each combination of parent values. Same as topology + local conditional probabilities. \n",
       "\n",
       "\n",
       "Joint distributions are implicitly encoded in Baynes nets, as products of local conditional distributions. Joint probability can be obtained by multiplying all conditional probabilities, assuming conditional independence.\n",
       "\n",
       "\n",
       "Baynes net does not need to be causal: arrows can represent correlation rather than causation. The topology in Baynes net should be taken to represent conditional independence (P(xi|x1,...,x(i-1))) = P(xi|parents(Xi)). \n",
       "\n",
       "\n",
       "An N-node net with up to k-parents per node has O(N 2^(k+1)) size. Space savings compared to having full joint probability model. \n",
       "\n",
       "\n",
       "In Bayes net, we assume $P(x_i | x_1..x_{i-1}) = P(x_i | parents(X_i))$. i.e. xi only depends on its defined parents. \n",
       "\n",
       "\n",
       "There are often additional assumptions that can be read off the graph. e.g. d-separation. \n",
       "\n",
       "\n",
       "Suppose we have causal chain where x -> y -> z. We can prove that X is indp of Z given y by showing p(z|x,y) = p(z|y). i.e. evidence along the chain blocks causal influence. \n",
       "\n",
       "\n",
       "Suppose we have common cause Y which influences X and Z. X is conditionally independent of Z given y by showing p(z|x,y) = p(z|y). i.e. seeing the cause blocks the influence between effects \n",
       "\n",
       "\n",
       "Suppose we have X and Y causing a common effect Z (v-strucutre). X and Y are independent. X and Y are not independent given Z (seeing effect activates influence between possible causes).  \n",
       "\n",
       "\n",
       "General question: are two variables independent in a given Bayes Net? We reason using the three cases above. Shade evidence nodes in the graph and examine resulting graph. \n",
       "\n",
       "\n",
       "Wrong approach: any two connected nodes not blocked by a shaded node is c. indp. Problem is with v-structures. \n",
       "\n",
       "\n",
       "Correct approach: define active triples: \n",
       "\n",
       "(U = unobserved, O = observed)\n",
       "\n",
       "1) U -> U -> U/O \n",
       "\n",
       "2) U <- U -> U \n",
       "\n",
       "3) U -> O <- U \n",
       "\n",
       "4) U -> (U->O) <- U \n",
       "\n",
       "vs inactive triples\n",
       "\n",
       "1) U -> O -> U\n",
       "\n",
       "2) U <- O -> U\n",
       "\n",
       "3) U -> U <- U \n",
       "\n",
       "\n",
       "Any path is active if each triple is active. \n",
       "\n",
       "\n",
       "If there are no active paths between X and Y then they are conditionally indp. i.e. d-separated. \n",
       "\n",
       "\n",
       "D separation algorithm: want to know if i, j are cond indp given evidence. Approach: check all undirected paths between i and j. If one or more paths are active, than indp not guaranteed. Otherwise guaranteed.\n",
       "\n",
       "\n",
       "Note that a path is active only if all triples along the path are active. \n",
       "\n",
       "\n",
       "Inference is the process of getting useful numbers from joint probabilty distribution e.g. posterior probability, most likely explanation. Inference by enumeration: sum hidden variables, calculate P(Q|e1,e2...). \n",
       "\n",
       "\n",
       "We can do inference by enumeration in bayes net but it will be expensive. Exponential time with respect to non observed variables. Reason is that we need to join the whole joint distribution before summing out hidden variables. To prevent that we can do variable elimination (np hard but faster), interleave joining and marginalizing.  \n",
       "\n",
       "\n",
       "## Module 11 Definitions\n",
       "* **Hidden variables** - Vars in Bayes net that are neither input nor output\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module11_sections = [\n",
    "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
    "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
    "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
    "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
    "]\n",
    "\n",
    "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
    "                         module11_sections, \n",
    "                         module11_definitions)\n",
    "display(Markdown(module11_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F3FPMzG9yUls"
   },
   "source": [
    "#Module 12 - Naive Bayes and Perceptrons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "iEIOpL7vyWcc"
   },
   "outputs": [],
   "source": [
    "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
    "\n",
    "chapter19_1 = \"\"\" Machine learning uses factored representation, \\\n",
    "representing input as a vector of attribute values. When output is \\\n",
    "one of finite set of values, then it is a classification problem. \\\n",
    "If it is a number, then a regression problem. \\n\n",
    "\n",
    "Three types of feedback: supervised learning (input -> label). \\\n",
    "Unsupervised learning e.g. clustering. Reinforcement learning \\\n",
    "agents have rewards and punishments. \"\"\"\n",
    "\n",
    "\n",
    "chapter19_2 = \"\"\" Overfitting models have high variance. Underfitting \\\n",
    "models have high bias. There is often a bias-variance tradeoff. \\n\n",
    "\n",
    "There is also a tradeoff between how expressive a hypo space is \\\n",
    "and the complexity of finding a good hypo within that space. \n",
    " \"\"\"\n",
    "\n",
    "chapter19_3 = \"\"\" It is intractable to find a guaranteed smallest decision \\\n",
    "tree. We can use a greedy divide-and-conquer strategy: testing most \\\n",
    "important attribtues first then recursively solve smaller subproblems. \\n\n",
    "\n",
    "We can evaluate performa nce using a learning curve, training test size vs \\\n",
    "proportion of correct answers. Learning curves are also called happy graphs. \\n\n",
    "\n",
    "We can measure importance in terms of information gain or entropy. \\\n",
    "An RV with only one possible value has entropy 0. A fair coin has 1 bit \\\n",
    "entropy. \\n\n",
    "\n",
    "Info gain is the expected reduction in entropy: \\\n",
    "Gain(A) = B(p/(p+n)) - Remainder(A). with B(q) being entropy of boolean RV with \\\n",
    "prob q. \\n\n",
    "\n",
    "Overfitting is more likely when number of attributes increase, and less \\\n",
    "liekly when number of training examples increase. \\n\n",
    "\n",
    "Decision tree pruning reduces overfitting, by elim nodes that are not \\\n",
    "necessarily relevant. A significance test using chi-squared statistic is used \\\n",
    "to determine if a node is significant. \\n\n",
    "\n",
    "Early stop (combining info gain and chi-squared pruning) has the risk of \\\n",
    "missing combination of attributes that are informative. \\n\n",
    "\n",
    "To deal with continuous variables, we can use a split point test (inequality \\\n",
    "test on value). For attributes with many finite values, we can \\\n",
    "use an information gain ratio, or use equality test. For numerical prediction \\\n",
    "we need a regression tree rather than a classification tree. \\n\n",
    "\n",
    "Decision tree can suffer from suboptimal policy because of greedy search, \\\n",
    "expensive runtime when it is too deep, and unstable since adding one \\\n",
    "example can change the whole tree. Random forest can mitigate these effects. \n",
    "  \"\"\"\n",
    "\n",
    "chapter19_4 = \"\"\" In machine learning we make the stationarity assumption: \\\n",
    "that each example has same prior prob distribution, and that it is indp from \\\n",
    "previous examples. Examples that satisfy these are called indp and identiiccally distributed \\\n",
    "or i.i.d. \\n\n",
    "\n",
    "Optimal fit = hypo that minimizes the error rate, measured by performance \\\n",
    "on test set. For hyperparameters (parameters of model class rather htan \\\n",
    "individual model) we need a dev set. \\n\n",
    "\n",
    "Extreme k-fold cross validation uses k = n, aka leave-one-out cross-validation or \\\n",
    "LOOCV. Each example will serve both as training and validation data but not \\\n",
    "at the same time. \\n\n",
    "\n",
    "Model selection algorithm: takes in as argument a learning algo, which in turns \\\n",
    "take in a hyperparameter (size) as the argument. It measures the error rate for \\\n",
    "each value of size. It picks the value that minimizes error. \\n\n",
    "\n",
    "Decision tree often has U shaped curves while CNNs often have \\\n",
    "decreasing curves. \\n\n",
    "\n",
    "A model that exactly fits training data has interpolated the data.\\n\n",
    "\n",
    "An improvement upon error rate is loss function, which allows us to emphasize \\\n",
    "what errors matter more. Most general formulation L(x, y, ypred) = \\\n",
    "Utility (result of using y given input x) - Utility (result of using ypred)\\n\n",
    "\n",
    "Best hypothesis is the one with minimum expected generalization loss \\\n",
    "defined as sum of Loss * probability. Since prob is not known, we use \\\n",
    "instead empirical loss as an estimate. \\n\n",
    "\n",
    "Four reasons why empirical loss may differ from true function: \\n\n",
    "\n",
    "1) Unrealizability: the hypo space does not contain the true f. \\n\n",
    "\n",
    "2) variance: algo returns different hypo for different sets of \\\n",
    "examples. \\n\n",
    "\n",
    "3) Noise: f may be nondeterminisitc. \\n\n",
    "\n",
    "4) Computational complexity: we cannot afford to search \\\n",
    "all possibilities. \\n\n",
    "\n",
    "Recent years use more large-scale learning than small-scall learning.\\n\n",
    " \"\"\"\n",
    "\n",
    "chapter19_5 = \"\"\" To figure out the number of samples needed is the question \\\n",
    "for computational learning theory. \\n\n",
    "\n",
    "Underlying priunciple of CLT is that any hypo consistent with a \\\n",
    "sufficiently large examples is unlikely to be seriously worng, or probably \\\n",
    "approximately correct (PAC). \\n\n",
    "\n",
    "PAC theorems assume stationarity. Hypothesis is called approximately \\\n",
    "correct if error(h) is smaller than or equal to a defined bound epsilon. \\\n",
    "It is close to the true function or lie inside an epsilon-ball around \\\n",
    "true functino f. \\n\n",
    "\n",
    "Number of required examples, as function of epsilon and delta (probabilty \\\n",
    "of H_bad containing at least one consistent hypo) is called \\\n",
    "the sample complexity of the learning algo.  \\n\n",
    "\n",
    "Dilemma between restricitng hypo space and not eliminating the true \\\n",
    "function. Three ways to mitigate: 1) bring prior knowledge, \\\n",
    "2) have the algo return a simple hypo, instead of complex 3) \\\n",
    "focus on learnable subset of the hypo space (decision lists). \\n\n",
    "\n",
    "k-DL: decision list with at most k conjunctions. We can show \\\n",
    "that k-DL is learnable, i.e. can be approximated accurately \\\n",
    "after training on suffiicent examples. We then find an efficent \\\n",
    "algo that returns a consistent decision list (e.g. Greedy algo \\\n",
    "decision list learning, repeatedly find test that agrees exactly \\\n",
    "with some subset of the training set).\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "chapter19_6 = \"\"\" Linear regression covers a diff hypo space: \\\n",
    "the calss of linear functions over continuously valued inputs. \\n\n",
    "\n",
    "We traiditonally use squared-error loss function L2, summed over \\\n",
    "all training examples. We find weights that minimize loss.\\n\n",
    "\n",
    "For univariate linear regression, the weight space is 2d. The loss functino \\\n",
    "is convex for every linreg problem with L2 loss functino. \\n\n",
    "\n",
    "Gradient descent is use to minimize loss. While not converged yet, \\\n",
    "we update each weight using $w = w - \\alpha $ (partial derivative of loss over w). \\n\n",
    "\n",
    "alpha s the step size, or rate at which we minimize los. Can be fixed or decay over\\\n",
    " time. For univariate, loss is quadratic so partial derivative is linear. \\n\n",
    "\n",
    " Batch/deterministic gradient descent: convex loss surface (no local minima), so \\\n",
    "will eventually converge, but may be slow.\\n\n",
    "\n",
    "Stochastic gradient descent: randomly select number (minibatch) of training examples at each step, \\\n",
    "May be hard to converge, but often faster. SGD is useful for online setting where \\\n",
    "data is coming in and stationarity may not hold. \\n\n",
    "\n",
    "For multi-variate regression there may be risk of overfitting, so we can use \\\n",
    "regularization. L1 regularization has the advantage that it tends to produce \\\n",
    "a sparse model, pushing many weights to zero. \\n\n",
    "\n",
    "Linear functions can be used for classfication by serving as decision boundaries. or linear\\\n",
    "separators. When data are linearly separable, percetron learning rule \\\n",
    "converges to a perfect linear separator. \\n\n",
    "\n",
    "An improvement upon perceptron is logistic regression, which uses the logistic function \\\n",
    "logistic(z) = $1 / (1 + e^{-z})$. \\n\n",
    "\n",
    "$h_w (x) = logistic(w \\dot x) = 1 / (1 + e^{-w \\dot x})$. \\n\n",
    "\"\"\"\n",
    "\n",
    "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
    "slp_chapter4_1 = \"\"\" Text categorization tasks: sentiment analysis, \\\n",
    "authorship attribution, spam detection, language id. Mostly done through \\\n",
    "supervised ML. \\n\n",
    "\n",
    "Probabilistic classifier: assign prob to observation belonging to classes. \\\n",
    "Generative classifier (e.g. naive Bayes): build model of how a class generates \\\n",
    "input data. Discriminative: learn what features are most useful. Generative \\\n",
    "classifiers are more popular. \\n\n",
    "\n",
    "\n",
    "Naive bayes model. Compute most probable class given document d, \\\n",
    "by choosing class with the highest product of prior probability p(c) and \\\n",
    "likelihood of document p(d|c). \\n\n",
    "\n",
    "Two simplifying assumptions: (1) bag of words assumption that positions do not \\\n",
    "matter. (2) naive Bayes assumption/ conditional independence assumption that \\\n",
    "probabilites $P(f_i | c)$ are independent given class c and hence can be \\\n",
    "naively multiplied. \\n\n",
    "\n",
    "Naive bayes calculations are also done in log space. In log space, predicted \\\n",
    "class is computed as linear function of input features. THerefore naive Bayes\\\n",
    " is also a linear classifier.  \"\"\"\n",
    "\n",
    "\n",
    "slp_chapter4_2 = \"\"\" P(c) = number of docs in the class / total num of docs \\n\n",
    "\n",
    "P(w|c) = count(w,c) / count of all words in c. \\n\n",
    "\n",
    "Problem of zero: one zero cond prob will cause product to be zero. \\\n",
    "Simple solution is add-one/ Laplace smoothing. \\n\n",
    "\n",
    "P(w|c) = (count(w,c) + 1) / ((count of all words in c) + |size of all words in all classes|)  \\n\n",
    "\n",
    "Solution for unknown words (words in test \\\n",
    "data but not training data): ignore them, remove from test document. \\n\n",
    "\n",
    "Stop words (very frequent words): either completely ignore or include \\\n",
    "(does not affect performance).\n",
    " \"\"\"\n",
    "\n",
    "slp_chapter4_3 = \"\"\" Worked example of naive bayes: drop unknown word \\\n",
    "in test set, then multiply P(w|c) for each words in test set and P(c). \"\"\"\n",
    "\n",
    "slp_chapter4_4 = \"\"\" Binary multinomial naive bayes: remove all duplicate \\\n",
    "words before concatenating them into big document. Emphasize existence \\\n",
    "of words to frequency. \\n\n",
    "\n",
    "Dealing with negatives: (1) prepend prefix NOT_ to words. (2) use \\\n",
    "parsing (more complext). \\n\n",
    "\n",
    "Dealing with insufficient training data: we can draw from sentiment lexicons, \\\n",
    "lists of words pre-annotated with positive/ negative sentiments. Popular lexicons \\\n",
    "include General Inquirer, LIWC, opinion lexicon and MPQA Subjectivity Lexicon. \\\n",
    "One way to use it is to add a feature tha is counted whenever a word \\\n",
    "in a lexicon occurs. \"\"\"\n",
    "\n",
    "slp_chapter4_5 = \"\"\" For tasks like spam detection, we can use \\\n",
    "predefined likely set of words/phrased as features, coupled with non-linguistic \\\n",
    "features. \\n\n",
    "\n",
    "For tasks like language id, most effective features are not words but \\\n",
    "character n-grams like zw and nya, or byte n-grams (which include spaces). \\\n",
    "Some system (langid.py) uses all possible n-grams of lenghts 1-4 and use feature \\\n",
    "selection to choose useful ones. \"\"\"\n",
    "\n",
    "slp_chapter4_6 = \"\"\" NB model can be seen as a set of class-specific unigram \\\n",
    "language models, wth the model of each class (e.g. positive and negative) \\\n",
    "instantiating a unigram language model.  \"\"\"\n",
    "\n",
    "slp_chapter4_7 = \"\"\" Human labels to evaluate models = gold labels. \\n\n",
    "\n",
    "Confusion matrix helps visualize how a model performs. Dimensions = \\\n",
    "gold labels and system output. \\n\n",
    "\n",
    "Accuracy = (tp + tn) / (tp + fp + tn + fn) \\n\n",
    "\n",
    "Precision = tp / (tp + fp) \\n\n",
    "\n",
    "Precision: percentage of items labeled as positives that are true positives. \\n\n",
    "\n",
    "Recall = tp / (tp + fn) \\n\n",
    "\n",
    "Recall = % of items actually present in the input that is correctly identified \\\n",
    "by the system. \\n\n",
    "\n",
    "\n",
    "F-measure incorporates both precision and recall. \\n\n",
    "\n",
    "$F_B = ((B^2 + 1)PR) / (B^2 P + R)$\\n\n",
    "\n",
    "B parameter weights the importance of recall and precision, with B > 1 favoriting \\\n",
    "recall and vice versa. B = 1 is the most freq used, called $F_1$. \\n\n",
    "\n",
    "F-measure is a weighted harmonic mean of precision and recall. It is a conservative \\\n",
    "metric - closer to the minimum of each number than arthmetic mean. \\n\n",
    "\n",
    "To compute metrics for more than two classes, we can use microaveraging \\\n",
    "(collect decision for all classes into a single confusion matrix, and compute \\\n",
    "precision and recall from that table), or macroaveraging (compute performance \\\n",
    "for each class then average over classes).\n",
    " \"\"\"\n",
    "\n",
    "slp_chapter4_8 = \"\"\" To be able to use some data for training and still use \\\n",
    "all for test, we can use cross-validation: randomly choose division of data, \\\n",
    "into training and test, train classifier, compute error rate on test, \\\n",
    "rinse and repeat. Do n times and average n runs to get average error rate = \\\n",
    "n-fold cross-validation. \\n\n",
    "\n",
    "It is common to create fixed training and test set, then do n-fold \\\n",
    "cross-validation inside training set, but comput error way normally in test set. \\n\n",
    "\n",
    " \"\"\"\n",
    "slp_chapter4_9 = \"\"\" To compare the performance of two system is to \\\n",
    "do statistical hypothesis testing. \\n\n",
    "\n",
    "p-value: probability that we would see an observed measure assuming null hypothesis. \\\n",
    "Statistically significant if p value is below a certain threshold. \\n\n",
    "\n",
    "NLP generally does not use simple parametric tests like t-tests or ANOVAS \\\n",
    "because of their assumptions. We use non-paremetric tests based on sampling \\\n",
    "e.g. approximate randomization and bootstrap test.  \\n\n",
    "\n",
    "Bootstrap test: can be applied to any metric.  Bootstrapping = repeatedly \\\n",
    "drwaining large numbers of smaller samples with replacement from an original \\\n",
    "larger sample. Assume that sample is representative of population. \\n\n",
    "\n",
    "We create b test sets from boostrapping, then compute statistics \\\n",
    "on how often A has accidental advantage. E.g. we can count how many \\\n",
    "times the delta exceeds the expected zero value by delta(x) or more. \n",
    "\"\"\"\n",
    "\n",
    "\n",
    "slp_chapter4_10 = \"\"\" ML systems replicate and even amplify biases in \\\n",
    "trainig data or labels. E.g. representational harms: harms by a system \\\n",
    "that demeans a social group, e.g. assigning lower sentiment scores. E.g. toxicity \\\n",
    "detection, which =can result in censorship.\\n\n",
    "\n",
    "To deal with harms we can release a model card, including information such as \\\n",
    "training algo and parameters, sources, processing, intended use, and performance over \\\n",
    "different demographics. \"\"\"\n",
    "\n",
    "# Lecture notes \n",
    "module12_lecture_notes = \"\"\" ML gives us a way to acquire a model from \\\n",
    "data, learning paremeters (e.g. probabilities), structure (e.g. Bayens net \\\n",
    "graphs), and hidden concepts (e.g. clustering). \\n\n",
    "\n",
    "In naive bayes model and perceptron our task is classification. \\\n",
    "Naive bayes is model-based approach: we build a model with random variables being the \\\n",
    "labels and features, then instantiate observed features, and query for the \\\n",
    "distribution of the label conditioned on the features. Major challenges are \\\n",
    "deciding the structure of the BN and deciding how we should learn the paremeters. \\n\n",
    "\n",
    "Naive Bayes model for digits: assume all features are independent effects of the \\\n",
    "label. Feature values are on or off depending on whether intensity > 0.5. Every \\\n",
    "input will be represented with a feature vector, and we predict \\\n",
    "P(Y|product(features)) = P(Y) * product (P(feature | Y)). (with independence \\\n",
    "assumption) \\n\n",
    "\n",
    "Full joint distribution of the left side has size |Y| |F|^n values,  with |F| being \\\n",
    "possible values for features and n being number of features. With independence \\\n",
    "assumption, the right side only has n |F| |Y| parameters, meaning that it is linear \\\n",
    "in number of features. \\n\n",
    "\n",
    "Steps for inference in Naive Bayes: (1) Get joint probability of label and \\\n",
    "evidence for each label P(y1,f1,...,fn). (2) sum to get probability of evidence. \\\n",
    "(3) Divide step (1) by step (2) to normalize. \\n\n",
    "\n",
    "Data we need: (1) P(Y): prior of labels, (2) P(Fi|Y) for each feature, i.e. evidence variable.\\\n",
    "These probabilities are called parameters / theta. \\n\n",
    "\n",
    "Data in ML: \\n\n",
    "\n",
    "(1) Training set\\n\n",
    "\n",
    "(2) Held-out data/ development set: optimize hyperparameters \\n\n",
    "\n",
    "(3) Test set \\n\n",
    "\n",
    "A typical experimentation cycle: learning paremeters on training set -> tune hyperparemeters on \\\n",
    "holdout set -> compute accuracy of test set (fraction of correct predictions). \\n\n",
    "\n",
    "Overfitting: fitting the training data too closely, but cannot generalize. \\\n",
    "E.g. if we have not seen a word (0 probability) we can get positive infinities \\\n",
    "In response we need to smooth or regularize the estimates. \\n\n",
    "\n",
    "Ways to estimate parameters: (1) ask a human, (2) learn from training data \\\n",
    "(empirical rate). Empricial rate = estimate (of the value of the feature) that \\\n",
    "maximizes the likelihood of the data. \\n\n",
    "\n",
    "Laplace smoothing: pretend we saw everything one more time than we did. \\\n",
    "Can derive the estimate with Dirichlet priors. If we use k instead of 1 for \\\n",
    "Laplace smoothing, k is called the strength of the prior. \\n\n",
    "\n",
    "Laplace smoothing does not do well for P(X|Y) when |X| or |Y| is very large. \\\n",
    "Linear interpolation: $P_{lin}(x|y) = alpha P(x|y) + (1 - alpha) P(x)$. \\n\n",
    "\n",
    "Hyperparameters: amount/ type of smoothing, k, alpha. We should not tune \\\n",
    "hyper-paremeters on training data since that way they would not generalize. \\n\n",
    "\n",
    "Baseline: simple straw man procedures that help determine how hard the task is \\\n",
    "and what is a good accuracy. E.g. labeling everything as spam for spam filters.\\n\n",
    "\n",
    "Perceptron: inputs * weights -> apply activation function and compare to threshold \\\n",
    "-> output. Instead of the threshold we can use a bias b = -threshold, so that \\\n",
    "output = 0 if w $\\dot$ x + b $\\leq$ 0. \\n\n",
    "\n",
    "In graphical represenation, classification decision will be \\\n",
    "represented by a decision boundary (line, plane, or hyperplane\\\n",
    "), orthogonal to the weight vector. If input vector has less than 90 degree \\\n",
    "distance to weight vector than positive, else negative. A bias would move the \\\n",
    "decision boundary away from the origin. \\n\n",
    "\n",
    "To learn weights, we start with weight = 0, classify with current weights \\\n",
    "then if wrong we update with w = w + y* $\\dot$ f. \\n\n",
    "\n",
    "With multiple classes we can have a weight vector for each class. \\\n",
    "Whichever class with highest score will win. If we have wrong predictions \\\n",
    "we subtract from weight of the wrong label and add to weight of right \\\n",
    "label. \\n\n",
    "\n",
    "Perceptron properties: \\n\n",
    "\n",
    "1) if the data is linearly separable, then we are guaranteed to \\\n",
    "converge on a set of parameters that offer right decisions. Separability \\\n",
    "= if there exists parameters that get the training data set perfectly \\\n",
    "correct. \\n\n",
    "\n",
    "2) if data is not separable, we can define mistake bound = maximum \\\n",
    "number of mistakes ((k)/(\\delta^2)$). delta^2 = margin = space \\\n",
    "between the two classes. \\n\n",
    "\n",
    "Problems with percetron: \\n\n",
    "\n",
    "1) If data is nonseparable, then weights may thrash, switching between \\\n",
    "non separable boundaries. We can use average perceptrons (averaging weihgt \\\n",
    "vectors over time)\\n\n",
    "\n",
    "2) Mediocre generalization: we can only find a barely separating solution. \\n\n",
    "\n",
    "3) Overtraining: can use held out set to adjust. Overtraining is kind of \\\n",
    "overfitting \\n\n",
    "\n",
    "MIRA: Margin infused Relaxed Algorithm, chooses updata size that fixes the current \\\n",
    "mistake. Scale back updates by a factor of $\\tau = \\frac{w_y - w_{y*} \\dot f + 1}{2f \\dot f}$ \\\n",
    "+1 in the formula helps us generalize. Sometimes we add a constant C to cap maximum \\\n",
    "value of tau. Helps perceptron to converge faster, esp. on noisy data. Cooresponds to \\\n",
    "optimization that assumes non separable data. \\n\n",
    "\n",
    "Support vector machine: decide which separator is better. Find the separator with \\\n",
    "max margin between two different classes (by maximizing margins between support vectors \\\n",
    "and separator). \\n\n",
    "\n",
    "Non linear separators: transform problems into higher dimensions, e.g. x into x^2, \\\n",
    "2-d to 3-d. Sometimes call the kernel trick.\\n\n",
    "\n",
    "Comparing Bayes with Perceptrons: Bayes builds a model, gives prediction probabilities, \\\n",
    "and have strong assumptions about feature independence. Need one pass through data. \\\n",
    "Perceptrons use less assumptions, is mistake-driven, and need multiple passes through data. \\\n",
    "Often more accurate. \"\"\"\n",
    "\n",
    "# Definitions\n",
    "module12_definitions = {\n",
    "\"Consistent hypothesis\" : \"Hypothesis such that each \\\n",
    "xi in training set has h(xi) = yi\",\n",
    "\"Bias\": \"tendency of a hypothesis to deviate from expected value \\\n",
    "when averaged over traininig sets\", \n",
    "\"variance\" : \"amount of change in hypo due to fluctuation in \\\n",
    "training data\", \n",
    "\"decision tree\" :\"function that maps a vector of attribute values \\\n",
    "to a single output value\",\n",
    "\"Regularization\" : \"process of penalizing complex hypothesis\",\n",
    "\"Feature selection\" : \"reduce dimensions by discarding irrelevant attributes \",\n",
    "\"Minimum description length\": \"minimizes number of bits required \\\n",
    "to specify empirical loss and complexity\", \n",
    "\"Grid search\" : \"Try all combinations of hyperparameter values\",\n",
    "\"Random search samples\" : \"alternative of grid search when there are \\\n",
    "too many combinations\", \n",
    "\"Bayesian optimization\" : \"use machine learning to learn \\\n",
    "good hyperparameters\",\n",
    "\"Population based training\" : \"Combination of random search and \\\n",
    "random mutations to find good hyperparameters\",\n",
    "\"decision lists\" :\"a series of tests, each of which is a conjunciton of literals\", \n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "KUc_LgzDvWf-",
    "outputId": "86966438-3549-41bb-e617-e37e63d701d8"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 12: Naive Bayes and Perceptrons\n",
       "## AIMA Chapter 19.1 Forms of Learning\n",
       " Machine learning uses factored representation, representing input as a vector of attribute values. When output is one of finite set of values, then it is a classification problem. If it is a number, then a regression problem. \n",
       "\n",
       "\n",
       "Three types of feedback: supervised learning (input -> label). Unsupervised learning e.g. clustering. Reinforcement learning agents have rewards and punishments. \n",
       "\n",
       "## AIMA Chapter 19.2 Supervised Learning\n",
       " Overfitting models have high variance. Underfitting models have high bias. There is often a bias-variance tradeoff. \n",
       "\n",
       "\n",
       "There is also a tradeoff between how expressive a hypo space is and the complexity of finding a good hypo within that space. \n",
       " \n",
       "\n",
       "## AIMA Chapter 19.3 Learning Decision Trees\n",
       " It is intractable to find a guaranteed smallest decision tree. We can use a greedy divide-and-conquer strategy: testing most important attribtues first then recursively solve smaller subproblems. \n",
       "\n",
       "\n",
       "We can evaluate performa nce using a learning curve, training test size vs proportion of correct answers. Learning curves are also called happy graphs. \n",
       "\n",
       "\n",
       "We can measure importance in terms of information gain or entropy. An RV with only one possible value has entropy 0. A fair coin has 1 bit entropy. \n",
       "\n",
       "\n",
       "Info gain is the expected reduction in entropy: Gain(A) = B(p/(p+n)) - Remainder(A). with B(q) being entropy of boolean RV with prob q. \n",
       "\n",
       "\n",
       "Overfitting is more likely when number of attributes increase, and less liekly when number of training examples increase. \n",
       "\n",
       "\n",
       "Decision tree pruning reduces overfitting, by elim nodes that are not necessarily relevant. A significance test using chi-squared statistic is used to determine if a node is significant. \n",
       "\n",
       "\n",
       "Early stop (combining info gain and chi-squared pruning) has the risk of missing combination of attributes that are informative. \n",
       "\n",
       "\n",
       "To deal with continuous variables, we can use a split point test (inequality test on value). For attributes with many finite values, we can use an information gain ratio, or use equality test. For numerical prediction we need a regression tree rather than a classification tree. \n",
       "\n",
       "\n",
       "Decision tree can suffer from suboptimal policy because of greedy search, expensive runtime when it is too deep, and unstable since adding one example can change the whole tree. Random forest can mitigate these effects. \n",
       "  \n",
       "\n",
       "## AIMA Chapter 19.4 Model Selection and Optimization\n",
       " In machine learning we make the stationarity assumption: that each example has same prior prob distribution, and that it is indp from previous examples. Examples that satisfy these are called indp and identiiccally distributed or i.i.d. \n",
       "\n",
       "\n",
       "Optimal fit = hypo that minimizes the error rate, measured by performance on test set. For hyperparameters (parameters of model class rather htan individual model) we need a dev set. \n",
       "\n",
       "\n",
       "Extreme k-fold cross validation uses k = n, aka leave-one-out cross-validation or LOOCV. Each example will serve both as training and validation data but not at the same time. \n",
       "\n",
       "\n",
       "Model selection algorithm: takes in as argument a learning algo, which in turns take in a hyperparameter (size) as the argument. It measures the error rate for each value of size. It picks the value that minimizes error. \n",
       "\n",
       "\n",
       "Decision tree often has U shaped curves while CNNs often have decreasing curves. \n",
       "\n",
       "\n",
       "A model that exactly fits training data has interpolated the data.\n",
       "\n",
       "\n",
       "An improvement upon error rate is loss function, which allows us to emphasize what errors matter more. Most general formulation L(x, y, ypred) = Utility (result of using y given input x) - Utility (result of using ypred)\n",
       "\n",
       "\n",
       "Best hypothesis is the one with minimum expected generalization loss defined as sum of Loss * probability. Since prob is not known, we use instead empirical loss as an estimate. \n",
       "\n",
       "\n",
       "Four reasons why empirical loss may differ from true function: \n",
       "\n",
       "\n",
       "1) Unrealizability: the hypo space does not contain the true f. \n",
       "\n",
       "\n",
       "2) variance: algo returns different hypo for different sets of examples. \n",
       "\n",
       "\n",
       "3) Noise: f may be nondeterminisitc. \n",
       "\n",
       "\n",
       "4) Computational complexity: we cannot afford to search all possibilities. \n",
       "\n",
       "\n",
       "Recent years use more large-scale learning than small-scall learning.\n",
       "\n",
       " \n",
       "\n",
       "## AIMA Chapter 19.5 The Theory of Learning\n",
       " To figure out the number of samples needed is the question for computational learning theory. \n",
       "\n",
       "\n",
       "Underlying priunciple of CLT is that any hypo consistent with a sufficiently large examples is unlikely to be seriously worng, or probably approximately correct (PAC). \n",
       "\n",
       "\n",
       "PAC theorems assume stationarity. Hypothesis is called approximately correct if error(h) is smaller than or equal to a defined bound epsilon. It is close to the true function or lie inside an epsilon-ball around true functino f. \n",
       "\n",
       "\n",
       "Number of required examples, as function of epsilon and delta (probabilty of H_bad containing at least one consistent hypo) is called the sample complexity of the learning algo.  \n",
       "\n",
       "\n",
       "Dilemma between restricitng hypo space and not eliminating the true function. Three ways to mitigate: 1) bring prior knowledge, 2) have the algo return a simple hypo, instead of complex 3) focus on learnable subset of the hypo space (decision lists). \n",
       "\n",
       "\n",
       "k-DL: decision list with at most k conjunctions. We can show that k-DL is learnable, i.e. can be approximated accurately after training on suffiicent examples. We then find an efficent algo that returns a consistent decision list (e.g. Greedy algo decision list learning, repeatedly find test that agrees exactly with some subset of the training set).\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.6 Linear Regression and Classification\n",
       " Linear regression covers a diff hypo space: the calss of linear functions over continuously valued inputs. \n",
       "\n",
       "\n",
       "We traiditonally use squared-error loss function L2, summed over all training examples. We find weights that minimize loss.\n",
       "\n",
       "\n",
       "For univariate linear regression, the weight space is 2d. The loss functino is convex for every linreg problem with L2 loss functino. \n",
       "\n",
       "\n",
       "Gradient descent is use to minimize loss. While not converged yet, we update each weight using $w = w - \u0007lpha $ (partial derivative of loss over w). \n",
       "\n",
       "\n",
       "alpha s the step size, or rate at which we minimize los. Can be fixed or decay over time. For univariate, loss is quadratic so partial derivative is linear. \n",
       "\n",
       "\n",
       " Batch/deterministic gradient descent: convex loss surface (no local minima), so will eventually converge, but may be slow.\n",
       "\n",
       "\n",
       "Stochastic gradient descent: randomly select number (minibatch) of training examples at each step, May be hard to converge, but often faster. SGD is useful for online setting where data is coming in and stationarity may not hold. \n",
       "\n",
       "\n",
       "For multi-variate regression there may be risk of overfitting, so we can use regularization. L1 regularization has the advantage that it tends to produce a sparse model, pushing many weights to zero. \n",
       "\n",
       "\n",
       "Linear functions can be used for classfication by serving as decision boundaries. or linearseparators. When data are linearly separable, percetron learning rule converges to a perfect linear separator. \n",
       "\n",
       "\n",
       "An improvement upon perceptron is logistic regression, which uses the logistic function logistic(z) = $1 / (1 + e^{-z})$. \n",
       "\n",
       "\n",
       "$h_w (x) = logistic(w \\dot x) = 1 / (1 + e^{-w \\dot x})$. \n",
       "\n",
       "\n",
       "\n",
       "## SLP Chapter 4.1 Naive Bayes Classifiers\n",
       " Text categorization tasks: sentiment analysis, authorship attribution, spam detection, language id. Mostly done through supervised ML. \n",
       "\n",
       "\n",
       "Probabilistic classifier: assign prob to observation belonging to classes. Generative classifier (e.g. naive Bayes): build model of how a class generates input data. Discriminative: learn what features are most useful. Generative classifiers are more popular. \n",
       "\n",
       "\n",
       "\n",
       "Naive bayes model. Compute most probable class given document d, by choosing class with the highest product of prior probability p(c) and likelihood of document p(d|c). \n",
       "\n",
       "\n",
       "Two simplifying assumptions: (1) bag of words assumption that positions do not matter. (2) naive Bayes assumption/ conditional independence assumption that probabilites $P(f_i | c)$ are independent given class c and hence can be naively multiplied. \n",
       "\n",
       "\n",
       "Naive bayes calculations are also done in log space. In log space, predicted class is computed as linear function of input features. THerefore naive Bayes is also a linear classifier.  \n",
       "\n",
       "## SLP Chapter 4.2 Training the Naive Bayes Classifier\n",
       " P(c) = number of docs in the class / total num of docs \n",
       "\n",
       "\n",
       "P(w|c) = count(w,c) / count of all words in c. \n",
       "\n",
       "\n",
       "Problem of zero: one zero cond prob will cause product to be zero. Simple solution is add-one/ Laplace smoothing. \n",
       "\n",
       "\n",
       "P(w|c) = (count(w,c) + 1) / ((count of all words in c) + |size of all words in all classes|)  \n",
       "\n",
       "\n",
       "Solution for unknown words (words in test data but not training data): ignore them, remove from test document. \n",
       "\n",
       "\n",
       "Stop words (very frequent words): either completely ignore or include (does not affect performance).\n",
       " \n",
       "\n",
       "## SLP Chapter 4.3 Worked Example\n",
       " Worked example of naive bayes: drop unknown word in test set, then multiply P(w|c) for each words in test set and P(c). \n",
       "\n",
       "## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n",
       " Binary multinomial naive bayes: remove all duplicate words before concatenating them into big document. Emphasize existence of words to frequency. \n",
       "\n",
       "\n",
       "Dealing with negatives: (1) prepend prefix NOT_ to words. (2) use parsing (more complext). \n",
       "\n",
       "\n",
       "Dealing with insufficient training data: we can draw from sentiment lexicons, lists of words pre-annotated with positive/ negative sentiments. Popular lexicons include General Inquirer, LIWC, opinion lexicon and MPQA Subjectivity Lexicon. One way to use it is to add a feature tha is counted whenever a word in a lexicon occurs. \n",
       "\n",
       "## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n",
       " For tasks like spam detection, we can use predefined likely set of words/phrased as features, coupled with non-linguistic features. \n",
       "\n",
       "\n",
       "For tasks like language id, most effective features are not words but character n-grams like zw and nya, or byte n-grams (which include spaces). Some system (langid.py) uses all possible n-grams of lenghts 1-4 and use feature selection to choose useful ones. \n",
       "\n",
       "## SLP Chapter 4.6 Naive Bayes as a Language Model\n",
       " NB model can be seen as a set of class-specific unigram language models, wth the model of each class (e.g. positive and negative) instantiating a unigram language model.  \n",
       "\n",
       "## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n",
       " Human labels to evaluate models = gold labels. \n",
       "\n",
       "\n",
       "Confusion matrix helps visualize how a model performs. Dimensions = gold labels and system output. \n",
       "\n",
       "\n",
       "Accuracy = (tp + tn) / (tp + fp + tn + fn) \n",
       "\n",
       "\n",
       "Precision = tp / (tp + fp) \n",
       "\n",
       "\n",
       "Precision: percentage of items labeled as positives that are true positives. \n",
       "\n",
       "\n",
       "Recall = tp / (tp + fn) \n",
       "\n",
       "\n",
       "Recall = % of items actually present in the input that is correctly identified by the system. \n",
       "\n",
       "\n",
       "\n",
       "F-measure incorporates both precision and recall. \n",
       "\n",
       "\n",
       "$F_B = ((B^2 + 1)PR) / (B^2 P + R)$\n",
       "\n",
       "\n",
       "B parameter weights the importance of recall and precision, with B > 1 favoriting recall and vice versa. B = 1 is the most freq used, called $F_1$. \n",
       "\n",
       "\n",
       "F-measure is a weighted harmonic mean of precision and recall. It is a conservative metric - closer to the minimum of each number than arthmetic mean. \n",
       "\n",
       "\n",
       "To compute metrics for more than two classes, we can use microaveraging (collect decision for all classes into a single confusion matrix, and compute precision and recall from that table), or macroaveraging (compute performance for each class then average over classes).\n",
       " \n",
       "\n",
       "## SLP Chapter 4.8 Test sets and Cross-validation\n",
       " To be able to use some data for training and still use all for test, we can use cross-validation: randomly choose division of data, into training and test, train classifier, compute error rate on test, rinse and repeat. Do n times and average n runs to get average error rate = n-fold cross-validation. \n",
       "\n",
       "\n",
       "It is common to create fixed training and test set, then do n-fold cross-validation inside training set, but comput error way normally in test set. \n",
       "\n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 4.9 Statistical Significance Testing\n",
       " To compare the performance of two system is to do statistical hypothesis testing. \n",
       "\n",
       "\n",
       "p-value: probability that we would see an observed measure assuming null hypothesis. Statistically significant if p value is below a certain threshold. \n",
       "\n",
       "\n",
       "NLP generally does not use simple parametric tests like t-tests or ANOVAS because of their assumptions. We use non-paremetric tests based on sampling e.g. approximate randomization and bootstrap test.  \n",
       "\n",
       "\n",
       "Bootstrap test: can be applied to any metric.  Bootstrapping = repeatedly drwaining large numbers of smaller samples with replacement from an original larger sample. Assume that sample is representative of population. \n",
       "\n",
       "\n",
       "We create b test sets from boostrapping, then compute statistics on how often A has accidental advantage. E.g. we can count how many times the delta exceeds the expected zero value by delta(x) or more. \n",
       "\n",
       "\n",
       "## SLP Chapter 4.10 Avoiding Harms in Classification\n",
       " ML systems replicate and even amplify biases in trainig data or labels. E.g. representational harms: harms by a system that demeans a social group, e.g. assigning lower sentiment scores. E.g. toxicity detection, which =can result in censorship.\n",
       "\n",
       "\n",
       "To deal with harms we can release a model card, including information such as training algo and parameters, sources, processing, intended use, and performance over different demographics. \n",
       "\n",
       "## Module 12 Lecture Notes\n",
       " ML gives us a way to acquire a model from data, learning paremeters (e.g. probabilities), structure (e.g. Bayens net graphs), and hidden concepts (e.g. clustering). \n",
       "\n",
       "\n",
       "In naive bayes model and perceptron our task is classification. Naive bayes is model-based approach: we build a model with random variables being the labels and features, then instantiate observed features, and query for the distribution of the label conditioned on the features. Major challenges are deciding the structure of the BN and deciding how we should learn the paremeters. \n",
       "\n",
       "\n",
       "Naive Bayes model for digits: assume all features are independent effects of the label. Feature values are on or off depending on whether intensity > 0.5. Every input will be represented with a feature vector, and we predict P(Y|product(features)) = P(Y) * product (P(feature | Y)). (with independence assumption) \n",
       "\n",
       "\n",
       "Full joint distribution of the left side has size |Y| |F|^n values,  with |F| being possible values for features and n being number of features. With independence assumption, the right side only has n |F| |Y| parameters, meaning that it is linear in number of features. \n",
       "\n",
       "\n",
       "Steps for inference in Naive Bayes: (1) Get joint probability of label and evidence for each label P(y1,f1,...,fn). (2) sum to get probability of evidence. (3) Divide step (1) by step (2) to normalize. \n",
       "\n",
       "\n",
       "Data we need: (1) P(Y): prior of labels, (2) P(Fi|Y) for each feature, i.e. evidence variable.These probabilities are called parameters / theta. \n",
       "\n",
       "\n",
       "Data in ML: \n",
       "\n",
       "\n",
       "(1) Training set\n",
       "\n",
       "\n",
       "(2) Held-out data/ development set: optimize hyperparameters \n",
       "\n",
       "\n",
       "(3) Test set \n",
       "\n",
       "\n",
       "A typical experimentation cycle: learning paremeters on training set -> tune hyperparemeters on holdout set -> compute accuracy of test set (fraction of correct predictions). \n",
       "\n",
       "\n",
       "Overfitting: fitting the training data too closely, but cannot generalize. E.g. if we have not seen a word (0 probability) we can get positive infinities In response we need to smooth or regularize the estimates. \n",
       "\n",
       "\n",
       "Ways to estimate parameters: (1) ask a human, (2) learn from training data (empirical rate). Empricial rate = estimate (of the value of the feature) that maximizes the likelihood of the data. \n",
       "\n",
       "\n",
       "Laplace smoothing: pretend we saw everything one more time than we did. Can derive the estimate with Dirichlet priors. If we use k instead of 1 for Laplace smoothing, k is called the strength of the prior. \n",
       "\n",
       "\n",
       "Laplace smoothing does not do well for P(X|Y) when |X| or |Y| is very large. Linear interpolation: $P_{lin}(x|y) = alpha P(x|y) + (1 - alpha) P(x)$. \n",
       "\n",
       "\n",
       "Hyperparameters: amount/ type of smoothing, k, alpha. We should not tune hyper-paremeters on training data since that way they would not generalize. \n",
       "\n",
       "\n",
       "Baseline: simple straw man procedures that help determine how hard the task is and what is a good accuracy. E.g. labeling everything as spam for spam filters.\n",
       "\n",
       "\n",
       "Perceptron: inputs * weights -> apply activation function and compare to threshold -> output. Instead of the threshold we can use a bias b = -threshold, so that output = 0 if w $\\dot$ x + b $\\leq$ 0. \n",
       "\n",
       "\n",
       "In graphical represenation, classification decision will be represented by a decision boundary (line, plane, or hyperplane), orthogonal to the weight vector. If input vector has less than 90 degree distance to weight vector than positive, else negative. A bias would move the decision boundary away from the origin. \n",
       "\n",
       "\n",
       "To learn weights, we start with weight = 0, classify with current weights then if wrong we update with w = w + y* $\\dot$ f. \n",
       "\n",
       "\n",
       "With multiple classes we can have a weight vector for each class. Whichever class with highest score will win. If we have wrong predictions we subtract from weight of the wrong label and add to weight of right label. \n",
       "\n",
       "\n",
       "Perceptron properties: \n",
       "\n",
       "\n",
       "1) if the data is linearly separable, then we are guaranteed to converge on a set of parameters that offer right decisions. Separability = if there exists parameters that get the training data set perfectly correct. \n",
       "\n",
       "\n",
       "2) if data is not separable, we can define mistake bound = maximum number of mistakes ((k)/(\\delta^2)$). delta^2 = margin = space between the two classes. \n",
       "\n",
       "\n",
       "Problems with percetron: \n",
       "\n",
       "\n",
       "1) If data is nonseparable, then weights may thrash, switching between non separable boundaries. We can use average perceptrons (averaging weihgt vectors over time)\n",
       "\n",
       "\n",
       "2) Mediocre generalization: we can only find a barely separating solution. \n",
       "\n",
       "\n",
       "3) Overtraining: can use held out set to adjust. Overtraining is kind of overfitting \n",
       "\n",
       "\n",
       "MIRA: Margin infused Relaxed Algorithm, chooses updata size that fixes the current mistake. Scale back updates by a factor of $\tau = \f",
       "rac{w_y - w_{y*} \\dot f + 1}{2f \\dot f}$ +1 in the formula helps us generalize. Sometimes we add a constant C to cap maximum value of tau. Helps perceptron to converge faster, esp. on noisy data. Cooresponds to optimization that assumes non separable data. \n",
       "\n",
       "\n",
       "Support vector machine: decide which separator is better. Find the separator with max margin between two different classes (by maximizing margins between support vectors and separator). \n",
       "\n",
       "\n",
       "Non linear separators: transform problems into higher dimensions, e.g. x into x^2, 2-d to 3-d. Sometimes call the kernel trick.\n",
       "\n",
       "\n",
       "Comparing Bayes with Perceptrons: Bayes builds a model, gives prediction probabilities, and have strong assumptions about feature independence. Need one pass through data. Perceptrons use less assumptions, is mistake-driven, and need multiple passes through data. Often more accurate. \n",
       "\n",
       "## Module 12 Definitions\n",
       "* **Consistent hypothesis** - Hypothesis such that each xi in training set has h(xi) = yi\n",
       "* **Bias** - tendency of a hypothesis to deviate from expected value when averaged over traininig sets\n",
       "* **Variance** - amount of change in hypo due to fluctuation in training data\n",
       "* **Decision tree** - function that maps a vector of attribute values to a single output value\n",
       "* **Regularization** - process of penalizing complex hypothesis\n",
       "* **Feature selection** - reduce dimensions by discarding irrelevant attributes \n",
       "* **Minimum description length** - minimizes number of bits required to specify empirical loss and complexity\n",
       "* **Grid search** - Try all combinations of hyperparameter values\n",
       "* **Random search samples** - alternative of grid search when there are too many combinations\n",
       "* **Bayesian optimization** - use machine learning to learn good hyperparameters\n",
       "* **Population based training** - Combination of random search and random mutations to find good hyperparameters\n",
       "* **Decision lists** - a series of tests, each of which is a conjunciton of literals\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module12_sections = [\n",
    "  (\"AIMA Chapter 19.1 Forms of Learning\", chapter19_1),\n",
    "  (\"AIMA Chapter 19.2 Supervised Learning\", chapter19_2),\n",
    "  (\"AIMA Chapter 19.3 Learning Decision Trees\", chapter19_3),\n",
    "  (\"AIMA Chapter 19.4 Model Selection and Optimization\", chapter19_4),\n",
    "  (\"AIMA Chapter 19.5 The Theory of Learning\", chapter19_5),\n",
    "  (\"AIMA Chapter 19.6 Linear Regression and Classification\", chapter19_6),\n",
    "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
    "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
    "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
    "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
    "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
    "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
    "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
    "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
    "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
    "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
    "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
    "]\n",
    "\n",
    "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
    "                         module12_sections, \n",
    "                         module12_definitions)\n",
    "display(Markdown(module12_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qVSQhTVSyYYO"
   },
   "source": [
    "# Module 13 - Neural Networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "Wm4mfW82yaRH"
   },
   "outputs": [],
   "source": [
    "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
    "\n",
    "slp_chapter5_1 = \"\"\" Log reg is a discriminative classifer since it identifies \\\n",
    "features that help identify classes. It directly compute P(c | d) rather \\\n",
    "than calculate it by multiplying likelihood and prior. \\n\n",
    "\n",
    "Components of ML system for classification: 1) a feature representation of the input \\\n",
    "2) a classification function that computes estimated class, 3) \\\n",
    "an objective function for learning (e.g. cross entropy loss), \\\n",
    "4) algo for optimizing objective function e.g. SGD. \\n\n",
    "\n",
    "Define z = w dot x + b. We apply simoid function on z to make it \\\n",
    "a value between 0 and 1. Sigmoid function squishes outliers to 0 or 1 \\\n",
    "is differentiable, and has defined range. \\n\n",
    "\n",
    "Advantage of logistic over naive bayes: naive bayes have overly \\\n",
    "strong cond indp assumptions. When there are many correlated features \\\n",
    "log reg works much better. However, on very small datasets naive bayes \\\n",
    "can work well, and it is very fast to train (no optimization).\n",
    "\"\"\"\n",
    "\n",
    "slp_chapter5_2 = \"\"\" We need a loss function that makes correct class \\\n",
    "labels more likely - a conditional maximum likelihood estimation. \\\n",
    "We want to maximize log probabilty of true y labels in the training data, \\\n",
    "so we reduce negative log likliehood loss = cross entropy loss function. \\n\n",
    " \"\"\"\n",
    "slp_chapter5_3 = \"\"\" Log reg has convex loss function with just one minimum. For multi layer \\\n",
    "neural networks we have non convex loss function. \"\"\"\n",
    "\n",
    "\n",
    "slp_chapter5_4 = \"\"\"Gradient: a vector that expresses the directional components of the sharpest \\\n",
    "slope along each of the N dimensions. \\n\n",
    "\n",
    "Gradient for log regression: $(\\sigmoid (w \\dot x + b) - y) + x_j$\\n\n",
    "\n",
    "Stochastic gradient descent: online algorithm, compute gradient after \\\n",
    "each training sample. 1) compute estimated output, 2) compute loss, \\\n",
    "3) derivative, 4) update. Stochastic because one random example is chosen \\\n",
    "at a time. \\n\n",
    "\n",
    "eta/ learning rate is a hyperparameter. We often start with a high one \\\n",
    "then slowly increase it, i.e. a function of iteration k. \\n\n",
    "\n",
    "Batch training, compute gradient over entire data set. Can be costly in time \\n\n",
    "\n",
    "Mini-batch: train m examples less than whole dataset. Cost function \\\n",
    "for mini-batch is average loss for each example. \"\"\"\n",
    "\n",
    "slp_chapter5_5 = \"\"\" Regularization term R(theta): added to objective function \\\n",
    "to avoid overfitting. \\n\n",
    "\n",
    "L2 (Ridge) regularization: quadratic function of weight values. L2 Norm = Euclidean distance \\\n",
    "of vector from origin. \\n\n",
    "\n",
    "L1 (Lasso) regularization: linear function of weights. L1 norm = Manhattan distance \\n\n",
    "\n",
    "L2 easier to optimize because of simple derivative. L1's derivative \\\n",
    "is non-continuous at zero. \\n\n",
    "\n",
    "L1 regularization leads to fewer features. \\n\n",
    "\n",
    "L1 and L2 have Bayesian interpretations as constraints on how weights should look.\n",
    "\"\"\"\n",
    "\n",
    "slp_chapter5_6 = \"\"\" For classification with more than 2 classes we do \\\n",
    "multinomial logistic regression or softmax regression. \\n\n",
    "\n",
    "Softmax is a generalization of sigmoid. Takes a vector and maps them to a probability \\\n",
    "distribution. \\n\n",
    "\n",
    "Softmax(z_i) = (exp(z_i) / $\\sum^k_{j=1} exp(z_j))$, for $1 \\leq i \\leq k$\\n\n",
    "\n",
    "Cross-entropy loss is the log of output prob corresponding to the correct class \\\n",
    "or the negative log likelihood loss. \\n\n",
    "\n",
    "L(ypred, y) = $- \\log (exp(w_k \\dot x + b_k) / \\sum^k_{j=1} exp(w_j \\dot x + b_j)) $, \\\n",
    "with k being the correct class.\n",
    " \"\"\"\n",
    "\n",
    "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
    "\n",
    "slp_chapter7_1 = \"\"\" Neural network differs from logistic regression in: 1) \\\n",
    "more powerful, minimum neural network with 1 hidden layer can learn any function \\\n",
    "2) do not need hand-derived features. \\n\n",
    "\n",
    "Neural units apply a non-linear function to z = w \\dot x + b. \\\n",
    "The ouput is called an activation value. Sigmoid is one candidate but \\\n",
    "not common. Tanh is usually better: \\n\n",
    "\n",
    "y = ($e^z - e^{-z}$)/ ($e^z + e^{-z}$) range from -1 to +1 \\n\n",
    "\n",
    "ReLu: y = max(x, 0)\\n\n",
    "\n",
    "tanh function is smoothly differntiable and map outlier values towards \\\n",
    "mean. ReLU has nice properties since it is close to linear. In sigmoid and tanh,\n",
    "high values of z results in saturated values of y (extremely close to 1 with \\\n",
    "derivatives close ot 0). \\n\n",
    "\n",
    "Rectifiers escape the prblem of vanishing gradient, which occurs when \\\n",
    "the gradient gets too close to 0, resulting in low error signals.\n",
    "\"\"\"\n",
    "\n",
    "slp_chapter7_2 = \"\"\" Proof that we need multi-layer networks: \\\n",
    "Minksy and Papert show that a single neural unit cannot compute \\\n",
    "simple functions of the input. E.g. a perceptron cannot compute XOR. \\n\n",
    "\n",
    "Intuition lies in fact that perceptron is a linear classifer, with the line \\\n",
    "acting as a decision boundary. XOR is not linearly separable. \\n\n",
    "\n",
    "In contrasta two layer network with 3 ReLu units can compute XOR. \n",
    " \"\"\"\n",
    "\n",
    "slp_chapter7_3 = \"\"\" Feedforward network: multilayer network where \\\n",
    "units are connected with no cycles. No outputs are passed back. Sometimes \\\n",
    "called multi-layer perceptrons, but a misnomer since they are not linear \\n\n",
    "\n",
    "Fully-connected network has unit in each layer takes as inputs the outputs \\\n",
    "from all units in previous layer, so that there is link between \\\n",
    "every pair of units in two adjacent layers \\n\n",
    "\n",
    "We can represent weights of a layer by a single matrix so that hidden layers \\\n",
    "can be computed with simple atrix operations. \\n\n",
    "\n",
    "We can apply the softmax function to normalize a vector of real values. \\n\n",
    "\n",
    "We can think of neural network as building a vector h (a hidden layer representation \\\n",
    "of the input) and then running standard logistic regression on the features in h. \\n\n",
    "\n",
    "WE can add a dummy node a_0 with value always = 1 at each layer as a replacement \\\n",
    "of the bias term. \n",
    " \"\"\"\n",
    "\n",
    "slp_chapter7_4 = \"\"\" We use error backpropogation or backprop t o\\\n",
    "compute the gradient of neural networks. It is the same as a more general \\\n",
    "procedure called backward differentiation, which depends on computation graphs. \\n\n",
    "\n",
    "Computation graphs: each node is an operation and each edge shows \\\n",
    "output to input relationship. \\n\n",
    "\n",
    "In backward pass we compute partial derivatives along each edge of computation \\\n",
    "graph from right to left. \\n\n",
    "\n",
    "Optimization for neural networks is non-convex. We need to initialize \\\n",
    "the weights with small random numbers, and (optionally) noramlize input values \\\n",
    "to have 0 mean and unit variance. \\n\n",
    "\n",
    "We can use dropout (randomly dropping some units and their connections) for \\\n",
    "regularization against overfitting.\\n\n",
    "\n",
    "PyTorch and TensorFlow build neural networks using computation graph formalisms that \\\n",
    "enable gradient computation and parallelization over vector-based GPUs. \"\"\"\n",
    "\n",
    "slp_chapter7_5 = \"\"\" Neural language models can have much higher \\\n",
    "accuracy than n-gram LMs, don't need smoothing, can handle longer histories \\\n",
    "and can generalize over contexts of similar words. However, they are slower \\\n",
    "to train. \\n\n",
    "\n",
    "Feedforward neural LM takes in a representaiton of previous words \\\n",
    "and output probabilty distributino of next words. \\n\n",
    "\n",
    "Prior context is represented by embeddings. We use a moving window \\\n",
    "with embedding vector representing each of k past words. We concatenate \\\n",
    "the vectors to product the input x.\\n\n",
    "\n",
    "We can train network with ready-made embeddings through pretraining, \\\n",
    "or learn embeddings simultaneously. \\n\n",
    "\n",
    "We use an embedding weight matrix (size $d * |V|$) with a column for each word, \\\n",
    "each a column vector of d dimensions. \\n\n",
    "\n",
    "Three step forward pass: 1) select three embeddings from E, multiply the 3 one\\hot ]\\\n",
    "vectors created by the embedding matrix, to form the projection layer. ]\n",
    "2) Multiply by W and add b and pass through the ReLu activation function to \\\n",
    "get hidden layer h. 3) Multiply h by U, 4) apply softmax. \\n\n",
    "\n",
    "Alternatively:\\n\n",
    "1) $e = (Ex_1, Ex_2, ..., Ex)$ \\n\n",
    "2) $h = \\sigmoid(We+b)$\\n\n",
    "3) z = Uh\\n\n",
    "4) ypred = softmax(z)\\n\n",
    "\n",
    "  \"\"\"\n",
    "\n",
    "# Lecture notes \n",
    "module13_lecture_notes = \"\"\" ML classifiers require a training corpus \\\n",
    "of M observations with (xi, yi) input/output pairs. Also need \\\n",
    "1) a feature representation of the input, 2) a classification function \\\n",
    "that computes estimated calss via p(y|x), 3) an objective function \\\n",
    "for learning, usually with minimizing error on training examples, and 4) \\\n",
    "an algorithm for optimizing the objective function (e.g. gradient descent). \\n\n",
    "\n",
    "It used to be the case that we concern ourselves with coming up with \\\n",
    "useful features, but now we are more concerned with choosing the right \\\n",
    "structure for the neural network. \\n\n",
    "\n",
    "Problem of sentiment classification. Logistic regression aims to learn \\\n",
    "vector of weights and bias term such that $z = \\sum_i w_i x_i + b$. With logistic \\\n",
    "regression we modify the range of z so that it can be interpreted as a probability. \\\n",
    "Apply sigmoid function $y = 1 / (1 + e ^{-z})$. Negative values for z = below 0.5 \\\n",
    "for y and vice versa. $P(y=1) = sigmoid(w \\dot x + b)$. P(y=0) = 1 - P(y=1). \\n\n",
    "\n",
    "Making decisions: decision = 1 if P(y=1|x) > 0.5 else 0. .5 is the decision boundary. \\n\n",
    "\n",
    "For learning the weights we need: (1) an objective function (e.g. cross-entropy \\\n",
    "loss). (2) an algorithm for optimizing objective function (e.g. gradient \\\n",
    "descent).\\n\n",
    "\n",
    "Loss function helps determine how close our output is to the actual \\\n",
    "output. E.g. mean squared error =  $L(\\hat{y}, y) = (\\hat{y}-y)^2 / 2 $. Not \\\n",
    "a good function for optimizing. \\n\n",
    "\n",
    "Conditional maximum likelihood estimation: choose w, b such that we maximize probabilities \\\n",
    "of correct labels in training data. Resulting loss function = cross entropy loss = \\\n",
    "negative log likelihood loss (to be minimized). \\n\n",
    "\n",
    "Derivation: $p(y|x) = \\hat{y}^y (1 - \\hat{y})^{1 - y}$. i.e. when y is 1, \\\n",
    "then p(y|x) = $\\hat{y}$ and when y = 0, p(y|x) = $ 1- \\hat{y}$. Then log both \\\n",
    "sides to get $\\log p(y|x) = y \\log \\hat{y} + (1 - y) \\log (1 - \\hat{y})$. Put \\\n",
    "negative signs on both sides to get $L = -[y \\log \\hat{y} + (1 - y) \\log (1 - \\hat{y})]$ \\n\n",
    "\n",
    "Correct prediction results in lower CE loss, while incorrect ones lead to \\\n",
    "higher CE losses. \\n\n",
    "\n",
    "Loss on all training examples = $- \\sum_{i=1}^m L_{ce} (\\hat{y}^i | y^i)$\\n\n",
    "\n",
    "Define theta = stuff we want to optimize (w, b). We compute $\\hat{\\theta}$ = \\\n",
    "$argmin_{\\theta} 1/m \\sum_{i=1}^m L_{ce} (y^i, x^i; \\theta) $. \\n\n",
    "\n",
    "Gradient descent is an algorithm to find minimum of a function by deciding \\\n",
    "which direction the function's slope is rising the most steeply, and moving in the \\\n",
    "opposite direction. (which way is down) \\n \n",
    "\n",
    "Loss function is guaranteed convex for logisitc regression -> so one min \\\n",
    "point. If we have small enough step size we are guaranteed \\\n",
    "to find minimum, else we may jum around. In non-convex problem we may be stuck in \\n\n",
    "\n",
    "Size of step = learning rate = eta. Update paremeter by: \\\n",
    "$w^{t+1} = w^t - \\eta d/dw f(loss, w)$. If derivative/slope is negative we \\\n",
    "increase weight, and vice versa.\\n\n",
    "\n",
    "Extending 2-D gradient descent to higher dimensions: for each weight vector, we \\\n",
    "take $d/dw L(f(x;\\theta), y)$. Collection of these derivatives is called \\\n",
    "$\\delta_{\\theta}$. We update $\\theta^{t+1} = \\theta^t - \\eta \\delta_{\\theta}$. \\n\n",
    "\n",
    "The derivative of loss function wrt each $w_i$ = $(\\hat{y} - y)$ $\\dot$ x. \\\n",
    "We can obtain the gradient for multiple data points by summing up individual ones. \\n\n",
    "\n",
    "Stochastic gradient descent: take a random training instance, then update \\\n",
    "based on it. Updates can fluctuate. \\n \n",
    "\n",
    "Batch training: Computes gradient over entire dataset. Takes long time \\\n",
    "but give better estimate of which direction to move. \\n\n",
    "\n",
    "We usually use mini batch training (batch size picked empirically, e.g. \\\n",
    "what we can fit into GPU card)\\n\n",
    "\n",
    "Problems: (1) overfitting results in poor generalization. (2) If features \\\n",
    "only appear in one class it gets high weights, we can use regularization \\\n",
    "to penalize large weights (add regularization term to loss function). \\n\n",
    "\n",
    "For multi-class classification, use multinomial logistic regression. \\\n",
    "Want to know the probability for each class. \\\n",
    "Can use generalization of sigmoid funciton called softmax function. \\\n",
    "Often the final step in neural network. Softmax takes input vectors \\\n",
    "and output vector of valeus normalized into probabilities. \\n\n",
    "\n",
    "(Feed forward) Neural network with many neurons can approximate many functions. \\\n",
    "Deep learning = neural network with many layers. \\n\n",
    "\n",
    "Perceptron has step function (0 or 1). Problem is that it is difficult to update \\\n",
    "weight vector - small change may cause huge swing btw 0 and 1. With sigmoid we \\\n",
    "have much smoother function: small changes in weight -> small changes in output. \\n\n",
    "\n",
    "Activation functions: sigmoid, ReLU, tanh. Neural network units = \\ \n",
    "perceptron + activation function\\n\n",
    "\n",
    "Totally connected: each node in next layer is connected to all inputs from \\\n",
    "previous layer.\\n\n",
    "\n",
    "Neural network requires different way of computing gradient: we need back \\\n",
    "propagation. Related to the general procedure called backward differentiation, \\\n",
    "which depends on computation graphs (representation of the process of computing \\\n",
    "a mathematical expression). Use chain rule to compute derivation of loss function L \\\n",
    "with respect to a, b, and c. \\n\n",
    "\n",
    "Chain rule: $\\frac{df}{dx} = \\frac{df}{da} \\frac{da}{dx}$. Get derivatives \\\n",
    "on each node of the graph, then multiply nodes along a path to get a \\\n",
    "destination derivative. \\n\n",
    "\n",
    " \"\"\"\n",
    "\n",
    "# Definitions\n",
    "module13_definitions = {\n",
    "\"Period disambiguation\" : \"decidingif the period is end of sentence \\\n",
    "or part of a word\",\n",
    "\"sigmoid\" : \"$1 / 1 + e ^ {-z}$\",\n",
    "\"Feature templates\" : \"ambstract specs of features that allow automatic \\\n",
    "creation of features\",     \n",
    "\"representation learning\" : \"ways to learn features automatically \\\n",
    "in an unsupervised way\",\n",
    "\"McCulloch-Pitts neuron\" : \"a simplified model of the neuron \\\n",
    "as a computing element describable in propositional logic\",\n",
    "\"Hard classification task\" : \"task where only one class is correct\",\n",
    "\"Pretraining\" : \"Relying on another algorithm to learn embedding \\\n",
    "representation for input words\"\n",
    "\n",
    "\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "kKWbCsbfxL3R",
    "outputId": "d482e871-5fc4-4294-8d3b-e27be7dddefd"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 13: Neural Networks\n",
       "## SLP Chapter 5.1 Classification: the sigmoid\n",
       " Log reg is a discriminative classifer since it identifies features that help identify classes. It directly compute P(c | d) rather than calculate it by multiplying likelihood and prior. \n",
       "\n",
       "\n",
       "Components of ML system for classification: 1) a feature representation of the input 2) a classification function that computes estimated class, 3) an objective function for learning (e.g. cross entropy loss), 4) algo for optimizing objective function e.g. SGD. \n",
       "\n",
       "\n",
       "Define z = w dot x + b. We apply simoid function on z to make it a value between 0 and 1. Sigmoid function squishes outliers to 0 or 1 is differentiable, and has defined range. \n",
       "\n",
       "\n",
       "Advantage of logistic over naive bayes: naive bayes have overly strong cond indp assumptions. When there are many correlated features log reg works much better. However, on very small datasets naive bayes can work well, and it is very fast to train (no optimization).\n",
       "\n",
       "\n",
       "## SLP Chapter 5.2 Learning in Logistic Regression\n",
       " We need a loss function that makes correct class labels more likely - a conditional maximum likelihood estimation. We want to maximize log probabilty of true y labels in the training data, so we reduce negative log likliehood loss = cross entropy loss function. \n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 5.3 The cross-entropy loss function\n",
       " Log reg has convex loss function with just one minimum. For multi layer neural networks we have non convex loss function. \n",
       "\n",
       "## SLP Chapter 5.4 Gradient Descent\n",
       "Gradient: a vector that expresses the directional components of the sharpest slope along each of the N dimensions. \n",
       "\n",
       "\n",
       "Gradient for log regression: $(\\sigmoid (w \\dot x + b) - y) + x_j$\n",
       "\n",
       "\n",
       "Stochastic gradient descent: online algorithm, compute gradient after each training sample. 1) compute estimated output, 2) compute loss, 3) derivative, 4) update. Stochastic because one random example is chosen at a time. \n",
       "\n",
       "\n",
       "eta/ learning rate is a hyperparameter. We often start with a high one then slowly increase it, i.e. a function of iteration k. \n",
       "\n",
       "\n",
       "Batch training, compute gradient over entire data set. Can be costly in time \n",
       "\n",
       "\n",
       "Mini-batch: train m examples less than whole dataset. Cost function for mini-batch is average loss for each example. \n",
       "\n",
       "## SLP Chapter 5.5 Regularization\n",
       " Regularization term R(theta): added to objective function to avoid overfitting. \n",
       "\n",
       "\n",
       "L2 (Ridge) regularization: quadratic function of weight values. L2 Norm = Euclidean distance of vector from origin. \n",
       "\n",
       "\n",
       "L1 (Lasso) regularization: linear function of weights. L1 norm = Manhattan distance \n",
       "\n",
       "\n",
       "L2 easier to optimize because of simple derivative. L1's derivative is non-continuous at zero. \n",
       "\n",
       "\n",
       "L1 regularization leads to fewer features. \n",
       "\n",
       "\n",
       "L1 and L2 have Bayesian interpretations as constraints on how weights should look.\n",
       "\n",
       "\n",
       "## SLP Chapter 5.6 Multinomial logistic regression\n",
       " For classification with more than 2 classes we do multinomial logistic regression or softmax regression. \n",
       "\n",
       "\n",
       "Softmax is a generalization of sigmoid. Takes a vector and maps them to a probability distribution. \n",
       "\n",
       "\n",
       "Softmax(z_i) = (exp(z_i) / $\\sum^k_{j=1} exp(z_j))$, for $1 \\leq i \\leq k$\n",
       "\n",
       "\n",
       "Cross-entropy loss is the log of output prob corresponding to the correct class or the negative log likelihood loss. \n",
       "\n",
       "\n",
       "L(ypred, y) = $- \\log (exp(w_k \\dot x + b_k) / \\sum^k_{j=1} exp(w_j \\dot x + b_j)) $, with k being the correct class.\n",
       " \n",
       "\n",
       "## SLP Chapter 7.1 Units\n",
       " Neural network differs from logistic regression in: 1) more powerful, minimum neural network with 1 hidden layer can learn any function 2) do not need hand-derived features. \n",
       "\n",
       "\n",
       "Neural units apply a non-linear function to z = w \\dot x + b. The ouput is called an activation value. Sigmoid is one candidate but not common. Tanh is usually better: \n",
       "\n",
       "\n",
       "y = ($e^z - e^{-z}$)/ ($e^z + e^{-z}$) range from -1 to +1 \n",
       "\n",
       "\n",
       "ReLu: y = max(x, 0)\n",
       "\n",
       "\n",
       "tanh function is smoothly differntiable and map outlier values towards mean. ReLU has nice properties since it is close to linear. In sigmoid and tanh,\n",
       "high values of z results in saturated values of y (extremely close to 1 with derivatives close ot 0). \n",
       "\n",
       "\n",
       "Rectifiers escape the prblem of vanishing gradient, which occurs when the gradient gets too close to 0, resulting in low error signals.\n",
       "\n",
       "\n",
       "## SLP Chapter 7.2 The XOR problem\n",
       " Proof that we need multi-layer networks: Minksy and Papert show that a single neural unit cannot compute simple functions of the input. E.g. a perceptron cannot compute XOR. \n",
       "\n",
       "\n",
       "Intuition lies in fact that perceptron is a linear classifer, with the line acting as a decision boundary. XOR is not linearly separable. \n",
       "\n",
       "\n",
       "In contrasta two layer network with 3 ReLu units can compute XOR. \n",
       " \n",
       "\n",
       "## SLP Chapter 7.3 Feedforward Neural Networks\n",
       " Feedforward network: multilayer network where units are connected with no cycles. No outputs are passed back. Sometimes called multi-layer perceptrons, but a misnomer since they are not linear \n",
       "\n",
       "\n",
       "Fully-connected network has unit in each layer takes as inputs the outputs from all units in previous layer, so that there is link between every pair of units in two adjacent layers \n",
       "\n",
       "\n",
       "We can represent weights of a layer by a single matrix so that hidden layers can be computed with simple atrix operations. \n",
       "\n",
       "\n",
       "We can apply the softmax function to normalize a vector of real values. \n",
       "\n",
       "\n",
       "We can think of neural network as building a vector h (a hidden layer representation of the input) and then running standard logistic regression on the features in h. \n",
       "\n",
       "\n",
       "WE can add a dummy node a_0 with value always = 1 at each layer as a replacement of the bias term. \n",
       " \n",
       "\n",
       "## SLP Chapter 7.4 Training Neural Nets\n",
       " We use error backpropogation or backprop t ocompute the gradient of neural networks. It is the same as a more general procedure called backward differentiation, which depends on computation graphs. \n",
       "\n",
       "\n",
       "Computation graphs: each node is an operation and each edge shows output to input relationship. \n",
       "\n",
       "\n",
       "In backward pass we compute partial derivatives along each edge of computation graph from right to left. \n",
       "\n",
       "\n",
       "Optimization for neural networks is non-convex. We need to initialize the weights with small random numbers, and (optionally) noramlize input values to have 0 mean and unit variance. \n",
       "\n",
       "\n",
       "We can use dropout (randomly dropping some units and their connections) for regularization against overfitting.\n",
       "\n",
       "\n",
       "PyTorch and TensorFlow build neural networks using computation graph formalisms that enable gradient computation and parallelization over vector-based GPUs. \n",
       "\n",
       "## SLP Chapter 7.5 Neural Language Models\n",
       " Neural language models can have much higher accuracy than n-gram LMs, don't need smoothing, can handle longer histories and can generalize over contexts of similar words. However, they are slower to train. \n",
       "\n",
       "\n",
       "Feedforward neural LM takes in a representaiton of previous words and output probabilty distributino of next words. \n",
       "\n",
       "\n",
       "Prior context is represented by embeddings. We use a moving window with embedding vector representing each of k past words. We concatenate the vectors to product the input x.\n",
       "\n",
       "\n",
       "We can train network with ready-made embeddings through pretraining, or learn embeddings simultaneously. \n",
       "\n",
       "\n",
       "We use an embedding weight matrix (size $d * |V|$) with a column for each word, each a column vector of d dimensions. \n",
       "\n",
       "\n",
       "Three step forward pass: 1) select three embeddings from E, multiply the 3 one\\hot ]vectors created by the embedding matrix, to form the projection layer. ]\n",
       "2) Multiply by W and add b and pass through the ReLu activation function to get hidden layer h. 3) Multiply h by U, 4) apply softmax. \n",
       "\n",
       "\n",
       "Alternatively:\n",
       "\n",
       "1) $e = (Ex_1, Ex_2, ..., Ex)$ \n",
       "\n",
       "2) $h = \\sigmoid(We+b)$\n",
       "\n",
       "3) z = Uh\n",
       "\n",
       "4) ypred = softmax(z)\n",
       "\n",
       "\n",
       "  \n",
       "\n",
       "## Module 13 Lecture Notes\n",
       " ML classifiers require a training corpus of M observations with (xi, yi) input/output pairs. Also need 1) a feature representation of the input, 2) a classification function that computes estimated calss via p(y|x), 3) an objective function for learning, usually with minimizing error on training examples, and 4) an algorithm for optimizing the objective function (e.g. gradient descent). \n",
       "\n",
       "\n",
       "It used to be the case that we concern ourselves with coming up with useful features, but now we are more concerned with choosing the right structure for the neural network. \n",
       "\n",
       "\n",
       "Problem of sentiment classification. Logistic regression aims to learn vector of weights and bias term such that $z = \\sum_i w_i x_i + b$. With logistic regression we modify the range of z so that it can be interpreted as a probability. Apply sigmoid function $y = 1 / (1 + e ^{-z})$. Negative values for z = below 0.5 for y and vice versa. $P(y=1) = sigmoid(w \\dot x + b)$. P(y=0) = 1 - P(y=1). \n",
       "\n",
       "\n",
       "Making decisions: decision = 1 if P(y=1|x) > 0.5 else 0. .5 is the decision boundary. \n",
       "\n",
       "\n",
       "For learning the weights we need: (1) an objective function (e.g. cross-entropy loss). (2) an algorithm for optimizing objective function (e.g. gradient descent).\n",
       "\n",
       "\n",
       "Loss function helps determine how close our output is to the actual output. E.g. mean squared error =  $L(\\hat{y}, y) = (\\hat{y}-y)^2 / 2 $. Not a good function for optimizing. \n",
       "\n",
       "\n",
       "Conditional maximum likelihood estimation: choose w, b such that we maximize probabilities of correct labels in training data. Resulting loss function = cross entropy loss = negative log likelihood loss (to be minimized). \n",
       "\n",
       "\n",
       "Derivation: $p(y|x) = \\hat{y}^y (1 - \\hat{y})^{1 - y}$. i.e. when y is 1, then p(y|x) = $\\hat{y}$ and when y = 0, p(y|x) = $ 1- \\hat{y}$. Then log both sides to get $\\log p(y|x) = y \\log \\hat{y} + (1 - y) \\log (1 - \\hat{y})$. Put negative signs on both sides to get $L = -[y \\log \\hat{y} + (1 - y) \\log (1 - \\hat{y})]$ \n",
       "\n",
       "\n",
       "Correct prediction results in lower CE loss, while incorrect ones lead to higher CE losses. \n",
       "\n",
       "\n",
       "Loss on all training examples = $- \\sum_{i=1}^m L_{ce} (\\hat{y}^i | y^i)$\n",
       "\n",
       "\n",
       "Define theta = stuff we want to optimize (w, b). We compute $\\hat{\theta}$ = $argmin_{\theta} 1/m \\sum_{i=1}^m L_{ce} (y^i, x^i; \theta) $. \n",
       "\n",
       "\n",
       "Gradient descent is an algorithm to find minimum of a function by deciding which direction the function's slope is rising the most steeply, and moving in the opposite direction. (which way is down) \n",
       " \n",
       "\n",
       "Loss function is guaranteed convex for logisitc regression -> so one min point. If we have small enough step size we are guaranteed to find minimum, else we may jum around. In non-convex problem we may be stuck in \n",
       "\n",
       "\n",
       "Size of step = learning rate = eta. Update paremeter by: $w^{t+1} = w^t - \\eta d/dw f(loss, w)$. If derivative/slope is negative we increase weight, and vice versa.\n",
       "\n",
       "\n",
       "Extending 2-D gradient descent to higher dimensions: for each weight vector, we take $d/dw L(f(x;\theta), y)$. Collection of these derivatives is called $\\delta_{\theta}$. We update $\theta^{t+1} = \theta^t - \\eta \\delta_{\theta}$. \n",
       "\n",
       "\n",
       "The derivative of loss function wrt each $w_i$ = $(\\hat{y} - y)$ $\\dot$ x. We can obtain the gradient for multiple data points by summing up individual ones. \n",
       "\n",
       "\n",
       "Stochastic gradient descent: take a random training instance, then update based on it. Updates can fluctuate. \n",
       " \n",
       "\n",
       "Batch training: Computes gradient over entire dataset. Takes long time but give better estimate of which direction to move. \n",
       "\n",
       "\n",
       "We usually use mini batch training (batch size picked empirically, e.g. what we can fit into GPU card)\n",
       "\n",
       "\n",
       "Problems: (1) overfitting results in poor generalization. (2) If features only appear in one class it gets high weights, we can use regularization to penalize large weights (add regularization term to loss function). \n",
       "\n",
       "\n",
       "For multi-class classification, use multinomial logistic regression. Want to know the probability for each class. Can use generalization of sigmoid funciton called softmax function. Often the final step in neural network. Softmax takes input vectors and output vector of valeus normalized into probabilities. \n",
       "\n",
       "\n",
       "(Feed forward) Neural network with many neurons can approximate many functions. Deep learning = neural network with many layers. \n",
       "\n",
       "\n",
       "Perceptron has step function (0 or 1). Problem is that it is difficult to update weight vector - small change may cause huge swing btw 0 and 1. With sigmoid we have much smoother function: small changes in weight -> small changes in output. \n",
       "\n",
       "\n",
       "Activation functions: sigmoid, ReLU, tanh. Neural network units = \\ \n",
       "perceptron + activation function\n",
       "\n",
       "\n",
       "Totally connected: each node in next layer is connected to all inputs from previous layer.\n",
       "\n",
       "\n",
       "Neural network requires different way of computing gradient: we need back propagation. Related to the general procedure called backward differentiation, which depends on computation graphs (representation of the process of computing a mathematical expression). Use chain rule to compute derivation of loss function L with respect to a, b, and c. \n",
       "\n",
       "\n",
       "Chain rule: $\f",
       "rac{df}{dx} = \f",
       "rac{df}{da} \f",
       "rac{da}{dx}$. Get derivatives on each node of the graph, then multiply nodes along a path to get a destination derivative. \n",
       "\n",
       "\n",
       " \n",
       "\n",
       "## Module 13 Definitions\n",
       "* **Period disambiguation** - decidingif the period is end of sentence or part of a word\n",
       "* **Sigmoid** - $1 / 1 + e ^ {-z}$\n",
       "* **Feature templates** - ambstract specs of features that allow automatic creation of features\n",
       "* **Representation learning** - ways to learn features automatically in an unsupervised way\n",
       "* **Mcculloch-pitts neuron** - a simplified model of the neuron as a computing element describable in propositional logic\n",
       "* **Hard classification task** - task where only one class is correct\n",
       "* **Pretraining** - Relying on another algorithm to learn embedding representation for input words\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module13_sections = [\n",
    "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
    "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
    "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
    "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
    "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
    "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
    "\n",
    "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
    "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
    "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
    "  (\"SLP Chapter 7.4 Training Neural Nets\", slp_chapter7_4),\n",
    "  (\"SLP Chapter 7.5 Neural Language Models\", slp_chapter7_5),\n",
    "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
    "]\n",
    "\n",
    "module13_markdown = format_module(13, \"Neural Networks\", \n",
    "                         module13_sections, \n",
    "                         module13_definitions)\n",
    "display(Markdown(module13_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RYESi2kMybL8"
   },
   "source": [
    "# Module 14 - Natural Language Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "qIRJ4YYCzNHL"
   },
   "outputs": [],
   "source": [
    "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
    "\n",
    "slp_chapter9_1 = \"\"\" A problem with using fixed-size contexts is that it limits \\\n",
    "the context from which information can be extracted. Moreover, it makes it difficult \\\n",
    "to learn systematic patterns arising from certain linguisitic phenomena. \\n\n",
    "\n",
    "Recurrent neural networks and transformer networks are two approaches that allow \\\n",
    "dealing with sequential nature of language, being able ot handle variable input \\\n",
    "lengths without using arbitrary fixed-sized windows and capture temporal nature \\\n",
    "of language.\n",
    "\n",
    "Autoregressive generation provides an alternative way \\\n",
    "to assess a language model. We have a LM generate random words and assess \\\n",
    "to what extent it matches the training data. \"\"\"\n",
    "\n",
    "slp_chapter9_2 = \"\"\" Recurrent neural network = network that contains a cycle. \\n\n",
    "\n",
    "The cycle link inform the hidden layer about information from a prceding point \\\n",
    "in time. The approach does not impose a fixed-length limit on the prior context \\n\n",
    "\n",
    "Backpropagation in RNN requires two pass. Frist pass: forward inference, accumulate \\\n",
    "loss at each step in time, and save value of hidden layer at each step. Second pass: \\\n",
    "process sequence in reverse, compute required gradient, and save error term. This is \\\n",
    "known as backpropagation through time. \\n\n",
    "\n",
    "We can also unroll a recurrent network into a feedforward computational graph to \\\n",
    "remove any explicit recurrences. We can then apply ordinary feedforward computation \\\\and backpropgation. \\n\n",
    "\n",
    "The hidden state in RNN embodies information all the way to the beginning, escaping problme \\\n",
    "of limited context. \\n\n",
    "\n",
    "To train RNN, we use a corpus of text + teacher forcing. Minimize error \\\n",
    "in predicting next word using cross-entropy loss. \\n\n",
    "\n",
    "Weight typing uses a single set of embeddings at input and softmax layers. \\\n",
    "It sets dimensionality of the final hidden layer to be the same \\\n",
    "d_h, and use same matrix for both layers. \\n\n",
    "\n",
    "RNNs are useful for sequenec labeling: assign label to each element \\\n",
    "of sentence, e.g. POS tagging and named entity resolution. \\n\n",
    "\n",
    "RNN can also classify entire sequences.\\n\n",
    "\n",
    "End-to-end training = training that uses loss from a downstream application to \\\n",
    "adjust weights all the way through the network. \\n\n",
    "\n",
    "Stacked RNNs: multiple RNN layers stacked on top of each other. \\\n",
    "The network is able to induce representation at different levels of abstraction across \\\n",
    "layers. \\n\n",
    "\n",
    "Bidirectional RNNs: Also consider context to the right of the input. \\\n",
    "Has two indepdnent RNNs whose outputs are combined into a single \\\n",
    "representation. \n",
    " \"\"\"\n",
    "\n",
    "slp_chapter9_3 = \"\"\" Two problems that RNN faces: 1) the hidden layers serve \\\n",
    "the dual functino of providing info for current decision and carrying forward \\\n",
    "information for future decisions, 2) in back-propagation, hidden layers are subject \\\n",
    "to repeaeted multiplications, driving gradients to zero, u.e. vanishing gradient. \\n\n",
    "\n",
    "Long short-term memory (LSTM) networks divide context management into two sub-problems: \\\n",
    "removing info no longer needed, and adding info likely to be needed. \\\n",
    "It adds an explicit context layer, using gates to control flow of info into and out \\\n",
    "of the units. Gates are implemented with additional weights that operate \\\n",
    "sequentially on the input. \\n\n",
    "\n",
    "Gates consist of a feedforward layer, a sigmoid activation function, and \\\n",
    "pointwise multiplication of layer being gated. Function is similar to a binary \\\n",
    "mask: no change to values close to one, and erase values with lower values. \\n\n",
    "\n",
    "Forget gate deletes info from context that is not needed. Computes a weighted \\\n",
    "sum of hidden layer and current input, pass it through sigmoid then multiply it \\\n",
    "with context. \\n\n",
    "\n",
    "Add gate selects info to add to current context. Output gate decides \\\n",
    "what info is required for current hidden state. \\n\n",
    "\n",
    "LSTM computation: inputs are current input x, previous hidden state, and previous \\\n",
    "context. Outputs are new hidden state and updated context. \\n\n",
    "\n",
    "Gated recurrent units reduce numbers of additional parameters required \\\n",
    "by LSTM, through using a separate context vector and reducing \\\n",
    "the number of gates to 2: a reset gate and an update gate. Reset gate is \\\n",
    "multiplied with previous hidden state to compute new hidden state.  \\\n",
    "UPdate gate uses values in the gate to interpolate between old hidden \\\n",
    "state and new one to determine which aspects of the output of update gate \\\n",
    "will be used directly in new hidden state, and which is needed int he future. \\n\n",
    "\n",
    "The increased complexity of LSTM and GRU are encapsulated within the \\\n",
    "neural units themselves, providing modularity. They can be substitutes \\\n",
    "for simple feedforward and RNN networks. \n",
    " \"\"\"\n",
    "\n",
    "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
    "\n",
    "slp_chapter6_1 = \"\"\" Words can be similar rather than synonymous. Similarity \\\n",
    "deals with words while synonymy with word senses.\\n\n",
    "\n",
    "Words can also be related/ associated, e.g. coffee and cup. One kind of \\\n",
    "relatedness is being in the same semantic field (set of words that \\\n",
    "cover a semantic domain). AKA topic models, e.g. Latent Dirichlet Allocation, using \\\n",
    "unsupervised ML on large sets to induce associated words.\\n\n",
    "\n",
    "More relations are hypernymy/ IS-A, antonymy, and meronymy. \\n\n",
    "\n",
    "Semantic frame: set of words that denote perspectives or participants \\\n",
    "in a type of event, e.g. transaction. \\n\n",
    "\n",
    "Connotations/ affective meanings: associated with setiments. Words \\\n",
    "vary along three dimensions for affective meaning: valence \\\n",
    "(pleasantness), arousal (intensity), dominance (Degree of control).  \"\"\"\n",
    "\n",
    "slp_chapter6_2 = \"\"\" Vector semantics represent word as a point in a \\\n",
    "multidimensional semantic space derived from distribution of word neighbors. \\n\n",
    "\n",
    "Vectors representing words are embeddings. \\n\n",
    "\n",
    "Two common models: tf-idf - meaning of word defined by a simple function of \\\n",
    "counts of nearby words. Results in long vectors that are sparse \\n\n",
    "\n",
    "Word2vec: construct short, dense vectors with useful semantic properties. \\n\n",
    "\n",
    "Cosine: standard way to use embeddings to calculate semantic similarity \\\n",
    "between two bodies of text.\n",
    " \"\"\"\n",
    "\n",
    "slp_chapter6_3 = \"\"\" Term document matrix: each row is a word and each \\\n",
    "document is a colun. Defined as  part of the vector space model of information \\\n",
    "retrieval. Each column is a vector. Has V rows and D columns in total. \\n\n",
    "\n",
    "Information retrieval: find document d from D documents that match query q. \\\n",
    "We represent a query by a vector and need to compute how similar two vectors \\\n",
    "are. \\n\n",
    "\n",
    "We can also use term-term matrix/ word-word matrix/ term-context matrix, \\\n",
    "with numbers being how often they appear together. WE can restric the size of \\\n",
    "columns to frequent words. Most numbers will be zero, creating a sparse matrix.\n",
    "\"\"\"\n",
    "\n",
    "slp_chapter6_4 = \"\"\" Normalized dot product (dot product divided by vector length ) =\\\n",
    "cosine of angles between two vectors = $(a \\dot b) / (|a||b|)$. Higher cosine = \\\n",
    "higher similarity. \"\"\"\n",
    "\n",
    "slp_chapter6_8 = \"\"\" Word2vec is fast, eeasy to train, and easily availble \\\n",
    "online with code and pre-trained embeddings. \\n\n",
    "\n",
    "Intuition behind word2vec: train a classifier on the prediction task: \\\n",
    "how often will word w show up near word y?  \\n\n",
    "\n",
    "Simpler than neural network: 1) simplifies task (classification instead \\\n",
    "of word prediction) 2) simplified architecture (log reg classifier rather than \\\n",
    "multi layer network). \\n\n",
    "\n",
    "Intuiton of skip-gram: 1) treat target word and neighbor context as \\\n",
    "positive examples. 2) randomly sample other words as negative \\\n",
    "examples. 3) Use log reg to train a classifier to distinguish. 4) \\\n",
    "use learned weights as embeddings. \\n\n",
    "\n",
    "To compute probability of two words being close, we see if their \\\n",
    "embeddings are similar. We apply sigmoid on the dot product of the two \\\n",
    "embeddings. It makes the assumption that all context words are independent. \\n\n",
    "\n",
    "Skip grams learn embeddings by startting with random vectors, then \\\n",
    "iteratively shifting embeddings of each word to be more like \\\n",
    "embeddings of words nearby. \\n\n",
    "\n",
    "SGNS uses more negative than positive examples, with ratio between them \\\n",
    "set by a parameter k. Negative examples have target word + noise word. \\\n",
    "Noise words are chosen according to htier weighted unigram frequency, with \\\n",
    "a weight alpha often set to 0.75. \\n\n",
    "\n",
    "Learned embeddings maximize similarity of target, context word pairs, \\\n",
    "and minimize similarity of negative examples. Two separate embeddings \\\n",
    "the target embedding and the context embedding are learned. \\n\n",
    "\n",
    "Other kinds of static embeddings: fasttext (extension of word2vec) \\\n",
    "deals with unknown words and sparsity using subword models. \n",
    "\"\"\"\n",
    "\n",
    "slp_chapter6_10 = \"\"\" Size of context window is usually \\\n",
    "between 1 and 10 words. Choice depends on the goal of representation: \\\n",
    "short windows lead to more syntactic representativions, while long windows \\\n",
    "lead to related but not similar words. \\n\n",
    "\n",
    "Two kinds of similarity: 1) first-order co-occurence/ syntagmatic association: \\\n",
    "occurs between two words typically nearby each other (e.g. book and wrote). \\\n",
    "2) second-order co-occurence/ paradigmatic association if two words \\\n",
    "have similar neighbors (e.g. wrote, said). \\n\n",
    "\n",
    "Parallelogram model can be used to solve simple analogy problems. \\\n",
    "Add vector from apple to tree to the word grape to get vine. \\n\n",
    "\n",
    "Embeddings can also be used to study how meaning changes over time. \"\"\"\n",
    "\n",
    "\n",
    "slp_chapter6_11 = \"\"\" Allocational harm occurs when system allocates resources \\\n",
    "unfairly to different groups. Can result from embeddings. \\n\n",
    "\n",
    "Embeddings can amplify bias: genedered terms can become more genedered than \\\n",
    "in the inputs. It can also encode implict associations, causing representational \\\n",
    "harm, caused by a system demeaning or ignoring certain social groups. \\n\n",
    "\n",
    "Debiasing can help reduce bias in embeddings. Historical \\\n",
    "embeddings can be used to measure biases in the past. \"\"\"\n",
    "\n",
    "slp_chapter6_12 = \"\"\" Intrinsic evaluation methods for embeddings: \\ \n",
    "1) most common - test performance on similarity: machine vs human evaluation \\\n",
    "on simliarity. E.g. TOEFL dataset. We can also include context \\\n",
    "in similarity tests. 2) analogy task. \\n \n",
    "\n",
    "Embedding algorithms can suffer from inherent variability: \\\n",
    "because of random negative sampling, different results can \\\n",
    "be generated from the same dataset.\"\"\"\n",
    "\n",
    "\n",
    "\n",
    "# Lecture notes \n",
    "module14_lecture_notes = \"\"\" \n",
    "Advantages of neural LMs: 1) don't need smoothing, 2) can handle longer \\\n",
    "histories, 3) Can generalize over contexts of similar words, 4) much higher accuracy\\n\n",
    "\n",
    "Proposed over 20 years ago by Joshua Bengio. 1) Associate each word with a vector \\\n",
    "representation, 2) express joint proabbility function in terms of word vectors, \\\n",
    "then 3) simultaneously learn word vectors and the parameters of the function. \\n\n",
    "\n",
    "Word vectors: low dimensional (d = 30-100) dense vectors. \\n\n",
    "\n",
    "Prob function is the product of conditional probs of the next word given the \\\n",
    "previous words.\\n\n",
    "\n",
    "Input: k-gram of words, output: prob distribution of next word. Each word is assocaited \\\n",
    "wth an embedding vector. Input vector x is concatenation of the vector of the k words. \\n\n",
    "\n",
    "The output layer will have output of size equal to the vocabulary. \\n\n",
    "\n",
    "The projected layer will contain a (learned) representation of each word. Projection \\\n",
    "layer of dog and puppy will be similar with each other. \\n\n",
    "\n",
    "Training examples: k-grams from corpus. Model is trained using cross-entropy loss. \\n\n",
    "\n",
    "What happen if we have unseen sequence of words? 1) Back off, 2) smoothing / interpolation, 3) \\\n",
    "generalizing. We can replace words with others with similar meaning. \\n\n",
    "\n",
    "Bengio's proposal: 1) use a vector space model with words = vectors with real values R^m,\\\n",
    " with m = 30, 60, 100. 2) Define a function that returns a joint probability of words \\\n",
    " in a sequence. 3) learn word representations and probabilty function simultaneously. \\n\n",
    "\n",
    "Lexical semantic models. A meaning representation we should encode: 1) \\\n",
    "synonyms, 2) antonyms, 3) connotations, 4) semantic roles, 5) entailment. \\n\n",
    "\n",
    "Downside of using formal logic as representation: need to acquire knowledge \\\n",
    "from somewhere else. Example: word net at Princeton, with synonym set. \\n\n",
    "\n",
    "Alternative to word net: build a vector space model. Allow automatic extraction from \\\n",
    "huge corpus. Each word represented in vector space, with simliar words occupying \\\n",
    "similar positions. \\n\n",
    "\n",
    "Distributional hypothesis: if two words have similar meanings then they will appear \\\n",
    "in similar sets of environments (Zellig Harris 1954). \\n\n",
    "\n",
    "First vector space models are developed in the SMART information retrieval system. \\\n",
    "Convert query into vector and retrieve everything with similar vectors. \\n\n",
    "\n",
    "Term document matrix: words as rows, documents as columns. Comparing two row vectors \\\n",
    "compare if two words are similar. \\n\n",
    "\n",
    "Euclidean distance is a poor estimate of similarity, because more frequent words \\\n",
    "will have higher distance from origin. We use L2 Normalization, divide by distance \\\n",
    "from origin. \\n\n",
    "\n",
    "Cosine distance: already built in the L2 Norm. Dot product of two word vectors, \\\n",
    "normalized by the product of their distance to origin.\\n\n",
    "\n",
    "Term-term matrix/ term-context matrix: square matrix. What a cell represent \\\n",
    "how often does the word X (col) appears in a context window surrounding the word Y (row)? \\n\n",
    "\n",
    "We can transform raw counts into something more useful. E.g. TF-IDF or PPMI. \\\n",
    "Counter Zipf's law: word distribution is inherently skewed. \\n\n",
    "\n",
    "TF-IDF: term frequency * inverse document frequency (1/num docuemtns where a word appears). \\\n",
    "Measures how \"informative\" a word appearance is. \\n\n",
    "\n",
    "Co-occurence matrix weighted by TF-IDF is long and sparse. We want to learn vectors \\\n",
    "that are short and dense. Use embedding instead. \\n\n",
    "\n",
    "To get dense vectors, we train a classifer. 1) We treat every word and its neighboring context as a positive training example. \\\n",
    "2) randomly sample other words as negative examples. 3) use log reg to train classifer to \\\n",
    "distinguish. 4) use the weights as embeddings. \\n\n",
    "\n",
    "Word2Vec model implements this. It is fast and easy to train, also available \\\n",
    "online in word2vec package. \\n\n",
    "\n",
    "Embedding can capture relational meaning. E.g. gender relationships. We can solve \\\n",
    "word analogy problems using word embeddings. \n",
    " \"\"\"\n",
    "\n",
    "# Definitions\n",
    "module14_definitions = {\n",
    "\"Distributional hypothesis\" : \"Linke between similarity in how words \\\n",
    "are distributed and what they mean\",\n",
    "\"One hot vector\" : \"vector with one value 1 and others 0 \",\n",
    "\"Vector semantics\" : \"Instantiation of the distributional hypothesis \\\n",
    "by learning meanings/embeddings from contexts\",  \n",
    "\"Represenation learning\": \"automatically (self-supervised) learning useful representations \\\n",
    "of input text\",\n",
    "\"Feature engineering\" : \"learning features by hand\",\n",
    "\"Lexical semantics\" : \"Linguistic study of word meaning\",\n",
    "\"Lemma/citation form\" : \"Word roots, as opposed to wordforms\",\n",
    "\"Word sense\" : \"a possible meaning of a word\",\n",
    "\"Polysemous\" : \"word having multiple senses\",\n",
    "\"Word sense disambiguation\" : \"tackling polysemous words\",\n",
    "\"Synonymy\" : \"two words are substitutable for one another without changing \\\n",
    "truth condition of the sentence\",\n",
    "\"Propositional meaning\" : \"Two words ahve same PM when they are synonymous\",\n",
    "\"Principle of contrast\" : \"A difference in linguistic form is always \\\n",
    "associated with some difference in meaning\",\n",
    "\"Skip gram with negative sampling (SGNS)\" : \"An algorithm for computing embeddings\", \n",
    "\"Contextual embeddings\" : \"BERT/ ELMO representations, where vector for each word \\\n",
    "is different in diff contexts\",\n",
    "\"Dense vector models\" : \"vector models with dimensionality 50-1000\",\n",
    "\"tf-idf\" : \"A sparse weighting which assigns weights by term frequency and \\\n",
    "inverse document frequency\"\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "lXNcHuKjyNXa",
    "outputId": "edae180e-90e6-439c-c5bb-2c304ecafb74"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 14: Natural Language Processing\n",
       "## SLP Chapter 9.1 Language Models Revisited\n",
       " A problem with using fixed-size contexts is that it limits the context from which information can be extracted. Moreover, it makes it difficult to learn systematic patterns arising from certain linguisitic phenomena. \n",
       "\n",
       "\n",
       "Recurrent neural networks and transformer networks are two approaches that allow dealing with sequential nature of language, being able ot handle variable input lengths without using arbitrary fixed-sized windows and capture temporal nature of language.\n",
       "\n",
       "Autoregressive generation provides an alternative way to assess a language model. We have a LM generate random words and assess to what extent it matches the training data. \n",
       "\n",
       "## SLP Chapter 9.2 Recurrent Neural Networks\n",
       " Recurrent neural network = network that contains a cycle. \n",
       "\n",
       "\n",
       "The cycle link inform the hidden layer about information from a prceding point in time. The approach does not impose a fixed-length limit on the prior context \n",
       "\n",
       "\n",
       "Backpropagation in RNN requires two pass. Frist pass: forward inference, accumulate loss at each step in time, and save value of hidden layer at each step. Second pass: process sequence in reverse, compute required gradient, and save error term. This is known as backpropagation through time. \n",
       "\n",
       "\n",
       "We can also unroll a recurrent network into a feedforward computational graph to remove any explicit recurrences. We can then apply ordinary feedforward computation \\and backpropgation. \n",
       "\n",
       "\n",
       "The hidden state in RNN embodies information all the way to the beginning, escaping problme of limited context. \n",
       "\n",
       "\n",
       "To train RNN, we use a corpus of text + teacher forcing. Minimize error in predicting next word using cross-entropy loss. \n",
       "\n",
       "\n",
       "Weight typing uses a single set of embeddings at input and softmax layers. It sets dimensionality of the final hidden layer to be the same d_h, and use same matrix for both layers. \n",
       "\n",
       "\n",
       "RNNs are useful for sequenec labeling: assign label to each element of sentence, e.g. POS tagging and named entity resolution. \n",
       "\n",
       "\n",
       "RNN can also classify entire sequences.\n",
       "\n",
       "\n",
       "End-to-end training = training that uses loss from a downstream application to adjust weights all the way through the network. \n",
       "\n",
       "\n",
       "Stacked RNNs: multiple RNN layers stacked on top of each other. The network is able to induce representation at different levels of abstraction across layers. \n",
       "\n",
       "\n",
       "Bidirectional RNNs: Also consider context to the right of the input. Has two indepdnent RNNs whose outputs are combined into a single representation. \n",
       " \n",
       "\n",
       "## SLP Chapter 9.3 RNNs as Language Models\n",
       " Two problems that RNN faces: 1) the hidden layers serve the dual functino of providing info for current decision and carrying forward information for future decisions, 2) in back-propagation, hidden layers are subject to repeaeted multiplications, driving gradients to zero, u.e. vanishing gradient. \n",
       "\n",
       "\n",
       "Long short-term memory (LSTM) networks divide context management into two sub-problems: removing info no longer needed, and adding info likely to be needed. It adds an explicit context layer, using gates to control flow of info into and out of the units. Gates are implemented with additional weights that operate sequentially on the input. \n",
       "\n",
       "\n",
       "Gates consist of a feedforward layer, a sigmoid activation function, and pointwise multiplication of layer being gated. Function is similar to a binary mask: no change to values close to one, and erase values with lower values. \n",
       "\n",
       "\n",
       "Forget gate deletes info from context that is not needed. Computes a weighted sum of hidden layer and current input, pass it through sigmoid then multiply it with context. \n",
       "\n",
       "\n",
       "Add gate selects info to add to current context. Output gate decides what info is required for current hidden state. \n",
       "\n",
       "\n",
       "LSTM computation: inputs are current input x, previous hidden state, and previous context. Outputs are new hidden state and updated context. \n",
       "\n",
       "\n",
       "Gated recurrent units reduce numbers of additional parameters required by LSTM, through using a separate context vector and reducing the number of gates to 2: a reset gate and an update gate. Reset gate is multiplied with previous hidden state to compute new hidden state.  UPdate gate uses values in the gate to interpolate between old hidden state and new one to determine which aspects of the output of update gate will be used directly in new hidden state, and which is needed int he future. \n",
       "\n",
       "\n",
       "The increased complexity of LSTM and GRU are encapsulated within the neural units themselves, providing modularity. They can be substitutes for simple feedforward and RNN networks. \n",
       " \n",
       "\n",
       "## SLP Chapter 6.1 Lexical Semantics\n",
       " Words can be similar rather than synonymous. Similarity deals with words while synonymy with word senses.\n",
       "\n",
       "\n",
       "Words can also be related/ associated, e.g. coffee and cup. One kind of relatedness is being in the same semantic field (set of words that cover a semantic domain). AKA topic models, e.g. Latent Dirichlet Allocation, using unsupervised ML on large sets to induce associated words.\n",
       "\n",
       "\n",
       "More relations are hypernymy/ IS-A, antonymy, and meronymy. \n",
       "\n",
       "\n",
       "Semantic frame: set of words that denote perspectives or participants in a type of event, e.g. transaction. \n",
       "\n",
       "\n",
       "Connotations/ affective meanings: associated with setiments. Words vary along three dimensions for affective meaning: valence (pleasantness), arousal (intensity), dominance (Degree of control).  \n",
       "\n",
       "## SLP Chapter 6.2 Vector Semantics\n",
       " Vector semantics represent word as a point in a multidimensional semantic space derived from distribution of word neighbors. \n",
       "\n",
       "\n",
       "Vectors representing words are embeddings. \n",
       "\n",
       "\n",
       "Two common models: tf-idf - meaning of word defined by a simple function of counts of nearby words. Results in long vectors that are sparse \n",
       "\n",
       "\n",
       "Word2vec: construct short, dense vectors with useful semantic properties. \n",
       "\n",
       "\n",
       "Cosine: standard way to use embeddings to calculate semantic similarity between two bodies of text.\n",
       " \n",
       "\n",
       "## SLP Chapter 6.3 Words and Vectors\n",
       " Term document matrix: each row is a word and each document is a colun. Defined as  part of the vector space model of information retrieval. Each column is a vector. Has V rows and D columns in total. \n",
       "\n",
       "\n",
       "Information retrieval: find document d from D documents that match query q. We represent a query by a vector and need to compute how similar two vectors are. \n",
       "\n",
       "\n",
       "We can also use term-term matrix/ word-word matrix/ term-context matrix, with numbers being how often they appear together. WE can restric the size of columns to frequent words. Most numbers will be zero, creating a sparse matrix.\n",
       "\n",
       "\n",
       "## SLP Chapter 6.4 Cosine for measuring similarity\n",
       " Normalized dot product (dot product divided by vector length ) =cosine of angles between two vectors = $(a \\dot b) / (|a||b|)$. Higher cosine = higher similarity. \n",
       "\n",
       "## SLP Chapter 6.8 Word2vec\n",
       " Word2vec is fast, eeasy to train, and easily availble online with code and pre-trained embeddings. \n",
       "\n",
       "\n",
       "Intuition behind word2vec: train a classifier on the prediction task: how often will word w show up near word y?  \n",
       "\n",
       "\n",
       "Simpler than neural network: 1) simplifies task (classification instead of word prediction) 2) simplified architecture (log reg classifier rather than multi layer network). \n",
       "\n",
       "\n",
       "Intuiton of skip-gram: 1) treat target word and neighbor context as positive examples. 2) randomly sample other words as negative examples. 3) Use log reg to train a classifier to distinguish. 4) use learned weights as embeddings. \n",
       "\n",
       "\n",
       "To compute probability of two words being close, we see if their embeddings are similar. We apply sigmoid on the dot product of the two embeddings. It makes the assumption that all context words are independent. \n",
       "\n",
       "\n",
       "Skip grams learn embeddings by startting with random vectors, then iteratively shifting embeddings of each word to be more like embeddings of words nearby. \n",
       "\n",
       "\n",
       "SGNS uses more negative than positive examples, with ratio between them set by a parameter k. Negative examples have target word + noise word. Noise words are chosen according to htier weighted unigram frequency, with a weight alpha often set to 0.75. \n",
       "\n",
       "\n",
       "Learned embeddings maximize similarity of target, context word pairs, and minimize similarity of negative examples. Two separate embeddings the target embedding and the context embedding are learned. \n",
       "\n",
       "\n",
       "Other kinds of static embeddings: fasttext (extension of word2vec) deals with unknown words and sparsity using subword models. \n",
       "\n",
       "\n",
       "## SLP Chapter 6.10 Semantic properties of embeddings\n",
       " Size of context window is usually between 1 and 10 words. Choice depends on the goal of representation: short windows lead to more syntactic representativions, while long windows lead to related but not similar words. \n",
       "\n",
       "\n",
       "Two kinds of similarity: 1) first-order co-occurence/ syntagmatic association: occurs between two words typically nearby each other (e.g. book and wrote). 2) second-order co-occurence/ paradigmatic association if two words have similar neighbors (e.g. wrote, said). \n",
       "\n",
       "\n",
       "Parallelogram model can be used to solve simple analogy problems. Add vector from apple to tree to the word grape to get vine. \n",
       "\n",
       "\n",
       "Embeddings can also be used to study how meaning changes over time. \n",
       "\n",
       "## SLP Chapter 6.11 Bias and Embeddings\n",
       " Allocational harm occurs when system allocates resources unfairly to different groups. Can result from embeddings. \n",
       "\n",
       "\n",
       "Embeddings can amplify bias: genedered terms can become more genedered than in the inputs. It can also encode implict associations, causing representational harm, caused by a system demeaning or ignoring certain social groups. \n",
       "\n",
       "\n",
       "Debiasing can help reduce bias in embeddings. Historical embeddings can be used to measure biases in the past. \n",
       "\n",
       "## SLP Chapter 6.12 Evaluating Vector Models\n",
       " Intrinsic evaluation methods for embeddings: \\ \n",
       "1) most common - test performance on similarity: machine vs human evaluation on simliarity. E.g. TOEFL dataset. We can also include context in similarity tests. 2) analogy task. \n",
       " \n",
       "\n",
       "Embedding algorithms can suffer from inherent variability: because of random negative sampling, different results can be generated from the same dataset.\n",
       "\n",
       "## Module 14 Lecture Notes\n",
       " \n",
       "Advantages of neural LMs: 1) don't need smoothing, 2) can handle longer histories, 3) Can generalize over contexts of similar words, 4) much higher accuracy\n",
       "\n",
       "\n",
       "Proposed over 20 years ago by Joshua Bengio. 1) Associate each word with a vector representation, 2) express joint proabbility function in terms of word vectors, then 3) simultaneously learn word vectors and the parameters of the function. \n",
       "\n",
       "\n",
       "Word vectors: low dimensional (d = 30-100) dense vectors. \n",
       "\n",
       "\n",
       "Prob function is the product of conditional probs of the next word given the previous words.\n",
       "\n",
       "\n",
       "Input: k-gram of words, output: prob distribution of next word. Each word is assocaited wth an embedding vector. Input vector x is concatenation of the vector of the k words. \n",
       "\n",
       "\n",
       "The output layer will have output of size equal to the vocabulary. \n",
       "\n",
       "\n",
       "The projected layer will contain a (learned) representation of each word. Projection layer of dog and puppy will be similar with each other. \n",
       "\n",
       "\n",
       "Training examples: k-grams from corpus. Model is trained using cross-entropy loss. \n",
       "\n",
       "\n",
       "What happen if we have unseen sequence of words? 1) Back off, 2) smoothing / interpolation, 3) generalizing. We can replace words with others with similar meaning. \n",
       "\n",
       "\n",
       "Bengio's proposal: 1) use a vector space model with words = vectors with real values R^m, with m = 30, 60, 100. 2) Define a function that returns a joint probability of words  in a sequence. 3) learn word representations and probabilty function simultaneously. \n",
       "\n",
       "\n",
       "Lexical semantic models. A meaning representation we should encode: 1) synonyms, 2) antonyms, 3) connotations, 4) semantic roles, 5) entailment. \n",
       "\n",
       "\n",
       "Downside of using formal logic as representation: need to acquire knowledge from somewhere else. Example: word net at Princeton, with synonym set. \n",
       "\n",
       "\n",
       "Alternative to word net: build a vector space model. Allow automatic extraction from huge corpus. Each word represented in vector space, with simliar words occupying similar positions. \n",
       "\n",
       "\n",
       "Distributional hypothesis: if two words have similar meanings then they will appear in similar sets of environments (Zellig Harris 1954). \n",
       "\n",
       "\n",
       "First vector space models are developed in the SMART information retrieval system. Convert query into vector and retrieve everything with similar vectors. \n",
       "\n",
       "\n",
       "Term document matrix: words as rows, documents as columns. Comparing two row vectors compare if two words are similar. \n",
       "\n",
       "\n",
       "Euclidean distance is a poor estimate of similarity, because more frequent words will have higher distance from origin. We use L2 Normalization, divide by distance from origin. \n",
       "\n",
       "\n",
       "Cosine distance: already built in the L2 Norm. Dot product of two word vectors, normalized by the product of their distance to origin.\n",
       "\n",
       "\n",
       "Term-term matrix/ term-context matrix: square matrix. What a cell represent how often does the word X (col) appears in a context window surrounding the word Y (row)? \n",
       "\n",
       "\n",
       "We can transform raw counts into something more useful. E.g. TF-IDF or PPMI. Counter Zipf's law: word distribution is inherently skewed. \n",
       "\n",
       "\n",
       "TF-IDF: term frequency * inverse document frequency (1/num docuemtns where a word appears). Measures how \"informative\" a word appearance is. \n",
       "\n",
       "\n",
       "Co-occurence matrix weighted by TF-IDF is long and sparse. We want to learn vectors that are short and dense. Use embedding instead. \n",
       "\n",
       "\n",
       "To get dense vectors, we train a classifer. 1) We treat every word and its neighboring context as a positive training example. 2) randomly sample other words as negative examples. 3) use log reg to train classifer to distinguish. 4) use the weights as embeddings. \n",
       "\n",
       "\n",
       "Word2Vec model implements this. It is fast and easy to train, also available online in word2vec package. \n",
       "\n",
       "\n",
       "Embedding can capture relational meaning. E.g. gender relationships. We can solve word analogy problems using word embeddings. \n",
       " \n",
       "\n",
       "## Module 14 Definitions\n",
       "* **Distributional hypothesis** - Linke between similarity in how words are distributed and what they mean\n",
       "* **One hot vector** - vector with one value 1 and others 0 \n",
       "* **Vector semantics** - Instantiation of the distributional hypothesis by learning meanings/embeddings from contexts\n",
       "* **Represenation learning** - automatically (self-supervised) learning useful representations of input text\n",
       "* **Feature engineering** - learning features by hand\n",
       "* **Lexical semantics** - Linguistic study of word meaning\n",
       "* **Lemma/citation form** - Word roots, as opposed to wordforms\n",
       "* **Word sense** - a possible meaning of a word\n",
       "* **Polysemous** - word having multiple senses\n",
       "* **Word sense disambiguation** - tackling polysemous words\n",
       "* **Synonymy** - two words are substitutable for one another without changing truth condition of the sentence\n",
       "* **Propositional meaning** - Two words ahve same PM when they are synonymous\n",
       "* **Principle of contrast** - A difference in linguistic form is always associated with some difference in meaning\n",
       "* **Skip gram with negative sampling (sgns)** - An algorithm for computing embeddings\n",
       "* **Contextual embeddings** - BERT/ ELMO representations, where vector for each word is different in diff contexts\n",
       "* **Dense vector models** - vector models with dimensionality 50-1000\n",
       "* **Tf-idf** - A sparse weighting which assigns weights by term frequency and inverse document frequency\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@title Preview of your summary\n",
    "#@markdown You don't need to modify anything in this cell. Just press play.\n",
    "module14_sections = [\n",
    "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
    "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
    "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
    "\n",
    "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
    "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
    "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
    "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
    "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
    "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
    "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
    "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
    "  (\"Module 14 Lecture Notes\", module14_lecture_notes),\n",
    "]\n",
    "\n",
    "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
    "                         module14_sections, \n",
    "                         module14_definitions)\n",
    "display(Markdown(module14_markdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vX7lbnJd4sT5"
   },
   "source": [
    "# Your Exam Notes\n",
    "\n",
    "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
    "\n",
    "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
    "* Run the cell below. \n",
    "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
    "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
    "* In the print options, change the layout from Landscape to Portrait. \n",
    "* Clicking the Save, will save a PDF that you can then print.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "T3cm1WV542_a",
    "outputId": "c81878f8-ab8f-41f7-afe3-18e4193a5674"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Module 8: Markov Decision Processes\n",
       "## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n",
       " $P(RESULT(a) = s') = \\sum_s P(s) P(s' | s, a)$ \n",
       "\n",
       "\n",
       "$Expected Utility = \\sum_s' P(RESULT(a) = s') U(s')$ \n",
       "\n",
       "\n",
       "MEU principle formalizes \"doing the right thing\" but does not operationalize: still need to estimate probability of states thruogh learning, and compute P(RESULT(a)=s') using a causal model.  \n",
       "\n",
       "## AIMA Chapter 16.2 The Basis of Utility Theory\n",
       " Constraints/axioms on rational preference relations: (1) orderability either two lotteries are indifferent or agent prefers one over other, (2) transitivity , (3) continuity: if B is between A and C in preference, there is some probability p s.t. agent will be indifferent between B for sure and A / C with probability (4) substitutability: If indifferent between A and B, then indifferent between two lotteries with only difference being A vs B, (5) Monotonicity: If A $\\succ$ B then agent prefers lottery with higher probability for A, (6) decomposability: Compound lotteries can be reduced to simpler ones\n",
       "\n",
       "\n",
       "von Neumann & Morgenstern: Existence of Utility function. If agent's preference obeys axiom, then there is a function s.t. U(A) > U(B) iff A $\\succ$ B and U(A) = U(B) if $A \\sim B$. \n",
       "\n",
       "## AIMA Chapter 16.3 Utility Functions\n",
       " Utility of money is not directly proportional to monetary value.Value of first million is very high, while marginal value decreases. s shaped curve. A billionnaire thus more likely to take a risky but beneficial move than a stable one \n",
       "\n",
       "\n",
       "Grayson study: utility of money almost exactly proportional to log of the amount. We typically have concave utility function for positive wealth (facing downwards),\n",
       "\n",
       "\n",
       "People in the slope-decreasing part of utility curve will assign higher utility to a guaranteed Expected Monetary Value than a lottery. Risk-avrese agents. Risk-seeking agents show up in the negative part of the wealth utility curve. If the changes in wealth are small, curve will be nearly linear, and agent will be risk-neutral \n",
       "\n",
       "\n",
       "Most rational action a = $argmax_a EU(a)$. Most of the time we can only estimate EU. Unbiased estimate: expected value of error is 0. Since we select action with highest EU estimate, we will have an optimistic bias/ optimizer's curse. Bias higher when we are choosing among more choices. If variance in utility estimate is high, what appears to be the best choice may not even be the best chioce: so make sure sample size is large enough for trials. We can avoid it by using a Bayesian model using a probability model of the error $P(EU_{est} | EU)$, updating EU based on our prior knowledge of it.\n",
       "Human preferences are predictably irrataional: there may be no utility function consistent with our choices. Possible reasons: (1) certainty effect (reduce computational burden, distrust stated probsaccoutning for emotion) (2) Ellsberg paradox: people have ambiguity aversion, prefer known probability. (3) framing effect, (4) anchoring effect. \n",
       "\n",
       "## AIMA Chapter 17.1 Sequential Decision Problems\n",
       " In sequential decision problems agent's utility is the result of a sequence of decisions. Markov decision process = sequential decision problem for fully observable, stochastic environment with a fully observable transition model and additive rewards. Solving MDPs usually require dynamic programming. \n",
       "\n",
       "\n",
       "If a game has finite time horizon, policy may change based on the time left. \n",
       "\n",
       "\n",
       "We use additive discount rewards to define utility for two reasons: (1) empirical evidence shows that humans value near-term rewards. (2) economic reasoning suggests early reward is momre valuable, (3) uncertainty about true rewards, (4) assumption of stationary preference, (5) avoid problems with infinite horizons. \n",
       "\n",
       "\n",
       "On point (5), three ways to avoid infinite utility. (1): use discounted rewards such that utility is bounded by $(R_{max}) / (1 - gamma)$. (2): design proper policies that guarantee agents will reach terminal states, save us from discouting (3): compare sequences using average rather than sum of reward. \n",
       "\n",
       "\n",
       "When we use discounted utilities with infinite horizons, the optimal policy is independent of the starting state. Thus, the utility of each state is the expected sum of discounted rewards if the agent carrys out an optimal policy, or U(s) \n",
       "\n",
       "\n",
       "Utility of one state = expected reward for next transition + discounted utility of the next state. This is the Bellman equation. \n",
       "\n",
       "\n",
       "An affine transformation of reward will leave the optimal policy unchanged. Moreoever the shaping theorem suggests that any transformation in this form will also produce no change to optimal policy: \n",
       "\n",
       "\n",
       "\t $R'(s, a, s') = R(s, a, s') + \\gamma \\phi (s') - \\phi (s)$ where $\\phi (s)$ is any function of state s. \n",
       "\n",
       "\n",
       "Function $\\phi (s)$ is called a potential. $ \\gamma \\phi (s') - \\phi (s)$ is the gradient of the potential. If $\\phi (s)$ is proportional to higher utility values, then adding $ \\gamma \\phi (s') - \\phi (s)$ will lead agent \"uphill\" in utility. \n",
       "\n",
       "\n",
       "Ways to represent MDPs: \n",
       "\n",
       "1) use 3-D tables of size $|S|^2 |A|$. Sparse tables with many 0's only have size $O(|S||A|)$. \n",
       "\n",
       "2) Dynamic decision networks (DDNs) with decision, reward, and utility nodes. Use factored represetnation, and have complexity exponential times better than atomic representation. \n",
       "\n",
       "\n",
       "## AIMA Chapter 17.2 Algorithms for MDPs\n",
       " Problem with solving Bellman equations: the MAX operator is not linear, so can't use linear system solver. Instead we use an iterative approach. \n",
       "\n",
       "\n",
       "Showing that the values would converge: \n",
       "\n",
       "Contraction: A function that, when apply to two values, return two values that are closer together by at least some constant factor. E.g. divide by four \n",
       "\n",
       "\n",
       "A contraction function has only one fixed point. The value gets closer to the fixed point each time contraction is applied. \n",
       "\n",
       "\n",
       "We can show that Bellman update is a contraction by a factor of $\\gamma$ on utility vectors. Define \"length\" of a vector as the abs value of its biggest component. Then difference between two vectors = max difference between two corresponding elements. Then we can show $||BU_i - BU'_i||$ $\\leq$ $\\gamma$ $||U_i - U'_i||$. \n",
       "\n",
       "\n",
       "On each iteration, the error decreases by a factor $\\geq \\gamma$. So value iteration converges in exponential rate. The maximum number of iterations = ceiling of $log (2R_{max} / \\epsilon (1 - \\gamma) / log (1 / \\gamma))$. Implies fast convergence if $\\gamma$ is small / horizon is short, but N does not depend too much on $\\epsilon / R_{max}$. \n",
       "\n",
       "\n",
       "It can be shown that in value iteration, policy often becomes optimal long before value converges. Policy loss is connected to error in value estimate by: \n",
       "\n",
       "\n",
       "if $||U_i - U|| < \\epsilon $ then $||U^{\\pi_i} - U|| < 2 \\epsilon$. \n",
       "\n",
       "\n",
       "Policy iteration is faster than solving Bellman equations: O(n^3) time since we don't have max operator, so equations are linear. To further save time, use modified policy iteration where do a small number of value iterations. \n",
       "\n",
       "\n",
       "Another alternative is asynchronous policy iteration: pick subset of states then apply either policy improvement or simplified value iteration. \n",
       " \n",
       "\n",
       "We can also convert MDPs into linear programming problems, by treating utilities as variables and seeking to minimize utilities under constraints. (bellman equations). Makes MDPs solvable in polynomial time in the number of states and actions. However, number of states may be quite large, so LP may be less efficient than DP. \n",
       "\n",
       "\n",
       "We can also solve MDPs using Expectimax trees, limiting depths by using the $\\epsilon$ horizon which is to cut the tree at a depth where sum of reward is lower than $\\epsilon$. We can also limit branching factor by sampling values instead of getting all of them. This approach is Real-Time Dynamic Programming (RTDP), similar to LRTA\\* algorithm. Useful in solving moderate sized problems,  but limited for large ones. \n",
       "\n",
       "\n",
       "Enhancements to RTDP: (1) apply reinforcement learning to generate better heuristic. (2) Look further ahead in MDP with Monte Carlo approach. \n",
       "\n",
       "\n",
       "## Module 8 Lecture Notes\n",
       " A MDP is a sequential, fully observable decision problem with additive rewards and stochastic transition model (i.e. nondeterministic). Need to be solved by expectimax search. \n",
       "\n",
       "\n",
       "MDP is defined by: (1) set of states, (2) set of actions, (3) transition function, specifying the probability (aka the model), (4) reward function, (5) initial state, and (6) (optional) terminal state. \n",
       " \n",
       "\n",
       "In MDPs rewards are (1) either positive or negative, but bounded. (2) specified on a sequence of actions since MDP is sequential, (3) can be received both at goal state or at each transition, (4) additive. \n",
       "\n",
       "\n",
       "A solution to a MDP is a policy: a mapping from state to action. vs solution to search problem is a plan. $\\pi$ denotes a policy and $\\pi *$ denotes the optimal policy. Each policy is associated with an expected utility. Results of same policy may vary according to the different environment history. \n",
       "\n",
       "\n",
       "Two reasons for discounting future rewards: (1) fit empirical observation and (2) help algorithm converges. \n",
       "\n",
       "\n",
       "Theorem: assuming stationary preferences (i.e. if agent prefers seq A to seq B, then agent prefers action r + seq A to action r + sequence B), we can only define utility as additive (simple sum) or discounted (sum weighted by discount). \n",
       "\n",
       "\n",
       "Three solutions for infinite games: (1) finite horizon (stop after T steps), which will cause nonstationary policies, (2) discounting (wiht smaller $\\gamma$ meaning smaller time horizon), and (3) absorbing state that every policy is guaranteed to reach. \n",
       "\n",
       "\n",
       "Value of state s = expected utility of starting in the state then acting optimally. $V^*(s) = max_a Q^*(s,a)$\n",
       "\n",
       "\n",
       "Value of Q state (s, a) = expected utility of taking a from s then acting optimally. $Q^*(s,a) = \\sum_{s'} T(s,a,s') [R(s,a,s') + $ $\\gamma$ $V^*(s')]$ \n",
       "\n",
       "\n",
       "We can use time limited values to reduce amount of work with expectimax. $V_k(s)$ = optimal value of s if game ends in k more time steps (i.e. value from k-depth expectimax on s). \n",
       "\n",
       "\n",
       "Value iteration starts from $V_0 (s) = 0$, then update using formula: $V_{k+1} (s) = max_a \\sum_{s'} T(s,a,s') [R(s,a,s') + $ $\\gamma$ $V_k(s')]$ Repeated until convergence (theorem guarantees that convergence will be reached). Each iteration is $O(S^2 A)$ complexity. \n",
       "\n",
       "\n",
       "Proof that values will converge: \n",
       "\n",
       "Case 1: if tree has max depth M then $V_M$ will hold actual values\n",
       "\n",
       "Case 2: if discount < 1, then the difference between $V_k$ and $V_{k+1}$ will be at most $\\gamma$ $^k$ $max|R|$ different. As k increases the values will converge. \n",
       "\n",
       "\n",
       "Alternative to value iteration is policy iteration. Define $V^{\\pi}(s)$ as the expected total award of starting in s and following $\\pi$. Update with formula: $V^{\\pi}_{k+1} (s) = \\sum_{s'} T(s, \\pi (s), s')] [R(s, \\pi (s),s') +$ $\\gamma$ $V_k^{\\pi}(s')]$ Each update is $O(s^2)$ in time complexity. \n",
       "\n",
       "\n",
       "We can also solve for V's under fixed policy using solvers for linear system, since there is no max operations involved. \n",
       "\n",
       "\n",
       "It is easier to select actions from q-values than from state-values. \n",
       "\n",
       "\n",
       "Three problems with value iteration: (1) slow runtime, (2) \"max\" at each state does not often change, and (3) policy can converge must quicker than values. \n",
       "\n",
       "\n",
       "Policy iteration steps: \n",
       "\n",
       "1) Policy evaluation: calculate utilities for fixed policy until values converge \n",
       "\n",
       "$V^{\\pi}_{k+1} (s) = \\sum_{s'} T(s, \\pi (s), s')] [R(s, \\pi (s),s') + $ $\\gamma$ $V_k^{\\pi}(s')]$\n",
       "\n",
       "\n",
       "2) Policy improvement: update policy with one-step lookahead, using resulting converged utilities as future values. \n",
       "\n",
       "$\\pi_{i+1}(s) = argmax_a \\sum_{s'} T(s,a,s')[R(s,a,s') + $ $\\gamma V^{\\pi_i}(s')$]\n",
       " \n",
       "3) Go back to step 1 until policy does not change \n",
       "\n",
       "\n",
       "Value iterations do not track the policy, only implicitly recompute it during the max operations. \n",
       "\n",
       "Theorem from Ramsey, von Neumann & Morgenstern: \n",
       "\n",
       "If preferences satisfy orderability, transitivity, continuity, substitutability, and monotonicity, then there is a utility function corresponding to it preserving the preferences. \n",
       "\n",
       "\n",
       "## Module 8 Definitions\n",
       "* **Markovian transitions** - Probability of arriving in s' only depends on the previous state\n",
       "* **Meu principle** - agent should choose action that maximizes expected utility.\n",
       "\n",
       "\n",
       "# Module 9: Reinforcement Learning\n",
       "## AIMA Chapter 17.3 Bandit Problems\n",
       " Bandit problem is a formal models for many important problems, e.g. deciding when to apply certain treatments / investments. \n",
       "\n",
       "\n",
       "Definition of Bandit Problems: \n",
       "\n",
       "(1) Each arm is a Markov Reward Process (MRP): MDP with only one possible action. \n",
       "\n",
       "(2) Overall bandit problem as an MDP with state space = Cartesian product of all states. \n",
       "\n",
       "(The arms are independent, and agent can only pull on at a time) \n",
       "\n",
       "\n",
       "Gittins index of a lever is defined as: \n",
       "\n",
       "$\\lambda$ = $max_{T > 0}$ $ (E(\\sum^{T-1}_{t=0} \\gamma ^t R_t)) / (E(\\sum^{T-1}_{t=0} \\gamma ^ t))$ \n",
       "\n",
       "\n",
       "Gittins index captures the maximum obtainable utility per unit of discounted time. Optimal policy for any bandit problem is to pull arm with higest GIndex then update the indices. Since index of arm only depends on info about that arm, optimal decision of first iteration can be calculated in O(n) time, and subsequent decision in O(1) time. \n",
       "\n",
       "\n",
       "At the point where optimal policy in indifferent between an arm and a fixed arm, the value of M is same as choosing an infinit sequence with reward = Gittens index. \n",
       "\n",
       "\n",
       "Restart MDP: replace the choice to enter an infinite sequence with lambda reward with choice to go back and restart M from initial state. Gittens index for M = (1 - $\\lambda$) times value of optimal policy for restart MDP. When we solve the MDP we can get Gittens index. \n",
       "\n",
       "\n",
       "Bernoulli bandit problem has each arm with reward 0 or 1. Cannot solve for Gittins index directly using restart MDP since state space is infinite, but we can approximate by truncating the MDP with a discount factor and a limit on numbers of trial. Result is preference for higher payoffs + an exploration bonus. \n",
       "\n",
       "\n",
       "Methods to generate almost optimal policies: (1) use Upper Confidence Bound (UCB) heuristic, using sampels to establish confidence interval for the payoff of each arm. Approximate index value for arm = \n",
       "\n",
       "\n",
       "$UCB(M_i) = \\mu_i + g(N) / \\sqrt(N_i)$ \n",
       "\n",
       "\n",
       "(2) Thompson sampling, which chooses an arm with probability related to the probability that the arm is optimal. Regret grows as O(log N) for both (1) and (2). \n",
       "\n",
       "\n",
       "Non-indexable problems: \n",
       "\n",
       "\n",
       "(1) Selection problem: choosing best options when there is fixed cost for each. No index function unlike bandit problems. Selection solvers will explore more than bandit solvers. \n",
       "\n",
       "\n",
       "(2) multitasking, or bandit superprocesses, or BSPs: Generalized bandit problems where each bandit is a full MDP in their own right. In BSPs globally optimal problem may require acting suboptimally locally. We often need to be greedier in each MDP. We can solve BSPs by modeling interactions between each MDP using opportunity cost. We can also convert a bandit arm into a MRP by coming up with an optimal dominating policy, which gives ceilign to the value of acting in the arm. Lower bound can be given by computing Gittens index. Then we can use search and recomputing bounds to solve BSPs.\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.1 Learning from Rewards\n",
       " As opposed to teaching agent the optimal moves, it is easier to give them rewards because (1) rewards are concise and easy to specify, and (2) we don't need to have the expertise. A problem is sparse rewards - when many states do not give reward at all. \n",
       "\n",
       "\n",
       "Two kinds of reinforcement learning appraoches:\n",
       "\n",
       "\n",
       "1) Model-based reinforcement learning: agent uses a transition model, learn a utility function U(s) in terms of sums of rewards from state onward.\n",
       "\n",
       "\n",
       "2) Model-free reinforcement learning: two forms. a) action utility learning. most common form is Q-learning - agent learns a Q-function with sum of rewards when taking certain action in certain state. b) Policy search: agent learns a policy, behaves like reflex agent. \n",
       "\n",
       "\n",
       "Passive reinforcement learning fixes the policy. Active RL requires agents to alos decide what to do/ i.e. exploration. \n",
       "\n",
       "\n",
       "## AIMA Chapter 22.2 Passive Reinforcement Learning\n",
       " Given fixed policy, agent tries to learn the expected discounted reward of beginning from state s. \n",
       "\n",
       "\n",
       "Direct utility estimatation: agent conducts trials, treat each resulting reward as a sample. Problem is that it ignores the connection between states (i.e. given by Bellma equation). Therefore it converges very slowly. \n",
       "\n",
       "\n",
       "Adaptive dynamic programming (model-based, passive, online): agent includes learned transition model into Bellman equation to calculate utilities. Since Bellman is linear, linear algebra packages can be used to solve them. \n",
       " \n",
       "\n",
       "\tdef passive_adp: \n",
       "\n",
       "\t(input: current state s' and reward r)\n",
       "\t(persistent: $\\pi$, mdp with model P, rewards R, actions A, discount $\\gamma$, table of utilities U, $N_{s'|s,a}$ table of outcome counts, s, and a (initially null)) \n",
       "\n",
       "  \t if s' is new then U[s'] = 0\n",
       "\n",
       "  \t if s is not null then:\n",
       "\n",
       "  \t\t increment $N_{s'|s,a}$[s,a][s']\n",
       "\n",
       "  \t\t R[s,a,s'] = r\n",
       "\n",
       "  \t\t add a to A[s]\n",
       "\n",
       "  \t\t P($\\dot$|s,a) = NORMALIZE($N_{s'|s,a}$[s,a])\n",
       "\n",
       "  \t\t U = policy_evaluation($\\pi$, U, mdp)\n",
       "\n",
       "  \t\t s,a = s', $\\pi$s' \n",
       "\n",
       "  \t\t return a \n",
       "\n",
       "\n",
       "We can also use modified policy iteration with simplified value iteration for updating utility estimates after each change to the learned model. \n",
       "\n",
       "\n",
       "Passive ADP provides a standard by which other RL algorithms are measured. \n",
       "\n",
       "It is impracticle for large state spaces. May take some time to \"discover\" rarely visited states \n",
       "\n",
       "\n",
       "Temporal difference learning (model-free, passive, offline): update utility of a state every time it transitions to a new state, with the function: \n",
       "\n",
       "\n",
       "$U^{\\pi}(s) = U^{\\pi}(s) +$ $a$ $[R(s, \\pi (s), s') +$ $\\gamma$ $U^{\\pi}(s') - U^{\\pi}(s)]$ \n",
       "\n",
       "\n",
       "$a$ stands for the learning rate. The term multiplied with alpha is an error signal. If $a$ becomes a function that decreases with the number of state visits, then $U^{\\pi}(s)$ will converge to the correct value. \n",
       "\n",
       "\n",
       "TD learning does not converge as fast, but is simpler and require fewer computations, since it does not require a transition model. TD adjusts utility based on observed successors, while ADP adjusts utility based on all successors. TD makes one adjustment every observed transition, while ADP makes as many as it needs. \n",
       "\n",
       "\n",
       "We can exstend TD appraoch to generate psuedoexperiences - to make the utility estimates closer to ADP, at the cost of higher computation time. \n",
       "\n",
       "\n",
       "We can also make ADP more efficient in the other way. Prioritized sweeping heuristic prioritizes adjusting states whose possible successors have got a large adjustment in utility estimates. Speeds up approximate ADP and allows it to handle larger state spaces than ADP. \n",
       "\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.3 Active Reinforcement Learning\n",
       " Greedy agents suffer from the problem of no exploration. To improve, we use GILE, or \"Greedy in the limit of infinite exploration\". GILE scheme tries each action in eact states certain times to avoid missing optimal actions. \n",
       "\n",
       "\n",
       "GILE scheme 1: random action at each point. Problem is slow conversion. \n",
       "\n",
       "\n",
       "GILE scheme 2: give weight to unexplored actions. Include exploration function f(u, n) which increases with u and decreases with n. E.g. we can define f(u, n) to return max reward if an action is not yet explored for k times, and return u otherwise. \n",
       "\n",
       "\n",
       "We should use $U^+$ (optimistic estimate of utility/ expected reward-to-go) for updates instead of U, since it propagates benefits of exploration. \n",
       "\n",
       "\n",
       "Need to handle the problem of entering absorbing states and exercising irreversible actions. We should therefore prefer policy that works well across all models, even if the policy is not optimal for the model with max probability. Three appraoches:\n",
       "\n",
       "\n",
       "1) Bayesian Reinforcement Learning: assumes a prior probaility, than update with each observation optimal policy is one with highest expected utility. Requires priors to be able to tell danger. Problem can be transformed into an exploration POMDP where belief states are distribution over models, and can be solved before agent starts moving. \n",
       "\n",
       "\n",
       "2) Robust control theory: defines optimal policy as one that performs best in worst case. Essentially maximin rule - may result in conservative policy \n",
       "\n",
       "\n",
       "3) Use human expertise: have human teachers do a demo, or write down constraints. \n",
       "\n",
       "\n",
       "Temporal difference Q-learning: model-free learning with agents learning Q(s,a) rather than U(s). It does not need a transition model. Update function is: \n",
       "\n",
       "\n",
       "$Q(s,a) = Q(s,a) + alpha [R(s,a,s') +$ $/gamma$  $max_{a'} Q(s',a') - Q(s,a)$\n",
       "\n",
       "\n",
       "Q-learning agent cannot look into the future, so if rewards are sparse it would find difficulty. It is off-policy, since it assumes stopping the current policy and adopting the best policy according to estimates. \n",
       "\n",
       "\n",
       "SARSA (state,action,reward,state,action): similar to Q-learning, except that SARSA waits until an action is taken, then update based on the taken action. If exploration gives negative reward, SARSA will punish the action but Q-learning will not. It is on-policy, since it assumes continuing with current policy. SARSA is more appropriate when we do not have full control over what the agent does. \n",
       "\n",
       "\n",
       "\n",
       "## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n",
       " In large gamespaces, convergence of utilities will be slow. We can approximate utility/Q-function instead, using weighted linear combination of features. \n",
       "\n",
       "\n",
       "E.g. Approximating direct utility estimation: parametrize the utility, then find best fit parameters (we use online learning, so we update parameters after each trial). Define error has half of the squared difference between predicted and actual utility. Use the Widrow-Hoff rule / delta rule for online least-squares. \\ \n",
       "If true utility function is linear than learning will be easy; if not, we can have features be nonlinear functions of state variables, can continue to use linear function approximators. \n",
       "\n",
       "\n",
       "Approximate TD learning, similar idea: vary the parameters and optimize. Converge in the case of TD learning and when function approximator is linear in features. When using active learning and non-linear functions, there may be problems with infinite parameters. Also problem with castastrophic forgetting - learning too well that it forgot previous bad experience. Potential solution is experience relay - remind self of bad experience at regular intervals. \n",
       "\n",
       "\n",
       "Deep RL goes beyond linear function approximators for 2 reasons: 1) there may not be good linear functions, 2) we may not be able to come up with features. More complex function but same gradients as before. Deep RL can play games well but may not handle varied environments. \n",
       "\n",
       "\n",
       "Credit assignment problem: agent needs to figure out where it does wrong. Solution is reward shaping: use additional pseudorewards. Need to ensure that optimal policy is not changed with their addition: we can use potential function $+ \\gamma \\phi (s') - \\phi (s)$ to reflect the aspect we want to keep track of. \n",
       "\n",
       "\n",
       "Hierarchical RL: break up long sequences of action into small pieces. Builds partial programs that extend normal programming languages and provide general know-how. \n",
       "\n",
       "\n",
       "HRL based on joint state space: (physical state, machine state) pairs. Define choice state as state at which program is at a choice point. Transform HRL into a Markovian decision problem, with state = choice states, actions = choices, reward = rewards for physical transitions. For transition model, if c chooses physical action, then use physical model if computational action taken, then modifies computational states. Solving MDP gets use the optimal policy consistent with partial program. \n",
       "\n",
       "\n",
       "Two additional advantages of HRL: 1) more opportunity for reward shaping, 2) provides additive decomposition of overall utility function. \n",
       "\n",
       "\n",
       "## Module 9 Lecture Notes\n",
       " \n",
       "In MDPs agents are given reward function and transition function (we know probability distribution of the states). In reinforcement learning agents must learn these. Tradeoffs between exploration and exploitation \n",
       "\n",
       "\n",
       "Double bandits (come from slot machines): actions are play blue vs play red states are win and lose. No discount, 100 time steps, and both states have the same value. If probabilities are specified, offline planning can solve MDP - we don't actually need to play. Without the probabilties, we need to learn rather than plan. Learning can be much harder than solving a known MDP. \n",
       "\n",
       "\n",
       "In RL, agents choose an action, receive reward from environment. Agents' utility is defined by reward function. Agent must learn to maximize expected rewards. All learning will be based on observed samples of outcomes. \\ We still assuming MDP with set of states, set of actions, model T, and reward function R, with T and R hidden initially. \n",
       "\n",
       "\n",
       "Two different kinds of learning: model-based learning and model-free learning \n",
       "\n",
       "\n",
       "Model based learning: learn an approximate model of R and T, then use MDP algo. Count outcomes for each action, normalize to estimate transition function, then discover rewards and come up with maximum likelihood estimates. E.g. fix input policy, start at different points, observe episodes. \n",
       "\n",
       "\n",
       "Passive RL: Every time we take an action, we compare what is expected vs what happens. Does not keep track of T and R. Given fixed policy, we want to learn state values. Execute policy and learn from experience. Goal is to compute values for every state under the action $\\pi$. Direct evaluation - average the observed sample values. \n",
       "\n",
       "\n",
       "In each passive RL training episode, we run through the policy to the end (from each state) and compute the value of state accordingly. \n",
       "\n",
       "\n",
       "Benefits of direct evaluation: \n",
       "\n",
       "1. easy to understand\n",
       "\n",
       "2. does not need to understand R and L\n",
       "\n",
       "3. Eventually compute the correct average values\n",
       "\n",
       "\n",
       "Disadvantages: \n",
       "\n",
       "1. Waste information about state connections (learn value of state wihtout looking at other states), \n",
       "\n",
       "2. each state learned separately \n",
       "\n",
       "3. So, takes a long time to learn\n",
       "\n",
       "\n",
       "We can improve using policy evaulation, which use connections between states. However, normal policy evaluation uses weighted average, which requires knowing the weights. Instead, we can use sample-based policy evaluation: for each (state, action) pair, collect sample resulting stages, sum up sample values and divide by num of samples. However, that requires \"Crystal ball\" simulation of results. \n",
       "\n",
       "\n",
       "Temporal difference learning: update V(s) every time we experience a transition that starts from s. Likely outcomes will lead to updates more often. Every time we have sample, we update V(s) as: $V^{\\pi}(s) = (1 - alpha) V^{\\pi}(s) + alpha * sample$. Alpha is a value we set which says how much our observations impact vs old V. \n",
       "\n",
       "\n",
       "We end up calculating an exponential moving average, which weights current observation more heavily than new observations. Forgets about the past (but the past is not as important). $x_n = (x_n + (1 - alpha) x_{n-1} + (1 - alpha)^2 x_{n-2} + ... ) / (1 + (1 - alpha) + (1 - alpha) ^ 2 + ...)$ \n",
       "\n",
       "\n",
       "TD value learning is model-free, mimicking Bellman updates with moving averages. \n",
       "\n",
       "\n",
       "Problems with TD value learning: we do not have Q-values, so we can't turn values into a new policy. \n",
       "\n",
       "\n",
       "Q-value iteration: given Qk, calculate k+1 q-values for all q-states. Formula: \n",
       "\n",
       "\t $Q_{k+1}(s,a) = \\sum_{s'} T(s, a, s') [R(s, a, s') + \\gamma max_{a'} Q_k (s', a')]$ \n",
       "\n",
       "\n",
       "Acting RL/ Q-learning: sample based Q value iteration. When we receive our sample, we compare to our previous esimate, then update. \n",
       "\n",
       "\n",
       "$Q(s,a) = (1 - a) Q(s,a) + a * [sample]$ \n",
       "\n",
       "\n",
       "Q-learning converges to optimal policy - example of off-policy learning. Caveats: we need to explore enoguh, we will eventually have to make learning rate small enough but not decrease it too quickly. \n",
       "\n",
       "\n",
       "Choosing when to explore: \n",
       "\n",
       "1. Simplest way: random actions ($\\epsilon$-greedy). WIth small probability = $\\epsilon$ we act randomly else act on current policy. Problem: value of random exploration decreases as we go. Solution 1: lower $\\epsilon$ over time. Solution 2: explicitly write a function \n",
       "\n",
       "2. Exploration functinos: explore areas whose value has not yet established. Function $f(u,n) = u + k / n$, u = value estimate, n = visit count, k: constant bonus, returns an optimistic utility. \n",
       "\n",
       "\n",
       "Modified Q-update: $Q(s,a) = R(s,a,s') + \\gamma max_{a'} f(Q(s',a'), N(s',a'))$. vs just using old Q value in regular Q update. \n",
       "\n",
       "\n",
       "We can quantify effectiveness of exploration strategy using regret. Regret measures total mistake cost: difference between expected rewards and optimal rewards. Minimize regret = requires us to reach optimal policy as quickly as we can. Random exploration has higher regret. \n",
       "\n",
       "\n",
       "Approximate Q-learning: Some problems have many states, so we need to generalize states. Basic q-learning keeps track of all q values, approx learning learn about small number of states, then generalize learning to new states. Similar to evaluating chess states using features. Use a linear value function that assigns weights to each feature. Pro: experienced summed up in a few numbers. Disadvantage: features may not adequately predict values. \n",
       "\n",
       "\n",
       "E.g. $Q(s,a) = w_1 f_1(s,a) + w_2 f_2(s,a) + .... $\n",
       "\n",
       "\n",
       "\t Transition = (s, a, r, s')\n",
       "\n",
       "\t Difference/error term = $[r + \\gamma max_{a'} Q(s', a')] - Q(s,a)$\n",
       "\n",
       "\t Q(s,a) = Q(s,a) + $alpha$ [difference]  (Exact Q's) \n",
       "\n",
       "\t\t $w_i = w_i + alpha$[difference]$f_i (s,a)$  (Approximate Q's)\n",
       "\n",
       "\t\t (we update each weight instead of the entire Q state) \n",
       "\n",
       "\n",
       "We adjust the weights of active features as we go. If something bad happens, blame them feature that were on.\n",
       "\n",
       "\n",
       "## Module 9 Definitions\n",
       "* **Prioritized sweeping heuristic** - Adjustment to ADP which prioritizes states whose likely successors have just undergone large adjustment in utilities\n",
       "* **Adaptive dynamic programming** - Estimates transition model from counts, then apply policy evaluation \n",
       "* **Temporal difference learning** - Learning that uses difference in utilities between successive states\n",
       "* **Greedy in the limit of infinite exploration (glie)** - Agent must try each action in each state an unbounded number of times\n",
       "* **Robust control theory** - Defines optimal policy as one that gives best outcome in the worst case\n",
       "* **Exploration** - trying previously unknown states to gain information\n",
       "* **Exploitation** - exercising currently known best action to obtain rewards\n",
       "* **Markovian transitions** - transitions depends only on the current state but not previous states.\n",
       "* **Environment history** - a sequence of states/actions on which the utility function is dependent\n",
       "* **Policy** - instruction that tells agents what kind of action they should carry out under each state\n",
       "* **Optimal policy** - Policy that yields the highest expected utility\n",
       "* **Finite horizon** - a fixed time point after which the game is done\n",
       "* **Nonstationary policy** - a policy that is dependent on time\n",
       "* **Infinite horizon** - games with an infinite state sequence (may still contain terminal state(s)\n",
       "* **Additive discounted rewards** - utility function taht defines utility as sum of rewards weighted by a discount factor\n",
       "* **Stationaray preferences** - if we prefer one future to another starting tomorrow, we would prefer it even if it starts today\n",
       "* **Proper policy** - a policy where an agent is guaranteed to reach a terminal state\n",
       "* **Bellman equation** - Utility of one state = expected reward for next transition + discounted utility of the next state. \n",
       "* **N-armed bandit** - slot machine with n levers\n",
       "* **Absorbing state** - state where no actions from agent has any effect and no rewards will be received\n",
       "* **Reward to-go** - expected reward starting from a certain state\n",
       "\n",
       "\n",
       "# Module 10: Probabilies and Language Models\n",
       "## AIMA Chapter 12.1 Acting under Uncertainty\n",
       " For real world problems, we often cannot generate a contingency plan for every possible state. To do so will result in (1) too many contingenicies considered, (2) belief-state with many unlikely possibilities, and (3) no way to compare value of plans when results are not guaranteed. Approach also suffer from qualification problem (we do not know what to exclude) \n",
       "\n",
       "\n",
       "Rational decision will depend on both importance of a goal and the probability at which it will be achieved. \n",
       "\n",
       "\n",
       "Problems with using logical rules to model the real world: 1) laziness (too much work to list all possibilities), 2) theoretical ignorance (do not have complete theory for the domain), 3) practical ignorance (have not tested knowledge under domain) \n",
       "\n",
       "\n",
       "To deal with degrees of belief we use probability theory. Logic and probability theory share the same ontological commitment (that the world is made of facts that are true or false in particular case), but differnet epistemological commitment. \n",
       "\n",
       "\n",
       "Decision theory = probability theory + utility theory. Fundamental idea is that a rational action chooses actions with teh highest expected utility, averaged over all possible outcomes of the action. (MEU principle)\n",
       "\n",
       "\n",
       "## AIMA Chapter 12.2 Basic Probability Notation\n",
       " Set of all possible world = sample space. Possible worlds are mutually exclusive and collectively exhaustive. Probability model is mapping between possible world and probabilities. Two basic axioms: prob between 0 and 1, and all add up to 1. \n",
       "\n",
       "\n",
       "Set of possible worlds is called outcomes (vs in logic, set of worlds = a proposition). \n",
       "\n",
       "\n",
       "Unconditional = prior probabilities = degrees of belief without extra info. \n",
       "\n",
       "\n",
       "Conditional = posterior probabilties = degrees of belief given evidence. \n",
       "\n",
       "\n",
       "We use (in this chapter) a factored representation to describe sets of possible worlds. \n",
       "\n",
       "\n",
       "Random variable = mapping from possible worlds to a range (i.e. set of possible values that it can take on). Bernoulli distribution has range {0, 1}. \n",
       "\n",
       "\n",
       "We can use bold P(RV), a vector of values, to represent probability of all values of a Random variable.  This will define a probability distribution for the RV. This will work for categorical distribution, where range is finite and discrete. \n",
       "\n",
       "\n",
       "For continuous variables , we define probability density function. Probabiltiy densities are not probabilites: they are measured with a unit. PDFs will show us how likely it is fall a value to fall close to a the value we enter. \n",
       "\n",
       "\n",
       "Joint probability distribution: table specifying distribution of multiple variables. \\ \n",
       "Full joint probability distribution: joint distribution of all random variables. \n",
       "\n",
       "\n",
       "From the two basic axioms we can derive: \n",
       "\n",
       "1) Probability of negation is 1 - probability \n",
       "\n",
       "2) Inclusion-exclusion principle: P(a or b) = P(a) + P(b) - P(a and b)\n",
       "\n",
       "\n",
       "Kolmogorov's axioms = inclusion-exclusion and two basic axioms \n",
       "\n",
       "\n",
       "Debates about whether our degrees of belief really need to obey axioms. De Finett's theorem: if agent's belief violates axioms of probabilty then they can be taken advantage of by another agent. \n",
       "\n",
       "\n",
       "Objection: what if one refuses to bet, rejoinder: one unavoidably participate in games like those. \n",
       "\n",
       "\n",
       "## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n",
       " Probabilistic inference is the process of computing posterior probabilities for query propositions, given observed evidence \n",
       "\n",
       "\n",
       "Marginalization/ summing out is the process of taking out variables by summing up all their possible values. \n",
       "\n",
       "\n",
       "Conditioning rule: $P(y) = \\sum_z P(y|z) P(z)$ \n",
       "\n",
       "\n",
       "Combining these rules yield the general inference function: \n",
       "\n",
       "$P(X|e) = a P(X,e) = a \\sum_y P(X,e,y)$ where E = list of evidence variables, e = observed values, Y = remaining unobserved variables. \n",
       "\n",
       "\n",
       "Using this approach with full joint distribution, we can answer prob queries for discrete variables. However, it takes input table of size $O(2^n)$ and time of $O(2^n)$ .\n",
       "\n",
       "\n",
       "## AIMA Chapter 12.4 Independence\n",
       "\n",
       "If a and b are independent then: \n",
       "\n",
       "P(a|b) = P(a) and P(b|a) = P(b) and P(a and b) = P(a)P(b)\n",
       "\n",
       "\n",
       "Independence can be used to deconstruct a joint distribution. E.g. distribution of coinflip can be represented as product of n single-variable distribution instead of a table with $2^n$ entries. \n",
       "\n",
       "\n",
       "## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n",
       " Bayes rule: $P(b|a) = P(a|b)P(b) / P(a)$\n",
       "\n",
       "\n",
       "Alternatively, P(cause|effect) = P(effect|cause)P(cause)/P(effect)\n",
       "\n",
       "\n",
       "P(effect|cause): quantifies relationship in causal direction. P(cause|effect) describes diagnostic direction. Diagnostic knowledge is often more fragile than causal knowledge. \n",
       "\n",
       "\n",
       "Conditional independence: P(X,Y|Z) = P(X|Z)P(Y|Z)\n",
       "\n",
       "\n",
       "We can also understand conditional independence as some common cause seperating two possible effects. It can be used to reduce representation from O(2^n) to O(n) and hence help prob systems scale up.\n",
       " \n",
       "\n",
       "## AIMA Chapter 12.6 Naive Bayes Models\n",
       " Naive bayes model uses equation: P(Cause, effect1, effect2,...) = P(cause) product of (P(effect|cause)).\n",
       "\n",
       "\n",
       "Run time of calculaiton is linear in number of observed effects and does not depend on unobserved ones. \n",
       "\n",
       "\n",
       "Naive Bayes assumes that words occur independently in documents.  \n",
       "\n",
       "## AIMA Chapter 12.7 The Wumpus World Revisited\n",
       " We can use naive bayes and conditional independence to calculate probabilities for unobserved aspects of Wumpus world.  \n",
       "\n",
       "## SLP Chapter 3.1 N-Grams\n",
       "Chain rule: $P(X_1, X_2, .... X_n) = P(X_1)P(X_2|X_1)P(X_3|X_{1:2})...P(X_n|X_{1:n-1})$.\n",
       "\n",
       "\n",
       "One way to come up with MLE is using relative frequency: Counts of n-gram in a corpus, divided/normalized by sum of n-grams that share the same first word(s). \n",
       "\n",
       "\n",
       "We can use log probabilities to prevent numerical underflow. p1p2p3p4 = exp(log p1+ log p2 + log p3 + log p4).\n",
       "  \n",
       "\n",
       "## SLP Chapter 3.2 Evaluating Language Models\n",
       " Extrinsic evaluation: apply LM in an application and see how much it improves. Intrinsic evaluation: evaluates without application. \n",
       "\n",
       "\n",
       "For intrinsic evaluation we need a test set, a training set, and a holdout set. We want to avoid training on the test set, which results in inaccurate perplexity. We may also want to introduce a development set / deveset which is a fresh test set that is never seen. We want a test size with a size that gives us enough statistical power to measure a stat. significant difference between 2 models. An example is 80% training, 10% dev and 10% test. \n",
       "\n",
       "\n",
       "Minimizing perplexity = maximizing test set probability given the language model. Perplexity is also same as weighted average branching factor of a language. With branching factor of language = num of words that can follow any word. If a word is very frequent perplexity will be lower. \n",
       "\n",
       "\n",
       "Using n-grams with higher n's give us more info and hence lower perplexity. If the n-gram model is constructed with knowledge of test set (which should be forbidden), the perplexity will be artificially low. \n",
       " \n",
       "\n",
       "## SLP Chapter 3.3 Sampling sentences from a language model\n",
       " We want to ensure that training and test set is in similar genres, and that the training data has appropriate dialect and variety. \n",
       "\n",
       "\n",
       "Problem of sparsity: some n-grams may have never appeared in training set (zeros). Two resulting problems: we may underestimate many probabilities, and we will estimate the prob of test set as 0. \n",
       "\n",
       "\n",
       "In more extreme case a word can never been seen (unknown words/ out of vocabulary (OOV) words). % of OOV words = OOV rate. \n",
       "\n",
       "\n",
       "Two ways to handle unknown word <UNK>: (1) turn problem into closed vocab. First choose a fixed vocab, then convert every OOV word to <UNK>, then estimate prob of <UNK> as a regular word. (2) Create an implicit vocab, by replacing words in training date by <UNK> based on their frequency. E.g. replace any word that appears fewer than k times by <UNK>, or choose top 50000 words and rest as <UNK>. \n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 3.4 Generalization and Zeros\n",
       " Smoothing/ discounting = taking away probability mass from high freq events and adding to unknown events. \n",
       "\n",
       "\n",
       "Laplace smoothing: adding one to all bigram counts. $P_{Laplace}(w_i) = (c_i + 1) / (N + V)$ Adjusted count $c* = (c_i + 1)$ $N / (N+V)$ can be turned into probability by normalizing by N. \n",
       "\n",
       "\n",
       "We can also see smoothing as discounting. Laplace smoothing = applying a discount $d_c = (c*)/c$.\n",
       "\n",
       "\n",
       "Downside of Lapalce smoothign: sharp changes in probabilities, since a lot of mass is moved to the zeros. \n",
       "\n",
       "\n",
       "Alternative: add-k smoothing, add a fractional count k. We can choose k by optimizing on a devset. \n",
       "\n",
       "\n",
       "Backoff: using n-gram if evidence is sufficient, else use n-1-gram, and n-2 and so on. We discount higher order n-grams to save some probability mass for lower order n-grams. Backoff with discount is also called Katz backoff. Katz backoff can be combined with a smoothing method called Good-Turing to become Good Turing backoff. \n",
       "\n",
       "\n",
       "Interpolation: mixing probability estimates from all n-gram estimators. Assign a $\\lambda$ weight to each n-gram, and select lambda by learning from held-out corpus (additiona corpus to learn hyperparameters). We can also use EM algorithm to find $\\lambda$. \n",
       "\n",
       "\n",
       "\n",
       "## SLP Chapter 3.5 Smoothing\n",
       " Absolute discounting: subtract a fixed discount d from all counts. High counts will not be affected much while small counts will be modified more. $P_{AbsoluteDiscounting}(w_i | w_{i-1})$ = $(C(w_{i-1}w_i - d))/(\\sum_v C(w_{i-1}v)) + \\lambda (w_{i-1}) P(w_i)$ \n",
       "\n",
       " \n",
       "Kneser-Ney smoothing: instead of getting unigram model, we seek the probability of a word appearing as a novel continuation. $P_{continuation} (w) = |{v: C(vw) > 0}|$ $ \\sum_{w'} |{v: C(vw') > 0}|$. Frequent word appearing in only a few contexts will have low continuation probability. \n",
       "\n",
       "\n",
       "P_{KN} (w_i | w_{i-1}) = max(C(w_{i-1}w_i) - d, 0) / C(w_{i-1}) + \\lambda (w_{i-1}) P_{continuation} \n",
       " \n",
       "\n",
       "\n",
       "Modified Kneser-Ney: use three different discounts for n-grams with counts of 1, 2, and 3+ \n",
       "\n",
       "## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n",
       " Given random variable X, entropy is: \n",
       "\n",
       "\n",
       "H(X) = - \\sum_{x \\in range(X)} p(x) log_2 p(x) \n",
       "\n",
       "\n",
       "Entropy calculated using log base 2 is measured in bits. \n",
       "\n",
       "\n",
       "Entropy = lower bound on number of bits that is needed to encode a decision given an optimal encoding scheme. \n",
       "\n",
       "\n",
       "Entropy for sequence (of words) = replace p(x) with $p(W^n_1)$. \n",
       "\n",
       "\n",
       "Entropy rate/ per-word entropy = entropy of sequence / number of words. \n",
       "\n",
       "\n",
       "To calculate entropy for a language that is regular, we can calculate the entropy of a single sequence that is long enough. By Shannon-McMillan-Breiman theorem. \n",
       "\n",
       "\n",
       "A stochastic process is stationary if probability it assigns to a sequence are invariant with respect to shifts in the time index. Natural language is not stationary, since there can be long-distance depedency. Markov models are stationary, and are therefore only approx of natural language \n",
       "\n",
       "\n",
       "Cross-entropy: defined on m, a model of probability p that we do not know. H(p, m) = $lim_{n -> \\inf} - 1 / n \\sum_{W \\in L} p(w1, ..., wn) log m (w1, ..., wn)$\n",
       "\n",
       "For stationary process it can be calculated as H(p, m) = $lim_{n -> \\inf} - 1 / n  log m (w1, ..., wn)$.\n",
       "\n",
       "\n",
       "Cross entropy H(p,m) is upper bound on entropy H(p) for any model m. The difference between H(p,m) and H(p) is the measure of the accuracy of a model. \n",
       "\n",
       "\n",
       "Perplexity of a model P on sequence W can be defined as the exponent of cross-entropy. Perplexity(W) = $2 ^ {H(W)}$\n",
       "\n",
       "\n",
       "## Module 10 Lecture Notes\n",
       " Agents often know (observed variables) evidence about the state of the world but need to reason about unobserved variables (states) of the world. A model is how the known variables related to the unknown. Probabilistic reasoning is the underlying process. \n",
       "\n",
       "\n",
       "Probability is about how likely a possible world is. Possible worlds = sample space, and they are mutually exclusive and collectively exhaustive. Fully specified probability model associates probability P(w) with each possible world w. \n",
       "\n",
       "\n",
       "Random variable is an aspect of the world that may be uncertain. Denoted with capital letters. \n",
       "Two axioms:\n",
       "\n",
       "1) Probability of any possible world is between 0 and 1\n",
       "\n",
       "2) Total probability of all possible worlds is 1\n",
       "\n",
       "\n",
       "Random variables have distributions. Distribution = table of probabilities of values, indexed by outcomes. If all domain entries are unique we can use shorthand: P(hot) = P(T=hot) \n",
       "\n",
       "\n",
       "Joint distribution specifies probability for each assignment to a set of RVs. Size of distribution with n variables and domain size d = $d^n$. \n",
       "\n",
       "\n",
       "Probabilistic model: joint distribution over a set of R.V.s. Assignments are called outcomes, probabilties normalized (sum to 1.0), and ideally only certain variables directly interact. (i.e. many will be independent). \n",
       "\n",
       "\n",
       "Event = set E of outcomes. P(E) = P(x1) + P(x2) + ... for all x in E. \n",
       "\n",
       "\n",
       "In many cases the events we care about are partial assignments. We can use marginal distributinons - subtables with variables eliminated. Process of marginalization = combining eliminated rows by adding. \n",
       "\n",
       "\n",
       "Definition of conditional probability: P(a|b) = P(a,b) / P(b). A given B. Defined in terms of joint probability. \n",
       "\n",
       "\n",
       "Probabilistic inference is to use known probablities to compute desired probabilities. Update beliefs based on evidence. \n",
       "\n",
       "\n",
       "Inference by enumeration: We have (1) evidence variables, (2) Query variable, and (3) hidden variables. Should sum up to all variables. We want P(Q | e1e2....ek). Steps: (1) select entries consistent with evidence, (2) sum H to get P(Q, e1e2...) = sum over h1h2h3... of P(Q and h1h2h3 and e1e2e3...), (3) normalize by dividing with $\\sum_q P(Q,e1e2...ek)$. \n",
       "\n",
       "\n",
       "To construct joint probability table, we use product rule P(y)P(x|y) = P(x and y). Generalize to chain rule: P(x1, x2, x3) = P(x1) P(x2|x1) P(x3 | x1,x2). \n",
       "\n",
       "\n",
       "Two ways to factor joint distribution over two vars: \n",
       "\n",
       "P(x,y) = P(x|y)P(y) = P(y|x)P(x)\n",
       "\n",
       "\n",
       "Bayes rule: $P(x|y) = \f",
       "rac{P(y|x)}{P(y)} P(x)$ Allows building one conditional from the reverse. \n",
       "\n",
       "\n",
       "Probabilistic language model: take each word position as an RV estimate probability distribution to the next RV. Goal is to compute prob of a sentence/ a sequence of words, or compute probability of upcoming word. A model that computes either P(W) or $P(w_n | w_1, w_2...)$ is a language model or a grammar. \n",
       "\n",
       "\n",
       "$P(w1,w2,...wn) = \\prod_i P(wi | w1,w2,w_{i-1})$. \n",
       "\n",
       "\n",
       "Maximum likelihood estimate: count and divide. Problem is that some sentences would never appear. We will not get enough data. \n",
       "\n",
       "\n",
       "Markov assumption: we can use the current state alone to predict the next state. In language models, this mean the next word will only depend on the last k words. \n",
       "\n",
       "\n",
       "unigram = no history used, bigram = 1 word, trigram = 2 words, 4-gram = 3 words as history. \n",
       "\n",
       "\n",
       "Markov built LM by counting up 20k letter in Onegin in 1913. 1948 Shannon approximates English by n-grams. 1956 Chomsky paper suggests that finite state Markov Model is not a good model for human languages. Fred Jelinek in 1980 uses n-gram for ASR, and in 1993 develops statistical machine translation. \n",
       "\n",
       "\n",
       "Formula for machine translation: $argmax_e p(e|f) = p(e) p(f|e)$\n",
       "\n",
       "\n",
       "## Module 10 Definitions\n",
       "* **Product rule** - P(a and b) = P(a|b) P(b)\n",
       "* **Normalization constant** - a constant that, when applied, ensures a     distribution will add up to 1\n",
       "* **Language models** - Models that assign probabilities to sequences     of words\n",
       "* **Markov assumption** - Prob of a word depends only on the previous word\n",
       "* **Maximum likelihood estimation (mle)** - Estimated paremeters of a probability     distribution, s.t. probability of observed data is the highest\n",
       "* **Perplexity** - Inverse probabilty of test set, normalized by number of words\n",
       "* **Closed vocabulary system** - test set can only contain words from a lexicon\n",
       "* **Open vocab system** - system in which we model unknown words as <UNK>\n",
       "\n",
       "\n",
       "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n",
       "## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n",
       " Bayesian network: each node is a random variable. No cycles, so a DAG. Each node has probability info that quantifies effect of parents on the node using a number of parameters. Intuitive meaning of an arrow is that X has a direct influence on Y. \n",
       "\n",
       "\n",
       "Local probability information in each node is represented in a conditional probability table. Each row contains cond prob for a conditioning case. \n",
       "\n",
       "\n",
       "Constructing Bayesian networks: (1) determine set of variables required to model the domain. (2) order them, with the network being more compact if causes precede effects in the order, (3) choose minimal set of parents for X_i such that P(X_i | X_(i-1)...X1) = P(X_i | Parents(X_i)), (4) insert a link from each parent to X_i, and (5) write down conditional probability table. \n",
       "\n",
       "\n",
       "The above construction method guarantees acyclicity, and no redundant prob values. So there cannot be inconsistency or violation of axioms of probability. \n",
       "\n",
       " \n",
       "\n",
       "## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n",
       " P(x1,...,xn) = product(P(x_i | parents(X_i))) defines the semantics of Bayesian networks. \n",
       "\n",
       "\n",
       "Bayesian networks are locally structured, or sparse systems. Each subcomponent interact directly only with a bounded number of other components, resulting in higher compactness. \n",
       "\n",
       "\n",
       "We can often suppose most RVs are influenced by at most k others for some k. THe conditional prob table will have at most 2^k numbers so complete network can be specified with 2^k n numbers vs 2^n \n",
       "\n",
       "\n",
       "Node ordering can affect how compact a NB is. WIth a causal model we will have to specify fewer numbers and numbers are easier to come up with. \n",
       "\n",
       "\n",
       "We can prove the property that in a NB each variable is cond independent of its non-descendents, given its parents. \n",
       "\n",
       "\n",
       "Also the property that a var is cond indp of all other notdes in the network, given its parents, children, and children's parents (i.e. its Markov blanket). \n",
       "\n",
       "\n",
       "Markov blanket property enables inference algos that use completely local and distributed stochastic sampling processes. \n",
       "\n",
       "\n",
       "To examine whether a set Z d-separates X and Y (X is cond indp \\of Y given Z), we:\n",
       "\n",
       "\n",
       "1) consider ancestral subgraph with just XYZ and ancestors. \n",
       "\n",
       "\n",
       "2) Add links between unliked pairs that share common child (create a moral graph)\n",
       "\n",
       "\n",
       "3) replace directed by undirected links.\n",
       "\n",
       "\n",
       "4) If Z blocks all paths between X and Y then Z d-separates X and Y.\n",
       "\n",
       "\n",
       "We can use canonical distribution to generalize the relatipnship between parent and children nodes. E.g. 1) deterministic nodes, 2) context specific indepdenecne (CSI), with two vars being cond indp provided certain values of others. \n",
       "\n",
       "\n",
       "We can use noisy logical relationships (e.g. noisy OR) to model uncertain relationships. The model assumes that all possible causes are listed (we can add a leak node to fill up miscell items), and that inhibition of each parent is indp from inhibition from others. Noisy relationships depending on k parents can be described using O(k) parameters instead of the full O(2^k). \n",
       "\n",
       "\n",
       "To model continuous vars we can: 1) use discretization, 2) define using standard prob density functions (e.g. Gaussian), 3) use non-paremetric representation, 4) use hybrid Bayesian Network with both discrete and continuous values. \n",
       "\n",
       "\n",
       "Example of distributions: linear-Gaussian conditional distribution, with mean varying linearly with value of parent and fixed s.d. Networking with only vars with linear-Gaussian distributions have joint distirbution that is a multivariate Gaussian distrbution. \n",
       "\n",
       "\n",
       "When discrete vars are added as paretns of cont vars, network defines a conditional Gaussian distribution.\n",
       "\n",
       "\n",
       "Probit and expit(inverse logit) model can model relationship with soft thresholds.\n",
       "  \n",
       "\n",
       "## AIMA 13.3 Exact Inference in Bayesian Networks\n",
       " Inference by enumeration: O(n 2^n) runtime with naive implementation, and O(2^n) if we take advantage of the nested structure of computation. We calculate P(X|e) = alpha * $\\sum_y P(X, e, y)$ Evaluation trees are calcualted in depth-first, left to right recurison, like backtracking and DPLL algorithm \n",
       "\n",
       "\n",
       "Variable elimination reduces repeated calculation. We evaluate expressions in right-to-left order, storing intermediate results. We can annotate each part of expression with the corresponding factor, a matrix indexed by values of its argument varible.s Then we can sum the factors up and get the pointwise product between them. Pointwise product yields a new factor whose varaibles are the union of vars of original factors, and elements are the product of corresponding elments. \n",
       "\n",
       "\n",
       "\n",
       "Different orderings chosen for the algorithm will generate differnet algo. Time and space requirements of var elimination is dominated by the largest factor constructed. \n",
       "\n",
       "\n",
       "Complexity of exact inference depends on structure of network. If there is at most one undirected path between two nodes, then the network if singly connected or called polytrees. The time and space complexity of exact inference is linear in size of network, with size being number of CPT entries. \n",
       "\n",
       "\n",
       "For multiply connected networks, var elimination can have exponential runtime in the worst case. Inference in Bayes nets is NP-hard (we can reduce satisfiability problems to BN inference problems). \n",
       "\n",
       "\n",
       "We can also reduce Bayes net inference to satisifability and use develoepd solvers. e.g. weighted model counting. \n",
       "\n",
       "\n",
       "Answering multiple queries (e.g. for all vars) can be done in O(n) time with clustering algorithms (or join tree algos). \n",
       "\n",
       "\n",
       "We can join indiv nodes to form clusters such that resulting network is a polytree. Nodes are replaced by meganodes. We then use a special purpose inference algo with a technique like constraint propagation to compute posterior probabilties in time lienar to size of clustered network. \n",
       "\n",
       "\n",
       "\n",
       "## Module 11 Lecture Notes\n",
       " X and Y are conditionally independent given Z if P(x,y|z) = P(x|z)P(y|z), or equivalently iff P(x|z,y) = P(x|z) \n",
       "\n",
       "\n",
       "Proof of equivalence: p(x|y,z) = p(x,y,z)/p(y,z) = p(z)p(x,y|z)/p(y,z) = p(z)p(x|z)p(y|z)/p(z)p(y|z) = P(x|z). \n",
       "\n",
       "C-independence is a property of a distribution. E.g. Alarm indp Fire given Smoke \n",
       "\n",
       "\n",
       "Unconditional independent is very rare, and does not help reasoning on probability. Conditional independence consistutes the most basic and robust form of knowledge about unknown environments. \n",
       "\n",
       "\n",
       "In Bayes net, nodes are variables with domains. Can be assigned/observed or unassigned (unobserved). Arcs are interactions between variables, showing direct influence/ conditional independence (but an arrow does not necessarily mean causal influence). \n",
       "\n",
       "\n",
       "Bayes net are direct, acyclic graph. Each node X with parents A1, A2, ... has a distribution for each combination of parent values. Same as topology + local conditional probabilities. \n",
       "\n",
       "\n",
       "Joint distributions are implicitly encoded in Baynes nets, as products of local conditional distributions. Joint probability can be obtained by multiplying all conditional probabilities, assuming conditional independence.\n",
       "\n",
       "\n",
       "Baynes net does not need to be causal: arrows can represent correlation rather than causation. The topology in Baynes net should be taken to represent conditional independence (P(xi|x1,...,x(i-1))) = P(xi|parents(Xi)). \n",
       "\n",
       "\n",
       "An N-node net with up to k-parents per node has O(N 2^(k+1)) size. Space savings compared to having full joint probability model. \n",
       "\n",
       "\n",
       "In Bayes net, we assume $P(x_i | x_1..x_{i-1}) = P(x_i | parents(X_i))$. i.e. xi only depends on its defined parents. \n",
       "\n",
       "\n",
       "There are often additional assumptions that can be read off the graph. e.g. d-separation. \n",
       "\n",
       "\n",
       "Suppose we have causal chain where x -> y -> z. We can prove that X is indp of Z given y by showing p(z|x,y) = p(z|y). i.e. evidence along the chain blocks causal influence. \n",
       "\n",
       "\n",
       "Suppose we have common cause Y which influences X and Z. X is conditionally independent of Z given y by showing p(z|x,y) = p(z|y). i.e. seeing the cause blocks the influence between effects \n",
       "\n",
       "\n",
       "Suppose we have X and Y causing a common effect Z (v-strucutre). X and Y are independent. X and Y are not independent given Z (seeing effect activates influence between possible causes).  \n",
       "\n",
       "\n",
       "General question: are two variables independent in a given Bayes Net? We reason using the three cases above. Shade evidence nodes in the graph and examine resulting graph. \n",
       "\n",
       "\n",
       "Wrong approach: any two connected nodes not blocked by a shaded node is c. indp. Problem is with v-structures. \n",
       "\n",
       "\n",
       "Correct approach: define active triples: \n",
       "\n",
       "(U = unobserved, O = observed)\n",
       "\n",
       "1) U -> U -> U/O \n",
       "\n",
       "2) U <- U -> U \n",
       "\n",
       "3) U -> O <- U \n",
       "\n",
       "4) U -> (U->O) <- U \n",
       "\n",
       "vs inactive triples\n",
       "\n",
       "1) U -> O -> U\n",
       "\n",
       "2) U <- O -> U\n",
       "\n",
       "3) U -> U <- U \n",
       "\n",
       "\n",
       "Any path is active if each triple is active. \n",
       "\n",
       "\n",
       "If there are no active paths between X and Y then they are conditionally indp. i.e. d-separated. \n",
       "\n",
       "\n",
       "D separation algorithm: want to know if i, j are cond indp given evidence. Approach: check all undirected paths between i and j. If one or more paths are active, than indp not guaranteed. Otherwise guaranteed.\n",
       "\n",
       "\n",
       "Note that a path is active only if all triples along the path are active. \n",
       "\n",
       "\n",
       "Inference is the process of getting useful numbers from joint probabilty distribution e.g. posterior probability, most likely explanation. Inference by enumeration: sum hidden variables, calculate P(Q|e1,e2...). \n",
       "\n",
       "\n",
       "We can do inference by enumeration in bayes net but it will be expensive. Exponential time with respect to non observed variables. Reason is that we need to join the whole joint distribution before summing out hidden variables. To prevent that we can do variable elimination (np hard but faster), interleave joining and marginalizing.  \n",
       "\n",
       "\n",
       "## Module 11 Definitions\n",
       "* **Hidden variables** - Vars in Bayes net that are neither input nor output\n",
       "\n",
       "\n",
       "# Module 12: Naive Bayes and Perceptrons\n",
       "## AIMA Chapter 19.1 Forms of Learning\n",
       " Machine learning uses factored representation, representing input as a vector of attribute values. When output is one of finite set of values, then it is a classification problem. If it is a number, then a regression problem. \n",
       "\n",
       "\n",
       "Three types of feedback: supervised learning (input -> label). Unsupervised learning e.g. clustering. Reinforcement learning agents have rewards and punishments. \n",
       "\n",
       "## AIMA Chapter 19.2 Supervised Learning\n",
       " Overfitting models have high variance. Underfitting models have high bias. There is often a bias-variance tradeoff. \n",
       "\n",
       "\n",
       "There is also a tradeoff between how expressive a hypo space is and the complexity of finding a good hypo within that space. \n",
       " \n",
       "\n",
       "## AIMA Chapter 19.3 Learning Decision Trees\n",
       " It is intractable to find a guaranteed smallest decision tree. We can use a greedy divide-and-conquer strategy: testing most important attribtues first then recursively solve smaller subproblems. \n",
       "\n",
       "\n",
       "We can evaluate performa nce using a learning curve, training test size vs proportion of correct answers. Learning curves are also called happy graphs. \n",
       "\n",
       "\n",
       "We can measure importance in terms of information gain or entropy. An RV with only one possible value has entropy 0. A fair coin has 1 bit entropy. \n",
       "\n",
       "\n",
       "Info gain is the expected reduction in entropy: Gain(A) = B(p/(p+n)) - Remainder(A). with B(q) being entropy of boolean RV with prob q. \n",
       "\n",
       "\n",
       "Overfitting is more likely when number of attributes increase, and less liekly when number of training examples increase. \n",
       "\n",
       "\n",
       "Decision tree pruning reduces overfitting, by elim nodes that are not necessarily relevant. A significance test using chi-squared statistic is used to determine if a node is significant. \n",
       "\n",
       "\n",
       "Early stop (combining info gain and chi-squared pruning) has the risk of missing combination of attributes that are informative. \n",
       "\n",
       "\n",
       "To deal with continuous variables, we can use a split point test (inequality test on value). For attributes with many finite values, we can use an information gain ratio, or use equality test. For numerical prediction we need a regression tree rather than a classification tree. \n",
       "\n",
       "\n",
       "Decision tree can suffer from suboptimal policy because of greedy search, expensive runtime when it is too deep, and unstable since adding one example can change the whole tree. Random forest can mitigate these effects. \n",
       "  \n",
       "\n",
       "## AIMA Chapter 19.4 Model Selection and Optimization\n",
       " In machine learning we make the stationarity assumption: that each example has same prior prob distribution, and that it is indp from previous examples. Examples that satisfy these are called indp and identiiccally distributed or i.i.d. \n",
       "\n",
       "\n",
       "Optimal fit = hypo that minimizes the error rate, measured by performance on test set. For hyperparameters (parameters of model class rather htan individual model) we need a dev set. \n",
       "\n",
       "\n",
       "Extreme k-fold cross validation uses k = n, aka leave-one-out cross-validation or LOOCV. Each example will serve both as training and validation data but not at the same time. \n",
       "\n",
       "\n",
       "Model selection algorithm: takes in as argument a learning algo, which in turns take in a hyperparameter (size) as the argument. It measures the error rate for each value of size. It picks the value that minimizes error. \n",
       "\n",
       "\n",
       "Decision tree often has U shaped curves while CNNs often have decreasing curves. \n",
       "\n",
       "\n",
       "A model that exactly fits training data has interpolated the data.\n",
       "\n",
       "\n",
       "An improvement upon error rate is loss function, which allows us to emphasize what errors matter more. Most general formulation L(x, y, ypred) = Utility (result of using y given input x) - Utility (result of using ypred)\n",
       "\n",
       "\n",
       "Best hypothesis is the one with minimum expected generalization loss defined as sum of Loss * probability. Since prob is not known, we use instead empirical loss as an estimate. \n",
       "\n",
       "\n",
       "Four reasons why empirical loss may differ from true function: \n",
       "\n",
       "\n",
       "1) Unrealizability: the hypo space does not contain the true f. \n",
       "\n",
       "\n",
       "2) variance: algo returns different hypo for different sets of examples. \n",
       "\n",
       "\n",
       "3) Noise: f may be nondeterminisitc. \n",
       "\n",
       "\n",
       "4) Computational complexity: we cannot afford to search all possibilities. \n",
       "\n",
       "\n",
       "Recent years use more large-scale learning than small-scall learning.\n",
       "\n",
       " \n",
       "\n",
       "## AIMA Chapter 19.5 The Theory of Learning\n",
       " To figure out the number of samples needed is the question for computational learning theory. \n",
       "\n",
       "\n",
       "Underlying priunciple of CLT is that any hypo consistent with a sufficiently large examples is unlikely to be seriously worng, or probably approximately correct (PAC). \n",
       "\n",
       "\n",
       "PAC theorems assume stationarity. Hypothesis is called approximately correct if error(h) is smaller than or equal to a defined bound epsilon. It is close to the true function or lie inside an epsilon-ball around true functino f. \n",
       "\n",
       "\n",
       "Number of required examples, as function of epsilon and delta (probabilty of H_bad containing at least one consistent hypo) is called the sample complexity of the learning algo.  \n",
       "\n",
       "\n",
       "Dilemma between restricitng hypo space and not eliminating the true function. Three ways to mitigate: 1) bring prior knowledge, 2) have the algo return a simple hypo, instead of complex 3) focus on learnable subset of the hypo space (decision lists). \n",
       "\n",
       "\n",
       "k-DL: decision list with at most k conjunctions. We can show that k-DL is learnable, i.e. can be approximated accurately after training on suffiicent examples. We then find an efficent algo that returns a consistent decision list (e.g. Greedy algo decision list learning, repeatedly find test that agrees exactly with some subset of the training set).\n",
       "\n",
       "\n",
       "## AIMA Chapter 19.6 Linear Regression and Classification\n",
       " Linear regression covers a diff hypo space: the calss of linear functions over continuously valued inputs. \n",
       "\n",
       "\n",
       "We traiditonally use squared-error loss function L2, summed over all training examples. We find weights that minimize loss.\n",
       "\n",
       "\n",
       "For univariate linear regression, the weight space is 2d. The loss functino is convex for every linreg problem with L2 loss functino. \n",
       "\n",
       "\n",
       "Gradient descent is use to minimize loss. While not converged yet, we update each weight using $w = w - \u0007lpha $ (partial derivative of loss over w). \n",
       "\n",
       "\n",
       "alpha s the step size, or rate at which we minimize los. Can be fixed or decay over time. For univariate, loss is quadratic so partial derivative is linear. \n",
       "\n",
       "\n",
       " Batch/deterministic gradient descent: convex loss surface (no local minima), so will eventually converge, but may be slow.\n",
       "\n",
       "\n",
       "Stochastic gradient descent: randomly select number (minibatch) of training examples at each step, May be hard to converge, but often faster. SGD is useful for online setting where data is coming in and stationarity may not hold. \n",
       "\n",
       "\n",
       "For multi-variate regression there may be risk of overfitting, so we can use regularization. L1 regularization has the advantage that it tends to produce a sparse model, pushing many weights to zero. \n",
       "\n",
       "\n",
       "Linear functions can be used for classfication by serving as decision boundaries. or linearseparators. When data are linearly separable, percetron learning rule converges to a perfect linear separator. \n",
       "\n",
       "\n",
       "An improvement upon perceptron is logistic regression, which uses the logistic function logistic(z) = $1 / (1 + e^{-z})$. \n",
       "\n",
       "\n",
       "$h_w (x) = logistic(w \\dot x) = 1 / (1 + e^{-w \\dot x})$. \n",
       "\n",
       "\n",
       "\n",
       "## SLP Chapter 4.1 Naive Bayes Classifiers\n",
       " Text categorization tasks: sentiment analysis, authorship attribution, spam detection, language id. Mostly done through supervised ML. \n",
       "\n",
       "\n",
       "Probabilistic classifier: assign prob to observation belonging to classes. Generative classifier (e.g. naive Bayes): build model of how a class generates input data. Discriminative: learn what features are most useful. Generative classifiers are more popular. \n",
       "\n",
       "\n",
       "\n",
       "Naive bayes model. Compute most probable class given document d, by choosing class with the highest product of prior probability p(c) and likelihood of document p(d|c). \n",
       "\n",
       "\n",
       "Two simplifying assumptions: (1) bag of words assumption that positions do not matter. (2) naive Bayes assumption/ conditional independence assumption that probabilites $P(f_i | c)$ are independent given class c and hence can be naively multiplied. \n",
       "\n",
       "\n",
       "Naive bayes calculations are also done in log space. In log space, predicted class is computed as linear function of input features. THerefore naive Bayes is also a linear classifier.  \n",
       "\n",
       "## SLP Chapter 4.2 Training the Naive Bayes Classifier\n",
       " P(c) = number of docs in the class / total num of docs \n",
       "\n",
       "\n",
       "P(w|c) = count(w,c) / count of all words in c. \n",
       "\n",
       "\n",
       "Problem of zero: one zero cond prob will cause product to be zero. Simple solution is add-one/ Laplace smoothing. \n",
       "\n",
       "\n",
       "P(w|c) = (count(w,c) + 1) / ((count of all words in c) + |size of all words in all classes|)  \n",
       "\n",
       "\n",
       "Solution for unknown words (words in test data but not training data): ignore them, remove from test document. \n",
       "\n",
       "\n",
       "Stop words (very frequent words): either completely ignore or include (does not affect performance).\n",
       " \n",
       "\n",
       "## SLP Chapter 4.3 Worked Example\n",
       " Worked example of naive bayes: drop unknown word in test set, then multiply P(w|c) for each words in test set and P(c). \n",
       "\n",
       "## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n",
       " Binary multinomial naive bayes: remove all duplicate words before concatenating them into big document. Emphasize existence of words to frequency. \n",
       "\n",
       "\n",
       "Dealing with negatives: (1) prepend prefix NOT_ to words. (2) use parsing (more complext). \n",
       "\n",
       "\n",
       "Dealing with insufficient training data: we can draw from sentiment lexicons, lists of words pre-annotated with positive/ negative sentiments. Popular lexicons include General Inquirer, LIWC, opinion lexicon and MPQA Subjectivity Lexicon. One way to use it is to add a feature tha is counted whenever a word in a lexicon occurs. \n",
       "\n",
       "## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n",
       " For tasks like spam detection, we can use predefined likely set of words/phrased as features, coupled with non-linguistic features. \n",
       "\n",
       "\n",
       "For tasks like language id, most effective features are not words but character n-grams like zw and nya, or byte n-grams (which include spaces). Some system (langid.py) uses all possible n-grams of lenghts 1-4 and use feature selection to choose useful ones. \n",
       "\n",
       "## SLP Chapter 4.6 Naive Bayes as a Language Model\n",
       " NB model can be seen as a set of class-specific unigram language models, wth the model of each class (e.g. positive and negative) instantiating a unigram language model.  \n",
       "\n",
       "## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n",
       " Human labels to evaluate models = gold labels. \n",
       "\n",
       "\n",
       "Confusion matrix helps visualize how a model performs. Dimensions = gold labels and system output. \n",
       "\n",
       "\n",
       "Accuracy = (tp + tn) / (tp + fp + tn + fn) \n",
       "\n",
       "\n",
       "Precision = tp / (tp + fp) \n",
       "\n",
       "\n",
       "Precision: percentage of items labeled as positives that are true positives. \n",
       "\n",
       "\n",
       "Recall = tp / (tp + fn) \n",
       "\n",
       "\n",
       "Recall = % of items actually present in the input that is correctly identified by the system. \n",
       "\n",
       "\n",
       "\n",
       "F-measure incorporates both precision and recall. \n",
       "\n",
       "\n",
       "$F_B = ((B^2 + 1)PR) / (B^2 P + R)$\n",
       "\n",
       "\n",
       "B parameter weights the importance of recall and precision, with B > 1 favoriting recall and vice versa. B = 1 is the most freq used, called $F_1$. \n",
       "\n",
       "\n",
       "F-measure is a weighted harmonic mean of precision and recall. It is a conservative metric - closer to the minimum of each number than arthmetic mean. \n",
       "\n",
       "\n",
       "To compute metrics for more than two classes, we can use microaveraging (collect decision for all classes into a single confusion matrix, and compute precision and recall from that table), or macroaveraging (compute performance for each class then average over classes).\n",
       " \n",
       "\n",
       "## SLP Chapter 4.8 Test sets and Cross-validation\n",
       " To be able to use some data for training and still use all for test, we can use cross-validation: randomly choose division of data, into training and test, train classifier, compute error rate on test, rinse and repeat. Do n times and average n runs to get average error rate = n-fold cross-validation. \n",
       "\n",
       "\n",
       "It is common to create fixed training and test set, then do n-fold cross-validation inside training set, but comput error way normally in test set. \n",
       "\n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 4.9 Statistical Significance Testing\n",
       " To compare the performance of two system is to do statistical hypothesis testing. \n",
       "\n",
       "\n",
       "p-value: probability that we would see an observed measure assuming null hypothesis. Statistically significant if p value is below a certain threshold. \n",
       "\n",
       "\n",
       "NLP generally does not use simple parametric tests like t-tests or ANOVAS because of their assumptions. We use non-paremetric tests based on sampling e.g. approximate randomization and bootstrap test.  \n",
       "\n",
       "\n",
       "Bootstrap test: can be applied to any metric.  Bootstrapping = repeatedly drwaining large numbers of smaller samples with replacement from an original larger sample. Assume that sample is representative of population. \n",
       "\n",
       "\n",
       "We create b test sets from boostrapping, then compute statistics on how often A has accidental advantage. E.g. we can count how many times the delta exceeds the expected zero value by delta(x) or more. \n",
       "\n",
       "\n",
       "## SLP Chapter 4.10 Avoiding Harms in Classification\n",
       " ML systems replicate and even amplify biases in trainig data or labels. E.g. representational harms: harms by a system that demeans a social group, e.g. assigning lower sentiment scores. E.g. toxicity detection, which =can result in censorship.\n",
       "\n",
       "\n",
       "To deal with harms we can release a model card, including information such as training algo and parameters, sources, processing, intended use, and performance over different demographics. \n",
       "\n",
       "## Module 12 Lecture Notes\n",
       " ML gives us a way to acquire a model from data, learning paremeters (e.g. probabilities), structure (e.g. Bayens net graphs), and hidden concepts (e.g. clustering). \n",
       "\n",
       "\n",
       "In naive bayes model and perceptron our task is classification. Naive bayes is model-based approach: we build a model with random variables being the labels and features, then instantiate observed features, and query for the distribution of the label conditioned on the features. Major challenges are deciding the structure of the BN and deciding how we should learn the paremeters. \n",
       "\n",
       "\n",
       "Naive Bayes model for digits: assume all features are independent effects of the label. Feature values are on or off depending on whether intensity > 0.5. Every input will be represented with a feature vector, and we predict P(Y|product(features)) = P(Y) * product (P(feature | Y)). (with independence assumption) \n",
       "\n",
       "\n",
       "Full joint distribution of the left side has size |Y| |F|^n values,  with |F| being possible values for features and n being number of features. With independence assumption, the right side only has n |F| |Y| parameters, meaning that it is linear in number of features. \n",
       "\n",
       "\n",
       "Steps for inference in Naive Bayes: (1) Get joint probability of label and evidence for each label P(y1,f1,...,fn). (2) sum to get probability of evidence. (3) Divide step (1) by step (2) to normalize. \n",
       "\n",
       "\n",
       "Data we need: (1) P(Y): prior of labels, (2) P(Fi|Y) for each feature, i.e. evidence variable.These probabilities are called parameters / theta. \n",
       "\n",
       "\n",
       "Data in ML: \n",
       "\n",
       "\n",
       "(1) Training set\n",
       "\n",
       "\n",
       "(2) Held-out data/ development set: optimize hyperparameters \n",
       "\n",
       "\n",
       "(3) Test set \n",
       "\n",
       "\n",
       "A typical experimentation cycle: learning paremeters on training set -> tune hyperparemeters on holdout set -> compute accuracy of test set (fraction of correct predictions). \n",
       "\n",
       "\n",
       "Overfitting: fitting the training data too closely, but cannot generalize. E.g. if we have not seen a word (0 probability) we can get positive infinities In response we need to smooth or regularize the estimates. \n",
       "\n",
       "\n",
       "Ways to estimate parameters: (1) ask a human, (2) learn from training data (empirical rate). Empricial rate = estimate (of the value of the feature) that maximizes the likelihood of the data. \n",
       "\n",
       "\n",
       "Laplace smoothing: pretend we saw everything one more time than we did. Can derive the estimate with Dirichlet priors. If we use k instead of 1 for Laplace smoothing, k is called the strength of the prior. \n",
       "\n",
       "\n",
       "Laplace smoothing does not do well for P(X|Y) when |X| or |Y| is very large. Linear interpolation: $P_{lin}(x|y) = alpha P(x|y) + (1 - alpha) P(x)$. \n",
       "\n",
       "\n",
       "Hyperparameters: amount/ type of smoothing, k, alpha. We should not tune hyper-paremeters on training data since that way they would not generalize. \n",
       "\n",
       "\n",
       "Baseline: simple straw man procedures that help determine how hard the task is and what is a good accuracy. E.g. labeling everything as spam for spam filters.\n",
       "\n",
       "\n",
       "Perceptron: inputs * weights -> apply activation function and compare to threshold -> output. Instead of the threshold we can use a bias b = -threshold, so that output = 0 if w $\\dot$ x + b $\\leq$ 0. \n",
       "\n",
       "\n",
       "In graphical represenation, classification decision will be represented by a decision boundary (line, plane, or hyperplane), orthogonal to the weight vector. If input vector has less than 90 degree distance to weight vector than positive, else negative. A bias would move the decision boundary away from the origin. \n",
       "\n",
       "\n",
       "To learn weights, we start with weight = 0, classify with current weights then if wrong we update with w = w + y* $\\dot$ f. \n",
       "\n",
       "\n",
       "With multiple classes we can have a weight vector for each class. Whichever class with highest score will win. If we have wrong predictions we subtract from weight of the wrong label and add to weight of right label. \n",
       "\n",
       "\n",
       "Perceptron properties: \n",
       "\n",
       "\n",
       "1) if the data is linearly separable, then we are guaranteed to converge on a set of parameters that offer right decisions. Separability = if there exists parameters that get the training data set perfectly correct. \n",
       "\n",
       "\n",
       "2) if data is not separable, we can define mistake bound = maximum number of mistakes ((k)/(\\delta^2)$). delta^2 = margin = space between the two classes. \n",
       "\n",
       "\n",
       "Problems with percetron: \n",
       "\n",
       "\n",
       "1) If data is nonseparable, then weights may thrash, switching between non separable boundaries. We can use average perceptrons (averaging weihgt vectors over time)\n",
       "\n",
       "\n",
       "2) Mediocre generalization: we can only find a barely separating solution. \n",
       "\n",
       "\n",
       "3) Overtraining: can use held out set to adjust. Overtraining is kind of overfitting \n",
       "\n",
       "\n",
       "MIRA: Margin infused Relaxed Algorithm, chooses updata size that fixes the current mistake. Scale back updates by a factor of $\tau = \f",
       "rac{w_y - w_{y*} \\dot f + 1}{2f \\dot f}$ +1 in the formula helps us generalize. Sometimes we add a constant C to cap maximum value of tau. Helps perceptron to converge faster, esp. on noisy data. Cooresponds to optimization that assumes non separable data. \n",
       "\n",
       "\n",
       "Support vector machine: decide which separator is better. Find the separator with max margin between two different classes (by maximizing margins between support vectors and separator). \n",
       "\n",
       "\n",
       "Non linear separators: transform problems into higher dimensions, e.g. x into x^2, 2-d to 3-d. Sometimes call the kernel trick.\n",
       "\n",
       "\n",
       "Comparing Bayes with Perceptrons: Bayes builds a model, gives prediction probabilities, and have strong assumptions about feature independence. Need one pass through data. Perceptrons use less assumptions, is mistake-driven, and need multiple passes through data. Often more accurate. \n",
       "\n",
       "## Module 12 Definitions\n",
       "* **Consistent hypothesis** - Hypothesis such that each xi in training set has h(xi) = yi\n",
       "* **Bias** - tendency of a hypothesis to deviate from expected value when averaged over traininig sets\n",
       "* **Variance** - amount of change in hypo due to fluctuation in training data\n",
       "* **Decision tree** - function that maps a vector of attribute values to a single output value\n",
       "* **Regularization** - process of penalizing complex hypothesis\n",
       "* **Feature selection** - reduce dimensions by discarding irrelevant attributes \n",
       "* **Minimum description length** - minimizes number of bits required to specify empirical loss and complexity\n",
       "* **Grid search** - Try all combinations of hyperparameter values\n",
       "* **Random search samples** - alternative of grid search when there are too many combinations\n",
       "* **Bayesian optimization** - use machine learning to learn good hyperparameters\n",
       "* **Population based training** - Combination of random search and random mutations to find good hyperparameters\n",
       "* **Decision lists** - a series of tests, each of which is a conjunciton of literals\n",
       "\n",
       "\n",
       "# Module 13: Neural Networks\n",
       "## SLP Chapter 5.1 Classification: the sigmoid\n",
       " Log reg is a discriminative classifer since it identifies features that help identify classes. It directly compute P(c | d) rather than calculate it by multiplying likelihood and prior. \n",
       "\n",
       "\n",
       "Components of ML system for classification: 1) a feature representation of the input 2) a classification function that computes estimated class, 3) an objective function for learning (e.g. cross entropy loss), 4) algo for optimizing objective function e.g. SGD. \n",
       "\n",
       "\n",
       "Define z = w dot x + b. We apply simoid function on z to make it a value between 0 and 1. Sigmoid function squishes outliers to 0 or 1 is differentiable, and has defined range. \n",
       "\n",
       "\n",
       "Advantage of logistic over naive bayes: naive bayes have overly strong cond indp assumptions. When there are many correlated features log reg works much better. However, on very small datasets naive bayes can work well, and it is very fast to train (no optimization).\n",
       "\n",
       "\n",
       "## SLP Chapter 5.2 Learning in Logistic Regression\n",
       " We need a loss function that makes correct class labels more likely - a conditional maximum likelihood estimation. We want to maximize log probabilty of true y labels in the training data, so we reduce negative log likliehood loss = cross entropy loss function. \n",
       "\n",
       " \n",
       "\n",
       "## SLP Chapter 5.3 The cross-entropy loss function\n",
       " Log reg has convex loss function with just one minimum. For multi layer neural networks we have non convex loss function. \n",
       "\n",
       "## SLP Chapter 5.4 Gradient Descent\n",
       "Gradient: a vector that expresses the directional components of the sharpest slope along each of the N dimensions. \n",
       "\n",
       "\n",
       "Gradient for log regression: $(\\sigmoid (w \\dot x + b) - y) + x_j$\n",
       "\n",
       "\n",
       "Stochastic gradient descent: online algorithm, compute gradient after each training sample. 1) compute estimated output, 2) compute loss, 3) derivative, 4) update. Stochastic because one random example is chosen at a time. \n",
       "\n",
       "\n",
       "eta/ learning rate is a hyperparameter. We often start with a high one then slowly increase it, i.e. a function of iteration k. \n",
       "\n",
       "\n",
       "Batch training, compute gradient over entire data set. Can be costly in time \n",
       "\n",
       "\n",
       "Mini-batch: train m examples less than whole dataset. Cost function for mini-batch is average loss for each example. \n",
       "\n",
       "## SLP Chapter 5.5 Regularization\n",
       " Regularization term R(theta): added to objective function to avoid overfitting. \n",
       "\n",
       "\n",
       "L2 (Ridge) regularization: quadratic function of weight values. L2 Norm = Euclidean distance of vector from origin. \n",
       "\n",
       "\n",
       "L1 (Lasso) regularization: linear function of weights. L1 norm = Manhattan distance \n",
       "\n",
       "\n",
       "L2 easier to optimize because of simple derivative. L1's derivative is non-continuous at zero. \n",
       "\n",
       "\n",
       "L1 regularization leads to fewer features. \n",
       "\n",
       "\n",
       "L1 and L2 have Bayesian interpretations as constraints on how weights should look.\n",
       "\n",
       "\n",
       "## SLP Chapter 5.6 Multinomial logistic regression\n",
       " For classification with more than 2 classes we do multinomial logistic regression or softmax regression. \n",
       "\n",
       "\n",
       "Softmax is a generalization of sigmoid. Takes a vector and maps them to a probability distribution. \n",
       "\n",
       "\n",
       "Softmax(z_i) = (exp(z_i) / $\\sum^k_{j=1} exp(z_j))$, for $1 \\leq i \\leq k$\n",
       "\n",
       "\n",
       "Cross-entropy loss is the log of output prob corresponding to the correct class or the negative log likelihood loss. \n",
       "\n",
       "\n",
       "L(ypred, y) = $- \\log (exp(w_k \\dot x + b_k) / \\sum^k_{j=1} exp(w_j \\dot x + b_j)) $, with k being the correct class.\n",
       " \n",
       "\n",
       "## SLP Chapter 7.1 Units\n",
       " Neural network differs from logistic regression in: 1) more powerful, minimum neural network with 1 hidden layer can learn any function 2) do not need hand-derived features. \n",
       "\n",
       "\n",
       "Neural units apply a non-linear function to z = w \\dot x + b. The ouput is called an activation value. Sigmoid is one candidate but not common. Tanh is usually better: \n",
       "\n",
       "\n",
       "y = ($e^z - e^{-z}$)/ ($e^z + e^{-z}$) range from -1 to +1 \n",
       "\n",
       "\n",
       "ReLu: y = max(x, 0)\n",
       "\n",
       "\n",
       "tanh function is smoothly differntiable and map outlier values towards mean. ReLU has nice properties since it is close to linear. In sigmoid and tanh,\n",
       "high values of z results in saturated values of y (extremely close to 1 with derivatives close ot 0). \n",
       "\n",
       "\n",
       "Rectifiers escape the prblem of vanishing gradient, which occurs when the gradient gets too close to 0, resulting in low error signals.\n",
       "\n",
       "\n",
       "## SLP Chapter 7.2 The XOR problem\n",
       " Proof that we need multi-layer networks: Minksy and Papert show that a single neural unit cannot compute simple functions of the input. E.g. a perceptron cannot compute XOR. \n",
       "\n",
       "\n",
       "Intuition lies in fact that perceptron is a linear classifer, with the line acting as a decision boundary. XOR is not linearly separable. \n",
       "\n",
       "\n",
       "In contrasta two layer network with 3 ReLu units can compute XOR. \n",
       " \n",
       "\n",
       "## SLP Chapter 7.3 Feedforward Neural Networks\n",
       " Feedforward network: multilayer network where units are connected with no cycles. No outputs are passed back. Sometimes called multi-layer perceptrons, but a misnomer since they are not linear \n",
       "\n",
       "\n",
       "Fully-connected network has unit in each layer takes as inputs the outputs from all units in previous layer, so that there is link between every pair of units in two adjacent layers \n",
       "\n",
       "\n",
       "We can represent weights of a layer by a single matrix so that hidden layers can be computed with simple atrix operations. \n",
       "\n",
       "\n",
       "We can apply the softmax function to normalize a vector of real values. \n",
       "\n",
       "\n",
       "We can think of neural network as building a vector h (a hidden layer representation of the input) and then running standard logistic regression on the features in h. \n",
       "\n",
       "\n",
       "WE can add a dummy node a_0 with value always = 1 at each layer as a replacement of the bias term. \n",
       " \n",
       "\n",
       "## SLP Chapter 7.4 Training Neural Nets\n",
       " We use error backpropogation or backprop t ocompute the gradient of neural networks. It is the same as a more general procedure called backward differentiation, which depends on computation graphs. \n",
       "\n",
       "\n",
       "Computation graphs: each node is an operation and each edge shows output to input relationship. \n",
       "\n",
       "\n",
       "In backward pass we compute partial derivatives along each edge of computation graph from right to left. \n",
       "\n",
       "\n",
       "Optimization for neural networks is non-convex. We need to initialize the weights with small random numbers, and (optionally) noramlize input values to have 0 mean and unit variance. \n",
       "\n",
       "\n",
       "We can use dropout (randomly dropping some units and their connections) for regularization against overfitting.\n",
       "\n",
       "\n",
       "PyTorch and TensorFlow build neural networks using computation graph formalisms that enable gradient computation and parallelization over vector-based GPUs. \n",
       "\n",
       "## SLP Chapter 7.5 Neural Language Models\n",
       " Neural language models can have much higher accuracy than n-gram LMs, don't need smoothing, can handle longer histories and can generalize over contexts of similar words. However, they are slower to train. \n",
       "\n",
       "\n",
       "Feedforward neural LM takes in a representaiton of previous words and output probabilty distributino of next words. \n",
       "\n",
       "\n",
       "Prior context is represented by embeddings. We use a moving window with embedding vector representing each of k past words. We concatenate the vectors to product the input x.\n",
       "\n",
       "\n",
       "We can train network with ready-made embeddings through pretraining, or learn embeddings simultaneously. \n",
       "\n",
       "\n",
       "We use an embedding weight matrix (size $d * |V|$) with a column for each word, each a column vector of d dimensions. \n",
       "\n",
       "\n",
       "Three step forward pass: 1) select three embeddings from E, multiply the 3 one\\hot ]vectors created by the embedding matrix, to form the projection layer. ]\n",
       "2) Multiply by W and add b and pass through the ReLu activation function to get hidden layer h. 3) Multiply h by U, 4) apply softmax. \n",
       "\n",
       "\n",
       "Alternatively:\n",
       "\n",
       "1) $e = (Ex_1, Ex_2, ..., Ex)$ \n",
       "\n",
       "2) $h = \\sigmoid(We+b)$\n",
       "\n",
       "3) z = Uh\n",
       "\n",
       "4) ypred = softmax(z)\n",
       "\n",
       "\n",
       "  \n",
       "\n",
       "## Module 13 Lecture Notes\n",
       " ML classifiers require a training corpus of M observations with (xi, yi) input/output pairs. Also need 1) a feature representation of the input, 2) a classification function that computes estimated calss via p(y|x), 3) an objective function for learning, usually with minimizing error on training examples, and 4) an algorithm for optimizing the objective function (e.g. gradient descent). \n",
       "\n",
       "\n",
       "It used to be the case that we concern ourselves with coming up with useful features, but now we are more concerned with choosing the right structure for the neural network. \n",
       "\n",
       "\n",
       "Problem of sentiment classification. Logistic regression aims to learn vector of weights and bias term such that $z = \\sum_i w_i x_i + b$. With logistic regression we modify the range of z so that it can be interpreted as a probability. Apply sigmoid function $y = 1 / (1 + e ^{-z})$. Negative values for z = below 0.5 for y and vice versa. $P(y=1) = sigmoid(w \\dot x + b)$. P(y=0) = 1 - P(y=1). \n",
       "\n",
       "\n",
       "Making decisions: decision = 1 if P(y=1|x) > 0.5 else 0. .5 is the decision boundary. \n",
       "\n",
       "\n",
       "For learning the weights we need: (1) an objective function (e.g. cross-entropy loss). (2) an algorithm for optimizing objective function (e.g. gradient descent).\n",
       "\n",
       "\n",
       "Loss function helps determine how close our output is to the actual output. E.g. mean squared error =  $L(\\hat{y}, y) = (\\hat{y}-y)^2 / 2 $. Not a good function for optimizing. \n",
       "\n",
       "\n",
       "Conditional maximum likelihood estimation: choose w, b such that we maximize probabilities of correct labels in training data. Resulting loss function = cross entropy loss = negative log likelihood loss (to be minimized). \n",
       "\n",
       "\n",
       "Derivation: $p(y|x) = \\hat{y}^y (1 - \\hat{y})^{1 - y}$. i.e. when y is 1, then p(y|x) = $\\hat{y}$ and when y = 0, p(y|x) = $ 1- \\hat{y}$. Then log both sides to get $\\log p(y|x) = y \\log \\hat{y} + (1 - y) \\log (1 - \\hat{y})$. Put negative signs on both sides to get $L = -[y \\log \\hat{y} + (1 - y) \\log (1 - \\hat{y})]$ \n",
       "\n",
       "\n",
       "Correct prediction results in lower CE loss, while incorrect ones lead to higher CE losses. \n",
       "\n",
       "\n",
       "Loss on all training examples = $- \\sum_{i=1}^m L_{ce} (\\hat{y}^i | y^i)$\n",
       "\n",
       "\n",
       "Define theta = stuff we want to optimize (w, b). We compute $\\hat{\theta}$ = $argmin_{\theta} 1/m \\sum_{i=1}^m L_{ce} (y^i, x^i; \theta) $. \n",
       "\n",
       "\n",
       "Gradient descent is an algorithm to find minimum of a function by deciding which direction the function's slope is rising the most steeply, and moving in the opposite direction. (which way is down) \n",
       " \n",
       "\n",
       "Loss function is guaranteed convex for logisitc regression -> so one min point. If we have small enough step size we are guaranteed to find minimum, else we may jum around. In non-convex problem we may be stuck in \n",
       "\n",
       "\n",
       "Size of step = learning rate = eta. Update paremeter by: $w^{t+1} = w^t - \\eta d/dw f(loss, w)$. If derivative/slope is negative we increase weight, and vice versa.\n",
       "\n",
       "\n",
       "Extending 2-D gradient descent to higher dimensions: for each weight vector, we take $d/dw L(f(x;\theta), y)$. Collection of these derivatives is called $\\delta_{\theta}$. We update $\theta^{t+1} = \theta^t - \\eta \\delta_{\theta}$. \n",
       "\n",
       "\n",
       "The derivative of loss function wrt each $w_i$ = $(\\hat{y} - y)$ $\\dot$ x. We can obtain the gradient for multiple data points by summing up individual ones. \n",
       "\n",
       "\n",
       "Stochastic gradient descent: take a random training instance, then update based on it. Updates can fluctuate. \n",
       " \n",
       "\n",
       "Batch training: Computes gradient over entire dataset. Takes long time but give better estimate of which direction to move. \n",
       "\n",
       "\n",
       "We usually use mini batch training (batch size picked empirically, e.g. what we can fit into GPU card)\n",
       "\n",
       "\n",
       "Problems: (1) overfitting results in poor generalization. (2) If features only appear in one class it gets high weights, we can use regularization to penalize large weights (add regularization term to loss function). \n",
       "\n",
       "\n",
       "For multi-class classification, use multinomial logistic regression. Want to know the probability for each class. Can use generalization of sigmoid funciton called softmax function. Often the final step in neural network. Softmax takes input vectors and output vector of valeus normalized into probabilities. \n",
       "\n",
       "\n",
       "(Feed forward) Neural network with many neurons can approximate many functions. Deep learning = neural network with many layers. \n",
       "\n",
       "\n",
       "Perceptron has step function (0 or 1). Problem is that it is difficult to update weight vector - small change may cause huge swing btw 0 and 1. With sigmoid we have much smoother function: small changes in weight -> small changes in output. \n",
       "\n",
       "\n",
       "Activation functions: sigmoid, ReLU, tanh. Neural network units = \\ \n",
       "perceptron + activation function\n",
       "\n",
       "\n",
       "Totally connected: each node in next layer is connected to all inputs from previous layer.\n",
       "\n",
       "\n",
       "Neural network requires different way of computing gradient: we need back propagation. Related to the general procedure called backward differentiation, which depends on computation graphs (representation of the process of computing a mathematical expression). Use chain rule to compute derivation of loss function L with respect to a, b, and c. \n",
       "\n",
       "\n",
       "Chain rule: $\f",
       "rac{df}{dx} = \f",
       "rac{df}{da} \f",
       "rac{da}{dx}$. Get derivatives on each node of the graph, then multiply nodes along a path to get a destination derivative. \n",
       "\n",
       "\n",
       " \n",
       "\n",
       "## Module 13 Definitions\n",
       "* **Period disambiguation** - decidingif the period is end of sentence or part of a word\n",
       "* **Sigmoid** - $1 / 1 + e ^ {-z}$\n",
       "* **Feature templates** - ambstract specs of features that allow automatic creation of features\n",
       "* **Representation learning** - ways to learn features automatically in an unsupervised way\n",
       "* **Mcculloch-pitts neuron** - a simplified model of the neuron as a computing element describable in propositional logic\n",
       "* **Hard classification task** - task where only one class is correct\n",
       "* **Pretraining** - Relying on another algorithm to learn embedding representation for input words\n",
       "\n",
       "\n",
       "# Module 14: Natural Language Processing\n",
       "## SLP Chapter 9.1 Language Models Revisited\n",
       " A problem with using fixed-size contexts is that it limits the context from which information can be extracted. Moreover, it makes it difficult to learn systematic patterns arising from certain linguisitic phenomena. \n",
       "\n",
       "\n",
       "Recurrent neural networks and transformer networks are two approaches that allow dealing with sequential nature of language, being able ot handle variable input lengths without using arbitrary fixed-sized windows and capture temporal nature of language.\n",
       "\n",
       "Autoregressive generation provides an alternative way to assess a language model. We have a LM generate random words and assess to what extent it matches the training data. \n",
       "\n",
       "## SLP Chapter 9.2 Recurrent Neural Networks\n",
       " Recurrent neural network = network that contains a cycle. \n",
       "\n",
       "\n",
       "The cycle link inform the hidden layer about information from a prceding point in time. The approach does not impose a fixed-length limit on the prior context \n",
       "\n",
       "\n",
       "Backpropagation in RNN requires two pass. Frist pass: forward inference, accumulate loss at each step in time, and save value of hidden layer at each step. Second pass: process sequence in reverse, compute required gradient, and save error term. This is known as backpropagation through time. \n",
       "\n",
       "\n",
       "We can also unroll a recurrent network into a feedforward computational graph to remove any explicit recurrences. We can then apply ordinary feedforward computation \\and backpropgation. \n",
       "\n",
       "\n",
       "The hidden state in RNN embodies information all the way to the beginning, escaping problme of limited context. \n",
       "\n",
       "\n",
       "To train RNN, we use a corpus of text + teacher forcing. Minimize error in predicting next word using cross-entropy loss. \n",
       "\n",
       "\n",
       "Weight typing uses a single set of embeddings at input and softmax layers. It sets dimensionality of the final hidden layer to be the same d_h, and use same matrix for both layers. \n",
       "\n",
       "\n",
       "RNNs are useful for sequenec labeling: assign label to each element of sentence, e.g. POS tagging and named entity resolution. \n",
       "\n",
       "\n",
       "RNN can also classify entire sequences.\n",
       "\n",
       "\n",
       "End-to-end training = training that uses loss from a downstream application to adjust weights all the way through the network. \n",
       "\n",
       "\n",
       "Stacked RNNs: multiple RNN layers stacked on top of each other. The network is able to induce representation at different levels of abstraction across layers. \n",
       "\n",
       "\n",
       "Bidirectional RNNs: Also consider context to the right of the input. Has two indepdnent RNNs whose outputs are combined into a single representation. \n",
       " \n",
       "\n",
       "## SLP Chapter 9.3 RNNs as Language Models\n",
       " Two problems that RNN faces: 1) the hidden layers serve the dual functino of providing info for current decision and carrying forward information for future decisions, 2) in back-propagation, hidden layers are subject to repeaeted multiplications, driving gradients to zero, u.e. vanishing gradient. \n",
       "\n",
       "\n",
       "Long short-term memory (LSTM) networks divide context management into two sub-problems: removing info no longer needed, and adding info likely to be needed. It adds an explicit context layer, using gates to control flow of info into and out of the units. Gates are implemented with additional weights that operate sequentially on the input. \n",
       "\n",
       "\n",
       "Gates consist of a feedforward layer, a sigmoid activation function, and pointwise multiplication of layer being gated. Function is similar to a binary mask: no change to values close to one, and erase values with lower values. \n",
       "\n",
       "\n",
       "Forget gate deletes info from context that is not needed. Computes a weighted sum of hidden layer and current input, pass it through sigmoid then multiply it with context. \n",
       "\n",
       "\n",
       "Add gate selects info to add to current context. Output gate decides what info is required for current hidden state. \n",
       "\n",
       "\n",
       "LSTM computation: inputs are current input x, previous hidden state, and previous context. Outputs are new hidden state and updated context. \n",
       "\n",
       "\n",
       "Gated recurrent units reduce numbers of additional parameters required by LSTM, through using a separate context vector and reducing the number of gates to 2: a reset gate and an update gate. Reset gate is multiplied with previous hidden state to compute new hidden state.  UPdate gate uses values in the gate to interpolate between old hidden state and new one to determine which aspects of the output of update gate will be used directly in new hidden state, and which is needed int he future. \n",
       "\n",
       "\n",
       "The increased complexity of LSTM and GRU are encapsulated within the neural units themselves, providing modularity. They can be substitutes for simple feedforward and RNN networks. \n",
       " \n",
       "\n",
       "## SLP Chapter 6.1 Lexical Semantics\n",
       " Words can be similar rather than synonymous. Similarity deals with words while synonymy with word senses.\n",
       "\n",
       "\n",
       "Words can also be related/ associated, e.g. coffee and cup. One kind of relatedness is being in the same semantic field (set of words that cover a semantic domain). AKA topic models, e.g. Latent Dirichlet Allocation, using unsupervised ML on large sets to induce associated words.\n",
       "\n",
       "\n",
       "More relations are hypernymy/ IS-A, antonymy, and meronymy. \n",
       "\n",
       "\n",
       "Semantic frame: set of words that denote perspectives or participants in a type of event, e.g. transaction. \n",
       "\n",
       "\n",
       "Connotations/ affective meanings: associated with setiments. Words vary along three dimensions for affective meaning: valence (pleasantness), arousal (intensity), dominance (Degree of control).  \n",
       "\n",
       "## SLP Chapter 6.2 Vector Semantics\n",
       " Vector semantics represent word as a point in a multidimensional semantic space derived from distribution of word neighbors. \n",
       "\n",
       "\n",
       "Vectors representing words are embeddings. \n",
       "\n",
       "\n",
       "Two common models: tf-idf - meaning of word defined by a simple function of counts of nearby words. Results in long vectors that are sparse \n",
       "\n",
       "\n",
       "Word2vec: construct short, dense vectors with useful semantic properties. \n",
       "\n",
       "\n",
       "Cosine: standard way to use embeddings to calculate semantic similarity between two bodies of text.\n",
       " \n",
       "\n",
       "## SLP Chapter 6.3 Words and Vectors\n",
       " Term document matrix: each row is a word and each document is a colun. Defined as  part of the vector space model of information retrieval. Each column is a vector. Has V rows and D columns in total. \n",
       "\n",
       "\n",
       "Information retrieval: find document d from D documents that match query q. We represent a query by a vector and need to compute how similar two vectors are. \n",
       "\n",
       "\n",
       "We can also use term-term matrix/ word-word matrix/ term-context matrix, with numbers being how often they appear together. WE can restric the size of columns to frequent words. Most numbers will be zero, creating a sparse matrix.\n",
       "\n",
       "\n",
       "## SLP Chapter 6.4 Cosine for measuring similarity\n",
       " Normalized dot product (dot product divided by vector length ) =cosine of angles between two vectors = $(a \\dot b) / (|a||b|)$. Higher cosine = higher similarity. \n",
       "\n",
       "## SLP Chapter 6.8 Word2vec\n",
       " Word2vec is fast, eeasy to train, and easily availble online with code and pre-trained embeddings. \n",
       "\n",
       "\n",
       "Intuition behind word2vec: train a classifier on the prediction task: how often will word w show up near word y?  \n",
       "\n",
       "\n",
       "Simpler than neural network: 1) simplifies task (classification instead of word prediction) 2) simplified architecture (log reg classifier rather than multi layer network). \n",
       "\n",
       "\n",
       "Intuiton of skip-gram: 1) treat target word and neighbor context as positive examples. 2) randomly sample other words as negative examples. 3) Use log reg to train a classifier to distinguish. 4) use learned weights as embeddings. \n",
       "\n",
       "\n",
       "To compute probability of two words being close, we see if their embeddings are similar. We apply sigmoid on the dot product of the two embeddings. It makes the assumption that all context words are independent. \n",
       "\n",
       "\n",
       "Skip grams learn embeddings by startting with random vectors, then iteratively shifting embeddings of each word to be more like embeddings of words nearby. \n",
       "\n",
       "\n",
       "SGNS uses more negative than positive examples, with ratio between them set by a parameter k. Negative examples have target word + noise word. Noise words are chosen according to htier weighted unigram frequency, with a weight alpha often set to 0.75. \n",
       "\n",
       "\n",
       "Learned embeddings maximize similarity of target, context word pairs, and minimize similarity of negative examples. Two separate embeddings the target embedding and the context embedding are learned. \n",
       "\n",
       "\n",
       "Other kinds of static embeddings: fasttext (extension of word2vec) deals with unknown words and sparsity using subword models. \n",
       "\n",
       "\n",
       "## SLP Chapter 6.10 Semantic properties of embeddings\n",
       " Size of context window is usually between 1 and 10 words. Choice depends on the goal of representation: short windows lead to more syntactic representativions, while long windows lead to related but not similar words. \n",
       "\n",
       "\n",
       "Two kinds of similarity: 1) first-order co-occurence/ syntagmatic association: occurs between two words typically nearby each other (e.g. book and wrote). 2) second-order co-occurence/ paradigmatic association if two words have similar neighbors (e.g. wrote, said). \n",
       "\n",
       "\n",
       "Parallelogram model can be used to solve simple analogy problems. Add vector from apple to tree to the word grape to get vine. \n",
       "\n",
       "\n",
       "Embeddings can also be used to study how meaning changes over time. \n",
       "\n",
       "## SLP Chapter 6.11 Bias and Embeddings\n",
       " Allocational harm occurs when system allocates resources unfairly to different groups. Can result from embeddings. \n",
       "\n",
       "\n",
       "Embeddings can amplify bias: genedered terms can become more genedered than in the inputs. It can also encode implict associations, causing representational harm, caused by a system demeaning or ignoring certain social groups. \n",
       "\n",
       "\n",
       "Debiasing can help reduce bias in embeddings. Historical embeddings can be used to measure biases in the past. \n",
       "\n",
       "## SLP Chapter 6.12 Evaluating Vector Models\n",
       " Intrinsic evaluation methods for embeddings: \\ \n",
       "1) most common - test performance on similarity: machine vs human evaluation on simliarity. E.g. TOEFL dataset. We can also include context in similarity tests. 2) analogy task. \n",
       " \n",
       "\n",
       "Embedding algorithms can suffer from inherent variability: because of random negative sampling, different results can be generated from the same dataset.\n",
       "\n",
       "## Module 14 Lecture Notes\n",
       " \n",
       "Advantages of neural LMs: 1) don't need smoothing, 2) can handle longer histories, 3) Can generalize over contexts of similar words, 4) much higher accuracy\n",
       "\n",
       "\n",
       "Proposed over 20 years ago by Joshua Bengio. 1) Associate each word with a vector representation, 2) express joint proabbility function in terms of word vectors, then 3) simultaneously learn word vectors and the parameters of the function. \n",
       "\n",
       "\n",
       "Word vectors: low dimensional (d = 30-100) dense vectors. \n",
       "\n",
       "\n",
       "Prob function is the product of conditional probs of the next word given the previous words.\n",
       "\n",
       "\n",
       "Input: k-gram of words, output: prob distribution of next word. Each word is assocaited wth an embedding vector. Input vector x is concatenation of the vector of the k words. \n",
       "\n",
       "\n",
       "The output layer will have output of size equal to the vocabulary. \n",
       "\n",
       "\n",
       "The projected layer will contain a (learned) representation of each word. Projection layer of dog and puppy will be similar with each other. \n",
       "\n",
       "\n",
       "Training examples: k-grams from corpus. Model is trained using cross-entropy loss. \n",
       "\n",
       "\n",
       "What happen if we have unseen sequence of words? 1) Back off, 2) smoothing / interpolation, 3) generalizing. We can replace words with others with similar meaning. \n",
       "\n",
       "\n",
       "Bengio's proposal: 1) use a vector space model with words = vectors with real values R^m, with m = 30, 60, 100. 2) Define a function that returns a joint probability of words  in a sequence. 3) learn word representations and probabilty function simultaneously. \n",
       "\n",
       "\n",
       "Lexical semantic models. A meaning representation we should encode: 1) synonyms, 2) antonyms, 3) connotations, 4) semantic roles, 5) entailment. \n",
       "\n",
       "\n",
       "Downside of using formal logic as representation: need to acquire knowledge from somewhere else. Example: word net at Princeton, with synonym set. \n",
       "\n",
       "\n",
       "Alternative to word net: build a vector space model. Allow automatic extraction from huge corpus. Each word represented in vector space, with simliar words occupying similar positions. \n",
       "\n",
       "\n",
       "Distributional hypothesis: if two words have similar meanings then they will appear in similar sets of environments (Zellig Harris 1954). \n",
       "\n",
       "\n",
       "First vector space models are developed in the SMART information retrieval system. Convert query into vector and retrieve everything with similar vectors. \n",
       "\n",
       "\n",
       "Term document matrix: words as rows, documents as columns. Comparing two row vectors compare if two words are similar. \n",
       "\n",
       "\n",
       "Euclidean distance is a poor estimate of similarity, because more frequent words will have higher distance from origin. We use L2 Normalization, divide by distance from origin. \n",
       "\n",
       "\n",
       "Cosine distance: already built in the L2 Norm. Dot product of two word vectors, normalized by the product of their distance to origin.\n",
       "\n",
       "\n",
       "Term-term matrix/ term-context matrix: square matrix. What a cell represent how often does the word X (col) appears in a context window surrounding the word Y (row)? \n",
       "\n",
       "\n",
       "We can transform raw counts into something more useful. E.g. TF-IDF or PPMI. Counter Zipf's law: word distribution is inherently skewed. \n",
       "\n",
       "\n",
       "TF-IDF: term frequency * inverse document frequency (1/num docuemtns where a word appears). Measures how \"informative\" a word appearance is. \n",
       "\n",
       "\n",
       "Co-occurence matrix weighted by TF-IDF is long and sparse. We want to learn vectors that are short and dense. Use embedding instead. \n",
       "\n",
       "\n",
       "To get dense vectors, we train a classifer. 1) We treat every word and its neighboring context as a positive training example. 2) randomly sample other words as negative examples. 3) use log reg to train classifer to distinguish. 4) use the weights as embeddings. \n",
       "\n",
       "\n",
       "Word2Vec model implements this. It is fast and easy to train, also available online in word2vec package. \n",
       "\n",
       "\n",
       "Embedding can capture relational meaning. E.g. gender relationships. We can solve word analogy problems using word embeddings. \n",
       " \n",
       "\n",
       "## Module 14 Definitions\n",
       "* **Distributional hypothesis** - Linke between similarity in how words are distributed and what they mean\n",
       "* **One hot vector** - vector with one value 1 and others 0 \n",
       "* **Vector semantics** - Instantiation of the distributional hypothesis by learning meanings/embeddings from contexts\n",
       "* **Represenation learning** - automatically (self-supervised) learning useful representations of input text\n",
       "* **Feature engineering** - learning features by hand\n",
       "* **Lexical semantics** - Linguistic study of word meaning\n",
       "* **Lemma/citation form** - Word roots, as opposed to wordforms\n",
       "* **Word sense** - a possible meaning of a word\n",
       "* **Polysemous** - word having multiple senses\n",
       "* **Word sense disambiguation** - tackling polysemous words\n",
       "* **Synonymy** - two words are substitutable for one another without changing truth condition of the sentence\n",
       "* **Propositional meaning** - Two words ahve same PM when they are synonymous\n",
       "* **Principle of contrast** - A difference in linguistic form is always associated with some difference in meaning\n",
       "* **Skip gram with negative sampling (sgns)** - An algorithm for computing embeddings\n",
       "* **Contextual embeddings** - BERT/ ELMO representations, where vector for each word is different in diff contexts\n",
       "* **Dense vector models** - vector models with dimensionality 50-1000\n",
       "* **Tf-idf** - A sparse weighting which assigns weights by term frequency and inverse document frequency\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# This will generate the full set of notes that you can use for your exam.\n",
    "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
    "                             module9_markdown, \n",
    "                             module10_markdown, \n",
    "                             module11_markdown, \n",
    "                             module12_markdown,\n",
    "                             module13_markdown,\n",
    "                             module14_markdown,]\n",
    "                       )\n",
    "display(Markdown(full_markdown))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
