{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2e2d9ef7-04d7-4be1-84f9-ccf9537f2cca"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "78fecd55-58e5-431e-ae74-470f90316665"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" Combining Beliefs and Desires under uncertainty\n",
        "Uncertain state and uncertain action results. \n",
        "Decision theory deals with immediate outcomes\n",
        "Utility function captures agent's preferences\n",
        "Expected Utility is the averaged utility value of the outcomes, weighted by the probability of the outcome\n",
        "MEU: Choose the action that maximizes the agent's expected utility\n",
        "The complex part is how to implement it\n",
        "\n",
        "\n",
        "\"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\" The Basis of Utility Theory\n",
        "Axions of utility theory: If an agent violates them, it'll be irrational\n",
        "Orderability: perefer either one, the other or both equally\n",
        "Transitivity: If A is prefered over B and B over C, then A is over C\n",
        "Continuity: If B is between A and C, then there is some probability that makes B equal to A or to C\n",
        "Susbtituitability: If indifferent to A and B, the if there is a C that is same as A or B, then C is also indifferent\n",
        "Monotonicity: If A is preferred over B, then the agent prefers a lottery with higher probability of A\n",
        "Decomposability: Two conscutive lotteries can be combined into a single equivalent one \n",
        "\n",
        "Consequences of the axioms:\n",
        "1)Existence of Utility Function: there exist an U(A)>U(B) if A is preferred to B and U(A)=U(B) if indifferent\n",
        "2)Expected Utility of a Lottery: It's the sum of the probability of each outcome times the utility of that outcome, therefore once we know the probabilities, the outcome is determined\n",
        "\n",
        "\"\"\"\n",
        "chapter16_3 =  \"\"\" Utility Functions\n",
        "\n",
        "They map from lotteries to real numbers. \n",
        "In the utility function, we use a normalized 0 - 1 values (0 bad, 1 good)\n",
        "To find the utility of S, we ask the agent to choose between S and a standard lottery with probabily p of good and 1-p of bad, and we adjust p\n",
        "until the agent is indifferent to both.\n",
        "\n",
        "Regarding money, it needs to be assigned utility, because the first million has more utility than the others\n",
        "\n",
        "rational way to choose best action is to maximize expected utility.\n",
        "\n",
        "However, the real outcome will be significantly worse, because the highest estimate is to the right of curve with all the probabilities\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\" In a MDP, we can end in any state, so the solution/policy needs to address how to act from any state\n",
        "we compare the policies by comparing the expected utilities\n",
        "The probabiliti distribution is determined by the initial state, the policy and the transition model.\n",
        "The optimal policy is independent of the starting state\n",
        "The true utility of a state is the expected sum of discounted rewards if the agent executes an optimal policy\n",
        "Reward Scales: we cna make the immediate rewards higher to affect what the agent should do\n",
        "Representing MDP: Using tables of size O(S^2 * A). We can also use Dynamic Decision Networks which are better for real world problems\"\"\"\n",
        "chapter17_2 =  \"\"\" Algorithms for MDP\n",
        "1) Value Iteration\n",
        "we can get an optimal policy even when the utility function is inaccurate, because we only care on what's better, not in the magnitude.\n",
        "Based on the Bellman equation, one for each state. The unknowns are the utitlities of the states.\n",
        "Ui+1(s)<- max Sum P(s'|s,a)[R(s,a,s\")+gamma U(s')]\n",
        "if we apply Bellman updates infintely, we find the optimal policy\n",
        "**Convergence of value iteration\n",
        "Contraction properties:\n",
        "1) It has only one fixed point\n",
        "2) When applied to an argument, the value must get closer to the fixed point.\n",
        "||BUi - BU'i|| <= gamma ||Ui-U'i|| The Bellman update is a contraction by a factor of gamma on the space of utility vector.\n",
        "If we replace with the true utilities U, where BU=U:\n",
        "||BUi - U|| <= gamma ||Ui-U'i|| , we obtain the rate of convergence\n",
        "We can converge faser if we give gamma a high value, but we miss the long-term effects of the agent's actions\n",
        "In practice, the policy becomes optimal long before utility converges.\n",
        "Value Iteration in practice:\n",
        "a-It converges to the correct utilities\n",
        "b-the error can be bound in the utility estimates if we stop after a finite number of iterations\n",
        "c-the policy loss can be bound\n",
        "\n",
        "2) Policy Iteration\n",
        "it alternates two steps:\n",
        "a-policy evaluation, calculate the utility if Pii is exectuted\n",
        "b-policy improvement, get a new MEU policy, using one step look ahead\n",
        "terminate when improvement is zero\n",
        "removing the max argument turns the problem into a linear equation that can be solved in O(n^3)\n",
        "\n",
        "3) Linear Programming\n",
        "Consider as variables in LP the utilies U(s) of each state: U(s) >= Sum P(s'|s,a)[R(s,a,s')+gamma U(s')] for every state s and every action a\n",
        "It runs on polynomial time, but even the simplest and mostuninformed search algorithms run in lineal time\n",
        "\n",
        "4)Online Algorithms\n",
        "For large MDP, offline is not possible. \n",
        "Use a simplified EXPECTIMINIMAX with chance nodes. \n",
        "Real time dynamic programming\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\" Stochastic or probabilistic environment: dtermine best action even wehn outcome is uncertain.\n",
        "MDP fully observable env, with outputs non deterministic. Policy takes a state and returns the best action from that state.\n",
        "Uses a transition model that is probabilistic.\n",
        "\n",
        "MDPs are:\n",
        "* Sequential\n",
        "* for a fully observable environment\n",
        "* with stochastic transtion model\n",
        "* and has additive rewards \n",
        "\n",
        "To find a solution to an MDP we need:\n",
        "* A set of states \n",
        "* a set of Actions\n",
        "* a transtion function T(s, a, s') The probability is called the model P(s'|s,a)\n",
        "* a rewards function R(s, a, s')\n",
        "* An initial state s0\n",
        "* Optionally, one or more terminal states\n",
        "\n",
        "In MDP, the solution is called a policy (the best action to take in the current state) Phi(S)=a. This is because I might end up in a state that I wasn't planning to be.\n",
        "Phi* is the best policy and it depends on the reward function.\n",
        "\n",
        "To penalize future rewards, we use discounting (Gamma) each time we descend a level in the expectimax tree.\n",
        "\n",
        "There are only two ways to define utilities, Additive or discounted.\n",
        "\n",
        "How to hanbdel infinite utilities:\n",
        "1) Finite horizon (Terminate after T steps) or nonstationary polycy (time left)\n",
        "2) Use discounting, with 0 < gamma < 1\n",
        "3) Absorving state: probabilistically will reach a terminal state\n",
        "\n",
        "T(s,a,s') = P(s'|s,a) * R(s,a,s')\n",
        "Main difference with expectimax tree:\n",
        "* The transtion function includes the probabily\n",
        "* the reward is given at each state in the tree as opposed to the terminal state\n",
        "\n",
        "Optimal Quantities:\n",
        "*Value utility of a state: V*(s) = Expected utility starting at s and acting optimally\n",
        "*Value utiltiy of a q-state: Q*(s,a) = expected utility having taken a from s and then acting optimally\n",
        "*Pi*(s) = optimal action from s\n",
        "\n",
        "V*(s) = maxQ*(s,a)\n",
        "Q*(s,a) = Sum T(s,a,s')[R(s,a,s') + gamma V*(s')]\n",
        "V*(s) = max sum T(s,a,s') [R(s,a,s') + gamma V*(s')]\n",
        "\n",
        "Problems:\n",
        "1) States are repeated (can use dynamic programming)\n",
        "2) Tree foes on forever (do a limited depth search)\n",
        "\n",
        "**Value Iteration**\n",
        "V0(s) = 0\n",
        "Vk+1(s) <-max Sum T(s,a,s')[R(s,a,s') + gamma Vk(s')]\n",
        "O(s^2A) \n",
        "guarantee to converge to unique optimal values\n",
        "\n",
        "Convergence\n",
        "1) if the tree has a meximum depth M, then Vm has the values\n",
        "2) if the discount is < 1\n",
        "\n",
        "**Policy Methods**\n",
        "How to evalue policies:\n",
        "Instead of calculatin the value of each state, just get the total of the policy.\n",
        "1) Do what Pi says. Calculate the utility\n",
        "V*(s) = sum T(s,PI(s),s') [R(s,PI(s),s') + gamma V*(s')] (There's no max!!!)\n",
        "O(s^2)\n",
        "\n",
        "**Policy Extraction**\n",
        "Given values, get the policy.\n",
        "Do a mini-expectimax (one step) \n",
        "Pi*(s) = arg max Sum T(s,a,s')[R(s,a,s') + gamma V*(s')]\n",
        "With Q values:\n",
        "Pi*(s) = arg max Q*(s,a)\n",
        "\n",
        "**Policy Iteration**\n",
        "Problems with Value Iteration:\n",
        "1) it's slow O(s^2A)\n",
        "2) the \"max\" in each state not changes frequently\n",
        "3) We can converge long before the last values\n",
        "\n",
        "Alt approach -> Policy Iteration\n",
        "1) Calculate all values in any policy until it converges\n",
        "2) Improve the policy with one step look ahead\n",
        "3) Repeat until converges\n",
        "policy is implicitly updated\n",
        "\n",
        "This is optimal\n",
        "\n",
        "Compared to value iteration, policy iteration often converges in fewer iterations but it's more costly (O(s^3) compared to O(A*S^2))\n",
        "\n",
        "**Comparison Value Iteration vs Policy Iteration**\n",
        "Both find optimal value, use MDP.\n",
        "In Value Iteration every iteration updates values and we don't explicitly keep track of the policy\n",
        "In Policy Iteration several passes that update the policy, and it's updated\n",
        "Both use dynamic programming\n",
        "\n",
        "To compute optimal values: Value Iteration or Policy Iteration\n",
        "values for one policy: Policy Extraction\n",
        "From values to Policy: Policy Extraction (one step lookahead)\n",
        "All are variants for the Bellman update/expectimax\n",
        "Main difference: if we use a fixed policy or if we go over all actions to find the best one\n",
        "\n",
        "**Maximum Expected Utility (MEU)**\n",
        "the agent should max the expected return (not worst case planning)\n",
        "For average-case expectimax, the magnitudes need to be meaningful, because they affect the choices.\n",
        "\n",
        "**Rational Preferences**\n",
        "-transitivity\n",
        "-Orderability\n",
        "-Susbtitutability\n",
        "-Continuity\n",
        "-Monotonicity\n",
        "To be rational, choose the action that maximizes expected utility\n",
        "**Utility Scales**\n",
        "-Micromort\n",
        "-QALYs: Quality adjusted life years\n",
        "Invariant under positive linear transformation\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"Stochastic Transition Modeo\" : \"p(s'|s,a) probability of ending in s' if we are in s and select action a\"  ,\n",
        "\"Markovian\" : \"The probability of arriving at s' depends only on s and a, not on the history\" , \n",
        "\"utility/reward function\" : \"it's instantenous, can be positive or negative, and are additive R(s, a, s')\",\n",
        "\"utility function\" : \"Takes a sequence of states and actions and adds the rewards\",\n",
        "\"expected utility\" : \"\",\n",
        "\"stationary preferences\" : \"If a prefer A over B, if I then add x to A and to B, I still prefer A+x to B+x\",\n",
        "\"lotteries\" : \"outcomes with probabilities attached\",\n",
        "\"value function/ordinal utility function\" : \"an agent needs only a preference ranking order\",\n",
        "\"preference elicitation\" : \"process to find out the human's utility function\",\n",
        "\"value of statistical life\" : \"10MM in the US\",\n",
        "\"monotonic preference for money\" : \"prefer more money to less\",\n",
        "\"certainty equivalent\" : \"value that an agent will accept in lieu of of a lottery\",\n",
        "\"insurance premium\" : \"difference between expected monetary value and certaninty equivalent\",\n",
        "\"risk neutral agent\" : \"one that has a linear curve\",\n",
        "\"normative theory\" : \"how it should act\",\n",
        "\"descriptive theory\" : \" how it acts\",\n",
        "\"transition model\": \"outcome of each action in each state\",\n",
        "\"optimal policy\":\"one that yields the highest expected utility. There could be multiple optimal policies\",\n",
        "\"nonstationary\" : \"a policy that depends on the time. It depends on the time left\",\n",
        "\"stationary\" : \"it only depends on the current state\",\n",
        "\"stationarity of preferences\" : \"if one prefers one future tomorrow, it still should prefer it today \",\n",
        "\"proper policy\":\"one that is guarantee to reach a terminal state\",\n",
        "\"Q Function/action-utility function\" : \"Expected utility of taking a given action in a given state\",\n",
        "\"contraction\":\"function of one argument that, when applied to two difference inputs, produces two output values that are 'closer together'\",\n",
        "\"max norm\" : \"measures the length of a vector by the absolute value of it's biggest component\",\n",
        "\"modified policy iteration\":\"Perform a some number of simplified velue iteration steps to get a good enough approximation of the utilities.\",\n",
        "\"asynchronous policy iteration\":\"intead of updating all states, select only a subset\",\n",
        "\n",
        "\n",
        "\"etc.\": \"your definition here\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "nXy5EcyAasov",
        "outputId": "8e2a38ae-083c-4147-dda5-6747cada8eaf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Combining Beliefs and Desires under uncertainty\nUncertain state and uncertain action results. \nDecision theory deals with immediate outcomes\nUtility function captures agent's preferences\nExpected Utility is the averaged utility value of the outcomes, weighted by the probability of the outcome\nMEU: Choose the action that maximizes the agent's expected utility\nThe complex part is how to implement it\n\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n The Basis of Utility Theory\nAxions of utility theory: If an agent violates them, it'll be irrational\nOrderability: perefer either one, the other or both equally\nTransitivity: If A is prefered over B and B over C, then A is over C\nContinuity: If B is between A and C, then there is some probability that makes B equal to A or to C\nSusbtituitability: If indifferent to A and B, the if there is a C that is same as A or B, then C is also indifferent\nMonotonicity: If A is preferred over B, then the agent prefers a lottery with higher probability of A\nDecomposability: Two conscutive lotteries can be combined into a single equivalent one \n\nConsequences of the axioms:\n1)Existence of Utility Function: there exist an U(A)>U(B) if A is preferred to B and U(A)=U(B) if indifferent\n2)Expected Utility of a Lottery: It's the sum of the probability of each outcome times the utility of that outcome, therefore once we know the probabilities, the outcome is determined\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n Utility Functions\n\nThey map from lotteries to real numbers. \nIn the utility function, we use a normalized 0 - 1 values (0 bad, 1 good)\nTo find the utility of S, we ask the agent to choose between S and a standard lottery with probabily p of good and 1-p of bad, and we adjust p\nuntil the agent is indifferent to both.\n\nRegarding money, it needs to be assigned utility, because the first million has more utility than the others\n\nrational way to choose best action is to maximize expected utility.\n\nHowever, the real outcome will be significantly worse, because the highest estimate is to the right of curve with all the probabilities\n\n\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n In a MDP, we can end in any state, so the solution/policy needs to address how to act from any state\nwe compare the policies by comparing the expected utilities\nThe probabiliti distribution is determined by the initial state, the policy and the transition model.\nThe optimal policy is independent of the starting state\nThe true utility of a state is the expected sum of discounted rewards if the agent executes an optimal policy\nReward Scales: we cna make the immediate rewards higher to affect what the agent should do\nRepresenting MDP: Using tables of size O(S^2 * A). We can also use Dynamic Decision Networks which are better for real world problems\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Algorithms for MDP\n1) Value Iteration\nwe can get an optimal policy even when the utility function is inaccurate, because we only care on what's better, not in the magnitude.\nBased on the Bellman equation, one for each state. The unknowns are the utitlities of the states.\nUi+1(s)<- max Sum P(s'|s,a)[R(s,a,s\")+gamma U(s')]\nif we apply Bellman updates infintely, we find the optimal policy\n**Convergence of value iteration\nContraction properties:\n1) It has only one fixed point\n2) When applied to an argument, the value must get closer to the fixed point.\n||BUi - BU'i|| <= gamma ||Ui-U'i|| The Bellman update is a contraction by a factor of gamma on the space of utility vector.\nIf we replace with the true utilities U, where BU=U:\n||BUi - U|| <= gamma ||Ui-U'i|| , we obtain the rate of convergence\nWe can converge faser if we give gamma a high value, but we miss the long-term effects of the agent's actions\nIn practice, the policy becomes optimal long before utility converges.\nValue Iteration in practice:\na-It converges to the correct utilities\nb-the error can be bound in the utility estimates if we stop after a finite number of iterations\nc-the policy loss can be bound\n\n2) Policy Iteration\nit alternates two steps:\na-policy evaluation, calculate the utility if Pii is exectuted\nb-policy improvement, get a new MEU policy, using one step look ahead\nterminate when improvement is zero\nremoving the max argument turns the problem into a linear equation that can be solved in O(n^3)\n\n3) Linear Programming\nConsider as variables in LP the utilies U(s) of each state: U(s) >= Sum P(s'|s,a)[R(s,a,s')+gamma U(s')] for every state s and every action a\nIt runs on polynomial time, but even the simplest and mostuninformed search algorithms run in lineal time\n\n4)Online Algorithms\nFor large MDP, offline is not possible. \nUse a simplified EXPECTIMINIMAX with chance nodes. \nReal time dynamic programming\n\n\n\n\n\n\n\n\n## Module 8 Lecture Notes\n Stochastic or probabilistic environment: dtermine best action even wehn outcome is uncertain.\nMDP fully observable env, with outputs non deterministic. Policy takes a state and returns the best action from that state.\nUses a transition model that is probabilistic.\n\nMDPs are:\n* Sequential\n* for a fully observable environment\n* with stochastic transtion model\n* and has additive rewards \n\nTo find a solution to an MDP we need:\n* A set of states \n* a set of Actions\n* a transtion function T(s, a, s') The probability is called the model P(s'|s,a)\n* a rewards function R(s, a, s')\n* An initial state s0\n* Optionally, one or more terminal states\n\nIn MDP, the solution is called a policy (the best action to take in the current state) Phi(S)=a. This is because I might end up in a state that I wasn't planning to be.\nPhi* is the best policy and it depends on the reward function.\n\nTo penalize future rewards, we use discounting (Gamma) each time we descend a level in the expectimax tree.\n\nThere are only two ways to define utilities, Additive or discounted.\n\nHow to hanbdel infinite utilities:\n1) Finite horizon (Terminate after T steps) or nonstationary polycy (time left)\n2) Use discounting, with 0 < gamma < 1\n3) Absorving state: probabilistically will reach a terminal state\n\nT(s,a,s') = P(s'|s,a) * R(s,a,s')\nMain difference with expectimax tree:\n* The transtion function includes the probabily\n* the reward is given at each state in the tree as opposed to the terminal state\n\nOptimal Quantities:\n*Value utility of a state: V*(s) = Expected utility starting at s and acting optimally\n*Value utiltiy of a q-state: Q*(s,a) = expected utility having taken a from s and then acting optimally\n*Pi*(s) = optimal action from s\n\nV*(s) = maxQ*(s,a)\nQ*(s,a) = Sum T(s,a,s')[R(s,a,s') + gamma V*(s')]\nV*(s) = max sum T(s,a,s') [R(s,a,s') + gamma V*(s')]\n\nProblems:\n1) States are repeated (can use dynamic programming)\n2) Tree foes on forever (do a limited depth search)\n\n**Value Iteration**\nV0(s) = 0\nVk+1(s) <-max Sum T(s,a,s')[R(s,a,s') + gamma Vk(s')]\nO(s^2A) \nguarantee to converge to unique optimal values\n\nConvergence\n1) if the tree has a meximum depth M, then Vm has the values\n2) if the discount is < 1\n\n**Policy Methods**\nHow to evalue policies:\nInstead of calculatin the value of each state, just get the total of the policy.\n1) Do what Pi says. Calculate the utility\nV*(s) = sum T(s,PI(s),s') [R(s,PI(s),s') + gamma V*(s')] (There's no max!!!)\nO(s^2)\n\n**Policy Extraction**\nGiven values, get the policy.\nDo a mini-expectimax (one step) \nPi*(s) = arg max Sum T(s,a,s')[R(s,a,s') + gamma V*(s')]\nWith Q values:\nPi*(s) = arg max Q*(s,a)\n\n**Policy Iteration**\nProblems with Value Iteration:\n1) it's slow O(s^2A)\n2) the \"max\" in each state not changes frequently\n3) We can converge long before the last values\n\nAlt approach -> Policy Iteration\n1) Calculate all values in any policy until it converges\n2) Improve the policy with one step look ahead\n3) Repeat until converges\npolicy is implicitly updated\n\nThis is optimal\n\nCompared to value iteration, policy iteration often converges in fewer iterations but it's more costly (O(s^3) compared to O(A*S^2))\n\n**Comparison Value Iteration vs Policy Iteration**\nBoth find optimal value, use MDP.\nIn Value Iteration every iteration updates values and we don't explicitly keep track of the policy\nIn Policy Iteration several passes that update the policy, and it's updated\nBoth use dynamic programming\n\nTo compute optimal values: Value Iteration or Policy Iteration\nvalues for one policy: Policy Extraction\nFrom values to Policy: Policy Extraction (one step lookahead)\nAll are variants for the Bellman update/expectimax\nMain difference: if we use a fixed policy or if we go over all actions to find the best one\n\n**Maximum Expected Utility (MEU)**\nthe agent should max the expected return (not worst case planning)\nFor average-case expectimax, the magnitudes need to be meaningful, because they affect the choices.\n\n**Rational Preferences**\n-transitivity\n-Orderability\n-Susbtitutability\n-Continuity\n-Monotonicity\nTo be rational, choose the action that maximizes expected utility\n**Utility Scales**\n-Micromort\n-QALYs: Quality adjusted life years\nInvariant under positive linear transformation\n\n\n\n\n\n\n\n## Module 8 Definitions\n* **Stochastic transition modeo** - p(s'|s,a) probability of ending in s' if we are in s and select action a\n* **Markovian** - The probability of arriving at s' depends only on s and a, not on the history\n* **Utility/reward function** - it's instantenous, can be positive or negative, and are additive R(s, a, s')\n* **Utility function** - Takes a sequence of states and actions and adds the rewards\n* **Expected utility** - \n* **Stationary preferences** - If a prefer A over B, if I then add x to A and to B, I still prefer A+x to B+x\n* **Lotteries** - outcomes with probabilities attached\n* **Value function/ordinal utility function** - an agent needs only a preference ranking order\n* **Preference elicitation** - process to find out the human's utility function\n* **Value of statistical life** - 10MM in the US\n* **Monotonic preference for money** - prefer more money to less\n* **Certainty equivalent** - value that an agent will accept in lieu of of a lottery\n* **Insurance premium** - difference between expected monetary value and certaninty equivalent\n* **Risk neutral agent** - one that has a linear curve\n* **Normative theory** - how it should act\n* **Descriptive theory** -  how it acts\n* **Transition model** - outcome of each action in each state\n* **Optimal policy** - one that yields the highest expected utility. There could be multiple optimal policies\n* **Nonstationary** - a policy that depends on the time. It depends on the time left\n* **Stationary** - it only depends on the current state\n* **Stationarity of preferences** - if one prefers one future tomorrow, it still should prefer it today \n* **Proper policy** - one that is guarantee to reach a terminal state\n* **Q function/action-utility function** - Expected utility of taking a given action in a given state\n* **Contraction** - function of one argument that, when applied to two difference inputs, produces two output values that are 'closer together'\n* **Max norm** - measures the length of a vector by the absolute value of it's biggest component\n* **Modified policy iteration** - Perform a some number of simplified velue iteration steps to get a good enough approximation of the utilities.\n* **Asynchronous policy iteration** - intead of updating all states, select only a subset\n* **Etc.** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\" Bandits Problem\n",
        "* Each arm Mi is a Markov Reward Process (an MDP with only one possible action), with Si states, Pi(s'|s,ai) transition model and reward Ri(s,a,s')\n",
        "* bandit problem is an MDP\n",
        "key property: the arms are independent \"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\" Learning from Rewards\n",
        " \"\"\"\n",
        "chapter22_2 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_3 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_4 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_5 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" Reinforcement Learning\n",
        "\n",
        "In MDP, the agent gets the rewards and the probability distribution. In RL, the agent needs to learn them.\n",
        "Exploration: Try unknowns to get information\n",
        "Exploitation: use what was learned\n",
        "Regret: mistakes\n",
        "Sampling: due to chance, due things several times\n",
        "Difficulty: harder than MDP\n",
        "\n",
        "Feedbacks comes as reward, an the agent learns to maximize it. All based on obseved samples of outcomes.\n",
        "\n",
        "*A set of states s\n",
        "*A set of actions per state A\n",
        "*A model T (s,a,s')\n",
        "*A reward function R(s,a,s')\n",
        "Looking for a policy Pi(s)\n",
        "\n",
        "T and R are unknown\n",
        "\n",
        "MDP is offline learning - RL is online learning\n",
        "\n",
        "Model Based Learning: (probability model of the transition function) learn an estimate of the transition model and \n",
        "learn the rewards and apply them to the model using MDP (like value iteration)\n",
        "\n",
        "**Model-Free learning - Passive RL**\n",
        "Intead of creating a model, we take an action and compare the actual output to the expected one, and if they're different\n",
        "we update the expected one. We don't keep track of the reward or transition model.\n",
        "we get a policy and need to lear the value of each state, without T or R\n",
        "This is called direct utility evaluation, we take the average reward of reaching the end state from every state using sample transitions\n",
        "Positives:\n",
        "1) Easy to understand\n",
        "2) Doesn't require knowledge of T or R\n",
        "3) Eventually gets the correct average values\n",
        "Negatives:\n",
        "1) Doesn't consider state connections\n",
        "2) Each state must be learned separately\n",
        "3) it takes long to learn\n",
        "\n",
        "**Use Policy Evaluation with samples**\n",
        "Do policy iteration and do actions several times and then average the samples to get the estimate value.\n",
        "This doesn't work because we may not end in the same state multiple times\n",
        "instead do:\n",
        "**Temporal Difference Learning (model-free)**\n",
        "The idea is we update the estimate every single time we go through S, keeping a running average\n",
        "Sample of  V(s):  sample = R(s,Pi(s),s') + gamma V^pi(s')\n",
        "Updated to V(s): V^pi(s)<- (1 - alpha) V^pi(s) + (alpha) sample\n",
        "Alpha controls the weight of the current outcome\n",
        "Use of an exponential moving average makes recent samples more important\n",
        "However, we can't turn the values into a new policy\n",
        "\n",
        "**Active Reinforcement Learning**\n",
        "Instead of getting a policy, the agent decides the actions\n",
        "\n",
        "Value Iteration: start at V0, then calculate the best action going up a level. Use Dynamic Programming\n",
        "**Q-Learning: sample-based Q-value iteration\n",
        "  To learn Q(s,a) as we go:\n",
        "  1-receive a sample (s,a,s',r)\n",
        "  2-take old estimate Q(s,a)\n",
        "  3-new sample estimate: sample = R(s,a,s') + gamma max Q(s',a')\n",
        "  4-incorporate new estimate into a running average\n",
        "\n",
        "  It converges to optimal policy - even if acting suboptimally (off-policy learning)\n",
        "  Required: enough exploration, make the learning rate small (<1) to keep knowldge about past\n",
        "\n",
        "**Exploration vs Exploitation**\n",
        "How to explore:\n",
        "1-random actions (epsilon greedy policy) random number explore, otherwise use current policy\n",
        "  issue: we may be repeating states that are bad. Idea, reduce epsilon over time\n",
        "2-exploration functions\n",
        "  fixed amount\n",
        "  explore areas that we're not certain are bad (be optimistic)\n",
        "  f(u,n) = u + k/n\n",
        "  u: current estimate of the utility\n",
        "  n: number of times the state was visited\n",
        "  k: constant bonus, that decreases over time\n",
        "  Modified Q function: Q(s,a) <- R(s,a,s') + gamma max f(Q(s',a'),N(s',a')) (we get bonus for unseen states and for states that lead to unknown states)\n",
        "\n",
        "To decide between epsilon learning or Exploration function, we can use Regret. Learn optimal policy as quickly as possible.\n",
        "\n",
        "**Approximate Q-Learning**\n",
        "There are so many states, we can't visit them all. we'll try to generalize to similar situations\n",
        "In practice, several features are taken and evalued to value a state\n",
        "Eval(s)=w1f1(s)+w2f2(s)+...+wnfn(s) This is a linear value function\n",
        "Q(s,a) = w1f1(s,a)+w2f2(s,a)+...+wnfn(s,a)\n",
        "Good: we can summarize state in few features\n",
        "Bad: some features may be shared, but with very different value\n",
        "the learning algorithm will adjust the weights\n",
        "take a transition function: transtion = (s,a,r,s')\n",
        "and calculate the difference with what I was expecting:\n",
        "difference = [r+gamma max Q(s',a')]-Q(s,a) This is an error term\n",
        "take the difference and update each weight base on the differnece and wehter the feature was active:\n",
        "wi <- w1 + alpha[difference]fi(s,a)\n",
        "\n",
        "Bellman equation: $$U(s)=R(s)+\\gamma \\max _{a} \\sum_{s^{\\prime}} P\\left(s^{\\prime} \\mid s, a\\right) U\\left(s^{\\prime}\\right)$$\n",
        "\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"regret\" : \"measure of the total cost of the mistakes made to learn a policy\",\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "db3e56cb-762b-4664-8ce4-ddc19ac561b3"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Bandits Problem\n* Each arm Mi is a Markov Reward Process (an MDP with only one possible action), with Si states, Pi(s'|s,ai) transition model and reward Ri(s,a,s')\n* bandit problem is an MDP\nkey property: the arms are independent \n\n## AIMA Chapter 22.1 Learning from Rewards\n Learning from Rewards\n \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n Reinforcement Learning\n\nIn MDP, the agent gets the rewards and the probability distribution. In RL, the agent needs to learn them.\nExploration: Try unknowns to get information\nExploitation: use what was learned\nRegret: mistakes\nSampling: due to chance, due things several times\nDifficulty: harder than MDP\n\nFeedbacks comes as reward, an the agent learns to maximize it. All based on obseved samples of outcomes.\n\n*A set of states s\n*A set of actions per state A\n*A model T (s,a,s')\n*A reward function R(s,a,s')\nLooking for a policy Pi(s)\n\nT and R are unknown\n\nMDP is offline learning - RL is online learning\n\nModel Based Learning: (probability model of the transition function) learn an estimate of the transition model and \nlearn the rewards and apply them to the model using MDP (like value iteration)\n\n**Model-Free learning - Passive RL**\nIntead of creating a model, we take an action and compare the actual output to the expected one, and if they're different\nwe update the expected one. We don't keep track of the reward or transition model.\nwe get a policy and need to lear the value of each state, without T or R\nThis is called direct utility evaluation, we take the average reward of reaching the end state from every state using sample transitions\nPositives:\n1) Easy to understand\n2) Doesn't require knowledge of T or R\n3) Eventually gets the correct average values\nNegatives:\n1) Doesn't consider state connections\n2) Each state must be learned separately\n3) it takes long to learn\n\n**Use Policy Evaluation with samples**\nDo policy iteration and do actions several times and then average the samples to get the estimate value.\nThis doesn't work because we may not end in the same state multiple times\ninstead do:\n**Temporal Difference Learning (model-free)**\nThe idea is we update the estimate every single time we go through S, keeping a running average\nSample of  V(s):  sample = R(s,Pi(s),s') + gamma V^pi(s')\nUpdated to V(s): V^pi(s)<- (1 - alpha) V^pi(s) + (alpha) sample\nAlpha controls the weight of the current outcome\nUse of an exponential moving average makes recent samples more important\nHowever, we can't turn the values into a new policy\n\n**Active Reinforcement Learning**\nInstead of getting a policy, the agent decides the actions\n\nValue Iteration: start at V0, then calculate the best action going up a level. Use Dynamic Programming\n**Q-Learning: sample-based Q-value iteration\n  To learn Q(s,a) as we go:\n  1-receive a sample (s,a,s',r)\n  2-take old estimate Q(s,a)\n  3-new sample estimate: sample = R(s,a,s') + gamma max Q(s',a')\n  4-incorporate new estimate into a running average\n\n  It converges to optimal policy - even if acting suboptimally (off-policy learning)\n  Required: enough exploration, make the learning rate small (<1) to keep knowldge about past\n\n**Exploration vs Exploitation**\nHow to explore:\n1-random actions (epsilon greedy policy) random number explore, otherwise use current policy\n  issue: we may be repeating states that are bad. Idea, reduce epsilon over time\n2-exploration functions\n  fixed amount\n  explore areas that we're not certain are bad (be optimistic)\n  f(u,n) = u + k/n\n  u: current estimate of the utility\n  n: number of times the state was visited\n  k: constant bonus, that decreases over time\n  Modified Q function: Q(s,a) <- R(s,a,s') + gamma max f(Q(s',a'),N(s',a')) (we get bonus for unseen states and for states that lead to unknown states)\n\nTo decide between epsilon learning or Exploration function, we can use Regret. Learn optimal policy as quickly as possible.\n\n**Approximate Q-Learning**\nThere are so many states, we can't visit them all. we'll try to generalize to similar situations\nIn practice, several features are taken and evalued to value a state\nEval(s)=w1f1(s)+w2f2(s)+...+wnfn(s) This is a linear value function\nQ(s,a) = w1f1(s,a)+w2f2(s,a)+...+wnfn(s,a)\nGood: we can summarize state in few features\nBad: some features may be shared, but with very different value\nthe learning algorithm will adjust the weights\ntake a transition function: transtion = (s,a,r,s')\nand calculate the difference with what I was expecting:\ndifference = [r+gamma max Q(s',a')]-Q(s,a) This is an error term\ntake the difference and update each weight base on the differnece and wehter the feature was active:\nwi <- w1 + alpha[difference]fi(s,a)\n\nBellman equation: $$U(s)=R(s)+\\gamma \\max _{a} \\sum_{s^{\\prime}} P\\left(s^{\\prime} \\mid s, a\right) U\\left(s^{\\prime}\right)$$\n\n\n \n\n## Module 9 Definitions\n* **Regret** - measure of the total cost of the mistakes made to learn a policy\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\" Acting under uncertainty\n",
        "An agent needs to handle uncertainty in real world situations. \n",
        "So far, the agent worked wtih belief states: a set of all possible sorld states and a contigency plan for them.\n",
        "drawbacks:\n",
        "1) Large belief-state as it has to consider every possible explanation for its sensor observations\n",
        "2) Plans to handle every eventuality become too large\n",
        "3) some plans don't guaratee the goal, so we need to be able to compare plans\n",
        " \"\"\"\n",
        "chapter12_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_6 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" The main problem with estimating the probabilities is taht new sentences are created all the time.\n",
        "To compute the probability of an entire sequence, we'll use the cahin rule:\n",
        "P(X1...Xn) = P(X1)P(X2|X1)P(X3|X1:2)...P(Xn|X1:n−1)\n",
        "\n",
        " \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" The branching factor of a language is the number of possi-\n",
        "ble next words that can follow any word \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" Probability\n",
        "**Review of Probabilities**\n",
        "A probability model associates a porbability P(w) to each possible world w\n",
        "**Axioms of probability:\n",
        "1) 0<=P(w)<=1 for every w\n",
        "2) Sum P(w) = 1\n",
        "\n",
        "Unobserved r.v. have a probability distribution\n",
        "\n",
        "**Joint Distributions:\n",
        "Specifies a real number for each assignment or outcome for a joint assignment\n",
        "P(X1=x1, X2=x2,...,Xn=xn)\n",
        "\n",
        "The size of a distribution of n variables with domain size d is d^n\n",
        "\n",
        "**Probabilitic Models\n",
        "It's a joint distribution over a set of random variables.\n",
        "contains:\n",
        "-Domain for r.v.\n",
        "-Outcomes\n",
        "-Joint distributions\n",
        "-Normalized (sum=1)\n",
        "\n",
        "**Event P(E) = Sum P(x1...xn)\n",
        "\n",
        "**Marginal Distributions\n",
        "Sub-tables where some variables were eliminated (they're sum up)\n",
        "P(t) = Sum_s P(t,s)\n",
        "\n",
        "**Conditional Probabilities\n",
        "P(a|b) = P(a,b) / P(b)\n",
        "It gives the probability of something we haven't seen\n",
        "in a Venn Diagram, it's the intersection\n",
        "\n",
        "\n",
        "**Conditional Distributions\n",
        "Probability distributions over some variables given fixed values of others\n",
        "\n",
        "\n",
        "**Probabilistic Inference**\n",
        "Compute a desired probability from other known probabilities\n",
        "New evidence can change the probability\n",
        "** Inference by Enumeration\n",
        "- SPlit r.v into three sets:\n",
        "1-Evidence variables: things we know\n",
        "2-Query variables: things we care about and want to know about\n",
        "3-Hidden variables: things that occure but we're not observing and we don't care \n",
        "We care about P(Q|E)\n",
        "\n",
        "Steps:\n",
        "1-take entries that are consistent with evidence\n",
        "2-sum out hidden variables\n",
        "3-Normalize it to get to add up to 1: Z = Sum_q P(Q, e_1, e_2...)\n",
        "\n",
        "**The Product Rule\n",
        "we get marginal or conditional probabilities and we want to construct the joint probability table\n",
        "Multiply the marginal and the conditional.\n",
        "P(y)P(x|y) = P(x,y)\n",
        "\n",
        "**Chain Rule\n",
        "P(x1, x2, x3) = P(x1)P(x2|x1)P(xe|x1,x2)\n",
        "P(x1,x2,...,xn) = MUL_i P(xi|x1...xi-1)\n",
        "\n",
        "**Bayes' Rule\n",
        "P(x|y) = P(y|x)/P(y) * P(x)\n",
        "\n",
        "**Probabilistic Language Model**\n",
        "Language Model is a moel that computes the probabilitiy of the next word in a sentence\n",
        "One problem is that a sentence might not exist in the corpus.\n",
        "To address this, we use Markov Assumption, limiting the amount of history\n",
        "An unigram will suggest the most used work in the corpus\n",
        "A bigram will look for a word given another word\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"sample space\" : \"mutually exclusive and exhaustive possible worlds\",\n",
        "\"random variable\" : \"aspect of the world that we're interested in. Use capital letters\",\n",
        "\"Markov assumption\" : \"the probability of a word depends only on the previous word\",\n",
        "\"extrinsic evaluation\" : \"embed a lnaguage model in an application and measure how much the application improves\",\n",
        "\"perplexity\" : \" of a language model on a test set is the inverse probability of the test set, normalized by the number of words.\"\n",
        "\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "2a03b5b2-cfec-4e65-c05b-6c6f83c51a42"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Acting under uncertainty\nAn agent needs to handle uncertainty in real world situations. \nSo far, the agent worked wtih belief states: a set of all possible sorld states and a contigency plan for them.\ndrawbacks:\n1) Large belief-state as it has to consider every possible explanation for its sensor observations\n2) Plans to handle every eventuality become too large\n3) some plans don't guaratee the goal, so we need to be able to compare plans\n \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n The main problem with estimating the probabilities is taht new sentences are created all the time.\nTo compute the probability of an entire sequence, we'll use the cahin rule:\nP(X1...Xn) = P(X1)P(X2|X1)P(X3|X1:2)...P(Xn|X1:n−1)\n\n \n\n## SLP Chapter 3.2 Evaluating Language Models\n The branching factor of a language is the number of possi-\nble next words that can follow any word \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Probability\n**Review of Probabilities**\nA probability model associates a porbability P(w) to each possible world w\n**Axioms of probability:\n1) 0<=P(w)<=1 for every w\n2) Sum P(w) = 1\n\nUnobserved r.v. have a probability distribution\n\n**Joint Distributions:\nSpecifies a real number for each assignment or outcome for a joint assignment\nP(X1=x1, X2=x2,...,Xn=xn)\n\nThe size of a distribution of n variables with domain size d is d^n\n\n**Probabilitic Models\nIt's a joint distribution over a set of random variables.\ncontains:\n-Domain for r.v.\n-Outcomes\n-Joint distributions\n-Normalized (sum=1)\n\n**Event P(E) = Sum P(x1...xn)\n\n**Marginal Distributions\nSub-tables where some variables were eliminated (they're sum up)\nP(t) = Sum_s P(t,s)\n\n**Conditional Probabilities\nP(a|b) = P(a,b) / P(b)\nIt gives the probability of something we haven't seen\nin a Venn Diagram, it's the intersection\n\n\n**Conditional Distributions\nProbability distributions over some variables given fixed values of others\n\n\n**Probabilistic Inference**\nCompute a desired probability from other known probabilities\nNew evidence can change the probability\n** Inference by Enumeration\n- SPlit r.v into three sets:\n1-Evidence variables: things we know\n2-Query variables: things we care about and want to know about\n3-Hidden variables: things that occure but we're not observing and we don't care \nWe care about P(Q|E)\n\nSteps:\n1-take entries that are consistent with evidence\n2-sum out hidden variables\n3-Normalize it to get to add up to 1: Z = Sum_q P(Q, e_1, e_2...)\n\n**The Product Rule\nwe get marginal or conditional probabilities and we want to construct the joint probability table\nMultiply the marginal and the conditional.\nP(y)P(x|y) = P(x,y)\n\n**Chain Rule\nP(x1, x2, x3) = P(x1)P(x2|x1)P(xe|x1,x2)\nP(x1,x2,...,xn) = MUL_i P(xi|x1...xi-1)\n\n**Bayes' Rule\nP(x|y) = P(y|x)/P(y) * P(x)\n\n**Probabilistic Language Model**\nLanguage Model is a moel that computes the probabilitiy of the next word in a sentence\nOne problem is that a sentence might not exist in the corpus.\nTo address this, we use Markov Assumption, limiting the amount of history\nAn unigram will suggest the most used work in the corpus\nA bigram will look for a word given another word\n\n\n\n\n\n\n\n\n\n\n\n\n\n## Module 10 Definitions\n* **Sample space** - mutually exclusive and exhaustive possible worlds\n* **Random variable** - aspect of the world that we're interested in. Use capital letters\n* **Markov assumption** - the probability of a word depends only on the previous word\n* **Extrinsic evaluation** - embed a lnaguage model in an application and measure how much the application improves\n* **Perplexity** -  of a language model on a test set is the inverse probability of the test set, normalized by the number of words.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter13_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter13_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" Bayes' Net\n",
        "\n",
        "Graph where nodes are random variables (w/domains - they can be observed or unobserved) and the arcs are the relationships (they encode conditional independence).\n",
        "\n",
        "Conditional independence are mediated by a third observed variable\n",
        "\n",
        "X and Y are independent if P(x,y) = P(x)P(y)\n",
        "X and Y are conditionally independent given Z if P(x,y|z) = P(x|z)P(y|z) or also as P(x|z,y) = P(x|z)\u0013\n",
        "P(Traffic, Rain, Umbrella) = P(Rain)P(Traffic|Rain)P(Umbrella|Rain,Traffic) = P(Rain)P(Traffic|Rain)P(Umbrella|Rain)\n",
        "\u0013Problems:\n",
        "  *The joint probability table gets too big with many variables\n",
        "  *Hard to learn empirivally when there are too many variables\n",
        "\n",
        "Baye's Net Semantics\n",
        "1-One node per variable\n",
        "2-Directed, acyclic graph\n",
        "3-A conditional distribution for each node (with any probabiliy ditribution)\n",
        "It implicitly encodes joint distributions as a product of local conditional distributions\n",
        "\n",
        "Not every BN represent every joint distribution\n",
        "\n",
        "BN need not be causal\n",
        "Topology encodes conditional independence\n",
        "\n",
        "Size of Bayes'Net:\n",
        "For N Boolean Variables: 2^N\n",
        "For an N-node with up to k parents: O(N*2^(k+1))\n",
        "\n",
        "BN is space saving\n",
        "\n",
        "Important for modelling: understand assumptions made when choosin Bayes net graph\n",
        "\n",
        "**Algorithm: D-Separation**\n",
        "To know if things are independent of each other\n",
        "*A - Causal Chains: X causes Y causes Z\n",
        "P(x,y,z) = P(x)P(y|x)P(z|y)\n",
        "Is X indenpendet of Z? A single examples is sufficient to shows that is not guaranteed\n",
        "Is X independet of Z given Y? P(z|x,y) = P(x,y,z)/P(x,y)\n",
        "Proof:\n",
        "P(z|x) or P(z|y) = P(x)P(y|x)P(z|y)/(P(x)P(y|x)) = P(z|y)\n",
        "Evidence along the chain blocks the influence (that is, if I know that Y is active, knowing that Z is active doesn't affect my knowledge of Z)\n",
        "\n",
        "*B - Common Cause: Y cause X and Y causes Z\n",
        "P(x,y,z) = P(y)P(x|y)P(z|y)\n",
        "Observing the cause blocks influence between effects.\n",
        "\n",
        "*C - Commmon Effect: X causes Z and Y causes Z\n",
        "Are X and Y independent of each other?\n",
        "There are two potential causes.\n",
        "Observing an effect activates influence between the possible causes.\n",
        "\n",
        "**Reachability and Path**\n",
        "How to know if two variables in a BN are indpendent? break the graph and use the three cases\n",
        "Active/Inactive Paths\n",
        "Are X and Y conditionally independent given {Z}? True if X and Y are d-separate by Z. If no active paths, then there's independence\n",
        "\n",
        "Active Path: Where all the triples are active:\n",
        "Causal chain: A->B->C, where B is unobserved\n",
        "Common Cause: A<-B->C, where B is unobserved\n",
        "Common Effect: A->B<-C, where B or one of its descendant is observed\n",
        "\n",
        "**D-Separation Algorithm**\n",
        "we want to know if Xi is guranteed conditionally independent of Xj, given observed variables Xk:\n",
        "1) Look for undirected paths between Xi and Xj. If one or more is active, the independence is not guaranteed\n",
        "2) If all paths are inactive, the independence is guaranteed\n",
        "\n",
        "**Inference**\n",
        "Posterior probability: P(Q|E1=e1,...,Ek=ek)\n",
        "Most Likely explanation: argmax P(Q=q|E1=e1...)\n",
        "The problem with this approach is that it's very expenseive, as it's exponential to the \n",
        "number of hidden variables. If they're binary, that's 2^(number of hidden variables)\n",
        "We have to join the whole joint distribution before summing up the hidden variables.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"Bayes' Net\" : \"A technique for describing complex joint distribution using simple, local distributions\u0013\", \n",
        "\"Observed nodes\" : \"We have evidence for\",\n",
        "\"Active nodes\" : \"They can still influence\",   \n",
        "}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "97e0a7c9-3ca2-4649-e58d-52e33b22f64b"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Bayes' Net\n\nGraph where nodes are random variables (w/domains - they can be observed or unobserved) and the arcs are the relationships (they encode conditional independence).\n\nConditional independence are mediated by a third observed variable\n\nX and Y are independent if P(x,y) = P(x)P(y)\nX and Y are conditionally independent given Z if P(x,y|z) = P(x|z)P(y|z) or also as P(x|z,y) = P(x|z)\u0013\nP(Traffic, Rain, Umbrella) = P(Rain)P(Traffic|Rain)P(Umbrella|Rain,Traffic) = P(Rain)P(Traffic|Rain)P(Umbrella|Rain)\n\u0013Problems:\n  *The joint probability table gets too big with many variables\n  *Hard to learn empirivally when there are too many variables\n\nBaye's Net Semantics\n1-One node per variable\n2-Directed, acyclic graph\n3-A conditional distribution for each node (with any probabiliy ditribution)\nIt implicitly encodes joint distributions as a product of local conditional distributions\n\nNot every BN represent every joint distribution\n\nBN need not be causal\nTopology encodes conditional independence\n\nSize of Bayes'Net:\nFor N Boolean Variables: 2^N\nFor an N-node with up to k parents: O(N*2^(k+1))\n\nBN is space saving\n\nImportant for modelling: understand assumptions made when choosin Bayes net graph\n\n**Algorithm: D-Separation**\nTo know if things are independent of each other\n*A - Causal Chains: X causes Y causes Z\nP(x,y,z) = P(x)P(y|x)P(z|y)\nIs X indenpendet of Z? A single examples is sufficient to shows that is not guaranteed\nIs X independet of Z given Y? P(z|x,y) = P(x,y,z)/P(x,y)\nProof:\nP(z|x) or P(z|y) = P(x)P(y|x)P(z|y)/(P(x)P(y|x)) = P(z|y)\nEvidence along the chain blocks the influence (that is, if I know that Y is active, knowing that Z is active doesn't affect my knowledge of Z)\n\n*B - Common Cause: Y cause X and Y causes Z\nP(x,y,z) = P(y)P(x|y)P(z|y)\nObserving the cause blocks influence between effects.\n\n*C - Commmon Effect: X causes Z and Y causes Z\nAre X and Y independent of each other?\nThere are two potential causes.\nObserving an effect activates influence between the possible causes.\n\n**Reachability and Path**\nHow to know if two variables in a BN are indpendent? break the graph and use the three cases\nActive/Inactive Paths\nAre X and Y conditionally independent given {Z}? True if X and Y are d-separate by Z. If no active paths, then there's independence\n\nActive Path: Where all the triples are active:\nCausal chain: A->B->C, where B is unobserved\nCommon Cause: A<-B->C, where B is unobserved\nCommon Effect: A->B<-C, where B or one of its descendant is observed\n\n**D-Separation Algorithm**\nwe want to know if Xi is guranteed conditionally independent of Xj, given observed variables Xk:\n1) Look for undirected paths between Xi and Xj. If one or more is active, the independence is not guaranteed\n2) If all paths are inactive, the independence is guaranteed\n\n**Inference**\nPosterior probability: P(Q|E1=e1,...,Ek=ek)\nMost Likely explanation: argmax P(Q=q|E1=e1...)\nThe problem with this approach is that it's very expenseive, as it's exponential to the \nnumber of hidden variables. If they're binary, that's 2^(number of hidden variables)\nWe have to join the whole joint distribution before summing up the hidden variables.\n\n\n\n\n\n\n\n\n\n\n\n \n\n## Module 11 Definitions\n* **Bayes' net** - A technique for describing complex joint distribution using simple, local distributions\u0013\n* **Observed nodes** - We have evidence for\n* **Active nodes** - They can still influence\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_6 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\" Naive Bayes Classifiers\n",
        " It returns the class which has the maximum posterior probability given a document.\n",
        " c_hat = argmax P(c|d) = argmax P(d|c)P(c)\n",
        " P(d|c) is the likelihood of the document and P(c) is the prior probability of the class.\n",
        " NB makes two simplifying assumptions:\n",
        " 1) Assume the order of the fetarues iw not important\n",
        " 2) The conditional probabilities of the featrues are independent, and can then be mulitiplied\n",
        " Calculations are done in log space to avoid undeflow and increase speed.\n",
        " \"\"\"\n",
        "slp_chapter4_2 = \"\"\" Training the Naive Bayes Classfier\n",
        "For P(c), check how many docs are in each category: P_hat(c) = N_c/N_doc\n",
        "P_hat(w_i|c) = count(w_i,c)/Sum_w count(w,v)\n",
        "The main problem is because the features are multiplied, if one of them doesn't appear in the training data, the class will get zero probability\n",
        "A possible solution is to use Laplace smoothing. the usion of all the word types needs to be accross categories.\n",
        "Unknow words: words that occur in the test data but not in our vocabulary? they should be excluded\n",
        "stop words: words like The, a, etc\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter4_3 = \"\"\" Worked example \"\"\"\n",
        "slp_chapter4_4 = \"\"\" Optimizing for Sentiment Analysis\n",
        "Sometimes, the presence of a word is more important than its frequency, so it's clipped at one for each document (Binary naive Bayes)\n",
        "To handle negation, preface with NOT_ every word util the next punctuation mark.\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Naive Bayes for other text classification tasks\n",
        "The classifier doens't need to use all the words in the training data as features.\n",
        "It can also use features other than words\n",
        "For language identification, useing character n-grams is effective ('zw', 'nya', 'ie z', 'thei')\n",
        "\"\"\"\n",
        "slp_chapter4_6 = \"\"\" Naive Bayes as a Language Model\n",
        "A NB model can be viewed as a set of class-specific unigram language models, where the model for each class instatiates a unigram laguage model, as each word is assigned a probability, then also each sentence has a probability assigned.\"\"\"\n",
        "slp_chapter4_7 = \"\"\"Evaluation: Preceision, Recall, F-measure\n",
        "accuracy is teh percentage of observations labeled correctly. Not used much for text classification, becaus it doesn't work well when the classes are unbalanced. It's not a good metric when we want to find something that is rare.\n",
        "Precision is use instead, which is the percentage of items detected that are positive.\n",
        "Recall measures the % of items present in the input that were identified.\n",
        "F-Measure: F_beta = (Beta^2+1)PR/(Beta^2P+R) Beta weights >1 favors recall and  beta < 1 favor precision. (F1= 2PR/(P+R))\n",
        "The Harmonic mean is closer to the minimum than the arithmetic mean.\n",
        "\n",
        "**Evaluating with more than two classes\n",
        "NB is already a multi-class classification algorithm\n",
        "To measure precision and recall, we need to use microaveraging, where we compute the performance for each class, and then average over classes.\n",
        "appropriate when all classes are equally important.\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter4_8 = \"\"\" Test sets and Cross-validation\n",
        "The problem with splitting the data in training, dev and test set, is that we may run out of data. We can use cross-validation to use all the data.\n",
        "We take a number k, and split the data in k parts. use one k fold as test set, and traing on the other k-1 folds. repeat 10 times and average for 10-fold cross-validation \"\"\"\n",
        "slp_chapter4_9 = \"\"\" Statistical Significance Testing\n",
        "How to compare the performance of two systems (statistical hypothesis testing)\n",
        "effect size: M(A,x)-M(B,x) If A or B is better given x.\n",
        "set up two hypothesis:\n",
        "H_0 : gamma(x) <= 0 (null hypothesis)\n",
        "H_1 : gamme(x) > 0\n",
        "To check if we can support H_1, we use p-value: P(gamma(X)>=gamma(x)|H_0 is true)\n",
        "A very small p-value means we can reject H_0\n",
        "To compute the probability, we don't use parametric tests, because they make assumptions about the ditribution that don't hold for NLP\n",
        "Use instad non=paramtric tests based on sampling (tests like 'approximate randomization' and 'boostrap test')\n",
        "**The Paired Boostrap Test\n",
        "can apply to any metric. draw large number of smaller samples with replacement from original large sample. \n",
        "\u0011\u0011\u0011\u0011 \"\"\"\n",
        "slp_chapter4_10 = \"\"\" Avoiding Harms in Classification\n",
        "representational harm: harms caused by a system that demeans a social group\n",
        "ML can amplify the biases in their training data casued by labelers, resources used or model architecture. \n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\" Machine Learning\n",
        "How to learn from examples/data\n",
        "**Classification Task**\n",
        "For problems where we don't know the probabilities.\n",
        "1) Learning parameters (probabilities)\n",
        "2) Learnign structure (Bayes Net graphs)\n",
        "3) Learnign hidden concepts (clustering)\n",
        "\n",
        "Input is represented as a feature vector\n",
        "Given a collection of training data, we want to learn a function that predicts a class\n",
        "\n",
        "Model base approach: build a mode where the class and the reatures are r.v.\n",
        "then, instantiate all the observed features and query the model (what class?)\n",
        "\n",
        "**Naive Bayes**\n",
        "Naive: Assume that all features are independent of each other, and only depend on the label\n",
        "P(Y,F1...Fn) is proportional to the P(Y) Product P(Fi|Y)\n",
        "|Y|*|F|^n values exponential to the number of features, but with the independent assumption we have n*|F|*|Y| (now is linear to the number of features)\n",
        "The model is simplistic because it assumes a lot of independence, but often works\n",
        "\n",
        "Objective: Compute pos dist ove label variable Y\n",
        "1) get joint probability of each each label of the label and evidence\n",
        "P(y1) MUL P(Fi|y1)\n",
        "P(y2) MUL P(Fi|y2)\n",
        "P(yk) MUL P(Fi|yk)\n",
        "2) then we sum the probabilities to get the probablity of the evidence\n",
        "and then 3)divide each one to normalize them (divide each row by the sum)\n",
        "\n",
        "We need to estimate the local conditional probabilities. \n",
        "\n",
        "**Basic Concpts on ML**\n",
        "Need to train the model. Partition data in three sets:\n",
        "1)Training: to estimate parameters. Use data labeled\n",
        "2)Heldout/Development: To optimize hyperparameters\n",
        "3)Test: To evaluate the model\n",
        "\n",
        "Overfitting: Relative frequency parameters will overfit the training data. Avoid given 0 probability to unseen eevents. Therefore, we smooth or regularize the data\n",
        "\n",
        "Parameter estimation: Elicitation or Empirically\n",
        "P_ML(x) = count(x)/total samples (P_ML is the probability of maximum likelihood)\n",
        "To know what estimate is better, we calculate the likelihood of data: L(x,theta) = MUL P_theta(x_i)\n",
        "\n",
        "**Maximum likelihood\n",
        "Pick the parameters that maximize the likelihood of data given the parameters\n",
        "the problem is unseen events, so we smooth the data\n",
        "\n",
        "**Laplace Smoothing\n",
        "Pretend every outcomes was seen once more than actually happened, or we can add an arbitrary number\n",
        "performs poorly with |Y| of |X| is very large for P(X|Y)\n",
        "\n",
        "**Linear Interpolation\n",
        "Use the current observed times alpha + prior estimate times 1-alpha\n",
        "\n",
        "**Tuning\n",
        "How to pick best parametrization\n",
        "paramters came from data traininf set\n",
        "hyperparameters come from another data set (to faciliate generalization)\n",
        "model accuracy on trainign data goes down as we increase k as the model with training was the best by definition\n",
        "\n",
        "Another option is to add new features\n",
        "\n",
        "Baseline: if I use an obvious label, what accuracy do I get?\n",
        "\n",
        "\n",
        "*** Pereceptrons *** \n",
        "Make decisions by weighting evidence\n",
        "\n",
        "Sum_j w_jc_j = w * x\n",
        "\n",
        "\n",
        "output: \n",
        "0 if w*x + b <= 0\n",
        "1 if w*x + b > 0\n",
        "b is the bias\n",
        "\n",
        "**Learning weights from examples\n",
        "Inputs are features representing something we want to classify. the weight provides a different classifier\n",
        "\n",
        "In binary decision rule, the bias allows to shift the boudary\n",
        "\n",
        "To learn the weights, we try several scenarios, if the output is correct, no change, if it's wrong, adjust the weight vector\n",
        "we start with all weights = 0 , and then classify the training instance. If it's wrong, we adjust the weight vector\n",
        "\n",
        "In multiclass scenarios:\n",
        "1. a w_y vector for each class\n",
        "2. an activation of a class y:  w_y * f(x)\n",
        "3. Prediction with the highest score wins: y = argmax w_y * f(x)\n",
        "Update the incorrect output and increase the correct output\n",
        "\n",
        "** Improvements on Perceptrons and SVM**\n",
        "If the data is separable, we're guaranteed perceptrons will converge on parameters\n",
        "If the data is not separable, we can bound the mistakes:\n",
        "mistakes < k/gamma^2 where gamma is the margin (degree of separability) and k is a constant\n",
        "\n",
        "Problems with Perecprons:\n",
        "1) Noise: If data is not separable, weights might thrash (go back and forth). It can be reduced by averaging the weights (averaged perceptron)\n",
        "2) Mediocre generalization: Barely separates\n",
        "3) Overtraining: type of overfitting\n",
        "\n",
        "Fixing the Perceptron:\n",
        "*MIRA: Margin Infused Relaxed Algorithm\n",
        "Update the weights by as little as possible. We scale the adjustment by Thau w_y = w'_y - Thau f(x)\n",
        "*Maximum Step Size: \n",
        "Limit Thau to no more than a constant\n",
        "Works very good with noisy data that is not easily to separate\n",
        "*Linear Separators (Support Vector Machine)\n",
        "Tries to create two vectors that maximize the boudary between the two classes\n",
        "*Non-Linear Separators\n",
        "Map data to higher dimensions and it becomes separable\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"Prior Probability\" : \"For a node in a Bayes Net without a parent\",\n",
        "\"Parameters of the model\" : \"Prior probabilities over labels and the conditional probabilities for each freature\",\n",
        "\"overfitting\":\"The model fits too well the training data but does not generalizes well\",\n",
        "\"Elicitation\":\"Ask a human to estimate a parameter\",\n",
        "\"Empirically\":\"Use training data\",\n",
        "\"supervised learning\":\"we have a data set of input observations, each associates with some correct output(a supervision signal)\",\n",
        "\"Generative classrifier\":\"buidl amodel of how a class could generate some input data\",\n",
        "\"Discriminative classifier\":\"like logistic regression, learn waht reatures from the input are most useful to discriminate between the different possible classes\",\n",
        "\"bag-of-word\":\"an unordered set of words with their positions ignored, keeping only their frequency in the document\",\n",
        "\"sentiment lexicons\":\"lists of words that are preannotated with positice or negative sentiment\",\n",
        "\"gold labels\":\"labels assigned by humans\",\n",
        "\"confusion matrix\":\"A table for visualizing how an algorithm performs with respect to the human gold label.\",\n",
        "\"effect size\" : \"Difference in F1 between two clasfiers\",\n",
        "\"toxicity detection\":\"the task of detecting hate speech, abuse, harassment or other kinds of toxic langauge.\"\n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "99a85411-5bda-4a63-b776-287458072f10"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Naive Bayes Classifiers\n It returns the class which has the maximum posterior probability given a document.\n c_hat = argmax P(c|d) = argmax P(d|c)P(c)\n P(d|c) is the likelihood of the document and P(c) is the prior probability of the class.\n NB makes two simplifying assumptions:\n 1) Assume the order of the fetarues iw not important\n 2) The conditional probabilities of the featrues are independent, and can then be mulitiplied\n Calculations are done in log space to avoid undeflow and increase speed.\n \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Training the Naive Bayes Classfier\nFor P(c), check how many docs are in each category: P_hat(c) = N_c/N_doc\nP_hat(w_i|c) = count(w_i,c)/Sum_w count(w,v)\nThe main problem is because the features are multiplied, if one of them doesn't appear in the training data, the class will get zero probability\nA possible solution is to use Laplace smoothing. the usion of all the word types needs to be accross categories.\nUnknow words: words that occur in the test data but not in our vocabulary? they should be excluded\nstop words: words like The, a, etc\n\n\n\n\n## SLP Chapter 4.3 Worked Example\n Worked example \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Optimizing for Sentiment Analysis\nSometimes, the presence of a word is more important than its frequency, so it's clipped at one for each document (Binary naive Bayes)\nTo handle negation, preface with NOT_ every word util the next punctuation mark.\n\n \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Naive Bayes for other text classification tasks\nThe classifier doens't need to use all the words in the training data as features.\nIt can also use features other than words\nFor language identification, useing character n-grams is effective ('zw', 'nya', 'ie z', 'thei')\n\n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Naive Bayes as a Language Model\nA NB model can be viewed as a set of class-specific unigram language models, where the model for each class instatiates a unigram laguage model, as each word is assigned a probability, then also each sentence has a probability assigned.\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\nEvaluation: Preceision, Recall, F-measure\naccuracy is teh percentage of observations labeled correctly. Not used much for text classification, becaus it doesn't work well when the classes are unbalanced. It's not a good metric when we want to find something that is rare.\nPrecision is use instead, which is the percentage of items detected that are positive.\nRecall measures the % of items present in the input that were identified.\nF-Measure: F_beta = (Beta^2+1)PR/(Beta^2P+R) Beta weights >1 favors recall and  beta < 1 favor precision. (F1= 2PR/(P+R))\nThe Harmonic mean is closer to the minimum than the arithmetic mean.\n\n**Evaluating with more than two classes\nNB is already a multi-class classification algorithm\nTo measure precision and recall, we need to use microaveraging, where we compute the performance for each class, and then average over classes.\nappropriate when all classes are equally important.\n\n \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Test sets and Cross-validation\nThe problem with splitting the data in training, dev and test set, is that we may run out of data. We can use cross-validation to use all the data.\nWe take a number k, and split the data in k parts. use one k fold as test set, and traing on the other k-1 folds. repeat 10 times and average for 10-fold cross-validation \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Statistical Significance Testing\nHow to compare the performance of two systems (statistical hypothesis testing)\neffect size: M(A,x)-M(B,x) If A or B is better given x.\nset up two hypothesis:\nH_0 : gamma(x) <= 0 (null hypothesis)\nH_1 : gamme(x) > 0\nTo check if we can support H_1, we use p-value: P(gamma(X)>=gamma(x)|H_0 is true)\nA very small p-value means we can reject H_0\nTo compute the probability, we don't use parametric tests, because they make assumptions about the ditribution that don't hold for NLP\nUse instad non=paramtric tests based on sampling (tests like 'approximate randomization' and 'boostrap test')\n**The Paired Boostrap Test\ncan apply to any metric. draw large number of smaller samples with replacement from original large sample. \n\u0011\u0011\u0011\u0011 \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Avoiding Harms in Classification\nrepresentational harm: harms caused by a system that demeans a social group\nML can amplify the biases in their training data casued by labelers, resources used or model architecture. \n\n \n\n## Module 12 Lecture Notes\n Machine Learning\nHow to learn from examples/data\n**Classification Task**\nFor problems where we don't know the probabilities.\n1) Learning parameters (probabilities)\n2) Learnign structure (Bayes Net graphs)\n3) Learnign hidden concepts (clustering)\n\nInput is represented as a feature vector\nGiven a collection of training data, we want to learn a function that predicts a class\n\nModel base approach: build a mode where the class and the reatures are r.v.\nthen, instantiate all the observed features and query the model (what class?)\n\n**Naive Bayes**\nNaive: Assume that all features are independent of each other, and only depend on the label\nP(Y,F1...Fn) is proportional to the P(Y) Product P(Fi|Y)\n|Y|*|F|^n values exponential to the number of features, but with the independent assumption we have n*|F|*|Y| (now is linear to the number of features)\nThe model is simplistic because it assumes a lot of independence, but often works\n\nObjective: Compute pos dist ove label variable Y\n1) get joint probability of each each label of the label and evidence\nP(y1) MUL P(Fi|y1)\nP(y2) MUL P(Fi|y2)\nP(yk) MUL P(Fi|yk)\n2) then we sum the probabilities to get the probablity of the evidence\nand then 3)divide each one to normalize them (divide each row by the sum)\n\nWe need to estimate the local conditional probabilities. \n\n**Basic Concpts on ML**\nNeed to train the model. Partition data in three sets:\n1)Training: to estimate parameters. Use data labeled\n2)Heldout/Development: To optimize hyperparameters\n3)Test: To evaluate the model\n\nOverfitting: Relative frequency parameters will overfit the training data. Avoid given 0 probability to unseen eevents. Therefore, we smooth or regularize the data\n\nParameter estimation: Elicitation or Empirically\nP_ML(x) = count(x)/total samples (P_ML is the probability of maximum likelihood)\nTo know what estimate is better, we calculate the likelihood of data: L(x,theta) = MUL P_theta(x_i)\n\n**Maximum likelihood\nPick the parameters that maximize the likelihood of data given the parameters\nthe problem is unseen events, so we smooth the data\n\n**Laplace Smoothing\nPretend every outcomes was seen once more than actually happened, or we can add an arbitrary number\nperforms poorly with |Y| of |X| is very large for P(X|Y)\n\n**Linear Interpolation\nUse the current observed times alpha + prior estimate times 1-alpha\n\n**Tuning\nHow to pick best parametrization\nparamters came from data traininf set\nhyperparameters come from another data set (to faciliate generalization)\nmodel accuracy on trainign data goes down as we increase k as the model with training was the best by definition\n\nAnother option is to add new features\n\nBaseline: if I use an obvious label, what accuracy do I get?\n\n\n*** Pereceptrons *** \nMake decisions by weighting evidence\n\nSum_j w_jc_j = w * x\n\n\noutput: \n0 if w*x + b <= 0\n1 if w*x + b > 0\nb is the bias\n\n**Learning weights from examples\nInputs are features representing something we want to classify. the weight provides a different classifier\n\nIn binary decision rule, the bias allows to shift the boudary\n\nTo learn the weights, we try several scenarios, if the output is correct, no change, if it's wrong, adjust the weight vector\nwe start with all weights = 0 , and then classify the training instance. If it's wrong, we adjust the weight vector\n\nIn multiclass scenarios:\n1. a w_y vector for each class\n2. an activation of a class y:  w_y * f(x)\n3. Prediction with the highest score wins: y = argmax w_y * f(x)\nUpdate the incorrect output and increase the correct output\n\n** Improvements on Perceptrons and SVM**\nIf the data is separable, we're guaranteed perceptrons will converge on parameters\nIf the data is not separable, we can bound the mistakes:\nmistakes < k/gamma^2 where gamma is the margin (degree of separability) and k is a constant\n\nProblems with Perecprons:\n1) Noise: If data is not separable, weights might thrash (go back and forth). It can be reduced by averaging the weights (averaged perceptron)\n2) Mediocre generalization: Barely separates\n3) Overtraining: type of overfitting\n\nFixing the Perceptron:\n*MIRA: Margin Infused Relaxed Algorithm\nUpdate the weights by as little as possible. We scale the adjustment by Thau w_y = w'_y - Thau f(x)\n*Maximum Step Size: \nLimit Thau to no more than a constant\nWorks very good with noisy data that is not easily to separate\n*Linear Separators (Support Vector Machine)\nTries to create two vectors that maximize the boudary between the two classes\n*Non-Linear Separators\nMap data to higher dimensions and it becomes separable\n\n \n\n## Module 12 Definitions\n* **Prior probability** - For a node in a Bayes Net without a parent\n* **Parameters of the model** - Prior probabilities over labels and the conditional probabilities for each freature\n* **Overfitting** - The model fits too well the training data but does not generalizes well\n* **Elicitation** - Ask a human to estimate a parameter\n* **Empirically** - Use training data\n* **Supervised learning** - we have a data set of input observations, each associates with some correct output(a supervision signal)\n* **Generative classrifier** - buidl amodel of how a class could generate some input data\n* **Discriminative classifier** - like logistic regression, learn waht reatures from the input are most useful to discriminate between the different possible classes\n* **Bag-of-word** - an unordered set of words with their positions ignored, keeping only their frequency in the document\n* **Sentiment lexicons** - lists of words that are preannotated with positice or negative sentiment\n* **Gold labels** - labels assigned by humans\n* **Confusion matrix** - A table for visualizing how an algorithm performs with respect to the human gold label.\n* **Effect size** - Difference in F1 between two clasfiers\n* **Toxicity detection** - the task of detecting hate speech, abuse, harassment or other kinds of toxic langauge.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\"\"\"A generative model uses likelihood, if we know that something is of a given class.\n",
        "A discrimitaive model by idneitfying what's unique, but if can't generate an exmaple of the class \"\"\"\n",
        "slp_chapter5_1 = \"\"\" Classification: the sigmoid\n",
        "The goal of binary LR is to train a classifier to decide between two classes.\n",
        "The sigmoid function takes a real-valued number and maps it to a range 0 - 1. It squashes ourlier values toward 0 or 1, and it's differentiable. \n",
        "Naive Bayes has a stron condtional independence assumption. If we incldue the same feature twice, NB will treat them as different and will mulitiply them.\n",
        "Logistic Regression is more robust and will identify if they're prefectly correlated. \n",
        "When there are many correlated featues, LR wil be more accurate than NB\n",
        "NB works really well with small datasets, it's easy to implement and fast to train \"\"\"\n",
        "slp_chapter5_2 = \"\"\" Learning on Logistic Regression\n",
        "We need two components: \n",
        "1) How close y_hat is to y (loss function)\n",
        "2) Optimization algorithm (stochastic gradient descent)\n",
        "\"\"\"\n",
        "slp_chapter5_3 = \"\"\" The cross-entorpy loss function\n",
        "Conditional maximum likelihood estimation: choose the parameters w,b that maximize the log probability of the true y labels in the training data/\n",
        " \"\"\"\n",
        "slp_chapter5_4 = \"\"\" Gradient Descent\n",
        "The goal of gradeint descent is to find the optimal weights that minimize the loss function defined for the model.\n",
        "Gradient descent algorithm finds the gradient of the loss function at the current point and moves in the opposite direction.\n",
        "The change we make in the parameters is the learning rate times the gradient \n",
        "*Mini Batch training\n",
        "Stochastic is called like that because it chooses a single randomg example every time\"\"\"\n",
        "slp_chapter5_5 = \"\"\" Regularization\n",
        "A setting of the weights that matches the training data perfectly (but uses many weights with high values to do so) will be penalized more than a setting that matches the data a little less well, but does so using smaller weights. \n",
        "L2 regularization uses the euclidena distance\n",
        "L1 regularization uses the Manhattan distance\n",
        "L2 prefers weight vectors with many small weights\n",
        "L1 prefers sparse solutions with some larger weights but many more weights set to zero. L! leads to sparser wieght vectors, fewer features\n",
        "\"\"\"\n",
        "slp_chapter5_6 = \"\"\" Multinomial logistic regression\n",
        "The softmax function is a generalization of the sigmoid function, where each class is mapped with a probability between 0 and 1, all summing up to 1.\n",
        "With softmax if one of the inputs is larger than the others, it will tendo to push its probabiliy toward 1 and suppress the probability of the smaller inputs\n",
        "\n",
        "**Features in Multinomial Logistic Regresssion**\n",
        "We'll need separate weight vectors (and biases) for each of the K classes. \n",
        "\n",
        "**Learning in Mulitnomial Logistic Regression**\n",
        "The loss function for multinomial logistic regresion generalizes the loss function for binary logistic regresson from 2 to K classes.\n",
        "\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\" \n",
        "** Logistic Regression **\n",
        "A supervised ML problem. Components:\n",
        "1) Feature representation of the input (vector of features)\n",
        "2) A classification function, that estimates y_hat via P(y|x)\n",
        "3) An objective function or loss function (tells how good or bad the function is, usually to minimize)\n",
        "4) An algorithm to optimize the objective function\n",
        "\n",
        "A difference between perceptrons and sentiment logistic regression is the range of values the classifier can output.\n",
        "In perceptron it can be anything.\n",
        "\n",
        "In order to cnovert to probability, we use the sigmoid function.\n",
        "\n",
        "P(y=1) = sigmoid funtion (w*x+b) = 1 / (1 + e ^-(wx+b))\n",
        "P(y=0) = 1 - sigmoid(wx+b)\n",
        "\n",
        "*Decision Boundary*\n",
        "y_hat = 1 if P(y=1|x) > 5 ; 0 otherwise\n",
        "\n",
        "*Learning in logistic regression*\n",
        "1) We need a loss function that tells us how far are we from the gold output (use cross-entropy loss)\n",
        "2) An algorithm to optimize the objective function (minimize the loss function)\n",
        "\n",
        "*Loss Functions*\n",
        "L(y_hat, y) = how much y_hat differs from true y\n",
        "One option is the mean squared error, but it's hard to optimize because it's not convex\n",
        "For probabilistic classification it's better to use \"Conditional Maximum Likelihood Estimation\":\n",
        "choose parameters that maximize the (log) probability of the true labels in the training data.\n",
        "This is the log likelihood loss (called cross entropy loss)\n",
        "\n",
        "We want to maximize the probability of the correct label:\n",
        "P(y|x) = y_hat ^ y(1-y_hat) ^(1-y)\n",
        "y can be either 1 or zero\n",
        "step 1: Change to log probability -> log p(x|y) = log[y_hat^y(-y_hat)^(1-y)] = y log y_hat + (1-y)log(1-y_hat)\n",
        "\n",
        "If our prediction is correct, the our CE loss is lower\n",
        "If our prediction is incorrect, then our CE loss is higher\n",
        "\n",
        "\n",
        "** Stochastic Gradient Descent**\n",
        "We want to find the parameters that minimize the loss function\n",
        "Theta is going to be the parameters and biases\n",
        "Theta_hat = argmin 1/m (Sum_i L_CE(y^i, c^i; Theta))\n",
        "Gradient Descent: Find the minimum of a function by finding the slope and moving towards the down slope\n",
        "\n",
        "A convex function has one minimum. There are no local minimum, so it's guranteed to find the minimum\n",
        "\n",
        "The step size is the learning rate and it's reprensented by greek letter 'eta'\n",
        "\n",
        "The new weight: w_t+1 = w_t - eta (derivative of the loss function with respect to the weight)\n",
        "\n",
        "*The Gradient is the difference between the true y and out estimate for x, multiplied by the input value x_j\n",
        "Average loss (this is to minimize, for every item in the training set):\n",
        "Cost(w,b) = 1/m Sum L_CE(y_hat, y)\n",
        "= -1/m Sum y log theta (wx+b) + (1-y) log (1-theta(wx+b))\n",
        "\n",
        "*Mini-batch training\n",
        "Classis stochastic descent chooses a single random example.\n",
        "With batch training, we compute the gradient over the entire dataset. It provides a better estimate.\n",
        "Mini-batch uses small batches of 512 or 1024.\n",
        "\n",
        "*Regularization (to avoid overfitting)\n",
        "Overfitting is a problem in ML.\n",
        "Regularization penalizes large weights. Lasso regularization uses L1 (sum of abs values, or manhattan distacnce), Ridge uses L2 (quadratic function).\n",
        "\n",
        "**Multinomial logistic regression\n",
        "We use a generalization of the sigmoid function called softmax function to use with multinomial LR\n",
        "usual the final step in NN\n",
        "Softmax: it takes an input vector and outputs a vector of values normalized into probabilities\n",
        "\n",
        "\n",
        "**Neural Networks**\n",
        "Difference between perceptrons and logistic regression is that in logistic regression we apply a sigmoid function.\n",
        "Also, perceptron outputs 0 or 1, which makes updating the weight vector difficult\n",
        "*Activation function\n",
        "The neuron unit ouputs a non-linear valued. Three functions are commonly used:\n",
        "1.Sigmoid\n",
        "2.Tanh\n",
        "3.Rectified Linear Unit (ReLU)\n",
        "\n",
        "*Feed Forward architecture: Multilayer network, fully-connected with no cycles\n",
        "\n",
        "*How to do the training:\n",
        "we need:\n",
        "1) A loss function\n",
        "2) An optimization algorithm (gradient descent)\n",
        "3) A way of computing the gradient of the loss function (error propagation)\n",
        "\n",
        "**Computing the gradient\n",
        "The derivatives of logistic regression only give correct updates for the last weight layer\n",
        "For deeper networks, we need to look back a previous layers. We use \"Error propagation algorithm\"\n",
        "*Backward differentiation: compute how each portion of a graph contributes to the end result\n",
        "\n",
        "**Neural Language Tools**\n",
        "Key components:\n",
        "1) NN - feedforward, with input layer, hidden layers, \n",
        "2) Training data\n",
        "3) Loss function\n",
        "We use specific activation functions that are diferentiable\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"loss function\" : \"how wrong the predictions are for each item in the training set.\",  \n",
        "\"training set\" : \"a vector of weights and a bias term\",\n",
        "\"hyperparameters\" : \"parameters selected by the designer of the algorithm that affect how the algorithm works\"\n",
        "\n",
        "}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a4086544-f8af-4e97-f61e-789382510c5b"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Classification: the sigmoid\nThe goal of binary LR is to train a classifier to decide between two classes.\nThe sigmoid function takes a real-valued number and maps it to a range 0 - 1. It squashes ourlier values toward 0 or 1, and it's differentiable. \nNaive Bayes has a stron condtional independence assumption. If we incldue the same feature twice, NB will treat them as different and will mulitiply them.\nLogistic Regression is more robust and will identify if they're prefectly correlated. \nWhen there are many correlated featues, LR wil be more accurate than NB\nNB works really well with small datasets, it's easy to implement and fast to train \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Learning on Logistic Regression\nWe need two components: \n1) How close y_hat is to y (loss function)\n2) Optimization algorithm (stochastic gradient descent)\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n The cross-entorpy loss function\nConditional maximum likelihood estimation: choose the parameters w,b that maximize the log probability of the true y labels in the training data/\n \n\n## SLP Chapter 5.4 Gradient Descent\n Gradient Descent\nThe goal of gradeint descent is to find the optimal weights that minimize the loss function defined for the model.\nGradient descent algorithm finds the gradient of the loss function at the current point and moves in the opposite direction.\nThe change we make in the parameters is the learning rate times the gradient \n*Mini Batch training\nStochastic is called like that because it chooses a single randomg example every time\n\n## SLP Chapter 5.5 Regularization\n Regularization\nA setting of the weights that matches the training data perfectly (but uses many weights with high values to do so) will be penalized more than a setting that matches the data a little less well, but does so using smaller weights. \nL2 regularization uses the euclidena distance\nL1 regularization uses the Manhattan distance\nL2 prefers weight vectors with many small weights\nL1 prefers sparse solutions with some larger weights but many more weights set to zero. L! leads to sparser wieght vectors, fewer features\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n Multinomial logistic regression\nThe softmax function is a generalization of the sigmoid function, where each class is mapped with a probability between 0 and 1, all summing up to 1.\nWith softmax if one of the inputs is larger than the others, it will tendo to push its probabiliy toward 1 and suppress the probability of the smaller inputs\n\n**Features in Multinomial Logistic Regresssion**\nWe'll need separate weight vectors (and biases) for each of the K classes. \n\n**Learning in Mulitnomial Logistic Regression**\nThe loss function for multinomial logistic regresion generalizes the loss function for binary logistic regresson from 2 to K classes.\n\n\n \n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n \n** Logistic Regression **\nA supervised ML problem. Components:\n1) Feature representation of the input (vector of features)\n2) A classification function, that estimates y_hat via P(y|x)\n3) An objective function or loss function (tells how good or bad the function is, usually to minimize)\n4) An algorithm to optimize the objective function\n\nA difference between perceptrons and sentiment logistic regression is the range of values the classifier can output.\nIn perceptron it can be anything.\n\nIn order to cnovert to probability, we use the sigmoid function.\n\nP(y=1) = sigmoid funtion (w*x+b) = 1 / (1 + e ^-(wx+b))\nP(y=0) = 1 - sigmoid(wx+b)\n\n*Decision Boundary*\ny_hat = 1 if P(y=1|x) > 5 ; 0 otherwise\n\n*Learning in logistic regression*\n1) We need a loss function that tells us how far are we from the gold output (use cross-entropy loss)\n2) An algorithm to optimize the objective function (minimize the loss function)\n\n*Loss Functions*\nL(y_hat, y) = how much y_hat differs from true y\nOne option is the mean squared error, but it's hard to optimize because it's not convex\nFor probabilistic classification it's better to use \"Conditional Maximum Likelihood Estimation\":\nchoose parameters that maximize the (log) probability of the true labels in the training data.\nThis is the log likelihood loss (called cross entropy loss)\n\nWe want to maximize the probability of the correct label:\nP(y|x) = y_hat ^ y(1-y_hat) ^(1-y)\ny can be either 1 or zero\nstep 1: Change to log probability -> log p(x|y) = log[y_hat^y(-y_hat)^(1-y)] = y log y_hat + (1-y)log(1-y_hat)\n\nIf our prediction is correct, the our CE loss is lower\nIf our prediction is incorrect, then our CE loss is higher\n\n\n** Stochastic Gradient Descent**\nWe want to find the parameters that minimize the loss function\nTheta is going to be the parameters and biases\nTheta_hat = argmin 1/m (Sum_i L_CE(y^i, c^i; Theta))\nGradient Descent: Find the minimum of a function by finding the slope and moving towards the down slope\n\nA convex function has one minimum. There are no local minimum, so it's guranteed to find the minimum\n\nThe step size is the learning rate and it's reprensented by greek letter 'eta'\n\nThe new weight: w_t+1 = w_t - eta (derivative of the loss function with respect to the weight)\n\n*The Gradient is the difference between the true y and out estimate for x, multiplied by the input value x_j\nAverage loss (this is to minimize, for every item in the training set):\nCost(w,b) = 1/m Sum L_CE(y_hat, y)\n= -1/m Sum y log theta (wx+b) + (1-y) log (1-theta(wx+b))\n\n*Mini-batch training\nClassis stochastic descent chooses a single random example.\nWith batch training, we compute the gradient over the entire dataset. It provides a better estimate.\nMini-batch uses small batches of 512 or 1024.\n\n*Regularization (to avoid overfitting)\nOverfitting is a problem in ML.\nRegularization penalizes large weights. Lasso regularization uses L1 (sum of abs values, or manhattan distacnce), Ridge uses L2 (quadratic function).\n\n**Multinomial logistic regression\nWe use a generalization of the sigmoid function called softmax function to use with multinomial LR\nusual the final step in NN\nSoftmax: it takes an input vector and outputs a vector of values normalized into probabilities\n\n\n**Neural Networks**\nDifference between perceptrons and logistic regression is that in logistic regression we apply a sigmoid function.\nAlso, perceptron outputs 0 or 1, which makes updating the weight vector difficult\n*Activation function\nThe neuron unit ouputs a non-linear valued. Three functions are commonly used:\n1.Sigmoid\n2.Tanh\n3.Rectified Linear Unit (ReLU)\n\n*Feed Forward architecture: Multilayer network, fully-connected with no cycles\n\n*How to do the training:\nwe need:\n1) A loss function\n2) An optimization algorithm (gradient descent)\n3) A way of computing the gradient of the loss function (error propagation)\n\n**Computing the gradient\nThe derivatives of logistic regression only give correct updates for the last weight layer\nFor deeper networks, we need to look back a previous layers. We use \"Error propagation algorithm\"\n*Backward differentiation: compute how each portion of a graph contributes to the end result\n\n**Neural Language Tools**\nKey components:\n1) NN - feedforward, with input layer, hidden layers, \n2) Training data\n3) Loss function\nWe use specific activation functions that are diferentiable\n\n\n\n\n\n## Module 13 Definitions\n* **Loss function** - how wrong the predictions are for each item in the training set.\n* **Training set** - a vector of weights and a bias term\n* **Hyperparameters** - parameters selected by the designer of the algorithm that affect how the algorithm works\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Language models revisited\n",
        "Probabilistic languages models use preceding context to predict the next word.\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter9_2 = \"\"\" Recurrent Neural Networks \"\"\"\n",
        "slp_chapter9_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_10 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_11 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\"\n",
        "Language models: estimate the probability of a sentence consisting of a word sequence \n",
        "\n",
        "Neural Language Models advantages:\n",
        "1) don't need smoothing\n",
        "2) can handle longer histories\n",
        "3) can generalize over context\n",
        "4) NLM are more accurate than n-gram LMs\n",
        "Disadvantages: slower to train\n",
        "\n",
        "NLM idea:\n",
        "1) Associate each words in a text with a vectorpresentation (similarity between words)\n",
        "2) Joint probabiliyt function of a word sequence in terms of the word vectos for the words in that sequence (using a multi-layer forward neural network)\n",
        "3) learn the word vectors adn the paremeters of the function\n",
        "\n",
        "Neural LMs\n",
        "Input: k-granm of words\n",
        "output: prob dist over the next word\n",
        "\n",
        "The input vector concatenates v(w) for each of the k words and it's then fed inot the NL\n",
        "\n",
        "y_hat = P(w_i|w_1:k) = LM(w1:k) = softmax(hW^2+b^2)\n",
        "\n",
        "The output network will have an output the size of the vocabulary of the language\n",
        "\n",
        "Training\n",
        "We use k-grams from the corpus.  The firest k+1 words are the features and the last word is the target label.\n",
        "It uses cross-entropy loss\n",
        "\n",
        "Esentially, it creates a table look up of probability of a word given its history. the problem is with unseen combination of words.\n",
        "\n",
        "*Alternate idea:\n",
        "generalizing concepts.\n",
        "A Neural Probabilistic LM:\n",
        "1) Use a vector space model where the w/words as vectors\n",
        "2) A function that returns a joint probability of words in a sequence based on a sequence of these vectors\n",
        "3) Learn word reprensentation and the probability from the data\n",
        "\n",
        "**Lexical Semantic Models**\n",
        "Word meaning to encode:\n",
        "1) Synonyms\n",
        "2) Antonyms\n",
        "3) Connotations (positive or negative)\n",
        "4) Semantic Roles (who bought who?)\n",
        "5) Entailments (dog is a subset of animal)\n",
        "Taxomonic representation\n",
        "\n",
        "Lexical Semantics\n",
        "Vector Space Models: Ability to extract knowledge from a corpus\n",
        "\n",
        "Word similarity: Not exaclty synonims (like cat and dog). \n",
        "Distributional Hypothesis: Two words occur in almost the same environments if they interchangable.\n",
        "relative frequency of such environments. \n",
        "\n",
        "\n",
        "Term-Document Matrix: Rwos are terms and columns are documents. Every cell represents how often that term occured in that document\n",
        "Document vectors can be sparsed (with most values as 0), as they don't use all the words in the English language.\n",
        "If documents are similar they can be retrieved together, or can check for plagiarism \n",
        "If we compare two row vectors tells how similar two words are. \n",
        "To compare vectors, there's a problem with using euclidean distance, because it uses frequency. Therefore, we have to normalize by the vector length using L2:\n",
        "LS = ||u|| = sqr(Sum_i+1_to_n u_i^2), then divide each vector by it's vector length\n",
        "Cosine distance is the angle between two vectors. the closer the angle, the closer the words\n",
        "\n",
        "\n",
        "Term-Term Matrix (AKA Term-contex matrix)\n",
        "Rows and Columns are words. The length of the vectors is now |V| instead of the number of documents. \n",
        "Each cell represents how often the word in the column appears in a context window surounding the work in the row.\n",
        "\n",
        "Context window: count the words that are k places to the right and to the left of a word.\n",
        "Long distance dependencies: looking at near words, we may miss information/meaning.\n",
        "Instead of window of words, we use dependency patterns to look for Subj-of-verb, adj-mod, obj-of-verb\n",
        "Raw word cound is transformed to avoid very common words. (Zipf distribution):\n",
        "* TF-IDF: Term frequency(how often a word occurred in a document) * inverse document frequency (1 over the number of documents that it occurred in)\n",
        "\n",
        "\n",
        "** Word Embedding**\n",
        "Sparse vs Dense Vectors\n",
        "How to get a dense vector:\n",
        "1) Treart the target word and a neighboring context word as positive examples\n",
        "2) Randomly sample other words for negative examples\n",
        "3) Use LR to id between positive and negative cases\n",
        "4) Use the weights as the embeddings\n",
        "\n",
        "* Word2Vec:\n",
        "It learnds the weights as part of the training process. \n",
        "Negative sampling: we want words that are related to the term to have high probability, and words that are not related to have low probability\n",
        "The direction of the vectors sometime encdoe relational meaning\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"perplexity\" : \"the perplexity (PP) of a model θ on an unseen test set is the inverse probability that θ assigns to the test set, normalized by the test set length\",    \n",
        "\"recurrent neural network (RNN)\":\"any network that contains a cycle within its network connections, meaning that the value of some unit is directly, or indirectly, dependent on its own earlier outputs as an input\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "67cca3dc-8315-490c-be86-4e863a7dee09"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 14 Lecture Notes\", module14_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "## WARNING: Your definition of 'perplexity' is too long!\nWARNING: Your definition of 'recurrent neural network (RNN)' is too long!\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Language models revisited\nProbabilistic languages models use preceding context to predict the next word.\n\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Recurrent Neural Networks \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 14 Lecture Notes\n\nLanguage models: estimate the probability of a sentence consisting of a word sequence \n\nNeural Language Models advantages:\n1) don't need smoothing\n2) can handle longer histories\n3) can generalize over context\n4) NLM are more accurate than n-gram LMs\nDisadvantages: slower to train\n\nNLM idea:\n1) Associate each words in a text with a vectorpresentation (similarity between words)\n2) Joint probabiliyt function of a word sequence in terms of the word vectos for the words in that sequence (using a multi-layer forward neural network)\n3) learn the word vectors adn the paremeters of the function\n\nNeural LMs\nInput: k-granm of words\noutput: prob dist over the next word\n\nThe input vector concatenates v(w) for each of the k words and it's then fed inot the NL\n\ny_hat = P(w_i|w_1:k) = LM(w1:k) = softmax(hW^2+b^2)\n\nThe output network will have an output the size of the vocabulary of the language\n\nTraining\nWe use k-grams from the corpus.  The firest k+1 words are the features and the last word is the target label.\nIt uses cross-entropy loss\n\nEsentially, it creates a table look up of probability of a word given its history. the problem is with unseen combination of words.\n\n*Alternate idea:\ngeneralizing concepts.\nA Neural Probabilistic LM:\n1) Use a vector space model where the w/words as vectors\n2) A function that returns a joint probability of words in a sequence based on a sequence of these vectors\n3) Learn word reprensentation and the probability from the data\n\n**Lexical Semantic Models**\nWord meaning to encode:\n1) Synonyms\n2) Antonyms\n3) Connotations (positive or negative)\n4) Semantic Roles (who bought who?)\n5) Entailments (dog is a subset of animal)\nTaxomonic representation\n\nLexical Semantics\nVector Space Models: Ability to extract knowledge from a corpus\n\nWord similarity: Not exaclty synonims (like cat and dog). \nDistributional Hypothesis: Two words occur in almost the same environments if they interchangable.\nrelative frequency of such environments. \n\n\nTerm-Document Matrix: Rwos are terms and columns are documents. Every cell represents how often that term occured in that document\nDocument vectors can be sparsed (with most values as 0), as they don't use all the words in the English language.\nIf documents are similar they can be retrieved together, or can check for plagiarism \nIf we compare two row vectors tells how similar two words are. \nTo compare vectors, there's a problem with using euclidean distance, because it uses frequency. Therefore, we have to normalize by the vector length using L2:\nLS = ||u|| = sqr(Sum_i+1_to_n u_i^2), then divide each vector by it's vector length\nCosine distance is the angle between two vectors. the closer the angle, the closer the words\n\n\nTerm-Term Matrix (AKA Term-contex matrix)\nRows and Columns are words. The length of the vectors is now |V| instead of the number of documents. \nEach cell represents how often the word in the column appears in a context window surounding the work in the row.\n\nContext window: count the words that are k places to the right and to the left of a word.\nLong distance dependencies: looking at near words, we may miss information/meaning.\nInstead of window of words, we use dependency patterns to look for Subj-of-verb, adj-mod, obj-of-verb\nRaw word cound is transformed to avoid very common words. (Zipf distribution):\n* TF-IDF: Term frequency(how often a word occurred in a document) * inverse document frequency (1 over the number of documents that it occurred in)\n\n\n** Word Embedding**\nSparse vs Dense Vectors\nHow to get a dense vector:\n1) Treart the target word and a neighboring context word as positive examples\n2) Randomly sample other words for negative examples\n3) Use LR to id between positive and negative cases\n4) Use the weights as the embeddings\n\n* Word2Vec:\nIt learnds the weights as part of the training process. \nNegative sampling: we want words that are related to the term to have high probability, and words that are not related to have low probability\nThe direction of the vectors sometime encdoe relational meaning\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n## Module 14 Definitions\n* **Perplexity** - the perplexity (PP) of a model θ on an unseen test set is the inverse probability that θ assigns to the test set, normalized by the test set length\n* **Recurrent neural network (rnn)** - any network that contains a cycle within its network connections, meaning that the value of some unit is directly, or indirectly, dependent on its own earlier outputs as an input\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T3cm1WV542_a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "cd1bfaad-47af-4165-cb8b-a491aaa60deb"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Combining Beliefs and Desires under uncertainty\nUncertain state and uncertain action results. \nDecision theory deals with immediate outcomes\nUtility function captures agent's preferences\nExpected Utility is the averaged utility value of the outcomes, weighted by the probability of the outcome\nMEU: Choose the action that maximizes the agent's expected utility\nThe complex part is how to implement it\n\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n The Basis of Utility Theory\nAxions of utility theory: If an agent violates them, it'll be irrational\nOrderability: perefer either one, the other or both equally\nTransitivity: If A is prefered over B and B over C, then A is over C\nContinuity: If B is between A and C, then there is some probability that makes B equal to A or to C\nSusbtituitability: If indifferent to A and B, the if there is a C that is same as A or B, then C is also indifferent\nMonotonicity: If A is preferred over B, then the agent prefers a lottery with higher probability of A\nDecomposability: Two conscutive lotteries can be combined into a single equivalent one \n\nConsequences of the axioms:\n1)Existence of Utility Function: there exist an U(A)>U(B) if A is preferred to B and U(A)=U(B) if indifferent\n2)Expected Utility of a Lottery: It's the sum of the probability of each outcome times the utility of that outcome, therefore once we know the probabilities, the outcome is determined\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n Utility Functions\n\nThey map from lotteries to real numbers. \nIn the utility function, we use a normalized 0 - 1 values (0 bad, 1 good)\nTo find the utility of S, we ask the agent to choose between S and a standard lottery with probabily p of good and 1-p of bad, and we adjust p\nuntil the agent is indifferent to both.\n\nRegarding money, it needs to be assigned utility, because the first million has more utility than the others\n\nrational way to choose best action is to maximize expected utility.\n\nHowever, the real outcome will be significantly worse, because the highest estimate is to the right of curve with all the probabilities\n\n\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n In a MDP, we can end in any state, so the solution/policy needs to address how to act from any state\nwe compare the policies by comparing the expected utilities\nThe probabiliti distribution is determined by the initial state, the policy and the transition model.\nThe optimal policy is independent of the starting state\nThe true utility of a state is the expected sum of discounted rewards if the agent executes an optimal policy\nReward Scales: we cna make the immediate rewards higher to affect what the agent should do\nRepresenting MDP: Using tables of size O(S^2 * A). We can also use Dynamic Decision Networks which are better for real world problems\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Algorithms for MDP\n1) Value Iteration\nwe can get an optimal policy even when the utility function is inaccurate, because we only care on what's better, not in the magnitude.\nBased on the Bellman equation, one for each state. The unknowns are the utitlities of the states.\nUi+1(s)<- max Sum P(s'|s,a)[R(s,a,s\")+gamma U(s')]\nif we apply Bellman updates infintely, we find the optimal policy\n**Convergence of value iteration\nContraction properties:\n1) It has only one fixed point\n2) When applied to an argument, the value must get closer to the fixed point.\n||BUi - BU'i|| <= gamma ||Ui-U'i|| The Bellman update is a contraction by a factor of gamma on the space of utility vector.\nIf we replace with the true utilities U, where BU=U:\n||BUi - U|| <= gamma ||Ui-U'i|| , we obtain the rate of convergence\nWe can converge faser if we give gamma a high value, but we miss the long-term effects of the agent's actions\nIn practice, the policy becomes optimal long before utility converges.\nValue Iteration in practice:\na-It converges to the correct utilities\nb-the error can be bound in the utility estimates if we stop after a finite number of iterations\nc-the policy loss can be bound\n\n2) Policy Iteration\nit alternates two steps:\na-policy evaluation, calculate the utility if Pii is exectuted\nb-policy improvement, get a new MEU policy, using one step look ahead\nterminate when improvement is zero\nremoving the max argument turns the problem into a linear equation that can be solved in O(n^3)\n\n3) Linear Programming\nConsider as variables in LP the utilies U(s) of each state: U(s) >= Sum P(s'|s,a)[R(s,a,s')+gamma U(s')] for every state s and every action a\nIt runs on polynomial time, but even the simplest and mostuninformed search algorithms run in lineal time\n\n4)Online Algorithms\nFor large MDP, offline is not possible. \nUse a simplified EXPECTIMINIMAX with chance nodes. \nReal time dynamic programming\n\n\n\n\n\n\n\n\n## Module 8 Lecture Notes\n Stochastic or probabilistic environment: dtermine best action even wehn outcome is uncertain.\nMDP fully observable env, with outputs non deterministic. Policy takes a state and returns the best action from that state.\nUses a transition model that is probabilistic.\n\nMDPs are:\n* Sequential\n* for a fully observable environment\n* with stochastic transtion model\n* and has additive rewards \n\nTo find a solution to an MDP we need:\n* A set of states \n* a set of Actions\n* a transtion function T(s, a, s') The probability is called the model P(s'|s,a)\n* a rewards function R(s, a, s')\n* An initial state s0\n* Optionally, one or more terminal states\n\nIn MDP, the solution is called a policy (the best action to take in the current state) Phi(S)=a. This is because I might end up in a state that I wasn't planning to be.\nPhi* is the best policy and it depends on the reward function.\n\nTo penalize future rewards, we use discounting (Gamma) each time we descend a level in the expectimax tree.\n\nThere are only two ways to define utilities, Additive or discounted.\n\nHow to hanbdel infinite utilities:\n1) Finite horizon (Terminate after T steps) or nonstationary polycy (time left)\n2) Use discounting, with 0 < gamma < 1\n3) Absorving state: probabilistically will reach a terminal state\n\nT(s,a,s') = P(s'|s,a) * R(s,a,s')\nMain difference with expectimax tree:\n* The transtion function includes the probabily\n* the reward is given at each state in the tree as opposed to the terminal state\n\nOptimal Quantities:\n*Value utility of a state: V*(s) = Expected utility starting at s and acting optimally\n*Value utiltiy of a q-state: Q*(s,a) = expected utility having taken a from s and then acting optimally\n*Pi*(s) = optimal action from s\n\nV*(s) = maxQ*(s,a)\nQ*(s,a) = Sum T(s,a,s')[R(s,a,s') + gamma V*(s')]\nV*(s) = max sum T(s,a,s') [R(s,a,s') + gamma V*(s')]\n\nProblems:\n1) States are repeated (can use dynamic programming)\n2) Tree foes on forever (do a limited depth search)\n\n**Value Iteration**\nV0(s) = 0\nVk+1(s) <-max Sum T(s,a,s')[R(s,a,s') + gamma Vk(s')]\nO(s^2A) \nguarantee to converge to unique optimal values\n\nConvergence\n1) if the tree has a meximum depth M, then Vm has the values\n2) if the discount is < 1\n\n**Policy Methods**\nHow to evalue policies:\nInstead of calculatin the value of each state, just get the total of the policy.\n1) Do what Pi says. Calculate the utility\nV*(s) = sum T(s,PI(s),s') [R(s,PI(s),s') + gamma V*(s')] (There's no max!!!)\nO(s^2)\n\n**Policy Extraction**\nGiven values, get the policy.\nDo a mini-expectimax (one step) \nPi*(s) = arg max Sum T(s,a,s')[R(s,a,s') + gamma V*(s')]\nWith Q values:\nPi*(s) = arg max Q*(s,a)\n\n**Policy Iteration**\nProblems with Value Iteration:\n1) it's slow O(s^2A)\n2) the \"max\" in each state not changes frequently\n3) We can converge long before the last values\n\nAlt approach -> Policy Iteration\n1) Calculate all values in any policy until it converges\n2) Improve the policy with one step look ahead\n3) Repeat until converges\npolicy is implicitly updated\n\nThis is optimal\n\nCompared to value iteration, policy iteration often converges in fewer iterations but it's more costly (O(s^3) compared to O(A*S^2))\n\n**Comparison Value Iteration vs Policy Iteration**\nBoth find optimal value, use MDP.\nIn Value Iteration every iteration updates values and we don't explicitly keep track of the policy\nIn Policy Iteration several passes that update the policy, and it's updated\nBoth use dynamic programming\n\nTo compute optimal values: Value Iteration or Policy Iteration\nvalues for one policy: Policy Extraction\nFrom values to Policy: Policy Extraction (one step lookahead)\nAll are variants for the Bellman update/expectimax\nMain difference: if we use a fixed policy or if we go over all actions to find the best one\n\n**Maximum Expected Utility (MEU)**\nthe agent should max the expected return (not worst case planning)\nFor average-case expectimax, the magnitudes need to be meaningful, because they affect the choices.\n\n**Rational Preferences**\n-transitivity\n-Orderability\n-Susbtitutability\n-Continuity\n-Monotonicity\nTo be rational, choose the action that maximizes expected utility\n**Utility Scales**\n-Micromort\n-QALYs: Quality adjusted life years\nInvariant under positive linear transformation\n\n\n\n\n\n\n\n## Module 8 Definitions\n* **Stochastic transition modeo** - p(s'|s,a) probability of ending in s' if we are in s and select action a\n* **Markovian** - The probability of arriving at s' depends only on s and a, not on the history\n* **Utility/reward function** - it's instantenous, can be positive or negative, and are additive R(s, a, s')\n* **Utility function** - Takes a sequence of states and actions and adds the rewards\n* **Expected utility** - \n* **Stationary preferences** - If a prefer A over B, if I then add x to A and to B, I still prefer A+x to B+x\n* **Lotteries** - outcomes with probabilities attached\n* **Value function/ordinal utility function** - an agent needs only a preference ranking order\n* **Preference elicitation** - process to find out the human's utility function\n* **Value of statistical life** - 10MM in the US\n* **Monotonic preference for money** - prefer more money to less\n* **Certainty equivalent** - value that an agent will accept in lieu of of a lottery\n* **Insurance premium** - difference between expected monetary value and certaninty equivalent\n* **Risk neutral agent** - one that has a linear curve\n* **Normative theory** - how it should act\n* **Descriptive theory** -  how it acts\n* **Transition model** - outcome of each action in each state\n* **Optimal policy** - one that yields the highest expected utility. There could be multiple optimal policies\n* **Nonstationary** - a policy that depends on the time. It depends on the time left\n* **Stationary** - it only depends on the current state\n* **Stationarity of preferences** - if one prefers one future tomorrow, it still should prefer it today \n* **Proper policy** - one that is guarantee to reach a terminal state\n* **Q function/action-utility function** - Expected utility of taking a given action in a given state\n* **Contraction** - function of one argument that, when applied to two difference inputs, produces two output values that are 'closer together'\n* **Max norm** - measures the length of a vector by the absolute value of it's biggest component\n* **Modified policy iteration** - Perform a some number of simplified velue iteration steps to get a good enough approximation of the utilities.\n* **Asynchronous policy iteration** - intead of updating all states, select only a subset\n* **Etc.** - your definition here\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Bandits Problem\n* Each arm Mi is a Markov Reward Process (an MDP with only one possible action), with Si states, Pi(s'|s,ai) transition model and reward Ri(s,a,s')\n* bandit problem is an MDP\nkey property: the arms are independent \n\n## AIMA Chapter 22.1 Learning from Rewards\n Learning from Rewards\n \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n Reinforcement Learning\n\nIn MDP, the agent gets the rewards and the probability distribution. In RL, the agent needs to learn them.\nExploration: Try unknowns to get information\nExploitation: use what was learned\nRegret: mistakes\nSampling: due to chance, due things several times\nDifficulty: harder than MDP\n\nFeedbacks comes as reward, an the agent learns to maximize it. All based on obseved samples of outcomes.\n\n*A set of states s\n*A set of actions per state A\n*A model T (s,a,s')\n*A reward function R(s,a,s')\nLooking for a policy Pi(s)\n\nT and R are unknown\n\nMDP is offline learning - RL is online learning\n\nModel Based Learning: (probability model of the transition function) learn an estimate of the transition model and \nlearn the rewards and apply them to the model using MDP (like value iteration)\n\n**Model-Free learning - Passive RL**\nIntead of creating a model, we take an action and compare the actual output to the expected one, and if they're different\nwe update the expected one. We don't keep track of the reward or transition model.\nwe get a policy and need to lear the value of each state, without T or R\nThis is called direct utility evaluation, we take the average reward of reaching the end state from every state using sample transitions\nPositives:\n1) Easy to understand\n2) Doesn't require knowledge of T or R\n3) Eventually gets the correct average values\nNegatives:\n1) Doesn't consider state connections\n2) Each state must be learned separately\n3) it takes long to learn\n\n**Use Policy Evaluation with samples**\nDo policy iteration and do actions several times and then average the samples to get the estimate value.\nThis doesn't work because we may not end in the same state multiple times\ninstead do:\n**Temporal Difference Learning (model-free)**\nThe idea is we update the estimate every single time we go through S, keeping a running average\nSample of  V(s):  sample = R(s,Pi(s),s') + gamma V^pi(s')\nUpdated to V(s): V^pi(s)<- (1 - alpha) V^pi(s) + (alpha) sample\nAlpha controls the weight of the current outcome\nUse of an exponential moving average makes recent samples more important\nHowever, we can't turn the values into a new policy\n\n**Active Reinforcement Learning**\nInstead of getting a policy, the agent decides the actions\n\nValue Iteration: start at V0, then calculate the best action going up a level. Use Dynamic Programming\n**Q-Learning: sample-based Q-value iteration\n  To learn Q(s,a) as we go:\n  1-receive a sample (s,a,s',r)\n  2-take old estimate Q(s,a)\n  3-new sample estimate: sample = R(s,a,s') + gamma max Q(s',a')\n  4-incorporate new estimate into a running average\n\n  It converges to optimal policy - even if acting suboptimally (off-policy learning)\n  Required: enough exploration, make the learning rate small (<1) to keep knowldge about past\n\n**Exploration vs Exploitation**\nHow to explore:\n1-random actions (epsilon greedy policy) random number explore, otherwise use current policy\n  issue: we may be repeating states that are bad. Idea, reduce epsilon over time\n2-exploration functions\n  fixed amount\n  explore areas that we're not certain are bad (be optimistic)\n  f(u,n) = u + k/n\n  u: current estimate of the utility\n  n: number of times the state was visited\n  k: constant bonus, that decreases over time\n  Modified Q function: Q(s,a) <- R(s,a,s') + gamma max f(Q(s',a'),N(s',a')) (we get bonus for unseen states and for states that lead to unknown states)\n\nTo decide between epsilon learning or Exploration function, we can use Regret. Learn optimal policy as quickly as possible.\n\n**Approximate Q-Learning**\nThere are so many states, we can't visit them all. we'll try to generalize to similar situations\nIn practice, several features are taken and evalued to value a state\nEval(s)=w1f1(s)+w2f2(s)+...+wnfn(s) This is a linear value function\nQ(s,a) = w1f1(s,a)+w2f2(s,a)+...+wnfn(s,a)\nGood: we can summarize state in few features\nBad: some features may be shared, but with very different value\nthe learning algorithm will adjust the weights\ntake a transition function: transtion = (s,a,r,s')\nand calculate the difference with what I was expecting:\ndifference = [r+gamma max Q(s',a')]-Q(s,a) This is an error term\ntake the difference and update each weight base on the differnece and wehter the feature was active:\nwi <- w1 + alpha[difference]fi(s,a)\n\nBellman equation: $$U(s)=R(s)+\\gamma \\max _{a} \\sum_{s^{\\prime}} P\\left(s^{\\prime} \\mid s, a\right) U\\left(s^{\\prime}\right)$$\n\n\n \n\n## Module 9 Definitions\n* **Regret** - measure of the total cost of the mistakes made to learn a policy\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Acting under uncertainty\nAn agent needs to handle uncertainty in real world situations. \nSo far, the agent worked wtih belief states: a set of all possible sorld states and a contigency plan for them.\ndrawbacks:\n1) Large belief-state as it has to consider every possible explanation for its sensor observations\n2) Plans to handle every eventuality become too large\n3) some plans don't guaratee the goal, so we need to be able to compare plans\n \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n The main problem with estimating the probabilities is taht new sentences are created all the time.\nTo compute the probability of an entire sequence, we'll use the cahin rule:\nP(X1...Xn) = P(X1)P(X2|X1)P(X3|X1:2)...P(Xn|X1:n−1)\n\n \n\n## SLP Chapter 3.2 Evaluating Language Models\n The branching factor of a language is the number of possi-\nble next words that can follow any word \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Probability\n**Review of Probabilities**\nA probability model associates a porbability P(w) to each possible world w\n**Axioms of probability:\n1) 0<=P(w)<=1 for every w\n2) Sum P(w) = 1\n\nUnobserved r.v. have a probability distribution\n\n**Joint Distributions:\nSpecifies a real number for each assignment or outcome for a joint assignment\nP(X1=x1, X2=x2,...,Xn=xn)\n\nThe size of a distribution of n variables with domain size d is d^n\n\n**Probabilitic Models\nIt's a joint distribution over a set of random variables.\ncontains:\n-Domain for r.v.\n-Outcomes\n-Joint distributions\n-Normalized (sum=1)\n\n**Event P(E) = Sum P(x1...xn)\n\n**Marginal Distributions\nSub-tables where some variables were eliminated (they're sum up)\nP(t) = Sum_s P(t,s)\n\n**Conditional Probabilities\nP(a|b) = P(a,b) / P(b)\nIt gives the probability of something we haven't seen\nin a Venn Diagram, it's the intersection\n\n\n**Conditional Distributions\nProbability distributions over some variables given fixed values of others\n\n\n**Probabilistic Inference**\nCompute a desired probability from other known probabilities\nNew evidence can change the probability\n** Inference by Enumeration\n- SPlit r.v into three sets:\n1-Evidence variables: things we know\n2-Query variables: things we care about and want to know about\n3-Hidden variables: things that occure but we're not observing and we don't care \nWe care about P(Q|E)\n\nSteps:\n1-take entries that are consistent with evidence\n2-sum out hidden variables\n3-Normalize it to get to add up to 1: Z = Sum_q P(Q, e_1, e_2...)\n\n**The Product Rule\nwe get marginal or conditional probabilities and we want to construct the joint probability table\nMultiply the marginal and the conditional.\nP(y)P(x|y) = P(x,y)\n\n**Chain Rule\nP(x1, x2, x3) = P(x1)P(x2|x1)P(xe|x1,x2)\nP(x1,x2,...,xn) = MUL_i P(xi|x1...xi-1)\n\n**Bayes' Rule\nP(x|y) = P(y|x)/P(y) * P(x)\n\n**Probabilistic Language Model**\nLanguage Model is a moel that computes the probabilitiy of the next word in a sentence\nOne problem is that a sentence might not exist in the corpus.\nTo address this, we use Markov Assumption, limiting the amount of history\nAn unigram will suggest the most used work in the corpus\nA bigram will look for a word given another word\n\n\n\n\n\n\n\n\n\n\n\n\n\n## Module 10 Definitions\n* **Sample space** - mutually exclusive and exhaustive possible worlds\n* **Random variable** - aspect of the world that we're interested in. Use capital letters\n* **Markov assumption** - the probability of a word depends only on the previous word\n* **Extrinsic evaluation** - embed a lnaguage model in an application and measure how much the application improves\n* **Perplexity** -  of a language model on a test set is the inverse probability of the test set, normalized by the number of words.\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Bayes' Net\n\nGraph where nodes are random variables (w/domains - they can be observed or unobserved) and the arcs are the relationships (they encode conditional independence).\n\nConditional independence are mediated by a third observed variable\n\nX and Y are independent if P(x,y) = P(x)P(y)\nX and Y are conditionally independent given Z if P(x,y|z) = P(x|z)P(y|z) or also as P(x|z,y) = P(x|z)\u0013\nP(Traffic, Rain, Umbrella) = P(Rain)P(Traffic|Rain)P(Umbrella|Rain,Traffic) = P(Rain)P(Traffic|Rain)P(Umbrella|Rain)\n\u0013Problems:\n  *The joint probability table gets too big with many variables\n  *Hard to learn empirivally when there are too many variables\n\nBaye's Net Semantics\n1-One node per variable\n2-Directed, acyclic graph\n3-A conditional distribution for each node (with any probabiliy ditribution)\nIt implicitly encodes joint distributions as a product of local conditional distributions\n\nNot every BN represent every joint distribution\n\nBN need not be causal\nTopology encodes conditional independence\n\nSize of Bayes'Net:\nFor N Boolean Variables: 2^N\nFor an N-node with up to k parents: O(N*2^(k+1))\n\nBN is space saving\n\nImportant for modelling: understand assumptions made when choosin Bayes net graph\n\n**Algorithm: D-Separation**\nTo know if things are independent of each other\n*A - Causal Chains: X causes Y causes Z\nP(x,y,z) = P(x)P(y|x)P(z|y)\nIs X indenpendet of Z? A single examples is sufficient to shows that is not guaranteed\nIs X independet of Z given Y? P(z|x,y) = P(x,y,z)/P(x,y)\nProof:\nP(z|x) or P(z|y) = P(x)P(y|x)P(z|y)/(P(x)P(y|x)) = P(z|y)\nEvidence along the chain blocks the influence (that is, if I know that Y is active, knowing that Z is active doesn't affect my knowledge of Z)\n\n*B - Common Cause: Y cause X and Y causes Z\nP(x,y,z) = P(y)P(x|y)P(z|y)\nObserving the cause blocks influence between effects.\n\n*C - Commmon Effect: X causes Z and Y causes Z\nAre X and Y independent of each other?\nThere are two potential causes.\nObserving an effect activates influence between the possible causes.\n\n**Reachability and Path**\nHow to know if two variables in a BN are indpendent? break the graph and use the three cases\nActive/Inactive Paths\nAre X and Y conditionally independent given {Z}? True if X and Y are d-separate by Z. If no active paths, then there's independence\n\nActive Path: Where all the triples are active:\nCausal chain: A->B->C, where B is unobserved\nCommon Cause: A<-B->C, where B is unobserved\nCommon Effect: A->B<-C, where B or one of its descendant is observed\n\n**D-Separation Algorithm**\nwe want to know if Xi is guranteed conditionally independent of Xj, given observed variables Xk:\n1) Look for undirected paths between Xi and Xj. If one or more is active, the independence is not guaranteed\n2) If all paths are inactive, the independence is guaranteed\n\n**Inference**\nPosterior probability: P(Q|E1=e1,...,Ek=ek)\nMost Likely explanation: argmax P(Q=q|E1=e1...)\nThe problem with this approach is that it's very expenseive, as it's exponential to the \nnumber of hidden variables. If they're binary, that's 2^(number of hidden variables)\nWe have to join the whole joint distribution before summing up the hidden variables.\n\n\n\n\n\n\n\n\n\n\n\n \n\n## Module 11 Definitions\n* **Bayes' net** - A technique for describing complex joint distribution using simple, local distributions\u0013\n* **Observed nodes** - We have evidence for\n* **Active nodes** - They can still influence\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Naive Bayes Classifiers\n It returns the class which has the maximum posterior probability given a document.\n c_hat = argmax P(c|d) = argmax P(d|c)P(c)\n P(d|c) is the likelihood of the document and P(c) is the prior probability of the class.\n NB makes two simplifying assumptions:\n 1) Assume the order of the fetarues iw not important\n 2) The conditional probabilities of the featrues are independent, and can then be mulitiplied\n Calculations are done in log space to avoid undeflow and increase speed.\n \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Training the Naive Bayes Classfier\nFor P(c), check how many docs are in each category: P_hat(c) = N_c/N_doc\nP_hat(w_i|c) = count(w_i,c)/Sum_w count(w,v)\nThe main problem is because the features are multiplied, if one of them doesn't appear in the training data, the class will get zero probability\nA possible solution is to use Laplace smoothing. the usion of all the word types needs to be accross categories.\nUnknow words: words that occur in the test data but not in our vocabulary? they should be excluded\nstop words: words like The, a, etc\n\n\n\n\n## SLP Chapter 4.3 Worked Example\n Worked example \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Optimizing for Sentiment Analysis\nSometimes, the presence of a word is more important than its frequency, so it's clipped at one for each document (Binary naive Bayes)\nTo handle negation, preface with NOT_ every word util the next punctuation mark.\n\n \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Naive Bayes for other text classification tasks\nThe classifier doens't need to use all the words in the training data as features.\nIt can also use features other than words\nFor language identification, useing character n-grams is effective ('zw', 'nya', 'ie z', 'thei')\n\n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Naive Bayes as a Language Model\nA NB model can be viewed as a set of class-specific unigram language models, where the model for each class instatiates a unigram laguage model, as each word is assigned a probability, then also each sentence has a probability assigned.\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\nEvaluation: Preceision, Recall, F-measure\naccuracy is teh percentage of observations labeled correctly. Not used much for text classification, becaus it doesn't work well when the classes are unbalanced. It's not a good metric when we want to find something that is rare.\nPrecision is use instead, which is the percentage of items detected that are positive.\nRecall measures the % of items present in the input that were identified.\nF-Measure: F_beta = (Beta^2+1)PR/(Beta^2P+R) Beta weights >1 favors recall and  beta < 1 favor precision. (F1= 2PR/(P+R))\nThe Harmonic mean is closer to the minimum than the arithmetic mean.\n\n**Evaluating with more than two classes\nNB is already a multi-class classification algorithm\nTo measure precision and recall, we need to use microaveraging, where we compute the performance for each class, and then average over classes.\nappropriate when all classes are equally important.\n\n \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Test sets and Cross-validation\nThe problem with splitting the data in training, dev and test set, is that we may run out of data. We can use cross-validation to use all the data.\nWe take a number k, and split the data in k parts. use one k fold as test set, and traing on the other k-1 folds. repeat 10 times and average for 10-fold cross-validation \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Statistical Significance Testing\nHow to compare the performance of two systems (statistical hypothesis testing)\neffect size: M(A,x)-M(B,x) If A or B is better given x.\nset up two hypothesis:\nH_0 : gamma(x) <= 0 (null hypothesis)\nH_1 : gamme(x) > 0\nTo check if we can support H_1, we use p-value: P(gamma(X)>=gamma(x)|H_0 is true)\nA very small p-value means we can reject H_0\nTo compute the probability, we don't use parametric tests, because they make assumptions about the ditribution that don't hold for NLP\nUse instad non=paramtric tests based on sampling (tests like 'approximate randomization' and 'boostrap test')\n**The Paired Boostrap Test\ncan apply to any metric. draw large number of smaller samples with replacement from original large sample. \n\u0011\u0011\u0011\u0011 \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Avoiding Harms in Classification\nrepresentational harm: harms caused by a system that demeans a social group\nML can amplify the biases in their training data casued by labelers, resources used or model architecture. \n\n \n\n## Module 12 Lecture Notes\n Machine Learning\nHow to learn from examples/data\n**Classification Task**\nFor problems where we don't know the probabilities.\n1) Learning parameters (probabilities)\n2) Learnign structure (Bayes Net graphs)\n3) Learnign hidden concepts (clustering)\n\nInput is represented as a feature vector\nGiven a collection of training data, we want to learn a function that predicts a class\n\nModel base approach: build a mode where the class and the reatures are r.v.\nthen, instantiate all the observed features and query the model (what class?)\n\n**Naive Bayes**\nNaive: Assume that all features are independent of each other, and only depend on the label\nP(Y,F1...Fn) is proportional to the P(Y) Product P(Fi|Y)\n|Y|*|F|^n values exponential to the number of features, but with the independent assumption we have n*|F|*|Y| (now is linear to the number of features)\nThe model is simplistic because it assumes a lot of independence, but often works\n\nObjective: Compute pos dist ove label variable Y\n1) get joint probability of each each label of the label and evidence\nP(y1) MUL P(Fi|y1)\nP(y2) MUL P(Fi|y2)\nP(yk) MUL P(Fi|yk)\n2) then we sum the probabilities to get the probablity of the evidence\nand then 3)divide each one to normalize them (divide each row by the sum)\n\nWe need to estimate the local conditional probabilities. \n\n**Basic Concpts on ML**\nNeed to train the model. Partition data in three sets:\n1)Training: to estimate parameters. Use data labeled\n2)Heldout/Development: To optimize hyperparameters\n3)Test: To evaluate the model\n\nOverfitting: Relative frequency parameters will overfit the training data. Avoid given 0 probability to unseen eevents. Therefore, we smooth or regularize the data\n\nParameter estimation: Elicitation or Empirically\nP_ML(x) = count(x)/total samples (P_ML is the probability of maximum likelihood)\nTo know what estimate is better, we calculate the likelihood of data: L(x,theta) = MUL P_theta(x_i)\n\n**Maximum likelihood\nPick the parameters that maximize the likelihood of data given the parameters\nthe problem is unseen events, so we smooth the data\n\n**Laplace Smoothing\nPretend every outcomes was seen once more than actually happened, or we can add an arbitrary number\nperforms poorly with |Y| of |X| is very large for P(X|Y)\n\n**Linear Interpolation\nUse the current observed times alpha + prior estimate times 1-alpha\n\n**Tuning\nHow to pick best parametrization\nparamters came from data traininf set\nhyperparameters come from another data set (to faciliate generalization)\nmodel accuracy on trainign data goes down as we increase k as the model with training was the best by definition\n\nAnother option is to add new features\n\nBaseline: if I use an obvious label, what accuracy do I get?\n\n\n*** Pereceptrons *** \nMake decisions by weighting evidence\n\nSum_j w_jc_j = w * x\n\n\noutput: \n0 if w*x + b <= 0\n1 if w*x + b > 0\nb is the bias\n\n**Learning weights from examples\nInputs are features representing something we want to classify. the weight provides a different classifier\n\nIn binary decision rule, the bias allows to shift the boudary\n\nTo learn the weights, we try several scenarios, if the output is correct, no change, if it's wrong, adjust the weight vector\nwe start with all weights = 0 , and then classify the training instance. If it's wrong, we adjust the weight vector\n\nIn multiclass scenarios:\n1. a w_y vector for each class\n2. an activation of a class y:  w_y * f(x)\n3. Prediction with the highest score wins: y = argmax w_y * f(x)\nUpdate the incorrect output and increase the correct output\n\n** Improvements on Perceptrons and SVM**\nIf the data is separable, we're guaranteed perceptrons will converge on parameters\nIf the data is not separable, we can bound the mistakes:\nmistakes < k/gamma^2 where gamma is the margin (degree of separability) and k is a constant\n\nProblems with Perecprons:\n1) Noise: If data is not separable, weights might thrash (go back and forth). It can be reduced by averaging the weights (averaged perceptron)\n2) Mediocre generalization: Barely separates\n3) Overtraining: type of overfitting\n\nFixing the Perceptron:\n*MIRA: Margin Infused Relaxed Algorithm\nUpdate the weights by as little as possible. We scale the adjustment by Thau w_y = w'_y - Thau f(x)\n*Maximum Step Size: \nLimit Thau to no more than a constant\nWorks very good with noisy data that is not easily to separate\n*Linear Separators (Support Vector Machine)\nTries to create two vectors that maximize the boudary between the two classes\n*Non-Linear Separators\nMap data to higher dimensions and it becomes separable\n\n \n\n## Module 12 Definitions\n* **Prior probability** - For a node in a Bayes Net without a parent\n* **Parameters of the model** - Prior probabilities over labels and the conditional probabilities for each freature\n* **Overfitting** - The model fits too well the training data but does not generalizes well\n* **Elicitation** - Ask a human to estimate a parameter\n* **Empirically** - Use training data\n* **Supervised learning** - we have a data set of input observations, each associates with some correct output(a supervision signal)\n* **Generative classrifier** - buidl amodel of how a class could generate some input data\n* **Discriminative classifier** - like logistic regression, learn waht reatures from the input are most useful to discriminate between the different possible classes\n* **Bag-of-word** - an unordered set of words with their positions ignored, keeping only their frequency in the document\n* **Sentiment lexicons** - lists of words that are preannotated with positice or negative sentiment\n* **Gold labels** - labels assigned by humans\n* **Confusion matrix** - A table for visualizing how an algorithm performs with respect to the human gold label.\n* **Effect size** - Difference in F1 between two clasfiers\n* **Toxicity detection** - the task of detecting hate speech, abuse, harassment or other kinds of toxic langauge.\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Classification: the sigmoid\nThe goal of binary LR is to train a classifier to decide between two classes.\nThe sigmoid function takes a real-valued number and maps it to a range 0 - 1. It squashes ourlier values toward 0 or 1, and it's differentiable. \nNaive Bayes has a stron condtional independence assumption. If we incldue the same feature twice, NB will treat them as different and will mulitiply them.\nLogistic Regression is more robust and will identify if they're prefectly correlated. \nWhen there are many correlated featues, LR wil be more accurate than NB\nNB works really well with small datasets, it's easy to implement and fast to train \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Learning on Logistic Regression\nWe need two components: \n1) How close y_hat is to y (loss function)\n2) Optimization algorithm (stochastic gradient descent)\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n The cross-entorpy loss function\nConditional maximum likelihood estimation: choose the parameters w,b that maximize the log probability of the true y labels in the training data/\n \n\n## SLP Chapter 5.4 Gradient Descent\n Gradient Descent\nThe goal of gradeint descent is to find the optimal weights that minimize the loss function defined for the model.\nGradient descent algorithm finds the gradient of the loss function at the current point and moves in the opposite direction.\nThe change we make in the parameters is the learning rate times the gradient \n*Mini Batch training\nStochastic is called like that because it chooses a single randomg example every time\n\n## SLP Chapter 5.5 Regularization\n Regularization\nA setting of the weights that matches the training data perfectly (but uses many weights with high values to do so) will be penalized more than a setting that matches the data a little less well, but does so using smaller weights. \nL2 regularization uses the euclidena distance\nL1 regularization uses the Manhattan distance\nL2 prefers weight vectors with many small weights\nL1 prefers sparse solutions with some larger weights but many more weights set to zero. L! leads to sparser wieght vectors, fewer features\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n Multinomial logistic regression\nThe softmax function is a generalization of the sigmoid function, where each class is mapped with a probability between 0 and 1, all summing up to 1.\nWith softmax if one of the inputs is larger than the others, it will tendo to push its probabiliy toward 1 and suppress the probability of the smaller inputs\n\n**Features in Multinomial Logistic Regresssion**\nWe'll need separate weight vectors (and biases) for each of the K classes. \n\n**Learning in Mulitnomial Logistic Regression**\nThe loss function for multinomial logistic regresion generalizes the loss function for binary logistic regresson from 2 to K classes.\n\n\n \n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n \n** Logistic Regression **\nA supervised ML problem. Components:\n1) Feature representation of the input (vector of features)\n2) A classification function, that estimates y_hat via P(y|x)\n3) An objective function or loss function (tells how good or bad the function is, usually to minimize)\n4) An algorithm to optimize the objective function\n\nA difference between perceptrons and sentiment logistic regression is the range of values the classifier can output.\nIn perceptron it can be anything.\n\nIn order to cnovert to probability, we use the sigmoid function.\n\nP(y=1) = sigmoid funtion (w*x+b) = 1 / (1 + e ^-(wx+b))\nP(y=0) = 1 - sigmoid(wx+b)\n\n*Decision Boundary*\ny_hat = 1 if P(y=1|x) > 5 ; 0 otherwise\n\n*Learning in logistic regression*\n1) We need a loss function that tells us how far are we from the gold output (use cross-entropy loss)\n2) An algorithm to optimize the objective function (minimize the loss function)\n\n*Loss Functions*\nL(y_hat, y) = how much y_hat differs from true y\nOne option is the mean squared error, but it's hard to optimize because it's not convex\nFor probabilistic classification it's better to use \"Conditional Maximum Likelihood Estimation\":\nchoose parameters that maximize the (log) probability of the true labels in the training data.\nThis is the log likelihood loss (called cross entropy loss)\n\nWe want to maximize the probability of the correct label:\nP(y|x) = y_hat ^ y(1-y_hat) ^(1-y)\ny can be either 1 or zero\nstep 1: Change to log probability -> log p(x|y) = log[y_hat^y(-y_hat)^(1-y)] = y log y_hat + (1-y)log(1-y_hat)\n\nIf our prediction is correct, the our CE loss is lower\nIf our prediction is incorrect, then our CE loss is higher\n\n\n** Stochastic Gradient Descent**\nWe want to find the parameters that minimize the loss function\nTheta is going to be the parameters and biases\nTheta_hat = argmin 1/m (Sum_i L_CE(y^i, c^i; Theta))\nGradient Descent: Find the minimum of a function by finding the slope and moving towards the down slope\n\nA convex function has one minimum. There are no local minimum, so it's guranteed to find the minimum\n\nThe step size is the learning rate and it's reprensented by greek letter 'eta'\n\nThe new weight: w_t+1 = w_t - eta (derivative of the loss function with respect to the weight)\n\n*The Gradient is the difference between the true y and out estimate for x, multiplied by the input value x_j\nAverage loss (this is to minimize, for every item in the training set):\nCost(w,b) = 1/m Sum L_CE(y_hat, y)\n= -1/m Sum y log theta (wx+b) + (1-y) log (1-theta(wx+b))\n\n*Mini-batch training\nClassis stochastic descent chooses a single random example.\nWith batch training, we compute the gradient over the entire dataset. It provides a better estimate.\nMini-batch uses small batches of 512 or 1024.\n\n*Regularization (to avoid overfitting)\nOverfitting is a problem in ML.\nRegularization penalizes large weights. Lasso regularization uses L1 (sum of abs values, or manhattan distacnce), Ridge uses L2 (quadratic function).\n\n**Multinomial logistic regression\nWe use a generalization of the sigmoid function called softmax function to use with multinomial LR\nusual the final step in NN\nSoftmax: it takes an input vector and outputs a vector of values normalized into probabilities\n\n\n**Neural Networks**\nDifference between perceptrons and logistic regression is that in logistic regression we apply a sigmoid function.\nAlso, perceptron outputs 0 or 1, which makes updating the weight vector difficult\n*Activation function\nThe neuron unit ouputs a non-linear valued. Three functions are commonly used:\n1.Sigmoid\n2.Tanh\n3.Rectified Linear Unit (ReLU)\n\n*Feed Forward architecture: Multilayer network, fully-connected with no cycles\n\n*How to do the training:\nwe need:\n1) A loss function\n2) An optimization algorithm (gradient descent)\n3) A way of computing the gradient of the loss function (error propagation)\n\n**Computing the gradient\nThe derivatives of logistic regression only give correct updates for the last weight layer\nFor deeper networks, we need to look back a previous layers. We use \"Error propagation algorithm\"\n*Backward differentiation: compute how each portion of a graph contributes to the end result\n\n**Neural Language Tools**\nKey components:\n1) NN - feedforward, with input layer, hidden layers, \n2) Training data\n3) Loss function\nWe use specific activation functions that are diferentiable\n\n\n\n\n\n## Module 13 Definitions\n* **Loss function** - how wrong the predictions are for each item in the training set.\n* **Training set** - a vector of weights and a bias term\n* **Hyperparameters** - parameters selected by the designer of the algorithm that affect how the algorithm works\n\n\n## WARNING: Your definition of 'perplexity' is too long!\nWARNING: Your definition of 'recurrent neural network (RNN)' is too long!\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Language models revisited\nProbabilistic languages models use preceding context to predict the next word.\n\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Recurrent Neural Networks \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 14 Lecture Notes\n\nLanguage models: estimate the probability of a sentence consisting of a word sequence \n\nNeural Language Models advantages:\n1) don't need smoothing\n2) can handle longer histories\n3) can generalize over context\n4) NLM are more accurate than n-gram LMs\nDisadvantages: slower to train\n\nNLM idea:\n1) Associate each words in a text with a vectorpresentation (similarity between words)\n2) Joint probabiliyt function of a word sequence in terms of the word vectos for the words in that sequence (using a multi-layer forward neural network)\n3) learn the word vectors adn the paremeters of the function\n\nNeural LMs\nInput: k-granm of words\noutput: prob dist over the next word\n\nThe input vector concatenates v(w) for each of the k words and it's then fed inot the NL\n\ny_hat = P(w_i|w_1:k) = LM(w1:k) = softmax(hW^2+b^2)\n\nThe output network will have an output the size of the vocabulary of the language\n\nTraining\nWe use k-grams from the corpus.  The firest k+1 words are the features and the last word is the target label.\nIt uses cross-entropy loss\n\nEsentially, it creates a table look up of probability of a word given its history. the problem is with unseen combination of words.\n\n*Alternate idea:\ngeneralizing concepts.\nA Neural Probabilistic LM:\n1) Use a vector space model where the w/words as vectors\n2) A function that returns a joint probability of words in a sequence based on a sequence of these vectors\n3) Learn word reprensentation and the probability from the data\n\n**Lexical Semantic Models**\nWord meaning to encode:\n1) Synonyms\n2) Antonyms\n3) Connotations (positive or negative)\n4) Semantic Roles (who bought who?)\n5) Entailments (dog is a subset of animal)\nTaxomonic representation\n\nLexical Semantics\nVector Space Models: Ability to extract knowledge from a corpus\n\nWord similarity: Not exaclty synonims (like cat and dog). \nDistributional Hypothesis: Two words occur in almost the same environments if they interchangable.\nrelative frequency of such environments. \n\n\nTerm-Document Matrix: Rwos are terms and columns are documents. Every cell represents how often that term occured in that document\nDocument vectors can be sparsed (with most values as 0), as they don't use all the words in the English language.\nIf documents are similar they can be retrieved together, or can check for plagiarism \nIf we compare two row vectors tells how similar two words are. \nTo compare vectors, there's a problem with using euclidean distance, because it uses frequency. Therefore, we have to normalize by the vector length using L2:\nLS = ||u|| = sqr(Sum_i+1_to_n u_i^2), then divide each vector by it's vector length\nCosine distance is the angle between two vectors. the closer the angle, the closer the words\n\n\nTerm-Term Matrix (AKA Term-contex matrix)\nRows and Columns are words. The length of the vectors is now |V| instead of the number of documents. \nEach cell represents how often the word in the column appears in a context window surounding the work in the row.\n\nContext window: count the words that are k places to the right and to the left of a word.\nLong distance dependencies: looking at near words, we may miss information/meaning.\nInstead of window of words, we use dependency patterns to look for Subj-of-verb, adj-mod, obj-of-verb\nRaw word cound is transformed to avoid very common words. (Zipf distribution):\n* TF-IDF: Term frequency(how often a word occurred in a document) * inverse document frequency (1 over the number of documents that it occurred in)\n\n\n** Word Embedding**\nSparse vs Dense Vectors\nHow to get a dense vector:\n1) Treart the target word and a neighboring context word as positive examples\n2) Randomly sample other words for negative examples\n3) Use LR to id between positive and negative cases\n4) Use the weights as the embeddings\n\n* Word2Vec:\nIt learnds the weights as part of the training process. \nNegative sampling: we want words that are related to the term to have high probability, and words that are not related to have low probability\nThe direction of the vectors sometime encdoe relational meaning\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n## Module 14 Definitions\n* **Perplexity** - the perplexity (PP) of a model θ on an unseen test set is the inverse probability that θ assigns to the test set, normalized by the test set length\n* **Recurrent neural network (rnn)** - any network that contains a cycle within its network connections, meaning that the value of some unit is directly, or indirectly, dependent on its own earlier outputs as an input\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}