{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "“Template_for CIS521_midterm_2_notes.ipynb”的副本",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1444e08-581f-48c5-9884-d8ec57da44ae"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "673d909e-6019-4f98-8e78-b5d45a92b3d6"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = False"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\"\n",
        "The MEU principle formalizes the general notion that an intelligent agent should “do the\n",
        "right thing,” but does not operationalize that advice.\\\\\n",
        " If an agent acts so as to maximize a utility function that correctly reflects the performance measure, then\n",
        "the agent will achieve the highest possible performance score (averaged over all the possible\n",
        "environments).\\\\\n",
        " The performance measure gives a score for a history—a sequence of states.\\\\\n",
        "\n",
        "\n",
        "\"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\"  any reasonable preference relation to obey:\\\\\n",
        "ORDERABILITY:Given any two lotteries, a rational agent must either prefer one or\n",
        "else rate them as equally preferable.\\\\\n",
        "Transitivity:Given any three lotteries, if an agent prefers A to B and prefers B to C,\n",
        "then the agent must prefer A to C\\\\\n",
        "Continuity:\n",
        "A ≻ B ≻ C ⇒ ∃ p [p, A; 1 − p, C] ∼ B\\\\\n",
        "Substitutability: If an agent is indifferent between two lotteries and , then the agent is indifferent between two more complex lotteries that are the same except that \n",
        "is substituted for in one of them.\n",
        "A ∼ B ⇒ [p, A; 1 − p, C] ∼ [p, B; 1 − p, C] .\\\\\n",
        "Monotonicity:\n",
        "A ≻ B ⇒ p > q ⇔ [p, A; 1 − p, B] ≻ [q, A; 1 − q, B]).\\\\\n",
        "Decomposability:\n",
        "[p, A; 1 − p, [q, B; 1 − q, C]] ∼ [p, A; (1 − p)q, B; (1 − p)(1 − q), C].\\\\\n",
        "\n",
        "EXISTENCE OF UTILITY FUNCTION:\n",
        "U(A) > U(B) ⇔ A ≻ B and U(A) = U(B) ⇔ A ∼ B.\\\\\n",
        "EXPECTED UTILITY OF A LOTTERY:\n",
        "U([p1, S1;…; pn, Sn]) = ∑i piU(Si).\\\\\n",
        " in a deterministic environment an agent needs only a preference ranking on states—the numbers don’t matter. This is called a value function or ordinal\n",
        "utility function.\\\\\n",
        " the existence of a utility function that describes an agent’s preference behavior does not necessarily mean that the agent is explicitly maximizing that\n",
        "utility function in its own deliberations.\n",
        "\n",
        "\n",
        " \"\"\"\n",
        "chapter16_3 =  \"\"\"Utility functions map from lotteries to real numbers.\\\\\n",
        "We say that the agent exhibits a monotonic preference for more money. This does\n",
        "not mean that money behaves as a utility function, because it says nothing about\n",
        "preferences between lotteries involving money.\\\\\n",
        " agents with curves of this shape are risk-averse: they prefer a sure thing with a\n",
        "payoff that is less than the expected monetary value of a gamble. On the other hand, in the\n",
        "“desperate” region at large negative wealth , the behavior is risk-seeking.\\\\\n",
        "The value an agent will accept in lieu of a lottery is called the certainty equivalent of the\n",
        "lottery.\\\\\n",
        "The difference between the EMV of a lottery and its certainty equivalent is called the\n",
        "insurance premium.\\\\\n",
        " An agent that has a linear curve is said to be risk-neutral.\\\\\n",
        " The rational way to choose the best action, , is to maximize expected utility\\\\\n",
        " This tendency for the estimated expected utility of the best choice to be too high is called the optimizer’s curse\\\\\n",
        " Decision theory is a normative theory: it describes how a rational agent should act.\\\\\n",
        "  A descriptive theory, on the other hand, describes how actual agents—for example, humans—\n",
        "really do act.\\\\\n",
        "One explanation for the apparently irrational preferences is the certainty effect (Kahneman\n",
        "and Tversky, 1979): people are strongly attracted to gains that are certain. There are several\n",
        "reasons why this may be so.\\\\\n",
        "First, people may prefer to reduce their computational burden;Second, people may distrust the legitimacy of the stated probabilities.;Third, people may be accounting for their emotional state as well as their financial state.\\\\\n",
        "people have ambiguity aversion\\\\\n",
        " framing effect: exact wording of a decision problem can have a big impact on the agent’s choices\\\\\n",
        "anchoring effect:People feel more comfortable making relative utility judgments rather than absolute ones.\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\"  sequential decision problems, in which the agent’s utility depends on a sequence of decisions.\\\\\n",
        " a sequential decision problem for a fully observable, stochastic environment\n",
        "with a Markovian transition model and additive rewards is called a Markov decision\n",
        "process, or MDP, and consists of a set of states (with an initial state s0);\\\\\n",
        " a set ACTIONS(s) of\n",
        "actions in each state; \\\\\n",
        "a transition model P(s′|s, a); \\\\\n",
        "and a reward function R(s, a, s′).\\\\\n",
        "A solution of this kind is called a policy.\\\\\n",
        "dynamic programming: simplifying a problem\n",
        "by recursively breaking it into smaller pieces and remembering the optimal solutions to the\n",
        "pieces.\\\\\n",
        "1)The first question to answer is whether there is a finite horizon or an infinite horizon for\n",
        "decision making: with a finite horizon, an optimal action in a given state may depend on how much time is left. A policy that\n",
        "depends on the time is called nonstationary.\\\\\n",
        "With no fixed time limit, on the other hand, there is no reason to behave differently in the same state at different times. Hence, an optimal action depends only on the current state,\n",
        "and the optimal policy is stationary.\\\\\n",
        " 2)The next question we must decide is how to calculate the utility of state sequences.\\\\\n",
        " additive discounted rewards: the utility of a history is Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + γR(s1, a1, s2) + γ2R(s2, a2, s3) + ⋯\\\\\n",
        "There are several reasons why additive discounted rewards make sense.\\\\\n",
        " One is empirical:\\\\\n",
        " Another is economic:\\\\\n",
        "A third reason is uncertainty about the true rewards: they may never arrive for all sorts of\n",
        "reasons that are not taken into account in the transition model. Under certain assumptions,\n",
        "a discount factor of gamma is equivalent to adding a probability 1 − γ of accidental\n",
        "termination at every time step, independent of the action taken.\\\\\n",
        "A fourth justification arises from a natural property of preferences over histories.\\\\\n",
        "A final justification for discounted rewards is that it conveniently makes some nasty\n",
        "infinities go away.:There are three solutions:1.1. With discounted rewards, the utility of an infinite sequence is finite.;2. If the environment contains terminal states and if the agent is guaranteed to get to one\n",
        "eventually, then we will never need to compare infinite sequences.3. Infinite sequences can be compared in terms of the average reward obtained per\n",
        "time step.\\\\\n",
        "the utility of a state is the expected reward for the next\n",
        "transition plus the discounted utility of the next state, assuming that the agent chooses the optimal\n",
        "action. \n",
        "That is, the utility of a state is given by  π∗(s) = argmax a∈A(s) \n",
        "U(s) = max a∈A(s)∑s′ P(s′|s, a)[R(s, a, s′) + γU(s′)].\n",
        "This is called the Bellman equation\\\\\n",
        "Another important quantity is the action-utility function, or Q-function: is the\n",
        "expected utility of taking a given action in a given state.\n",
        "Q(s, a) = ∑s′ P(s′|s, a)[R(s, a, s′) + γ U(s′)] = ∑s′ P(s′|s, a)[R(s, a, s′) + γ max a′ Q(s′, a′)]\n",
        " \"\"\"\n",
        "chapter17_2 =  \"\"\" algorithms for solving MDPs. \\\\\n",
        "The first three, value iteration, policy iteration, and linear programming, generate exact solutions offline. The fourth is a family of online approximate algorithms that includes Monte Carlo planning\\\\\n",
        "Value Iteration: Convergence of value iteration:The basic concept used in showing that value iteration converges is the notion of a\n",
        "contraction.\n",
        "The iteration step, called a Bellman update:\\\\\n",
        "Ui+1(s) ← max ∑P(s′|s, a)[R(s, a, s′) + γ Ui(s′)] \\\\\n",
        "POLICY EVALUATION: given a policy πi, calculate Ui = Uπi , the utility of each state if πi were to be executed.\n",
        "POLICY IMPROVEMENT: Calculate a new MEU policy πi+1, using one-step look-ahead\n",
        "based on Ui\n",
        "Linear programming or LP, which was mentioned briefly in, is a\n",
        "general approach for formulating constrained optimization problems, and there are many\n",
        "industrial-strength LP solvers available.\n",
        "Here we will consider online algorithms:The most straightforward approach is actually a simplification of the EXPECTIMINIMAX\n",
        "algorithm for game trees with chance nodes: the EXPECTIMAX algorithm builds a tree of\n",
        "alternating max and chance nodes\n",
        "\n",
        "\n",
        " \"\"\"\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\" Stochastic Transition Model\n",
        "The transition function is sometimes written as T(s, a, s’), or explicitly as a probability: p( s’ | s, a )\\\\\n",
        "Reward function: R( s, a, s’ )\\\\\n",
        "A Markov decision process or MDP is\\\\\n",
        "• a sequential decision problem\\\\\n",
        "• for a fully observable environment\\\\\n",
        "• with a stochastic transition model\\\\\n",
        "• that has additive rewards\\\\\n",
        "MDPs are non-deterministic search problems. One way of solving them is via expectimax search.\\\\\n",
        "there are only two ways to define utilities: \\\\\n",
        "• Additive utility:  Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + R(s1, a1, s2) + 2R(s2, a2, s3) + ⋯\\\\\n",
        "• Discounted utility: Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + γR(s1, a1, s2) + γ^2R(s2, a2, s3) + ⋯\\\\\n",
        "Infinite Utilities:\\\\\n",
        "Solutions:\\\\\n",
        "1) Finite horizon: (similar to depth-limited search) \\\\\n",
        "   ▪ Terminate episodes after a fixed T steps (e.g. life)\\\\\n",
        "   ▪ Gives nonstationary policies (depends on time left)\\\\\n",
        "2) Discounting: use 0 < γ< 1\\\\\n",
        "  ▪ Smaller  means smaller “horizon” – shorter term focus\\\\\n",
        "3) Absorbing state: guarantee that for every policy, a terminal state will eventually be reached (like “overheated” for racing)\\\\\n",
        "\n",
        "\n",
        "Value Iteration:\n",
        "Vi+1(s) ← max a ∑ s' T(s′|s, a)[R(s, a, s′) + γ Vi(s′)] \\\\\n",
        "Problem 1: It’s slow – O(S2A) per iteration\n",
        "Problem 2: The “max” at each state rarely changes\n",
        "Problem 3: The policy often converges long before the values\\\\\n",
        "\n",
        "Policy Iteration: \n",
        "Alternative approach for optimal values:\\\\\n",
        "• Step 1: Policy evaluation: calculate utilities for some fixed policy (not optimal utilities!) until convergence\n",
        "• Step 2: Policy improvement: update policy using one-step look- ahead with resulting converged (but not optimal!) utilities as future values\n",
        "• Repeat steps until policy converges\\\\\n",
        "Evaluation: For fixed current policy , find values with policy evaluation:\\\\\n",
        "Improvement: For fixed values, get a better policy using policy extraction\\\\\n",
        "Comparison\\\\\n",
        "In value iteration:\n",
        "• Every iteration updates both the values and (implicitly) the policy\n",
        "• We don’t track the policy, but taking the max over actions implicitly recomputes it\n",
        "In policy iteration:\n",
        "• We do several passes that update utilities with fixed policy (each pass is fast because we consider only one action, not all of them)\n",
        "• After the policy is evaluated, a new policy is chosen (slow like a value iteration pass)\n",
        "$V^\\pi_{k+1}(s)=\\Sigma_{s'}T(s,\\pi(s),s')[R(s,\\pi(s),s')+\\gamma V_k^\\pi(s')]$\n",
        "• The new policy will be better (or we’re done)\\\\\n",
        "Utilities are functions from outcomes (states of the world) to real numbers that describe an agent’s preferences\\\\\n",
        "MEU Principle\\\\\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any bolded term in the textbook, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \" a single number to express the desirability of a state\",\n",
        "\"expected utility\" : \"average utility value of the outcomes, weighted by the probability that the outcome occurs\",\n",
        "\"MEU\": \"choose the action maximizes the agent’s expected utility\",\n",
        "\"Markovian\": \"the probability of reaching s′ from s depends only on s and not on the history of earlier states.\"\n",
        "}\n",
        "\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "c751c7ae-8cbf-47f1-9090-a2657a25a00d"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nThe MEU principle formalizes the general notion that an intelligent agent should “do the\nright thing,” but does not operationalize that advice.\\\n If an agent acts so as to maximize a utility function that correctly reflects the performance measure, then\nthe agent will achieve the highest possible performance score (averaged over all the possible\nenvironments).\\\n The performance measure gives a score for a history—a sequence of states.\\\n\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n  any reasonable preference relation to obey:\\ORDERABILITY:Given any two lotteries, a rational agent must either prefer one or\nelse rate them as equally preferable.\\\nTransitivity:Given any three lotteries, if an agent prefers A to B and prefers B to C,\nthen the agent must prefer A to C\\\nContinuity:\nA ≻ B ≻ C ⇒ ∃ p [p, A; 1 − p, C] ∼ B\\\nSubstitutability: If an agent is indifferent between two lotteries and , then the agent is indifferent between two more complex lotteries that are the same except that \nis substituted for in one of them.\nA ∼ B ⇒ [p, A; 1 − p, C] ∼ [p, B; 1 − p, C] .\\\nMonotonicity:\nA ≻ B ⇒ p > q ⇔ [p, A; 1 − p, B] ≻ [q, A; 1 − q, B]).\\\nDecomposability:\n[p, A; 1 − p, [q, B; 1 − q, C]] ∼ [p, A; (1 − p)q, B; (1 − p)(1 − q), C].\\\n\nEXISTENCE OF UTILITY FUNCTION:\nU(A) > U(B) ⇔ A ≻ B and U(A) = U(B) ⇔ A ∼ B.\\\nEXPECTED UTILITY OF A LOTTERY:\nU([p1, S1;…; pn, Sn]) = ∑i piU(Si).\\\n in a deterministic environment an agent needs only a preference ranking on states—the numbers don’t matter. This is called a value function or ordinal\nutility function.\\\n the existence of a utility function that describes an agent’s preference behavior does not necessarily mean that the agent is explicitly maximizing that\nutility function in its own deliberations.\n\n\n \n\n## AIMA Chapter 16.3 Utility Functions\nUtility functions map from lotteries to real numbers.\\We say that the agent exhibits a monotonic preference for more money. This does\nnot mean that money behaves as a utility function, because it says nothing about\npreferences between lotteries involving money.\\\n agents with curves of this shape are risk-averse: they prefer a sure thing with a\npayoff that is less than the expected monetary value of a gamble. On the other hand, in the\n“desperate” region at large negative wealth , the behavior is risk-seeking.\\\nThe value an agent will accept in lieu of a lottery is called the certainty equivalent of the\nlottery.\\\nThe difference between the EMV of a lottery and its certainty equivalent is called the\ninsurance premium.\\\n An agent that has a linear curve is said to be risk-neutral.\\\n The rational way to choose the best action, , is to maximize expected utility\\\n This tendency for the estimated expected utility of the best choice to be too high is called the optimizer’s curse\\\n Decision theory is a normative theory: it describes how a rational agent should act.\\\n  A descriptive theory, on the other hand, describes how actual agents—for example, humans—\nreally do act.\\\nOne explanation for the apparently irrational preferences is the certainty effect (Kahneman\nand Tversky, 1979): people are strongly attracted to gains that are certain. There are several\nreasons why this may be so.\\\nFirst, people may prefer to reduce their computational burden;Second, people may distrust the legitimacy of the stated probabilities.;Third, people may be accounting for their emotional state as well as their financial state.\\\npeople have ambiguity aversion\\\n framing effect: exact wording of a decision problem can have a big impact on the agent’s choices\\\nanchoring effect:People feel more comfortable making relative utility judgments rather than absolute ones.\n\n \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n  sequential decision problems, in which the agent’s utility depends on a sequence of decisions.\\ a sequential decision problem for a fully observable, stochastic environment\nwith a Markovian transition model and additive rewards is called a Markov decision\nprocess, or MDP, and consists of a set of states (with an initial state s0);\\\n a set ACTIONS(s) of\nactions in each state; \\\na transition model P(s′|s, a); \\\nand a reward function R(s, a, s′).\\\nA solution of this kind is called a policy.\\\ndynamic programming: simplifying a problem\nby recursively breaking it into smaller pieces and remembering the optimal solutions to the\npieces.\\\n1)The first question to answer is whether there is a finite horizon or an infinite horizon for\ndecision making: with a finite horizon, an optimal action in a given state may depend on how much time is left. A policy that\ndepends on the time is called nonstationary.\\\nWith no fixed time limit, on the other hand, there is no reason to behave differently in the same state at different times. Hence, an optimal action depends only on the current state,\nand the optimal policy is stationary.\\\n 2)The next question we must decide is how to calculate the utility of state sequences.\\\n additive discounted rewards: the utility of a history is Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + γR(s1, a1, s2) + γ2R(s2, a2, s3) + ⋯\\\nThere are several reasons why additive discounted rewards make sense.\\\n One is empirical:\\\n Another is economic:\\\nA third reason is uncertainty about the true rewards: they may never arrive for all sorts of\nreasons that are not taken into account in the transition model. Under certain assumptions,\na discount factor of gamma is equivalent to adding a probability 1 − γ of accidental\ntermination at every time step, independent of the action taken.\\\nA fourth justification arises from a natural property of preferences over histories.\\\nA final justification for discounted rewards is that it conveniently makes some nasty\ninfinities go away.:There are three solutions:1.1. With discounted rewards, the utility of an infinite sequence is finite.;2. If the environment contains terminal states and if the agent is guaranteed to get to one\neventually, then we will never need to compare infinite sequences.3. Infinite sequences can be compared in terms of the average reward obtained per\ntime step.\\\nthe utility of a state is the expected reward for the next\ntransition plus the discounted utility of the next state, assuming that the agent chooses the optimal\naction. \nThat is, the utility of a state is given by  π∗(s) = argmax a∈A(s) \nU(s) = max a∈A(s)∑s′ P(s′|s, a)[R(s, a, s′) + γU(s′)].\nThis is called the Bellman equation\\\nAnother important quantity is the action-utility function, or Q-function: is the\nexpected utility of taking a given action in a given state.\nQ(s, a) = ∑s′ P(s′|s, a)[R(s, a, s′) + γ U(s′)] = ∑s′ P(s′|s, a)[R(s, a, s′) + γ max a′ Q(s′, a′)]\n \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n algorithms for solving MDPs. \\The first three, value iteration, policy iteration, and linear programming, generate exact solutions offline. The fourth is a family of online approximate algorithms that includes Monte Carlo planning\\Value Iteration: Convergence of value iteration:The basic concept used in showing that value iteration converges is the notion of a\ncontraction.\nThe iteration step, called a Bellman update:\\\nUi+1(s) ← max ∑P(s′|s, a)[R(s, a, s′) + γ Ui(s′)] \\\nPOLICY EVALUATION: given a policy πi, calculate Ui = Uπi , the utility of each state if πi were to be executed.\nPOLICY IMPROVEMENT: Calculate a new MEU policy πi+1, using one-step look-ahead\nbased on Ui\nLinear programming or LP, which was mentioned briefly in, is a\ngeneral approach for formulating constrained optimization problems, and there are many\nindustrial-strength LP solvers available.\nHere we will consider online algorithms:The most straightforward approach is actually a simplification of the EXPECTIMINIMAX\nalgorithm for game trees with chance nodes: the EXPECTIMAX algorithm builds a tree of\nalternating max and chance nodes\n\n\n \n\n## Module 8 Lecture Notes\n Stochastic Transition Model\nThe transition function is sometimes written as T(s, a, s’), or explicitly as a probability: p( s’ | s, a )\\\nReward function: R( s, a, s’ )\\\nA Markov decision process or MDP is\\\n• a sequential decision problem\\\n• for a fully observable environment\\\n• with a stochastic transition model\\\n• that has additive rewards\\\nMDPs are non-deterministic search problems. One way of solving them is via expectimax search.\\\nthere are only two ways to define utilities: \\\n• Additive utility:  Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + R(s1, a1, s2) + 2R(s2, a2, s3) + ⋯\\\n• Discounted utility: Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + γR(s1, a1, s2) + γ^2R(s2, a2, s3) + ⋯\\\nInfinite Utilities:\\\nSolutions:\\\n1) Finite horizon: (similar to depth-limited search) \\\n   ▪ Terminate episodes after a fixed T steps (e.g. life)\\\n   ▪ Gives nonstationary policies (depends on time left)\\\n2) Discounting: use 0 < γ< 1\\\n  ▪ Smaller  means smaller “horizon” – shorter term focus\\\n3) Absorbing state: guarantee that for every policy, a terminal state will eventually be reached (like “overheated” for racing)\\\n\n\nValue Iteration:\nVi+1(s) ← max a ∑ s' T(s′|s, a)[R(s, a, s′) + γ Vi(s′)] \\\nProblem 1: It’s slow – O(S2A) per iteration\nProblem 2: The “max” at each state rarely changes\nProblem 3: The policy often converges long before the values\\\n\nPolicy Iteration: \nAlternative approach for optimal values:\\\n• Step 1: Policy evaluation: calculate utilities for some fixed policy (not optimal utilities!) until convergence\n• Step 2: Policy improvement: update policy using one-step look- ahead with resulting converged (but not optimal!) utilities as future values\n• Repeat steps until policy converges\\\nEvaluation: For fixed current policy , find values with policy evaluation:\\\nImprovement: For fixed values, get a better policy using policy extraction\\\nComparison\\\nIn value iteration:\n• Every iteration updates both the values and (implicitly) the policy\n• We don’t track the policy, but taking the max over actions implicitly recomputes it\nIn policy iteration:\n• We do several passes that update utilities with fixed policy (each pass is fast because we consider only one action, not all of them)\n• After the policy is evaluated, a new policy is chosen (slow like a value iteration pass)\n$V^\\pi_{k+1}(s)=\\Sigma_{s'}T(s,\\pi(s),s')[R(s,\\pi(s),s')+\\gamma V_k^\\pi(s')]$\n• The new policy will be better (or we’re done)\\\nUtilities are functions from outcomes (states of the world) to real numbers that describe an agent’s preferences\\\nMEU Principle\\\n\n\n\n## Module 8 Definitions\n* **Utility function** -  a single number to express the desirability of a state\n* **Expected utility** - average utility value of the outcomes, weighted by the probability that the outcome occurs\n* **Meu** - choose the action maximizes the agent’s expected utility\n* **Markovian** - the probability of reaching s′ from s depends only on s and not on the history of earlier states.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\"bandit problems:\n",
        "Each arm Mi is a Markov reward process or MRP, that is, an MDP with only one\n",
        "possible action ai. It has states Si, transition model Pi(s′|s, ai), and reward Ri(s, ai, s′).\n",
        "The arm defines a distribution over sequences of rewards Ri,0, Ri,1, Ri,2,…, where each \n",
        "Ri,t is a random variable.\n",
        "The overall bandit problem is an MDP: the state space is given by the Cartesian product \n",
        "S = S1 × ⋯ × Sn; the actions are a1,…, an; the transition model updates the state of\n",
        "whichever arm Mi is selected, according to its specific transition model, leaving the\n",
        "other arms unchanged; and the discount factor is γ. \"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\"An alternative is reinforcement learning (RL), in which an agent interacts with the world\n",
        "and periodically receives rewards (or, in the terminology of psychology, reinforcements).\\\\\n",
        "\n",
        "Model-free reinforcement learning: two varieties:action-utility learning:1)most common form is Q-learning.2)Policy search: learns a policy π(s) that maps directly from states to\n",
        "actions.\\\\\n",
        "passive reinforcement learning, where the agent’s policy is\n",
        "fixed and the task is to learn the utilities of states (or of state–action pairs); this could also\n",
        "involve learning a model of the environment.\\\\\n",
        "active reinforcement learning, where the agent must also figure out what to do. The\n",
        "principal issue is exploration: an agent must experience as much as possible of its\n",
        "environment in order to learn how to behave in it.\\\\\n",
        "\n",
        " \"\"\"\n",
        "chapter22_2 =  \"\"\"  Passive Reinforcement Learning: We start with the simple case of a fully observable environment with a small number of\n",
        "actions and states, in which an agent already has a fixed policy that determines its\n",
        "actions.\\\\\n",
        "The agent is trying to learn the utility function —the expected total discounted\n",
        "reward if policy is executed beginning in state $U^\\pi(s)$.\n",
        "The passive learning task is similar to the policy evaluation task\\\\\n",
        "\n",
        " 1)direct utility estimation: direct utility estimation is that the utility of a state is defined as the expected\n",
        "total reward from that state onward (called the expected reward-to-go): we can view direct utility estimation\n",
        "as searching for in a hypothesis space that is much larger than it needs to be, in that it\n",
        "includes many functions that violate the Bellman equations. For this reason, the algorithm\n",
        "often converges very slowly.\\\\\n",
        "2)adaptive dynamic programming (or ADP):  For a passive\n",
        "learning agent, this means plugging the learned transition model P(s′|s,π(s)) and the\n",
        "observed rewards R(s,π(s),s′) into Equation (22.2):\n",
        "Ui(s) = ∑s′ P(s′∣∣∣∣s,πi(s))[R(s,πi(s),s′) + γUi(s′)].\n",
        " to calculate the utilities of the states.\n",
        "modified policy iteration: using a simplified value iteration process to update the utility estimates after each change to the\n",
        "learned model. \\\\\n",
        "The transition model \n",
        "P(s′|s,a) is represented as a table and it is estimated directly from the counts that are\n",
        "accumulated in Ns′|sa.The counts record how often state s′\n",
        " is reached when executing a in s.\\\\\n",
        "temporal-difference (TD): this update rule uses the difference in utilities between successive states (and thus successive times).\n",
        "Uπ(s) ← Uπ(s) + α[R(s,π(s),s′) + γUπ(s′) − Uπ(s)].\n",
        "Notice that TD does not need a transition model to perform its updates. The\n",
        "environment itself supplies the connection between neighboring states in the form of\n",
        "observed transitions.\n",
        " The prioritized\n",
        "sweeping heuristic prefers to make adjustments to states whose likely successors have just\n",
        "undergone a large adjustment in their own utility estimates.\n",
        "\"\"\"\n",
        "chapter22_3 =  \"\"\" Active Reinforcement Learning\n",
        ", an agent must make a tradeoff between exploitation of\n",
        "the current best action to maximize its short-term reward and exploration of previously\n",
        "unknown states to gain information that can lead to a change in policy (and to greater\n",
        "rewards in the future).\\\\\n",
        "Exploration:\\\\\n",
        "“greedy in the limit of infinite exploration,” or GLIE\n",
        "GLIE schemes: 1) choose a random\n",
        "action at time step t with probability 1/t and to follow the greedy policy otherwise. While\n",
        "this does eventually converge to an optimal policy, it can be slow. 2)better:\n",
        "give some weight to actions that the agent has not tried very often, while tending to avoid\n",
        "actions that are believed to be of low utility \\\\\n",
        "\n",
        "\n",
        "Here, f is the exploration function. The function determines how greed (preference\n",
        "for high values of the utility ) is traded off against curiosity (preference for actions that\n",
        "have not been tried often and have a low count ).\\\\\n",
        "U +(s) ← maxa f (∑s′ P(s′|s,a)[R(s,a,s′) + γU +(s′)], N(s,a)) .\n",
        "f(u,n) =  R+ (if n < Ne)/ u (otherwise)  R+ is an optimistic estimate of the best possible reward obtainable in any state\\\\\n",
        "\n",
        "Safe exploration\\\\\n",
        "The first approach, Bayesian reinforcement learning.  Let \n",
        "be the expected utility, averaged over all possible start states, obtained by executing policy \n",
        "in model h . Then we have\n",
        "π∗ = argmax π ∑h P(h|e)Uπh .\\\\\n",
        "\n",
        "The second approach, derived from robust control theory, allows for a set of possible\n",
        "models without assigning probabilities to them, and defines an optimal robust policy as\n",
        "one that gives the best outcome in the worst case over :\n",
        "π∗ = argmax π minh Uπh.\\\\\n",
        "\n",
        "\n",
        " Temporal-difference Q-learning\n",
        "  a TD Q-learning agent does\n",
        "not need a transition model, either for learning or for action selection.\n",
        "Q(s,a) ← Q(s,a) + α[R(s,a,s′) + γ max a′ Q(s′,a′) − Q(s,a)].\\\\\n",
        "An exploratory Q-learning agent. It is an active learner that learns the value Q(s,a) of each action in each\n",
        "situation. It uses the same exploration function f as the exploratory ADP agent, but avoids having to\n",
        "learn the transition model.\\\\\n",
        "\n",
        "Q-learning has a close relative called SARSA (for state, action, reward, state, action).\n",
        "difference: When exploration is happening, however, they differ: if the\n",
        "exploration yields a negative reward, SARSA penalizes the action, while Q-learning does\n",
        "not.\\\\\n",
        "Q-learning is an off-policy learning algorithm, ” SARSA is an on-policy algorithm; Q-learning is more flexible than SARSA, a Q-learning agent can\n",
        "learn how to behave well when under the control of a wide variety of exploration policies.\n",
        "On the other hand, SARSA is appropriate if the overall policy is even partly controlled by\n",
        "other agents or programs, in which case it is better to learn a Q-function for what will\n",
        "actually happen rather than what would happen if the agent got to pick estimated best\n",
        "actions.\n",
        " \"\"\"\n",
        "chapter22_4 =  \"\"\"  Generalization in Reinforcement Learning\\\\\n",
        "1) Approximating direct utility estimation\n",
        " The state and the sum of rewards received constitute a training example for a\n",
        "supervised learning algorithm.\n",
        "θi ← θi − α = θi + α[uj(s) − Uˆθ(s)] . ∂Ej(s) ∂θi ∂Uˆθ(s) ∂θi\n",
        "This is called the Widrow–Hoff rule, or the delta rule, for online least-squares.\n",
        "2)Approximating temporal-difference learning\n",
        "θi ← θi + α[R(s,a,s′) + γUˆθ(s′) − Uˆθ(s)] ∂Uˆθ(s) ∂θi\n",
        "for utilities and\n",
        "θi ← θi + α[R(s,a,s′) + γ max a′ Qˆθ(s′,a′) − Qˆθ(s,a)] ∂Qˆθ(s,a) ∂θi\n",
        "for Q-values.\n",
        "3)Deep reinforcement learning\n",
        "4)Reward shaping: R′(s,a,s′) = R(s,a,s′) + γΦ(s′) − Φ(s).The potential fuction Φ can be constructed to reflect any desirable aspects of the state, such\n",
        "as achievement of subgoals or distance to a desired terminal state.\\\\\n",
        "5)Hierarchical reinforcement learning:Another way to cope with very long action sequences is to break them up into a few smaller\n",
        "pieces, and then break those into smaller pieces still, and so on until the action sequences\n",
        "are short enough to make learning easy.\n",
        "\"\"\"\n",
        "chapter22_5 =  \"\"\" \n",
        " Policy Search:\n",
        "policy search is the simplest of all the methods in this chapter: the\n",
        "idea is to keep twiddling the policy as long as its performance improves, then stop.\\\\\n",
        "In Q-learning with function approximation, the algorithm finds a value of such that is\n",
        "“close” to , the optimal Q-function. Policy search, on the other hand, finds a value of \n",
        "that results in good performance\n",
        "Another clear instance\n",
        "of the difference is the case where is calculated using, say, depth-10 look-ahead search\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" Important ideas in reinforcement learning that came up\n",
        "Exploration: you have to try unknown actions to get information\\\\\n",
        " Exploitation: eventually, you have to use what you know\\\\\n",
        "Regret: even if you learn intelligently, you make mistakes \\\\\n",
        "Sampling: because of chance, you have to try things repeatedly \\\\\n",
        "Difficulty: learning can be much harder than solving a known MDP\\\\\n",
        "\n",
        "Model-Based Idea:\\\\\n",
        "Learn an approximate model based on experiences Solve for values as if the learned model were correct\n",
        "Step 1: Learn empirical MDP model\n",
        "Count outcomes s’ for each s, a\n",
        "Normalize to give an estimate of T(s,a,s′)\n",
        "Discover each R(s,a,s′)  when we experience (s, a, s’)\n",
        "Step 2: Solve the learned MDP\n",
        "For example, use value iteration, as before\\\\\n",
        "\n",
        "\n",
        "\n",
        "Model-Free Learning:\\\\\n",
        "Passive Reinforcement Learning：\\\\\n",
        "Simplified task: policy evaluation\n",
        "Input: a fixed policy π(s)\n",
        "You don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’)\\\\\n",
        " Goal: learn the state values\\\\\n",
        "Learner is “along for the ride”\n",
        "No choice about what actions to take\n",
        "Just execute the policy and learn from experience\n",
        "This is NOT offline planning! You actually take actions in the world.\n",
        "Direct Evaluation:\\\\\n",
        "Goal: Compute values for each state under π\\\\\n",
        "Idea: Average together observed sample values\\\\\n",
        "Act according to π\n",
        "Every time you visit a state, write down what the sum of discounted rewards turned out to be\n",
        "Average those samples\\\\\n",
        "What’s good about direct evaluation?\n",
        "It’s easy to understand\n",
        "It doesn’t require any knowledge of T, R\n",
        "It eventually computes the correct average values, using just sample transitions\\\\\n",
        "What bad about it?\n",
        "It wastes information about state connections Each state must be learned separately\n",
        "So, it takes a long time to learn\\\\\n",
        "Simplified Bellman updates calculate V for a fixed policy:\n",
        "Each round, replace V with a one-step-look-ahead layer over V\\\\\n",
        "$V^\\pi_{k+1}(s)=\\Sigma_{s'}T(s,\\pi(s),s')[R(s,\\pi(s),s')+\\gamma V_k^\\pi(s')]$\n",
        "This approach fully exploited the connections between the states Unfortunately, we need T and R to do it!\\\\\n",
        "Idea: Take samples of outcomes s’ (by doing the action!) and average\\\\\n",
        "$V^\\pi_{k+1}(s)=n/1\\Sigma_isample_i$\n",
        "\\\\\n",
        "Temporal Difference Learning\\\\\n",
        "Big idea: learn from every experience!\n",
        "Update V(s) each time we experience a transition (s, a, s’, r) Likely outcomes s’ will contribute updates more often\\\\\n",
        "Temporal difference learning of values\\\\\n",
        "  Policy still fixed, still doing evaluation!\\\\\n",
        "Move values toward value of whatever successor occurs: running average\\\\\n",
        "Sample of V(s): R(s,a,s′) + γVπ(s′)\\\\\n",
        "Update to V(s):Vπ(s) ← Vπ(s) + α[sample− Vπ(s)].\n",
        "\\\\\n",
        "Exponential moving average\n",
        "The running interpolation update: Makes recent samples more important,Forgets about the past (distant past values were wrong anyway)\n",
        "\\\\\n",
        "Decreasing learning rate (alpha) can give converging averages\\\\\n",
        "Problems with TD Value Learning\\\\\n",
        "TD value leaning is a model-free way to do policy evaluation, mimicking Bellman updates with running sample averages\n",
        "However, if we want to turn values into a (new) policy, we’re sunk:Idea: learn Q-values, not values\n",
        "Makes action selection model-free too!\\\\\n",
        "\n",
        "\n",
        "Active Reinforcement Learning\\\\\n",
        "Full reinforcement learning: optimal policies (like value iteration)\n",
        "You don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’) You choose the actions now\\\\\n",
        "Goal: learn the optimal policy / values\\\\\n",
        "Learner makes choices!\\\\\n",
        "Fundamental tradeoff: exploration vs. exploitation\\\\\n",
        "This is NOT offline planning! You actually take actions in the world and find out what happens...\\\\\n",
        "\n",
        "Detour: Q-Value Iteration\n",
        "Q-Learning:sample-based Q-value iteration\n",
        "$Q_{k+1}(s,a) =\\Sigma_{s'} T(s,a,s′)[R(s,a,s′) + \\gamma max_{a′} Q_k(s′,a′) ].$\\\\\n",
        "Exploration vs. Exploitation\\\\\n",
        "Several schemes for forcing exploration\n",
        "Simplest: random actions (ε-greedy) Every time step, flip a coin\\\\\n",
        "With (small) probability ε, act randomly\\\\\n",
        "With (large) probability 1-ε, act on current policy\\\\\n",
        "Another solution: exploration functions\\\\\n",
        "Takes a value estimate u and a visit count n, and returns an optimistic utility\n",
        "e.g. f(u,n)=u+k/n\\\\\n",
        "Regular Q-Update:\n",
        "$ Q_{k+1}(s,a) =\\alpha R(s,a,s′) + \\gamma max_{a'}Q(s',a') $\\\\\n",
        "Modified Q-Update:\n",
        "$Q_{k+1}(s,a) =\\alpha R(s,a,s′) +\\gamma max_{a'}f(Q(s',a'),N(s',a')) $\\\\\n",
        "Regret is a measure of your total mistake cost: the difference between your (expected) rewards, including youthful suboptimality, and optimal (expected) rewards\n",
        "Minimizing regret goes beyond learning to be optimal – it requires optimally learning to be optimal\\\\\n",
        "Approximate Q-Learning:\n",
        "Ideal function: returns the actual minimax value of the position In practice: typically weighted linear sum of features:\\\\\n",
        "$Eval(s)=w_1f_(s)+w_2f_2(s)+\\dots$\n",
        " Advantage: our experience is summed up in a few powerful numbers Disadvantage: states may share features but actually be very different in value!\n",
        "\\\\\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"term\" : \"your definition here\",\n",
        "}"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "7fb964db-e0ee-4a00-cd28-fbf128cabacd"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\nbandit problems:\nEach arm Mi is a Markov reward process or MRP, that is, an MDP with only one\npossible action ai. It has states Si, transition model Pi(s′|s, ai), and reward Ri(s, ai, s′).\nThe arm defines a distribution over sequences of rewards Ri,0, Ri,1, Ri,2,…, where each \nRi,t is a random variable.\nThe overall bandit problem is an MDP: the state space is given by the Cartesian product \nS = S1 × ⋯ × Sn; the actions are a1,…, an; the transition model updates the state of\nwhichever arm Mi is selected, according to its specific transition model, leaving the\nother arms unchanged; and the discount factor is γ. \n\n## AIMA Chapter 22.1 Learning from Rewards\nAn alternative is reinforcement learning (RL), in which an agent interacts with the world\nand periodically receives rewards (or, in the terminology of psychology, reinforcements).\\\n\nModel-free reinforcement learning: two varieties:action-utility learning:1)most common form is Q-learning.2)Policy search: learns a policy π(s) that maps directly from states to\nactions.\\\npassive reinforcement learning, where the agent’s policy is\nfixed and the task is to learn the utilities of states (or of state–action pairs); this could also\ninvolve learning a model of the environment.\\\nactive reinforcement learning, where the agent must also figure out what to do. The\nprincipal issue is exploration: an agent must experience as much as possible of its\nenvironment in order to learn how to behave in it.\\\n\n \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n  Passive Reinforcement Learning: We start with the simple case of a fully observable environment with a small number of\nactions and states, in which an agent already has a fixed policy that determines its\nactions.\\\nThe agent is trying to learn the utility function —the expected total discounted\nreward if policy is executed beginning in state $U^\\pi(s)$.\nThe passive learning task is similar to the policy evaluation task\\\n\n 1)direct utility estimation: direct utility estimation is that the utility of a state is defined as the expected\ntotal reward from that state onward (called the expected reward-to-go): we can view direct utility estimation\nas searching for in a hypothesis space that is much larger than it needs to be, in that it\nincludes many functions that violate the Bellman equations. For this reason, the algorithm\noften converges very slowly.\\\n2)adaptive dynamic programming (or ADP):  For a passive\nlearning agent, this means plugging the learned transition model P(s′|s,π(s)) and the\nobserved rewards R(s,π(s),s′) into Equation (22.2):\nUi(s) = ∑s′ P(s′∣∣∣∣s,πi(s))[R(s,πi(s),s′) + γUi(s′)].\n to calculate the utilities of the states.\nmodified policy iteration: using a simplified value iteration process to update the utility estimates after each change to the\nlearned model. \\\nThe transition model \nP(s′|s,a) is represented as a table and it is estimated directly from the counts that are\naccumulated in Ns′|sa.The counts record how often state s′\n is reached when executing a in s.\\\ntemporal-difference (TD): this update rule uses the difference in utilities between successive states (and thus successive times).\nUπ(s) ← Uπ(s) + α[R(s,π(s),s′) + γUπ(s′) − Uπ(s)].\nNotice that TD does not need a transition model to perform its updates. The\nenvironment itself supplies the connection between neighboring states in the form of\nobserved transitions.\n The prioritized\nsweeping heuristic prefers to make adjustments to states whose likely successors have just\nundergone a large adjustment in their own utility estimates.\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Active Reinforcement Learning\n, an agent must make a tradeoff between exploitation of\nthe current best action to maximize its short-term reward and exploration of previously\nunknown states to gain information that can lead to a change in policy (and to greater\nrewards in the future).\\\nExploration:\\\n“greedy in the limit of infinite exploration,” or GLIE\nGLIE schemes: 1) choose a random\naction at time step t with probability 1/t and to follow the greedy policy otherwise. While\nthis does eventually converge to an optimal policy, it can be slow. 2)better:\ngive some weight to actions that the agent has not tried very often, while tending to avoid\nactions that are believed to be of low utility \\\n\n\nHere, f is the exploration function. The function determines how greed (preference\nfor high values of the utility ) is traded off against curiosity (preference for actions that\nhave not been tried often and have a low count ).\\\nU +(s) ← maxa f (∑s′ P(s′|s,a)[R(s,a,s′) + γU +(s′)], N(s,a)) .\nf(u,n) =  R+ (if n < Ne)/ u (otherwise)  R+ is an optimistic estimate of the best possible reward obtainable in any state\\\n\nSafe exploration\\\nThe first approach, Bayesian reinforcement learning.  Let \nbe the expected utility, averaged over all possible start states, obtained by executing policy \nin model h . Then we have\nπ∗ = argmax π ∑h P(h|e)Uπh .\\\n\nThe second approach, derived from robust control theory, allows for a set of possible\nmodels without assigning probabilities to them, and defines an optimal robust policy as\none that gives the best outcome in the worst case over :\nπ∗ = argmax π minh Uπh.\\\n\n\n Temporal-difference Q-learning\n  a TD Q-learning agent does\nnot need a transition model, either for learning or for action selection.\nQ(s,a) ← Q(s,a) + α[R(s,a,s′) + γ max a′ Q(s′,a′) − Q(s,a)].\\\nAn exploratory Q-learning agent. It is an active learner that learns the value Q(s,a) of each action in each\nsituation. It uses the same exploration function f as the exploratory ADP agent, but avoids having to\nlearn the transition model.\\\n\nQ-learning has a close relative called SARSA (for state, action, reward, state, action).\ndifference: When exploration is happening, however, they differ: if the\nexploration yields a negative reward, SARSA penalizes the action, while Q-learning does\nnot.\\\nQ-learning is an off-policy learning algorithm, ” SARSA is an on-policy algorithm; Q-learning is more flexible than SARSA, a Q-learning agent can\nlearn how to behave well when under the control of a wide variety of exploration policies.\nOn the other hand, SARSA is appropriate if the overall policy is even partly controlled by\nother agents or programs, in which case it is better to learn a Q-function for what will\nactually happen rather than what would happen if the agent got to pick estimated best\nactions.\n \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n  Generalization in Reinforcement Learning\u0001) Approximating direct utility estimation\n The state and the sum of rewards received constitute a training example for a\nsupervised learning algorithm.\nθi ← θi − α = θi + α[uj(s) − Uˆθ(s)] . ∂Ej(s) ∂θi ∂Uˆθ(s) ∂θi\nThis is called the Widrow–Hoff rule, or the delta rule, for online least-squares.\n2)Approximating temporal-difference learning\nθi ← θi + α[R(s,a,s′) + γUˆθ(s′) − Uˆθ(s)] ∂Uˆθ(s) ∂θi\nfor utilities and\nθi ← θi + α[R(s,a,s′) + γ max a′ Qˆθ(s′,a′) − Qˆθ(s,a)] ∂Qˆθ(s,a) ∂θi\nfor Q-values.\n3)Deep reinforcement learning\n4)Reward shaping: R′(s,a,s′) = R(s,a,s′) + γΦ(s′) − Φ(s).The potential fuction Φ can be constructed to reflect any desirable aspects of the state, such\nas achievement of subgoals or distance to a desired terminal state.\\\n5)Hierarchical reinforcement learning:Another way to cope with very long action sequences is to break them up into a few smaller\npieces, and then break those into smaller pieces still, and so on until the action sequences\nare short enough to make learning easy.\n\n\n## Module 9 Lecture Notes\n Important ideas in reinforcement learning that came up\nExploration: you have to try unknown actions to get information\\\n Exploitation: eventually, you have to use what you know\\\nRegret: even if you learn intelligently, you make mistakes \\\nSampling: because of chance, you have to try things repeatedly \\\nDifficulty: learning can be much harder than solving a known MDP\\\n\nModel-Based Idea:\\\nLearn an approximate model based on experiences Solve for values as if the learned model were correct\nStep 1: Learn empirical MDP model\nCount outcomes s’ for each s, a\nNormalize to give an estimate of T(s,a,s′)\nDiscover each R(s,a,s′)  when we experience (s, a, s’)\nStep 2: Solve the learned MDP\nFor example, use value iteration, as before\\\n\n\n\nModel-Free Learning:\\\nPassive Reinforcement Learning：\\\nSimplified task: policy evaluation\nInput: a fixed policy π(s)\nYou don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’)\\\n Goal: learn the state values\\\nLearner is “along for the ride”\nNo choice about what actions to take\nJust execute the policy and learn from experience\nThis is NOT offline planning! You actually take actions in the world.\nDirect Evaluation:\\\nGoal: Compute values for each state under π\\\nIdea: Average together observed sample values\\\nAct according to π\nEvery time you visit a state, write down what the sum of discounted rewards turned out to be\nAverage those samples\\\nWhat’s good about direct evaluation?\nIt’s easy to understand\nIt doesn’t require any knowledge of T, R\nIt eventually computes the correct average values, using just sample transitions\\\nWhat bad about it?\nIt wastes information about state connections Each state must be learned separately\nSo, it takes a long time to learn\\\nSimplified Bellman updates calculate V for a fixed policy:\nEach round, replace V with a one-step-look-ahead layer over V\\\n$V^\\pi_{k+1}(s)=\\Sigma_{s'}T(s,\\pi(s),s')[R(s,\\pi(s),s')+\\gamma V_k^\\pi(s')]$\nThis approach fully exploited the connections between the states Unfortunately, we need T and R to do it!\\\nIdea: Take samples of outcomes s’ (by doing the action!) and average\\\n$V^\\pi_{k+1}(s)=n/1\\Sigma_isample_i$\n\\\nTemporal Difference Learning\\\nBig idea: learn from every experience!\nUpdate V(s) each time we experience a transition (s, a, s’, r) Likely outcomes s’ will contribute updates more often\\\nTemporal difference learning of values\\\n  Policy still fixed, still doing evaluation!\\\nMove values toward value of whatever successor occurs: running average\\\nSample of V(s): R(s,a,s′) + γVπ(s′)\\\nUpdate to V(s):Vπ(s) ← Vπ(s) + α[sample− Vπ(s)].\n\\\nExponential moving average\nThe running interpolation update: Makes recent samples more important,Forgets about the past (distant past values were wrong anyway)\n\\\nDecreasing learning rate (alpha) can give converging averages\\\nProblems with TD Value Learning\\\nTD value leaning is a model-free way to do policy evaluation, mimicking Bellman updates with running sample averages\nHowever, if we want to turn values into a (new) policy, we’re sunk:Idea: learn Q-values, not values\nMakes action selection model-free too!\\\n\n\nActive Reinforcement Learning\\\nFull reinforcement learning: optimal policies (like value iteration)\nYou don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’) You choose the actions now\\\nGoal: learn the optimal policy / values\\\nLearner makes choices!\\\nFundamental tradeoff: exploration vs. exploitation\\\nThis is NOT offline planning! You actually take actions in the world and find out what happens...\\\n\nDetour: Q-Value Iteration\nQ-Learning:sample-based Q-value iteration\n$Q_{k+1}(s,a) =\\Sigma_{s'} T(s,a,s′)[R(s,a,s′) + \\gamma max_{a′} Q_k(s′,a′) ].$\\\nExploration vs. Exploitation\\\nSeveral schemes for forcing exploration\nSimplest: random actions (ε-greedy) Every time step, flip a coin\\\nWith (small) probability ε, act randomly\\\nWith (large) probability 1-ε, act on current policy\\\nAnother solution: exploration functions\\\nTakes a value estimate u and a visit count n, and returns an optimistic utility\ne.g. f(u,n)=u+k/n\\\nRegular Q-Update:\n$ Q_{k+1}(s,a) =\u0007lpha R(s,a,s′) + \\gamma max_{a'}Q(s',a') $\\\nModified Q-Update:\n$Q_{k+1}(s,a) =\u0007lpha R(s,a,s′) +\\gamma max_{a'}f(Q(s',a'),N(s',a')) $\\\nRegret is a measure of your total mistake cost: the difference between your (expected) rewards, including youthful suboptimality, and optimal (expected) rewards\nMinimizing regret goes beyond learning to be optimal – it requires optimally learning to be optimal\\\nApproximate Q-Learning:\nIdeal function: returns the actual minimax value of the position In practice: typically weighted linear sum of features:\\\n$Eval(s)=w_1f_(s)+w_2f_2(s)+\\dots$\n Advantage: our experience is summed up in a few powerful numbers Disadvantage: states may share features but actually be very different in value!\n\\\n\n\n\n## Module 9 Definitions\n* **Term** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\" Agents in the real world need to handle uncertainty, whether due to partial observability,\n",
        "nondeterminism, or adversaries. An agent may never know for sure what state it is in now\n",
        "or where it will end up after a sequence of actions.\n",
        "The right thing to do—the rational\n",
        "decision—therefore depends on both the relative importance of various goals and the likelihood that,\n",
        "and degree to which, they will be achieved.\\\\\n",
        "use logic to cope with a domain like\n",
        "medical diagnosis thus fails for three main reasons:\n",
        "LAZINESS: It is too much work to list the complete set of antecedents or consequents\n",
        "needed to ensure an exceptionless rule and too hard to use such rules.\n",
        "THEORETICAL IGNORANCE: Medical science has no complete theory for the domain.\n",
        "PRACTICAL IGNORANCE: Even if we know all the rules, we might be uncertain about\n",
        "a particular patient because not all the necessary tests have been or can be run.\\\\\n",
        "The agent’s\n",
        "knowledge can at best provide only a degree of belief in the relevant sentences. Our main\n",
        "tool for dealing with degrees of belief is probability theory.\n",
        " In the terminology, the ontological commitments of logic and probability theory are the same—that the\n",
        "world is composed of facts that do or do not hold in any particular case—but the\n",
        "epistemological commitments are different: a logical agent believes each sentence to be\n",
        "true or false or has no opinion, whereas a probabilistic agent may have a numerical degree\n",
        "of belief between 0 (for sentences that are certainly false) and 1 (certainly true).\n",
        "The theory of probability provides a way of summarizing the uncertainty that comes from our\n",
        "laziness and ignorance, thereby solving the qualification problem.\\\\\n",
        "Preferences, as expressed by utilities, are combined with probabilities in the general theory\n",
        "of rational decisions called decision theory: Decision theory = probability theory + utility theory.\\\\\n",
        "The fundamental idea of decision theory is that an agent is rational if and only if it chooses the\n",
        "action that yields the highest expected utility, averaged over all the possible outcomes of the action.\n",
        "This is called the principle of maximum expected utility (MEU).\\\\\n",
        " The primary difference\n",
        "is that the decision-theoretic agent’s belief state represents not just the possibilities for world\n",
        "states but also their probabilities.\n",
        "\"\"\"\n",
        "chapter12_2 = \"\"\" Basic Probability Notation\n",
        " In probability\n",
        "theory, the set of all possible worlds is called the sample space. The possible worlds are\n",
        "mutually exclusive and exhaustive—two possible worlds cannot both be the case, and one\n",
        "possible world must be the case.\\\\\n",
        "A fully specified probability model associates a numerical probability P(ω) with each\n",
        "possible world.1\n",
        "\n",
        " unconditional or prior\n",
        "probabilities (and sometimes just “priors” for short): refer to degrees of belief in\n",
        "propositions in the absence of any other information.\n",
        "\n",
        "\n",
        " \"\"\"\n",
        "chapter12_3 = \"\"\" Inference Using Full Joint Distributions:\n",
        " Let E be the list of\n",
        "evidence variables (just Toothache in the example), let e be the list of observed values for\n",
        "them, and let Y be the remaining unobserved variables (just Catch in the example). The\n",
        "query is P(X | e) and can be evaluated as\n",
        " P(X | e) = α P(X, e) = α∑P(X, e, y),\"\"\"\n",
        "chapter12_4 = \"\"\" Independence:\n",
        "P(a | b) = P(a) or P(b | a) = P(b) or P(a ∧ b) = P(a)P(b).\n",
        "\"\"\"\n",
        "chapter12_5 = \"\"\" Bayes’ Rule and Its Use:\n",
        "P(Y | X) = α P(X | Y )P(Y ),\n",
        "where is the normalization constant needed to make the entries in sum to 1.\n",
        "Using Bayes’ rule: Combining evidence\n",
        "The general definition of conditional independence of two variables X and Y , given a third\n",
        "variable Z, is\n",
        "P(X, Y |Z) = P(X |Z)P(Y |Z).\n",
        " As with absolute\n",
        "independence in Equation (12.11) , the equivalent forms：\n",
        "P(X | Y ,Z) = P(X |Z) and P(Y | X,Z) = P(Y |Z)\n",
        "In general, for symptoms that are all conditionally independent given \"\" , the size of\n",
        "the representation grows as O(n) instead of O(2n). That means that conditional independence\n",
        "assertions can allow probabilistic systems to scale up; moreover, they are much more commonly\n",
        "available than absolute independence assertions.\n",
        "\"\"\"\n",
        "chapter12_6 = \"\"\" Naive Bayes Models: \n",
        "P(Cause, Effect1,…, Effectn) = P(Cause)∏P(Effecti | Cause).\n",
        "Such a probability distribution is called a naive Bayes model—“naive” because it is often\n",
        "used (as a simplifying assumption) in cases where the “effect” variables are not strictly\n",
        "independent given the cause variable.\n",
        "The naive Bayes model is sometimes called a\n",
        "Bayesian classifier\\\\\n",
        " Call the observed effects E = e, while the remaining\n",
        "effect variables Y are unobserved. Then the standard method for inference from the joint\n",
        "distribution (Equation (12.9) ) can be applied:\n",
        "P(Cause | e) = α∑y P(Cause, e, y).\\\\\n",
        "P(Cause | e) = α∑y P (Cause) P (y|Cause) (∏j P (ej ∣ Cause)) \\\\\n",
        "= α P (Cause) (∏j P (ej∣ Cause))∑y P (y| Cause)\\\\\n",
        " = α P (Cause)∏j P (ej ∣∣ Cause)\n",
        "\n",
        "\n",
        "  Here the “cause”\n",
        "is the Category variable, and the “effect” variables are the presence or absence of certain key\n",
        "words, HasWordi.\n",
        "The naive Bayes model assumes that words occur independently in documents, with\n",
        "frequencies determined by the document category.\n",
        "\"\"\"\n",
        "chapter12_7 = \"\"\" The Wumpus World Revisited \n",
        "P(P1,3|known, b) = α ∑frontier P(b|known, P1,3,frontier) ∑other P(P1,3)P(known)P(frontier)P(other)\n",
        " = αP(known)P(P1,3) ∑frontier P(b| known, P1,3,frontier)P(frontier) ∑other P(other) \n",
        " = α′P(P1,3) ∑frontier P(b| known, P1,3,frontier)P(frontier),\n",
        "\n",
        "Summary:\n",
        "Uncertainty arises because of both laziness and ignorance. It is inescapable in complex,\n",
        "nondeterministic, or partially observable environments.\\\\\n",
        "Probabilities express the agent’s inability to reach a definite decision regarding the truth\n",
        "of a sentence. Probabilities summarize the agent’s beliefs relative to the evidence.\\\\\n",
        "Decision theory combines the agent’s beliefs and desires, defining the best action as the\n",
        "one that maximizes expected utility.\\\\\n",
        "Basic probability statements include prior or unconditional probabilities and posterior\n",
        "or conditional probabilities over simple and complex propositions.\\\\\n",
        "The axioms of probability constrain the probabilities of logically related propositions.\n",
        "An agent that violates the axioms must behave irrationally in some cases.\\\\\n",
        "The full joint probability distribution specifies the probability of each complete\n",
        "assignment of values to random variables. It is usually too large to create or use in its\n",
        "explicit form, but when it is available it can be used to answer queries simply by adding\n",
        "up entries for the possible worlds corresponding to the query propositions.\\\\\n",
        "Absolute independence between subsets of random variables allows the full joint\n",
        "distribution to be factored into smaller joint distributions, greatly reducing its\n",
        "complexity.\\\\\n",
        "Bayes’ rule allows unknown probabilities to be computed from known conditional\n",
        "probabilities, usually in the causal direction. Applying Bayes’ rule with many pieces of\n",
        "evidence runs into the same scaling problems as does the full joint distribution.\\\\\n",
        "Conditional independence brought about by direct causal relationships in the domain\n",
        "allows the full joint distribution to be factored into smaller, conditional distributions.\n",
        "The naive Bayes model assumes the conditional independence of all effect variables,\n",
        "given a single cause variable; its size grows linearly with the number of effects.\\\\\n",
        "A wumpus-world agent can calculate probabilities for unobserved aspects of the world,\n",
        "thereby improving on the decisions of a purely logical agent. Conditional independence\n",
        "makes these calculations tractable.\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" N-Grams:\n",
        "how can we compute probabilities of entire sequences like P(w1 , w2 , ..., wn )? One thing we can do is decompose this probability using the chain rule of proba- bility:\n",
        "P(X1...Xn) = P(X1)P(X2|X1)P(X3|X1:2)...P(Xn|X1:n−1) n\n",
        "= 􏰃P(Xk|X1:k−1)\\\\\n",
        "Applying the chain rule to words, we get\n",
        "P(w1:n) = P(w1)P(w2|w1)P(w3|w1:2)...P(wn|w1:n−1) n\n",
        "= 􏰃P(wk|w1:k−1)\\\\\n",
        "The intuition of the n-gram model is that instead of computing the probability of a word given its entire history, we can approximate the history by just the last few words.\\\\\n",
        "How do we estimate these bigram or n-gram probabilities? An intuitive way to\n",
        "estimate probabilities is called maximum likelihood estimation or MLE. \\\\\n",
        "Equation 3.12 (like Eq. 3.11) estimates the n-gram probability by dividing the\n",
        "observed frequency of a particular sequence by the observed frequency of a prefix.\n",
        "This ratio is called a relative frequency. \\\\\n",
        "\n",
        " \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" Evaluating Language Models\n",
        "The best way to evaluate the performance of a language model is to embed it in an application and measure how much the application improves. Such end-to-end evaluation is called extrinsic evaluation.\\\\\n",
        "An intrinsic evaluation metric is one that mea- sures the quality of a model independent of any application.\\\\\n",
        "The perplexity (sometimes called PP for short) of a language model on a test set is the inverse probability of the test set, normalized by the number of words. \\\\\n",
        "the higher the conditional probabil- ity of the word sequence, the lower the perplexity.\\\\\n",
        "\n",
        " \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Sampling sentences from a language model\\\\\n",
        "Sampling from a distribution means to choose random points\\\\\n",
        "Thus sampling from a language model—which rep- resents a distribution over sentences—means to generate some sentences, choosing each sentence according to its likelihood as defined by the model. \\\\\n",
        " \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" Generalization and Zeros:\n",
        "It is equally important to get training data in the appropriate dialect or variety, especially when processing social media posts or spoken transcripts.\\\\\n",
        "Matching genres and dialects is still not sufficient. Our models may still be subject to the problem of sparsity. \\\\\n",
        "Unknown Words:\n",
        "we have to deal with words we haven’t seen before, which we’ll\n",
        "OOV call unknown words, or out of vocabulary (OOV) words\\\\\n",
        "\n",
        "There are two common ways to train the probabilities of the unknown word model <UNK>. The first one is to turn the problem back into a closed vocabulary one by choosing a fixed vocabulary in advance:\\\\\n",
        "1. Choose a vocabulary (word list) that is fixed in advance.\\\\\n",
        "2. Convert in the training set any word that is not in this set (any OOV word) to\n",
        "the unknown word token <UNK> in a text normalization step.\\\\\n",
        "3. Estimatetheprobabilitiesfor<UNK>fromitscountsjustlikeanyotherregular\n",
        "word in the training set.\n",
        " \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Smoothing:\n",
        "This modification is called smoothing or discounting. In this section and the fol- lowing ones we’ll introduce a variety of ways to do smoothing: Laplace (add-one) smoothing, add-k smoothing, stupid backoff, and Kneser-Ney smoothing.\\\\\n",
        "Laplace Smoothing:\n",
        "Laplace smoothing merely adds one to each count (hence its alternate name add- one smoothing). \\\\\n",
        "A related way to view smoothing is as discounting (lowering) some non-zero counts in order to get the probability mass that will be assigned to the zero counts. Thus, instead of referring to the discounted counts c∗, we might describe a smooth- ing algorithm in terms of a relative discount dc, the ratio of the discounted counts to the original counts:\n",
        "dc = c∗ c\\\\\n",
        "One alternative to add-one smoothing is to move a bit less of the probability mass from the seen to the unseen events. \\\\\n",
        "In other words, sometimes using less context is a good thing, helping to general- ize more for contexts that the model hasn’t learned much about. There are two ways to use this n-gram “hierarchy”. In backoff, we use the trigram if the evidence is sufficient, otherwise we use the bigram, otherwise the unigram. In other words, we only “back off” to a lower-order n-gram if we have zero evidence for a higher-order n-gram. By contrast, in interpolation, we always mix the probability estimates from all the n-gram estimators, weighing and combining the trigram, bigram, and unigram counts.\\\\\n",
        "This kind of backoff with discounting is also called Katz backoff. In Katz back- off we rely on a discounted probability P∗ if we’ve seen this n-gram before (i.e., if we have non-zero counts). Otherwise, we recursively back off to the Katz probabil- ity for the shorter-history (N-1)-gram.\n",
        " \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Huge Language Models and Stupid Backoff\\\\\n",
        "Another option is to build approximate language models using techniques like Bloom filters\\\\\n",
        "Although with these toolkits it is possible to build web-scale language models using full Kneser-Ney smoothing, Brants et al. (2007) show that with very large lan- guage models a much simpler algorithm may be sufficient. The algorithm is called stupid backoff.\\\\\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\"\n",
        " Probability Distributions:\\\\\n",
        "Marginal Distributions: $P(X_1=x_1)=\\Sigma_{x_2} P(X_1=x_1,X_2=x_2)$\\\\\n",
        "Conditional Probabilities: $P(a|b)=P(a,b)/p(b)$\\\\\n",
        "Probabilistic Inference\\\\\n",
        "General case:\n",
        "• Evidencevariables:  $E_1 \\dots E_k$\\\\\n",
        "• Query*variable: Q\\\\\n",
        "• Hidden variables: $H_1 \\dots H_k$\\\\\n",
        "Step 1: Select the entries consistent with the evidence\\\\\n",
        "Step 2: Sum out H to get joint of Query and evidence\\\\\n",
        "Step 3: Normalize\\\\\n",
        "The Product Rule:\n",
        "P(y)P(x|y)=P(x,y)\n",
        "The Chain Rule:\n",
        "$P(x_1,x_2,x_3)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)$\\\\\n",
        "$P(x_1,x_2,\\dots ,x_n)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)$\\\\\n",
        "Bayes’ Rule:\n",
        "$P(x,y)=P(x|y)P(y)=P(x|y)P(y|x)P(x)$\\\\\n",
        "$P(x,y)=P(x|y)=(P(y|x)P(x))/P(y)$\\\\\n",
        "Probabilistic Language Models:Probabilistic Language Models\n",
        "One goal: assign a probability to a sentence\\\\\n",
        "Joint probability of words in sentence:\n",
        "$P(w_1,w_2,\\dots ,w_n)=\\Pi_i P(w_i|w_1w_2\\dots)$\\\\\n",
        "How to estimate these probabilities:\\\\\n",
        "Simplifying Assumption = Markov Assumption:\n",
        "Simplest case: Unigram model: $P(w_1,w_2,\\dots ,w_n)=\\Pi_i $\\\\\n",
        "Bigram model:$P(w_i|w_1,w_2,\\dots ,w_i-1)=\\Pi_i P(w_i|w_{i-1})$\\\\\n",
        "N-gram models: In general this is an insufficient model of language\n",
        "• because language has long-distance dependencies\\\\\n",
        "Language Modeling:Estimating N-gram Probabilities\\\\\n",
        "Estimating bigram probabilities\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"conditional probabilities\" : \"P(a | b) = P(a ∧ b)/ P(b)\",\n",
        "\"product rule\" : \"P(a ∧ b) = P(a | b)P(b)\",\n",
        "\"full joint probability distribution\":\"it follows that a probability model is completely determined by the joint distribution for all of the random variables\",\n",
        " \"inclusion–exclusion principle\":\"P(a ∨ b) = P(a) + P(b) − P(a ∧ b)\",\n",
        "\"marginalization, or summing out\":\"P (Y) = ∑P (Y, Z = z)\",\n",
        "\"conditioning\":\"P(Y) = ∑Z P(Y | Z)P(Z)\",\n",
        "\"product rule\":\"P(a ∧ b) = P(a | b)P(b) and P(a ∧ b) = P(b | a)P(a).\"\n",
        "}"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "11664c92-a9a0-4c5e-83bb-07e18f585ac6"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Agents in the real world need to handle uncertainty, whether due to partial observability,\nnondeterminism, or adversaries. An agent may never know for sure what state it is in now\nor where it will end up after a sequence of actions.\nThe right thing to do—the rational\ndecision—therefore depends on both the relative importance of various goals and the likelihood that,\nand degree to which, they will be achieved.\\\nuse logic to cope with a domain like\nmedical diagnosis thus fails for three main reasons:\nLAZINESS: It is too much work to list the complete set of antecedents or consequents\nneeded to ensure an exceptionless rule and too hard to use such rules.\nTHEORETICAL IGNORANCE: Medical science has no complete theory for the domain.\nPRACTICAL IGNORANCE: Even if we know all the rules, we might be uncertain about\na particular patient because not all the necessary tests have been or can be run.\\\nThe agent’s\nknowledge can at best provide only a degree of belief in the relevant sentences. Our main\ntool for dealing with degrees of belief is probability theory.\n In the terminology, the ontological commitments of logic and probability theory are the same—that the\nworld is composed of facts that do or do not hold in any particular case—but the\nepistemological commitments are different: a logical agent believes each sentence to be\ntrue or false or has no opinion, whereas a probabilistic agent may have a numerical degree\nof belief between 0 (for sentences that are certainly false) and 1 (certainly true).\nThe theory of probability provides a way of summarizing the uncertainty that comes from our\nlaziness and ignorance, thereby solving the qualification problem.\\\nPreferences, as expressed by utilities, are combined with probabilities in the general theory\nof rational decisions called decision theory: Decision theory = probability theory + utility theory.\\\nThe fundamental idea of decision theory is that an agent is rational if and only if it chooses the\naction that yields the highest expected utility, averaged over all the possible outcomes of the action.\nThis is called the principle of maximum expected utility (MEU).\\\n The primary difference\nis that the decision-theoretic agent’s belief state represents not just the possibilities for world\nstates but also their probabilities.\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n Basic Probability Notation\n In probability\ntheory, the set of all possible worlds is called the sample space. The possible worlds are\nmutually exclusive and exhaustive—two possible worlds cannot both be the case, and one\npossible world must be the case.\\\nA fully specified probability model associates a numerical probability P(ω) with each\npossible world.1\n\n unconditional or prior\nprobabilities (and sometimes just “priors” for short): refer to degrees of belief in\npropositions in the absence of any other information.\n\n\n \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Inference Using Full Joint Distributions:\n Let E be the list of\nevidence variables (just Toothache in the example), let e be the list of observed values for\nthem, and let Y be the remaining unobserved variables (just Catch in the example). The\nquery is P(X | e) and can be evaluated as\n P(X | e) = α P(X, e) = α∑P(X, e, y),\n\n## AIMA Chapter 12.4 Independence\n Independence:\nP(a | b) = P(a) or P(b | a) = P(b) or P(a ∧ b) = P(a)P(b).\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Bayes’ Rule and Its Use:\nP(Y | X) = α P(X | Y )P(Y ),\nwhere is the normalization constant needed to make the entries in sum to 1.\nUsing Bayes’ rule: Combining evidence\nThe general definition of conditional independence of two variables X and Y , given a third\nvariable Z, is\nP(X, Y |Z) = P(X |Z)P(Y |Z).\n As with absolute\nindependence in Equation (12.11) , the equivalent forms：\nP(X | Y ,Z) = P(X |Z) and P(Y | X,Z) = P(Y |Z)\nIn general, for symptoms that are all conditionally independent given \"\" , the size of\nthe representation grows as O(n) instead of O(2n). That means that conditional independence\nassertions can allow probabilistic systems to scale up; moreover, they are much more commonly\navailable than absolute independence assertions.\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n Naive Bayes Models: \nP(Cause, Effect1,…, Effectn) = P(Cause)∏P(Effecti | Cause).\nSuch a probability distribution is called a naive Bayes model—“naive” because it is often\nused (as a simplifying assumption) in cases where the “effect” variables are not strictly\nindependent given the cause variable.\nThe naive Bayes model is sometimes called a\nBayesian classifier\\\n Call the observed effects E = e, while the remaining\neffect variables Y are unobserved. Then the standard method for inference from the joint\ndistribution (Equation (12.9) ) can be applied:\nP(Cause | e) = α∑y P(Cause, e, y).\\\nP(Cause | e) = α∑y P (Cause) P (y|Cause) (∏j P (ej ∣ Cause)) \\\n= α P (Cause) (∏j P (ej∣ Cause))∑y P (y| Cause)\\\n = α P (Cause)∏j P (ej ∣∣ Cause)\n\n\n  Here the “cause”\nis the Category variable, and the “effect” variables are the presence or absence of certain key\nwords, HasWordi.\nThe naive Bayes model assumes that words occur independently in documents, with\nfrequencies determined by the document category.\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n The Wumpus World Revisited \nP(P1,3|known, b) = α ∑frontier P(b|known, P1,3,frontier) ∑other P(P1,3)P(known)P(frontier)P(other)\n = αP(known)P(P1,3) ∑frontier P(b| known, P1,3,frontier)P(frontier) ∑other P(other) \n = α′P(P1,3) ∑frontier P(b| known, P1,3,frontier)P(frontier),\n\nSummary:\nUncertainty arises because of both laziness and ignorance. It is inescapable in complex,\nnondeterministic, or partially observable environments.\\\nProbabilities express the agent’s inability to reach a definite decision regarding the truth\nof a sentence. Probabilities summarize the agent’s beliefs relative to the evidence.\\\nDecision theory combines the agent’s beliefs and desires, defining the best action as the\none that maximizes expected utility.\\\nBasic probability statements include prior or unconditional probabilities and posterior\nor conditional probabilities over simple and complex propositions.\\\nThe axioms of probability constrain the probabilities of logically related propositions.\nAn agent that violates the axioms must behave irrationally in some cases.\\\nThe full joint probability distribution specifies the probability of each complete\nassignment of values to random variables. It is usually too large to create or use in its\nexplicit form, but when it is available it can be used to answer queries simply by adding\nup entries for the possible worlds corresponding to the query propositions.\\\nAbsolute independence between subsets of random variables allows the full joint\ndistribution to be factored into smaller joint distributions, greatly reducing its\ncomplexity.\\\nBayes’ rule allows unknown probabilities to be computed from known conditional\nprobabilities, usually in the causal direction. Applying Bayes’ rule with many pieces of\nevidence runs into the same scaling problems as does the full joint distribution.\\\nConditional independence brought about by direct causal relationships in the domain\nallows the full joint distribution to be factored into smaller, conditional distributions.\nThe naive Bayes model assumes the conditional independence of all effect variables,\ngiven a single cause variable; its size grows linearly with the number of effects.\\\nA wumpus-world agent can calculate probabilities for unobserved aspects of the world,\nthereby improving on the decisions of a purely logical agent. Conditional independence\nmakes these calculations tractable.\n\n\n## SLP Chapter 3.1 N-Grams\n N-Grams:\nhow can we compute probabilities of entire sequences like P(w1 , w2 , ..., wn )? One thing we can do is decompose this probability using the chain rule of proba- bility:\nP(X1...Xn) = P(X1)P(X2|X1)P(X3|X1:2)...P(Xn|X1:n−1) n\n= 􏰃P(Xk|X1:k−1)\\\nApplying the chain rule to words, we get\nP(w1:n) = P(w1)P(w2|w1)P(w3|w1:2)...P(wn|w1:n−1) n\n= 􏰃P(wk|w1:k−1)\\\nThe intuition of the n-gram model is that instead of computing the probability of a word given its entire history, we can approximate the history by just the last few words.\\\nHow do we estimate these bigram or n-gram probabilities? An intuitive way to\nestimate probabilities is called maximum likelihood estimation or MLE. \\\nEquation 3.12 (like Eq. 3.11) estimates the n-gram probability by dividing the\nobserved frequency of a particular sequence by the observed frequency of a prefix.\nThis ratio is called a relative frequency. \\\n\n \n\n## SLP Chapter 3.2 Evaluating Language Models\n Evaluating Language Models\nThe best way to evaluate the performance of a language model is to embed it in an application and measure how much the application improves. Such end-to-end evaluation is called extrinsic evaluation.\\\nAn intrinsic evaluation metric is one that mea- sures the quality of a model independent of any application.\\\nThe perplexity (sometimes called PP for short) of a language model on a test set is the inverse probability of the test set, normalized by the number of words. \\\nthe higher the conditional probabil- ity of the word sequence, the lower the perplexity.\\\n\n \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Sampling sentences from a language model\\Sampling from a distribution means to choose random points\\Thus sampling from a language model—which rep- resents a distribution over sentences—means to generate some sentences, choosing each sentence according to its likelihood as defined by the model. \\ \n\n## SLP Chapter 3.4 Generalization and Zeros\n Generalization and Zeros:\nIt is equally important to get training data in the appropriate dialect or variety, especially when processing social media posts or spoken transcripts.\\\nMatching genres and dialects is still not sufficient. Our models may still be subject to the problem of sparsity. \\\nUnknown Words:\nwe have to deal with words we haven’t seen before, which we’ll\nOOV call unknown words, or out of vocabulary (OOV) words\\\n\nThere are two common ways to train the probabilities of the unknown word model <UNK>. The first one is to turn the problem back into a closed vocabulary one by choosing a fixed vocabulary in advance:\\\n1. Choose a vocabulary (word list) that is fixed in advance.\\\n2. Convert in the training set any word that is not in this set (any OOV word) to\nthe unknown word token <UNK> in a text normalization step.\\\n3. Estimatetheprobabilitiesfor<UNK>fromitscountsjustlikeanyotherregular\nword in the training set.\n \n\n## SLP Chapter 3.5 Smoothing\n Smoothing:\nThis modification is called smoothing or discounting. In this section and the fol- lowing ones we’ll introduce a variety of ways to do smoothing: Laplace (add-one) smoothing, add-k smoothing, stupid backoff, and Kneser-Ney smoothing.\\\nLaplace Smoothing:\nLaplace smoothing merely adds one to each count (hence its alternate name add- one smoothing). \\\nA related way to view smoothing is as discounting (lowering) some non-zero counts in order to get the probability mass that will be assigned to the zero counts. Thus, instead of referring to the discounted counts c∗, we might describe a smooth- ing algorithm in terms of a relative discount dc, the ratio of the discounted counts to the original counts:\ndc = c∗ c\\\nOne alternative to add-one smoothing is to move a bit less of the probability mass from the seen to the unseen events. \\\nIn other words, sometimes using less context is a good thing, helping to general- ize more for contexts that the model hasn’t learned much about. There are two ways to use this n-gram “hierarchy”. In backoff, we use the trigram if the evidence is sufficient, otherwise we use the bigram, otherwise the unigram. In other words, we only “back off” to a lower-order n-gram if we have zero evidence for a higher-order n-gram. By contrast, in interpolation, we always mix the probability estimates from all the n-gram estimators, weighing and combining the trigram, bigram, and unigram counts.\\\nThis kind of backoff with discounting is also called Katz backoff. In Katz back- off we rely on a discounted probability P∗ if we’ve seen this n-gram before (i.e., if we have non-zero counts). Otherwise, we recursively back off to the Katz probabil- ity for the shorter-history (N-1)-gram.\n \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Huge Language Models and Stupid Backoff\\Another option is to build approximate language models using techniques like Bloom filters\\Although with these toolkits it is possible to build web-scale language models using full Kneser-Ney smoothing, Brants et al. (2007) show that with very large lan- guage models a much simpler algorithm may be sufficient. The algorithm is called stupid backoff.\n\n## Module 10 Lecture Notes\n\n Probability Distributions:\\\nMarginal Distributions: $P(X_1=x_1)=\\Sigma_{x_2} P(X_1=x_1,X_2=x_2)$\\\nConditional Probabilities: $P(a|b)=P(a,b)/p(b)$\\\nProbabilistic Inference\\\nGeneral case:\n• Evidencevariables:  $E_1 \\dots E_k$\\\n• Query*variable: Q\\\n• Hidden variables: $H_1 \\dots H_k$\\\nStep 1: Select the entries consistent with the evidence\\\nStep 2: Sum out H to get joint of Query and evidence\\\nStep 3: Normalize\\\nThe Product Rule:\nP(y)P(x|y)=P(x,y)\nThe Chain Rule:\n$P(x_1,x_2,x_3)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)$\\\n$P(x_1,x_2,\\dots ,x_n)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)$\\\nBayes’ Rule:\n$P(x,y)=P(x|y)P(y)=P(x|y)P(y|x)P(x)$\\\n$P(x,y)=P(x|y)=(P(y|x)P(x))/P(y)$\\\nProbabilistic Language Models:Probabilistic Language Models\nOne goal: assign a probability to a sentence\\\nJoint probability of words in sentence:\n$P(w_1,w_2,\\dots ,w_n)=\\Pi_i P(w_i|w_1w_2\\dots)$\\\nHow to estimate these probabilities:\\\nSimplifying Assumption = Markov Assumption:\nSimplest case: Unigram model: $P(w_1,w_2,\\dots ,w_n)=\\Pi_i $\\\nBigram model:$P(w_i|w_1,w_2,\\dots ,w_i-1)=\\Pi_i P(w_i|w_{i-1})$\\\nN-gram models: In general this is an insufficient model of language\n• because language has long-distance dependencies\\\nLanguage Modeling:Estimating N-gram Probabilities\\\nEstimating bigram probabilities\n\n \n\n## Module 10 Definitions\n* **Conditional probabilities** - P(a | b) = P(a ∧ b)/ P(b)\n* **Product rule** - P(a ∧ b) = P(a | b)P(b) and P(a ∧ b) = P(b | a)P(a).\n* **Full joint probability distribution** - it follows that a probability model is completely determined by the joint distribution for all of the random variables\n* **Inclusion–exclusion principle** - P(a ∨ b) = P(a) + P(b) − P(a ∧ b)\n* **Marginalization, or summing out** - P (Y) = ∑P (Y, Z = z)\n* **Conditioning** - P(Y) = ∑Z P(Y | Z)P(Z)\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "\n",
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" A Bayesian network is a directed graph in which each node is annotated with quantitative\n",
        "probability information. The full specification is as follows:\n",
        "1. Each node corresponds to a random variable, which may be discrete or continuous.\\\\\n",
        "2. Directed links or arrows connect pairs of nodes. If there is an arrow from node X to\n",
        "node Y , X is said to be a parent of Y . The graph has no directed cycles and hence is\n",
        "a directed acyclic graph, or DAG.\\\\\n",
        "3. Each node Xi has associated probability information θ(Xi|Parents(Xi)) that\n",
        "quantifies the effect of the parents on the node using a finite number of parameters.\\\\\n",
        "The local probability information attached to each node in Figure takes the form of a\n",
        "conditional probability table (CPT).\\\\\n",
        "Each row in a CPT contains the conditional probability of each node value for a\n",
        "conditioning case.\n",
        " \"\"\"\n",
        "chapter13_2 = \"\"\" The Semantics of Bayesian Networks\n",
        " The semantics of Bayes nets defines each entry in the joint distribution as follows:\n",
        " P(x1,…,xn) = n∏i=1 θ(xi|parents(Xi)),\\\\\n",
        " P(x1,…,xn) = n∏i=1 P(xi|parents(Xi)).\\\\\n",
        "  product rule: P(x1,…,xn) = P(xn|xn−1,…,x1)P(xn−1,…,x1).\\\\\n",
        "   Chain rule:\n",
        "  P(x1,…,xn) = P(xn|xn−1,…,x1)P(xn−1|xn−2,…,x1) ⋯ P(x2|x1)P(x1)  = n∏i=1 P(xi|xi−1,…,x1). \\\\\n",
        "  P(Xi|Xi−1,…,X1) = P(Xi|Parents(Xi))\\\\\n",
        "\n",
        "\n",
        "  Compactness and node ordering\\\\\n",
        "  The most general conditional independence question one might ask in a Bayes net is\n",
        "whether a set of nodes X is conditionally independent of another set Y, given a third set Z.\n",
        "This can be determined efficiently by examining the Bayes net to see whether Z d-separates\n",
        "X and Y. The process works as follows:\\\\\n",
        "1. Consider just the ancestral subgraph consisting of X, Y, Z, and their ancestors.\\\\\n",
        "2. Add links between any unlinked pair of nodes that share a common child; now we\n",
        "have the so-called moral graph.\\\\\n",
        "3. Replace all directed links by undirected links.\\\\\n",
        "4. If Z blocks all paths between X and Y in the resulting graph, then Z d-separates X\n",
        "and Y. In that case, X is conditionally independent of Y, given Z. Otherwise, the\n",
        "original Bayes net does not require conditional independence.\\\\\n",
        "In brief, then, d-separation means separation in the undirected, moralized, ancestral\n",
        "subgraph.\\\\\n",
        "\n",
        " In fact, this is a worst-case scenario in which the relationship between\n",
        "the parents and the child is completely arbitrary. Usually, such relationships are describable\n",
        "by a canonical distribution that fits some standard pattern.\\\\\n",
        "Another important pattern that occurs often in practice is context-specific independence or\n",
        "CSI.\n",
        "\\\\\n",
        "Bayesian nets with continuous variables:\n",
        "1)One way to handle continuous variables is with discretization—that is, dividing up\n",
        "the possible values into a fixed set of intervals.\\\\\n",
        "2)another solution—sometimes called a nonparametric representation—is to define the\n",
        "conditional distribution implicitly with a collection of instances, each containing specific\n",
        "values of the parent and child variables.\\\\\n",
        "  hybrid Bayesian network.: The most common choice is the linear–\n",
        "Gaussian conditional distribution, in which the child has a Gaussian distribution whose\n",
        "mean μ varies linearly with the value of the parent and whose standard deviation σ is fixed.\\\\\n",
        "Each variable is conditionally independent of its non-descendants, given its parents.\\\\\n",
        " When discrete variables are added as parents (not as children) of\n",
        "continuous variables, the network defines a conditional Gaussian, or CG, distribution:\n",
        "given any assignment to the discrete variables, the distribution over the continuous\n",
        "variables is a multivariate Gaussian.\\\\\n",
        "the distributions for discrete variables with continuous parents:1)   One way to make soft thresholds is to use the\n",
        "integral of the standard normal distribution\\\\\n",
        "2)An alternative to the probit model is the expit or inverse logit model.\n",
        " \"\"\"\n",
        "chapter13_3 = \"\"\" Exact Inference in Bayesian Networks\\\\\n",
        "  Inference by enumeration\\\\\n",
        " a query can be answered using a Bayes net by computing sums of products of conditional\n",
        "probabilities from the network.\n",
        "there will be$O(2^n)$ terms in the sum, each a product of $O(n)$probability values. A naive implementation\n",
        "would therefore have complexity $O(n2^n)$\\\\\n",
        "This can be reduced to by taking advantage of the nested structure of the\n",
        "computation.\\\\\n",
        "The ENUMERATION-ASK algorithm: you will see that it contains\n",
        "repeated subexpressions. $O(2^n)$\\\\\n",
        " 2)The variable elimination algorithm:  do the calculation\n",
        "once and save the results for later use.,This is a form of dynamic programming.\\\\\n",
        " e.g. variable elimination algorithm,：Variable ordering and variable relevance： Every choice of ordering yields a valid algorithm, but different\n",
        "orderings cause different intermediate factors to be generated during the calculation.\\\\\n",
        "In general, the time and space requirements of variable elimination are dominated by the\n",
        "size of the largest factor constructed during the operation of the algorithm. This in turn is\n",
        "determined by the order of elimination of variables and by the structure of the network.\\\\\n",
        " 3)The complexity of exact inference\n",
        "The complexity of exact inference in Bayes nets depends strongly on the structure of the\n",
        "network.\\\\\n",
        " These are called singly connected networks or polytrees, and\n",
        "they have a particularly nice property: The time and space complexity of exact inference in\n",
        "polytrees is linear in the size of the network.\\\\\n",
        "As well as reducing satisfiability problems to Bayes net inference, we can reduce Bayes net\n",
        "inference to satisfiability, which allows us to take advantage of the powerful machinery\n",
        "developed for SAT-solving (see Chapter 7 ). In this case, the reduction is to a particular\n",
        "form of SAT solving called weighted model counting (WMC).\\\\\n",
        "4)Clustering algorithms: Using clustering algorithms (also\n",
        "known as join tree algorithms), the time can be reduced to O(n).\\\\\n",
        "The two Boolean nodes are replaced by a meganode that takes on four possible\n",
        "values: tt, tf, ft, and ff. The meganode has only one parent.\n",
        "\\\\\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" \n",
        " Independence:\n",
        " $P(x,y)=P(x)P(y)$ or $P(x|y)=P(x)$\\\\\n",
        " Conditional Independence:\n",
        " $P(x|z,y)=P(x)P(z)$ or $P(x,y|z)=P(x|z)P(x=y|z)$\\\\\n",
        " Bayes’ Nets(graphical models): a technique for describing complex joint distributions (models) using simple, local distributions (conditional probabilities)\\\\\n",
        "A Bayes net = Topology (graph) + Local Conditional Probabilities\\\\\n",
        "Probabilities in BNs: $P(x_1,x_2,\\dots ,x_n)=\\Pi_{i=1}^{n} P(x_i|Parents(x_{i}))$\\\\\n",
        "Causal Chains: x->y->z $P(x,y,z)=P(x)P(y|x)P(z|y)$ Not Guaranteed X independent of Z ; Guaranteed X independent of Z given Y\\\\\n",
        "Common Cause: x<-y->z $P(x,y,z)=P(y)P(x|y)P(z|y)$ Not Guaranteed X independent of Z ; Guaranteed X independent of Z given Y\\\\\n",
        "Common Effect(v-structures): x->z<-y $P(x,y,z)=P(y)P(x|y)P(z|y)$ Guaranteed X independent of Z ; Not  X and Y independent given Z\\\\\n",
        "Reachability\\\\\n",
        "Active Triples\n",
        "\n",
        "\n",
        "Consider all (undirected) paths from X to Y: if No active paths = independence\\\\\n",
        "A path is active if each triple is active\\\\\n",
        "1)Causal chain A -> B -> C where B is unobserved (either direction)\n",
        "2)Common cause A<-B -> C where B is unobserved\n",
        "3) Common effect (aka v-structure)\n",
        "A -> B <- C where B or one of its descendents is observed\\\\\n",
        "\n",
        "1)If one or more paths is active, then independence not guaranteed \\\\\n",
        "2)if all paths are inactive then independence is guaranteed\\\\\n",
        "3)A path is active if each triple is active\\\\\n",
        "Inference: calculating some useful quantity from a joint probability distribution\\\\\n",
        "Inference by Enumeration:\\\\\n",
        "We want $P(Q|e_1,e_2\\dots)$\\\\\n",
        "Step 1: Select the entries consistent with the evidence\\\\\n",
        "Step 2: Sum out H to get joint of Query and evidence: $P(Q,e_1,e_2\\dots, e_k)=\\Sigma_{h_1\\dots h_r}P(Q,h_1\\dots h_r,e_1,e_2\\dots, e_k)$\\\\\n",
        "Step 3: Normalize\\\\\n",
        " $P(Q|e_1,e_2\\dots, e_k)=1/Z*P(Q,e_1,e_2\\dots, e_k)$\\\\\n",
        " Inference by Enumeration:slow->You join up the whole joint distribution before you sum out the hidden variables\\\\\n",
        "  Advanced technique: Variable Elimination\n",
        "§ Interleave joining and marginalizing\n",
        "§ Still NP-hard, but usually much faster than inference by enumeration\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"Independence\" : \"$P(x,y)=P(x)P(y)$ or $P(x|y)=P(x)$\",  \n",
        "\"Conditional Independence\" : \"$P(x|z,y)=P(x)P(z)$ or $P(x,y|z)=P(x|z)P(x=y|z)$\",    \n",
        "}\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "beaad810-8d16-43f6-efd5-c932dc350d2b"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n A Bayesian network is a directed graph in which each node is annotated with quantitative\nprobability information. The full specification is as follows:\n1. Each node corresponds to a random variable, which may be discrete or continuous.\\\n2. Directed links or arrows connect pairs of nodes. If there is an arrow from node X to\nnode Y , X is said to be a parent of Y . The graph has no directed cycles and hence is\na directed acyclic graph, or DAG.\\\n3. Each node Xi has associated probability information θ(Xi|Parents(Xi)) that\nquantifies the effect of the parents on the node using a finite number of parameters.\\\nThe local probability information attached to each node in Figure takes the form of a\nconditional probability table (CPT).\\\nEach row in a CPT contains the conditional probability of each node value for a\nconditioning case.\n \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n The Semantics of Bayesian Networks\n The semantics of Bayes nets defines each entry in the joint distribution as follows:\n P(x1,…,xn) = n∏i=1 θ(xi|parents(Xi)),\\\n P(x1,…,xn) = n∏i=1 P(xi|parents(Xi)).\\\n  product rule: P(x1,…,xn) = P(xn|xn−1,…,x1)P(xn−1,…,x1).\\\n   Chain rule:\n  P(x1,…,xn) = P(xn|xn−1,…,x1)P(xn−1|xn−2,…,x1) ⋯ P(x2|x1)P(x1)  = n∏i=1 P(xi|xi−1,…,x1). \\\n  P(Xi|Xi−1,…,X1) = P(Xi|Parents(Xi))\\\n\n\n  Compactness and node ordering\\\n  The most general conditional independence question one might ask in a Bayes net is\nwhether a set of nodes X is conditionally independent of another set Y, given a third set Z.\nThis can be determined efficiently by examining the Bayes net to see whether Z d-separates\nX and Y. The process works as follows:\\\n1. Consider just the ancestral subgraph consisting of X, Y, Z, and their ancestors.\\\n2. Add links between any unlinked pair of nodes that share a common child; now we\nhave the so-called moral graph.\\\n3. Replace all directed links by undirected links.\\\n4. If Z blocks all paths between X and Y in the resulting graph, then Z d-separates X\nand Y. In that case, X is conditionally independent of Y, given Z. Otherwise, the\noriginal Bayes net does not require conditional independence.\\\nIn brief, then, d-separation means separation in the undirected, moralized, ancestral\nsubgraph.\\\n\n In fact, this is a worst-case scenario in which the relationship between\nthe parents and the child is completely arbitrary. Usually, such relationships are describable\nby a canonical distribution that fits some standard pattern.\\\nAnother important pattern that occurs often in practice is context-specific independence or\nCSI.\n\\\nBayesian nets with continuous variables:\n1)One way to handle continuous variables is with discretization—that is, dividing up\nthe possible values into a fixed set of intervals.\\\n2)another solution—sometimes called a nonparametric representation—is to define the\nconditional distribution implicitly with a collection of instances, each containing specific\nvalues of the parent and child variables.\\\n  hybrid Bayesian network.: The most common choice is the linear–\nGaussian conditional distribution, in which the child has a Gaussian distribution whose\nmean μ varies linearly with the value of the parent and whose standard deviation σ is fixed.\\\nEach variable is conditionally independent of its non-descendants, given its parents.\\\n When discrete variables are added as parents (not as children) of\ncontinuous variables, the network defines a conditional Gaussian, or CG, distribution:\ngiven any assignment to the discrete variables, the distribution over the continuous\nvariables is a multivariate Gaussian.\\\nthe distributions for discrete variables with continuous parents:1)   One way to make soft thresholds is to use the\nintegral of the standard normal distribution\\\n2)An alternative to the probit model is the expit or inverse logit model.\n \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Exact Inference in Bayesian Networks\\  Inference by enumeration\\ a query can be answered using a Bayes net by computing sums of products of conditional\nprobabilities from the network.\nthere will be$O(2^n)$ terms in the sum, each a product of $O(n)$probability values. A naive implementation\nwould therefore have complexity $O(n2^n)$\\\nThis can be reduced to by taking advantage of the nested structure of the\ncomputation.\\\nThe ENUMERATION-ASK algorithm: you will see that it contains\nrepeated subexpressions. $O(2^n)$\\\n 2)The variable elimination algorithm:  do the calculation\nonce and save the results for later use.,This is a form of dynamic programming.\\\n e.g. variable elimination algorithm,：Variable ordering and variable relevance： Every choice of ordering yields a valid algorithm, but different\norderings cause different intermediate factors to be generated during the calculation.\\\nIn general, the time and space requirements of variable elimination are dominated by the\nsize of the largest factor constructed during the operation of the algorithm. This in turn is\ndetermined by the order of elimination of variables and by the structure of the network.\\\n 3)The complexity of exact inference\nThe complexity of exact inference in Bayes nets depends strongly on the structure of the\nnetwork.\\\n These are called singly connected networks or polytrees, and\nthey have a particularly nice property: The time and space complexity of exact inference in\npolytrees is linear in the size of the network.\\\nAs well as reducing satisfiability problems to Bayes net inference, we can reduce Bayes net\ninference to satisfiability, which allows us to take advantage of the powerful machinery\ndeveloped for SAT-solving (see Chapter 7 ). In this case, the reduction is to a particular\nform of SAT solving called weighted model counting (WMC).\\\n4)Clustering algorithms: Using clustering algorithms (also\nknown as join tree algorithms), the time can be reduced to O(n).\\\nThe two Boolean nodes are replaced by a meganode that takes on four possible\nvalues: tt, tf, ft, and ff. The meganode has only one parent.\n\\\n \n\n## Module 11 Lecture Notes\n \n Independence:\n $P(x,y)=P(x)P(y)$ or $P(x|y)=P(x)$\\\n Conditional Independence:\n $P(x|z,y)=P(x)P(z)$ or $P(x,y|z)=P(x|z)P(x=y|z)$\\\n Bayes’ Nets(graphical models): a technique for describing complex joint distributions (models) using simple, local distributions (conditional probabilities)\\\nA Bayes net = Topology (graph) + Local Conditional Probabilities\\\nProbabilities in BNs: $P(x_1,x_2,\\dots ,x_n)=\\Pi_{i=1}^{n} P(x_i|Parents(x_{i}))$\\\nCausal Chains: x->y->z $P(x,y,z)=P(x)P(y|x)P(z|y)$ Not Guaranteed X independent of Z ; Guaranteed X independent of Z given Y\\\nCommon Cause: x<-y->z $P(x,y,z)=P(y)P(x|y)P(z|y)$ Not Guaranteed X independent of Z ; Guaranteed X independent of Z given Y\\\nCommon Effect(v-structures): x->z<-y $P(x,y,z)=P(y)P(x|y)P(z|y)$ Guaranteed X independent of Z ; Not  X and Y independent given Z\\\nReachability\\\nActive Triples\n\n\nConsider all (undirected) paths from X to Y: if No active paths = independence\\\nA path is active if each triple is active\\\n1)Causal chain A -> B -> C where B is unobserved (either direction)\n2)Common cause A<-B -> C where B is unobserved\n3) Common effect (aka v-structure)\nA -> B <- C where B or one of its descendents is observed\\\n\n1)If one or more paths is active, then independence not guaranteed \\\n2)if all paths are inactive then independence is guaranteed\\\n3)A path is active if each triple is active\\\nInference: calculating some useful quantity from a joint probability distribution\\\nInference by Enumeration:\\\nWe want $P(Q|e_1,e_2\\dots)$\\\nStep 1: Select the entries consistent with the evidence\\\nStep 2: Sum out H to get joint of Query and evidence: $P(Q,e_1,e_2\\dots, e_k)=\\Sigma_{h_1\\dots h_r}P(Q,h_1\\dots h_r,e_1,e_2\\dots, e_k)$\\\nStep 3: Normalize\\\n $P(Q|e_1,e_2\\dots, e_k)=1/Z*P(Q,e_1,e_2\\dots, e_k)$\\\n Inference by Enumeration:slow->You join up the whole joint distribution before you sum out the hidden variables\\\n  Advanced technique: Variable Elimination\n§ Interleave joining and marginalizing\n§ Still NP-hard, but usually much faster than inference by enumeration\n \n\n## Module 11 Definitions\n* **Independence** - $P(x,y)=P(x)P(y)$ or $P(x|y)=P(x)$\n* **Conditional independence** - $P(x|z,y)=P(x)P(z)$ or $P(x,y|z)=P(x|z)P(x=y|z)$\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" Forms of Learning：\n",
        "When the output is one of a finite set of values (such as sunny/cloudy/rainy or true/false),\n",
        "the learning problem is called classification. When it is a number (such as tomorrow’s\n",
        "temperature, measured either as an integer or a real number), the learning problem has the\n",
        "(admittedly obscure1) name regression.\\\\\n",
        "There are three types of feedback that can accompany the inputs, and that determine the\n",
        "three main types of learning:\\\\\n",
        "supervised learning\\\\\n",
        "unsupervised learning: The most common unsupervised learning task is clustering\\\\\n",
        " reinforcement learning\n",
        "\n",
        " \"\"\"\n",
        "chapter19_2 = \"\"\" Supervised Learning:\n",
        "The function h is called a hypothesis about the world. It is drawn from a hypothesis space\n",
        "H of possible functions.\\\\\n",
        "With alternative vocabulary, we can say that h is a model of the data, drawn from a model\n",
        "class H, or we can say a function drawn from a function class. We call the output yi the\n",
        "ground truth—the true answer we are asking our model to predict.\\\\\n",
        " we can perform exploratory data analysis:examining the data with statistical tests and visualizations—histograms, scatter plots, box\n",
        "plots—to get a feel for the data, and some insight into what hypothesis space might be\n",
        "appropriate.\\\\\n",
        " How do we choose a good hypothesis from within the hypothesis space? We could hope for\n",
        "a consistent hypothesis: and h such that each xi in the training set has h(xi) = yi. With\n",
        "continuous-valued outputs we can’t expect an exact match to the ground truth; instead we\n",
        "look for a best-fit function for which each h(xi) is close to yi\\\\\n",
        "One way to analyze hypothesis spaces is by the bias they impose (regardless of the training\n",
        "data set) and the variance they produce (from one training set to another).\\\\\n",
        "One way to analyze hypothesis spaces is by the bias they impose (regardless of the training\n",
        "data set) and the variance they produce (from one training set to another).\\\\\n",
        " the small difference in the data set translates into a small difference in the\n",
        "hypothesis. We call that low variance.\\\\\n",
        "Often there is a bias–variance tradeoff: a choice between more complex, low-bias\n",
        "hypotheses that fit the training data well and simpler, low-variance hypotheses that may\n",
        "generalize better.\\\\\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "chapter19_3 = \"\"\" Learning Decision Trees\n",
        "A decision tree is a representation of a function that maps a vector of attribute values to a\n",
        "single output value—a “decision.”\\\\\n",
        "  Learning decision trees from examples:\n",
        "LEARN-DECISION-TREE algorithm adopts a greedy divide-and-conquer strategy: always test the\n",
        "most important attribute first, then recursively solve the smaller subproblems that are\n",
        "defined by the possible results of the test.\\\\\n",
        " There are\n",
        "four cases to consider for these recursive subproblems:\\\\\n",
        "1. If the remaining examples are all positive (or all negative), then we are done: we\n",
        "can answer Yes or No.\\\\\n",
        "2. If there are some positive and some negative examples, then choose the best\n",
        "attribute to split them.\\\\\n",
        "3. If there are no examples left, it means that no example has been observed for this\n",
        "combination of attribute values, and we return the most common output value from\n",
        "the set of examples that were used in constructing the node’s parent.\\\\\n",
        "4. If there are no attributes left, but both positive and negative examples, it means that\n",
        "these examples have exactly the same description, but different classifications. This\n",
        "can happen because there is an error or noise in the data\\\\\n",
        "We can evaluate the performance of a learning algorithm with a learning curve,\\\\\n",
        " Choosing attribute tests:The decision tree learning algorithm chooses the attribute with the highest IMPORTANCE. We\n",
        "will now show how to measure importance, using the notion of information gain, which is\n",
        "defined in terms of entropy\\\\\n",
        "Entropy: H(V ) = −∑k P(vk)log2 P(vk).\\\\\n",
        " define as the entropy of a Boolean random variable that is true with\n",
        "probability:B(q) = −(q log2 q + (1 − q)log2(1 − q)).\\\\\n",
        " If a training\n",
        "set contains positive examples and negative examples:\n",
        "H(Output) = B(p/(p+n) ).\n",
        "Remainder(A) = d∑k=1  pk + nk/ p + n B(pk/ pk + nk ). \\\\\n",
        "Gain(A) = B( p/p + n) − Remainder(A).\\\\\n",
        " Generalization and overfitting: For decision trees, a technique called decision tree pruning combats overfitting.:\n",
        "  how large a gain should we require in order to\n",
        "split on a particular attribute?using a statistical significance test.\\\\\n",
        " combine them using an approach called early stopping\\\\\n",
        " Broadening the applicability of decision trees:\n",
        " 1, MISSING DATA;2, CONTINUOUS AND MULTIVALUED INPUT ATTRIBUTES:\\\\\n",
        " CONTINUOUS-VALUED OUTPUT ATTRIBUTE:The name CART, standing for Classification And Regression Trees, is used to cover\n",
        "both classes.\n",
        " \"\"\"\n",
        "chapter19_4 = \"\"\" Model Selection and Optimization:\n",
        "First we will make the assumption that the future examples will be like the past. We call this\n",
        "the stationarity assumption;;\\\\\n",
        " a training set to create the hypothesis, and a test set to evaluate it.;\\\\\n",
        " we might want to compare two completely\n",
        "different machine learning models: We call these “knobs” hyperparameters—parameters\n",
        "of the model class, not of the individual model.\\\\\n",
        "1. A training set to train candidate models.\n",
        "2. A validation set, also known as a development set or dev set, to evaluate the\n",
        "candidate models and choose the best one.\\\\\n",
        "What if we don’t have enough data to make all three of these data sets? We can squeeze\n",
        "more out of the data using a technique called k-fold cross-validation\\\\\n",
        "The extreme is k = n, also known as leave-one-out cross-validation or LOOCV. Even with\n",
        "cross-validation, we still need a separate test set.\\\\\n",
        " We can think of the task of finding a good hypothesis as two\n",
        "subtasks: model selection4 chooses a good hypothesis space, and optimization (also called\n",
        "training) finds the best hypothesis within that space.\\\\\n",
        "\n",
        "Model selection\\\\\n",
        "We say that a model that exactly fits all the training data has interpolated the data\n",
        "\n",
        "\n",
        " Regularization\\\\\n",
        "Another way to simplify models is to reduce the dimensions that the models work with. A\n",
        "process of feature selection\\\\\n",
        "\n",
        "Hyperparameter tuning:The simplest approach to hyperparameter tuning is hand-tuning: guess some parameter\n",
        "values based on past experience, train a model, measure its performance on the validation\n",
        "data, analyze the results, and use your intuition to suggest new parameter values.\\\\\n",
        "a more systematic approach called grid search is appropriate:\n",
        " Bayesian optimization treats the task of choosing good hyperparameter values as\n",
        "a machine learning problem in itself.\\\\\n",
        "An alternative to Bayesian optimization is population-based training (PBT). PBT starts by\n",
        "using random search to train (in parallel) a population of models, each with different\n",
        "hyperparameter values.\n",
        "\n",
        "\"\"\"\n",
        "chapter19_5 = \"\"\"The Theory of Learning:\n",
        "any hypothesis that is consistent with a sufficiently large set of training examples is\n",
        "unlikely to be seriously wrong: that is, it must be probably approximately correct (PAC).\\\\\n",
        " One can think of an\n",
        "approximately correct hypothesis as being “close” to the true function in hypothesis space: it\n",
        "lies inside what is called the ϵ-ball around the true function f. The hypothesis space outside\n",
        "this ball is called H bad.']\"\"\"\n",
        "chapter19_6 = \"\"\" 19.6 Linear Regression and Classification\\\\\n",
        " Univariate linear regression\\\\\n",
        " The task of\n",
        "finding the hw that best fits these data is called linear regression.\\\\\n",
        "We see that the loss function is convex, this is\n",
        "true for every linear regression problem with an loss function, and implies that there are\n",
        "no local minima.\\\\\n",
        " \n",
        " Gradient descent\\\\\n",
        " The parameter which we called the step size in Section 4.2 , is usually called the\n",
        "learning rate when we are trying to minimize loss in a learning problem.\\\\\n",
        "For univariate regression, the loss is quadratic, so the partial derivative will be linear. (The\n",
        "only calculus you need to know is the chain rule\\\\\n",
        "These updates constitute the batch gradient descent learning rule for univariate linear\n",
        "regression (also called deterministic gradient descent).\\\\\n",
        "A faster variant is called stochastic gradient descent or SGD: it randomly selects a small\n",
        "number of training examples at each step, and updates\\\\\n",
        "A step that covers all the training examples is called an epoch.\\\\\n",
        "\n",
        "Multivariable linear regression\\\\\n",
        "We call the expression (X⊤X)−1X⊤\n",
        " the pseudoinverse of the data matrix, w∗ = (X⊤X)−1X⊤y. is called the normal equation.\\\\\n",
        "With univariate linear regression we didn’t have to worry about overfitting. But with\n",
        "multivariable linear regression in high-dimensional spaces it is possible that some\n",
        "dimension that is actually irrelevant appears by chance to be useful, resulting in overfitting.\\\\\n",
        "$Complexity(h_w) = \\Sigma |w_i|^q. $\n",
        "As with loss functions, with q = 1 we have L1 regularization9, which minimizes the sum of\n",
        "the absolute values; with q = 2, L2 regularization minimizes the sum of squares. Which\n",
        "regularization function should you pick? That depends on the specific problem, but L1\n",
        "regularization has an important advantage: it tends to produce a sparse model. That is, it\n",
        "often sets many weights to zero, effectively declaring the corresponding attributes to be\n",
        "completely irrelevant\\\\\n",
        "\n",
        " Linear classifiers with a hard threshold\\\\\n",
        "A decision boundary is a line (or a surface, in higher dimensions) that separates the two\n",
        "classes.   A linear decision\n",
        "boundary is called a linear separator and data that admit such a separator are called\n",
        "linearly separable. \\\\\n",
        "\n",
        "Alternatively, we can think of h as the result of passing the linear function w ⋅ x through a\n",
        "threshold function\\\\\n",
        "a simple weight update rule that converges to a solution—that is, to a\n",
        "linear separator that classifies the data perfectly—provided the data are linearly separable.\n",
        "For a single example (x, y), we have\n",
        "wi ← wi + α (y − hw(x)) × xi\\\\\n",
        "This\n",
        "rule is called the perceptron learning rule\\\\\n",
        "\n",
        " Linear classification with logistic regression\\\\\n",
        " We have seen that passing the output of a linear function through the threshold function\n",
        "creates a linear classifier; yet the hard nature of the threshold causes some problems\\\\\n",
        "All of these issues can be resolved to a large extent by softening the threshold function—\n",
        "approximating the hard threshold with a continuous, differentiable function.\\\\\n",
        "\n",
        "  \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\"  \n",
        "Naive Bayes Classifiers\n",
        " multinomial naive Bayes classifier:\n",
        "The intuition of the classifier is shown in Fig. 4.1. We represent a text document\n",
        "as if it were a bag-of-words, that is, an unordered set of words with their position ignored, keeping only their frequency in the document.\\\\\n",
        "\n",
        "Naive Bayes is a probabilistic classifier, meaning that for a document d, out of all classes c ∈ C the classifier returns the class cˆ which has the maximum posterior ˆ probability given the document. \\\\\n",
        "cˆ = argmaxP(c|d)\\\\\n",
        "cˆ = argmaxP(c|d) = argmaxP(d|c)P(c)\\\\\n",
        "\n",
        "Naive Bayes classifiers therefore make two simplifying assumptions:The first is the bag of words assumption discussed intuitively above,; The second is commonly called the naive Bayes assumption: this is the condi- tional independence assumption \n",
        "$c_{NB} = argmax P(c) 􏰄\\Pi P(f|c)$\n",
        "cNB = argmax P(c) 􏰄 P(wi |c)\n",
        " Classifiers that use a linear combination of the inputs to make a classification decision —like naive Bayes and also logistic regression— are called linear classifiers.\n",
        " \"\"\"  \n",
        "\n",
        "slp_chapter4_2 = \"\"\" Training the Naive Bayes Classifier \n",
        "Note once again that it is crucial that the vocabulary V consists of the union of all the word types in all classes, not just the words in one class c (try to convince yourself why this must be true\\\\\n",
        "What do we do about words that occur in our test data but are not in our vocab- ulary at all because they did not occur in any training document in any class? The solution for such unknown words is to ignore them—remove them from the test document and not include any probability for them at all.\\\\\n",
        "Finally, some systems choose to completely ignore another class of words: stop words\\\\\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\" Optimizing for Sentiment Analysis：First, for sentiment classification and a number of other text classification tasks, whether a word occurs or not seems to matter more than its frequency.、、\n",
        " This variant is called binary\n",
        "multinomial naive Bayes or binary NB\n",
        " \n",
        " \n",
        " \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Naive Bayes for other text classification tasks：\n",
        "the most effective naive Bayes features are not words at all, but character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’, ‘thei’), or, even simpler byte n-grams,；\n",
        " \"\"\"\n",
        "slp_chapter4_6 = \"\"\" Naive Bayes as a Language Model:\\\\\n",
        "Specifically, a naive Bayes model can be viewed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. \\\\\n",
        "Since the likelihood features from the naive Bayes model assign a probability to each word P(word|c), the model also assigns a probability to each sentence:\n",
        "$P(s|c) = 􏰄 \\Pi P(w_i|c)  i∈ positions$\n",
        "once we multiply in the prior a full naive Bayes model might well make a different classification decision.\\\\\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter4_7 = \"\"\" Evaluation: Precision, Recall, F-measure;\\\\\n",
        "refer to these human labels as the gold labels. \\\\\n",
        "To evaluate any system for detecting things, we start by building a confusion matrix\\\\\n",
        "e.g.true positives are documents that are indeed spam (indicated by human-created gold labels) that our system correctly said were spam. False negatives are documents that are indeed spam but our system incorrectly labeled as non-spam.\\\\\n",
        "That’s why instead of accuracy we generally turn to two other metrics shown in Fig. 4.4: precision and recall. Precision measures the percentage of the items that the system detected (i.e., the system labeled as positive) that are in fact positive (i.e., are positive according to the human gold labels). Precision is defined as\n",
        "Precision =\n",
        " true positives/\n",
        "true positives + false positives\\\\\n",
        "Recall measures the percentage of items actually present in the input that were correctly identified by the system. Recall is defined as\\\\\n",
        "Recall =true positives/\n",
        "true positives + false negatives\\\\\n",
        "The simplest of these combinations is the F-measure F1 = 2PR/(P+R)\\\\\n",
        " In macroaveraging, we compute the performance for each class, and then average over classes. In microaveraging, we collect the de- cisions for all classes into a single confusion matrix, and then compute precision and recall from that table. \\\\\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter4_8 = \"\"\" Test sets and Cross-validation\n",
        "we use the training set to train the model, then use the development test set (also called a devset) to perhaps tune some parameters\\\\\n",
        "in order to save lots of data for training, the test set (or devset) might not be large enough to be representative. Wouldn’t it be better if we could somehow use all our data for training and still use\n",
        "all our data for test? We can do this by cross-validation.\\\\\n",
        "In cross-validation, we choose a number k, and partition our data into k disjoint\n",
        "subsets called folds. \n",
        " \"\"\"\n",
        "slp_chapter4_9 = \"\"\" Statistical Significance Testing:\\\\\n",
        "In the paradigm of statistical hypothesis testing, we test this by formalizing two hypotheses.\n",
        "H0 :δ(x)≤0\n",
        "H1 : δ(x)>0 (4.20)\n",
        "The hypothesis H0, called the null hypothesis, supposes that δ(x) is actually nega- tive or zero, meaning that A is not better than B. We would like to know if we can confidently rule out this hypothesis, and instead support H1, that A is better.\n",
        "\"\"\"\n",
        "slp_chapter4_10 = \"\"\" Avoiding Harms in Classification\\\\\n",
        " \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\"\n",
        "Naïve Bayes: Assume all features are independent effects of the label\\\\\n",
        "A general Naive Bayes model:$P(Y,F_1,\\dots F_n)=P(Y)\\Pi_i P(F_i|Y)$\\\\\n",
        "$|Y|*|F|^n$ values->$|Y|*|F|*n$\n",
        "Total number of parameters is linear in number of features Model is very simplistic\n",
        " Inference for Naïve Bayes\n",
        " Goal: compute posterior distribution over label variable Y \n",
        " • Step 1: get joint probability of label and evidence for each label\n",
        "      $P(Y,F_1, \\dots F_n)=\\\\\n",
        "      [P(y_1,f_1, \\dots f_n)\\\\\n",
        "      P(y_2,f_1, \\dots f_n)\\\\\n",
        "      \\dots\\\\\n",
        "      P(y_k,f_1, \\dots f_n)\n",
        "      ]$=\\\\\n",
        "      $[P(y_1)\\Pi_i P(f_i|y_1)\\\\\n",
        "      P(y_2)\\Pi_i P(f_i|y_2)\\\\\n",
        "      \\dots\\\\\n",
        "      P(y_k)\\Pi_i P(f_i|y_k)]$\\\\\n",
        "  \n",
        "  $P(Y,f_1, \\dots f_n)/P(f_1, \\dots f_n)=P(Y|f_1, \\dots f_n)$\\\\\n",
        "  • Step 2: sum to get probability of evidence\\\\\n",
        "• Step3:normalizebydividingStep1byStep2\\\\\n",
        "Naïve Bayes for Text:\n",
        "Bag-of-words Naïve Bayes:\\\\\n",
        "• Features: Wi is the word at positon i\\\\\n",
        "• As before: predict label conditioned on feature variables (spam vs. ham) • As before: assume features are conditionally independent given label\\\\\n",
        "• New: each Wi is identically distributed\\\\\n",
        "“Tied” distributions and bag-of-words\\\\\n",
        "• Usually, each variable gets its own conditional probability distribution P(F|Y)\\\\\n",
        " • In a bag-of-words model\\\\\n",
        "• Each position is identically distributed\\\\\n",
        "• All positions share the same conditional probs P(W|Y)\\\\\n",
        "• Called “bag-of-words” because model is insensitive to word order or reordering\\\\\n",
        "\n",
        "\n",
        "Important Concepts:\\\\\n",
        "\n",
        "Data: labeled instances, e.g. emails marked spam/ham\\\\\n",
        "• Training set\n",
        "• Held out set\n",
        "• Test set\\\\\n",
        "Features: attribute-value pairs which characterize each x\\\\\n",
        "Experimentation cycle\\\\\n",
        "• Learn parameters (e.g. model probabilities) on training set\n",
        "• (Tune hyperparameters on held-out set)\n",
        "• Compute accuracy of test set\n",
        "• Very important: never “peek” at the test set!\\\\\n",
        "Evaluation\\\\\n",
        "• Accuracy: fraction of instances predicted correctly\\\\\n",
        "Overfitting and generalization\\\\\n",
        "• Want a classifier which does well on test data\n",
        "• Overfitting: fitting the training data very closely, but not generalizing well\n",
        "• We’ll investigate overfitting and generalization formally in a few lectures\\\\\n",
        "\n",
        "Generalization and Overfitting:\n",
        "Relative frequency parameters will overfit the training data!\\\\\n",
        "To generalize better: we need to smooth or regularize the estimates\\\\\n",
        "\n",
        "Parameter Estimation:Estimating the distribution of a random variable\\\\\n",
        "Empirically: use training data (learning!)\\\\\n",
        "This is the estimate that maximizes the likelihood of the data:\n",
        "$P_{ML}(x)=count(x)/total samples$\\\\\n",
        "This is the estimate that maximizes the likelihood of the data\\\\\n",
        "$L(x,\\theta))=\\Pi_i P(x_i)$\n",
        "Maximum Likelihood: Relative frequencies are the maximum likelihood estimates\n",
        " Laplace Smoothing:Pretendyousaweveryoutcome once more than you actually did\\\\\n",
        " Laplace’s estimate (extended):Pretend you saw every outcome k extra times\n",
        "Confidences from a Classifier\\\\\n",
        "The confidence of a probabilistic classifier: \n",
        "• Posterior over the top label:\n",
        "$confidence(x)=max_y P(y|x)$\n",
        "• Represent how sure the classifier is of the classification\n",
        "• Any probabilistic model will have confidences\n",
        "• No guarantee confidence is correct\n",
        "Summary\n",
        "Bayes rule lets us do diagnostic queries with causal probabilities\\\\\n",
        "The naïve Bayes assumption takes all features to be independent given the class label\\\\\n",
        "We can build classifiers out of a naïve Bayes model using training data Smoothing estimates is important in real systems\\\\\n",
        "Classifier confidences are useful, when you can get them\\\\\n",
        "\n",
        "Perceptrons\\\\\n",
        "Linear Classifiers\n",
        "Inputs are feature values; Each feature has a weight; Sum is the activation\\\\\n",
        "If the activation is:\n",
        "• Positive, output +1 • Negative, output -1\n",
        "Weights:\n",
        "Weight Updates:Start with weights = 0\n",
        "For each training instance:\n",
        "• Classify with current weights: y=+1 if $wf(x)>=0$ else y=-1\\\\\n",
        "• If correct (i.e., y=y*), no change!\\\\\n",
        "• If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1. $w=w+y^*f$\\\\\n",
        "Multiclass Decision Rule:  Prediction highest score wins: $y=argmax_y w_y f(x)$\n",
        "\n",
        "Learning: Multiclass Perceptron\n",
        "Start with all weights = 0\n",
        "Pick up training examples one by one Predict with current weights\n",
        "If wrong: lower score of wrong answer, raise score of right answer: $w_y=w_y+f(x)$, $w_{y^*}=w_{y^*}+f(x)$\\\\\n",
        "\n",
        "Properties of Perceptrons: Separability: true if some parameters get the training set perfectly correct\\\\\n",
        "Convergence: if the training is separable, perceptron will eventually converge (binary case)\\\\\n",
        "Mistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separability\\\\\n",
        "\n",
        "Problems with the Perceptron:\\\\\n",
        "Noise: if the data isn’t separable, weights might thrash\n",
        "• Averaging weight vectors over time can help (averaged perceptron)\\\\\n",
        "Mediocre generalization: finds a “barely” separating solution\\\\\n",
        "Overtraining: test / held-out accuracy usually rises, then falls\n",
        "• Overtraining is a kind of overfitting\\\\\n",
        "\n",
        "Idea: adjust the weight update to mitigate these effects\n",
        "MIRA*: choose an update size that fixes the current mistake... but, minimizes the change to w\\\\\n",
        "$w_y=w'_y+\\taut f(x)$, $w'_{y^*}=w'_{y^*}+\\taut f(x)$\\\\\n",
        "Minimum Correcting Update\n",
        "Maximum Step Size:You may not have enough features\n",
        "• Solution: cap the maximum possible value of t with some constant C\n",
        "$\\taut^*=min((w'_y-w_{y^*}f+1)/2ff),C) $\\\\\n",
        "Support Vector Machines: Maximizing the margin\\\\\n",
        "Basically, SVMs are MIRA where you optimize over all examples atonce\\\\\n",
        "Classification: Comparison\n",
        "Naïve Bayes\\\\\n",
        "• Builds a model training data\n",
        "• Gives prediction probabilities\n",
        "• Strong assumptions about feature independence • One pass through data (counting)\\\\\n",
        "Perceptrons / MIRA:\\\\\n",
        "• Makes less assumptions about data\n",
        "• Mistake-driven learning\n",
        "• Multiple passes through data (prediction) • Often more accurate\\\\\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "9b4d2966-fda0-499d-f081-3f3cb94df172"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Forms of Learning：\nWhen the output is one of a finite set of values (such as sunny/cloudy/rainy or true/false),\nthe learning problem is called classification. When it is a number (such as tomorrow’s\ntemperature, measured either as an integer or a real number), the learning problem has the\n(admittedly obscure1) name regression.\\\nThere are three types of feedback that can accompany the inputs, and that determine the\nthree main types of learning:\\\nsupervised learning\\\nunsupervised learning: The most common unsupervised learning task is clustering\\\n reinforcement learning\n\n \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Supervised Learning:\nThe function h is called a hypothesis about the world. It is drawn from a hypothesis space\nH of possible functions.\\\nWith alternative vocabulary, we can say that h is a model of the data, drawn from a model\nclass H, or we can say a function drawn from a function class. We call the output yi the\nground truth—the true answer we are asking our model to predict.\\\n we can perform exploratory data analysis:examining the data with statistical tests and visualizations—histograms, scatter plots, box\nplots—to get a feel for the data, and some insight into what hypothesis space might be\nappropriate.\\\n How do we choose a good hypothesis from within the hypothesis space? We could hope for\na consistent hypothesis: and h such that each xi in the training set has h(xi) = yi. With\ncontinuous-valued outputs we can’t expect an exact match to the ground truth; instead we\nlook for a best-fit function for which each h(xi) is close to yi\\\nOne way to analyze hypothesis spaces is by the bias they impose (regardless of the training\ndata set) and the variance they produce (from one training set to another).\\\nOne way to analyze hypothesis spaces is by the bias they impose (regardless of the training\ndata set) and the variance they produce (from one training set to another).\\\n the small difference in the data set translates into a small difference in the\nhypothesis. We call that low variance.\\\nOften there is a bias–variance tradeoff: a choice between more complex, low-bias\nhypotheses that fit the training data well and simpler, low-variance hypotheses that may\ngeneralize better.\\\n\n\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Learning Decision Trees\nA decision tree is a representation of a function that maps a vector of attribute values to a\nsingle output value—a “decision.”\\\n  Learning decision trees from examples:\nLEARN-DECISION-TREE algorithm adopts a greedy divide-and-conquer strategy: always test the\nmost important attribute first, then recursively solve the smaller subproblems that are\ndefined by the possible results of the test.\\\n There are\nfour cases to consider for these recursive subproblems:\\\n1. If the remaining examples are all positive (or all negative), then we are done: we\ncan answer Yes or No.\\\n2. If there are some positive and some negative examples, then choose the best\nattribute to split them.\\\n3. If there are no examples left, it means that no example has been observed for this\ncombination of attribute values, and we return the most common output value from\nthe set of examples that were used in constructing the node’s parent.\\\n4. If there are no attributes left, but both positive and negative examples, it means that\nthese examples have exactly the same description, but different classifications. This\ncan happen because there is an error or noise in the data\\\nWe can evaluate the performance of a learning algorithm with a learning curve,\\\n Choosing attribute tests:The decision tree learning algorithm chooses the attribute with the highest IMPORTANCE. We\nwill now show how to measure importance, using the notion of information gain, which is\ndefined in terms of entropy\\\nEntropy: H(V ) = −∑k P(vk)log2 P(vk).\\\n define as the entropy of a Boolean random variable that is true with\nprobability:B(q) = −(q log2 q + (1 − q)log2(1 − q)).\\\n If a training\nset contains positive examples and negative examples:\nH(Output) = B(p/(p+n) ).\nRemainder(A) = d∑k=1  pk + nk/ p + n B(pk/ pk + nk ). \\\nGain(A) = B( p/p + n) − Remainder(A).\\\n Generalization and overfitting: For decision trees, a technique called decision tree pruning combats overfitting.:\n  how large a gain should we require in order to\nsplit on a particular attribute?using a statistical significance test.\\\n combine them using an approach called early stopping\\\n Broadening the applicability of decision trees:\n 1, MISSING DATA;2, CONTINUOUS AND MULTIVALUED INPUT ATTRIBUTES:\\\n CONTINUOUS-VALUED OUTPUT ATTRIBUTE:The name CART, standing for Classification And Regression Trees, is used to cover\nboth classes.\n \n\n## AIMA Chapter 5.5 Stochastic Games\n Model Selection and Optimization:\nFirst we will make the assumption that the future examples will be like the past. We call this\nthe stationarity assumption;;\\\n a training set to create the hypothesis, and a test set to evaluate it.;\\\n we might want to compare two completely\ndifferent machine learning models: We call these “knobs” hyperparameters—parameters\nof the model class, not of the individual model.\\\n1. A training set to train candidate models.\n2. A validation set, also known as a development set or dev set, to evaluate the\ncandidate models and choose the best one.\\\nWhat if we don’t have enough data to make all three of these data sets? We can squeeze\nmore out of the data using a technique called k-fold cross-validation\\\nThe extreme is k = n, also known as leave-one-out cross-validation or LOOCV. Even with\ncross-validation, we still need a separate test set.\\\n We can think of the task of finding a good hypothesis as two\nsubtasks: model selection4 chooses a good hypothesis space, and optimization (also called\ntraining) finds the best hypothesis within that space.\\\n\nModel selection\\\nWe say that a model that exactly fits all the training data has interpolated the data\n\n\n Regularization\\\nAnother way to simplify models is to reduce the dimensions that the models work with. A\nprocess of feature selection\\\n\nHyperparameter tuning:The simplest approach to hyperparameter tuning is hand-tuning: guess some parameter\nvalues based on past experience, train a model, measure its performance on the validation\ndata, analyze the results, and use your intuition to suggest new parameter values.\\\na more systematic approach called grid search is appropriate:\n Bayesian optimization treats the task of choosing good hyperparameter values as\na machine learning problem in itself.\\\nAn alternative to Bayesian optimization is population-based training (PBT). PBT starts by\nusing random search to train (in parallel) a population of models, each with different\nhyperparameter values.\n\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\nThe Theory of Learning:\nany hypothesis that is consistent with a sufficiently large set of training examples is\nunlikely to be seriously wrong: that is, it must be probably approximately correct (PAC).\\\n One can think of an\napproximately correct hypothesis as being “close” to the true function in hypothesis space: it\nlies inside what is called the ϵ-ball around the true function f. The hypothesis space outside\nthis ball is called H bad.']\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n 19.6 Linear Regression and Classification\\ Univariate linear regression\\ The task of\nfinding the hw that best fits these data is called linear regression.\\\nWe see that the loss function is convex, this is\ntrue for every linear regression problem with an loss function, and implies that there are\nno local minima.\\\n \n Gradient descent\\\n The parameter which we called the step size in Section 4.2 , is usually called the\nlearning rate when we are trying to minimize loss in a learning problem.\\\nFor univariate regression, the loss is quadratic, so the partial derivative will be linear. (The\nonly calculus you need to know is the chain rule\\\nThese updates constitute the batch gradient descent learning rule for univariate linear\nregression (also called deterministic gradient descent).\\\nA faster variant is called stochastic gradient descent or SGD: it randomly selects a small\nnumber of training examples at each step, and updates\\\nA step that covers all the training examples is called an epoch.\\\n\nMultivariable linear regression\\\nWe call the expression (X⊤X)−1X⊤\n the pseudoinverse of the data matrix, w∗ = (X⊤X)−1X⊤y. is called the normal equation.\\\nWith univariate linear regression we didn’t have to worry about overfitting. But with\nmultivariable linear regression in high-dimensional spaces it is possible that some\ndimension that is actually irrelevant appears by chance to be useful, resulting in overfitting.\\\n$Complexity(h_w) = \\Sigma |w_i|^q. $\nAs with loss functions, with q = 1 we have L1 regularization9, which minimizes the sum of\nthe absolute values; with q = 2, L2 regularization minimizes the sum of squares. Which\nregularization function should you pick? That depends on the specific problem, but L1\nregularization has an important advantage: it tends to produce a sparse model. That is, it\noften sets many weights to zero, effectively declaring the corresponding attributes to be\ncompletely irrelevant\\\n\n Linear classifiers with a hard threshold\\\nA decision boundary is a line (or a surface, in higher dimensions) that separates the two\nclasses.   A linear decision\nboundary is called a linear separator and data that admit such a separator are called\nlinearly separable. \\\n\nAlternatively, we can think of h as the result of passing the linear function w ⋅ x through a\nthreshold function\\\na simple weight update rule that converges to a solution—that is, to a\nlinear separator that classifies the data perfectly—provided the data are linearly separable.\nFor a single example (x, y), we have\nwi ← wi + α (y − hw(x)) × xi\\\nThis\nrule is called the perceptron learning rule\\\n\n Linear classification with logistic regression\\\n We have seen that passing the output of a linear function through the threshold function\ncreates a linear classifier; yet the hard nature of the threshold causes some problems\\\nAll of these issues can be resolved to a large extent by softening the threshold function—\napproximating the hard threshold with a continuous, differentiable function.\\\n\n  \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n  \nNaive Bayes Classifiers\n multinomial naive Bayes classifier:\nThe intuition of the classifier is shown in Fig. 4.1. We represent a text document\nas if it were a bag-of-words, that is, an unordered set of words with their position ignored, keeping only their frequency in the document.\\\n\nNaive Bayes is a probabilistic classifier, meaning that for a document d, out of all classes c ∈ C the classifier returns the class cˆ which has the maximum posterior ˆ probability given the document. \\\ncˆ = argmaxP(c|d)\\\ncˆ = argmaxP(c|d) = argmaxP(d|c)P(c)\\\n\nNaive Bayes classifiers therefore make two simplifying assumptions:The first is the bag of words assumption discussed intuitively above,; The second is commonly called the naive Bayes assumption: this is the condi- tional independence assumption \n$c_{NB} = argmax P(c) 􏰄\\Pi P(f|c)$\ncNB = argmax P(c) 􏰄 P(wi |c)\n Classifiers that use a linear combination of the inputs to make a classification decision —like naive Bayes and also logistic regression— are called linear classifiers.\n \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Training the Naive Bayes Classifier \nNote once again that it is crucial that the vocabulary V consists of the union of all the word types in all classes, not just the words in one class c (try to convince yourself why this must be true\\\nWhat do we do about words that occur in our test data but are not in our vocab- ulary at all because they did not occur in any training document in any class? The solution for such unknown words is to ignore them—remove them from the test document and not include any probability for them at all.\\\nFinally, some systems choose to completely ignore another class of words: stop words\\\n\n\n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Optimizing for Sentiment Analysis：First, for sentiment classification and a number of other text classification tasks, whether a word occurs or not seems to matter more than its frequency.、、\n This variant is called binary\nmultinomial naive Bayes or binary NB\n \n \n \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Naive Bayes for other text classification tasks：\nthe most effective naive Bayes features are not words at all, but character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’, ‘thei’), or, even simpler byte n-grams,；\n \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Naive Bayes as a Language Model:\\Specifically, a naive Bayes model can be viewed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. \\Since the likelihood features from the naive Bayes model assign a probability to each word P(word|c), the model also assigns a probability to each sentence:\n$P(s|c) = 􏰄 \\Pi P(w_i|c)  i∈ positions$\nonce we multiply in the prior a full naive Bayes model might well make a different classification decision.\\\n\n\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Evaluation: Precision, Recall, F-measure;\refer to these human labels as the gold labels. \\To evaluate any system for detecting things, we start by building a confusion matrix\\e.g.true positives are documents that are indeed spam (indicated by human-created gold labels) that our system correctly said were spam. False negatives are documents that are indeed spam but our system incorrectly labeled as non-spam.\\That’s why instead of accuracy we generally turn to two other metrics shown in Fig. 4.4: precision and recall. Precision measures the percentage of the items that the system detected (i.e., the system labeled as positive) that are in fact positive (i.e., are positive according to the human gold labels). Precision is defined as\nPrecision =\n true positives/\ntrue positives + false positives\\\nRecall measures the percentage of items actually present in the input that were correctly identified by the system. Recall is defined as\\\nRecall =true positives/\ntrue positives + false negatives\\\nThe simplest of these combinations is the F-measure F1 = 2PR/(P+R)\\\n In macroaveraging, we compute the performance for each class, and then average over classes. In microaveraging, we collect the de- cisions for all classes into a single confusion matrix, and then compute precision and recall from that table. \\\n\n\n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Test sets and Cross-validation\nwe use the training set to train the model, then use the development test set (also called a devset) to perhaps tune some parameters\\\nin order to save lots of data for training, the test set (or devset) might not be large enough to be representative. Wouldn’t it be better if we could somehow use all our data for training and still use\nall our data for test? We can do this by cross-validation.\\\nIn cross-validation, we choose a number k, and partition our data into k disjoint\nsubsets called folds. \n \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Statistical Significance Testing:\\In the paradigm of statistical hypothesis testing, we test this by formalizing two hypotheses.\nH0 :δ(x)≤0\nH1 : δ(x)>0 (4.20)\nThe hypothesis H0, called the null hypothesis, supposes that δ(x) is actually nega- tive or zero, meaning that A is not better than B. We would like to know if we can confidently rule out this hypothesis, and instead support H1, that A is better.\n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Avoiding Harms in Classification\\ \n\n## Module 12 Lecture Notes\n\nNaïve Bayes: Assume all features are independent effects of the label\\\nA general Naive Bayes model:$P(Y,F_1,\\dots F_n)=P(Y)\\Pi_i P(F_i|Y)$\\\n$|Y|*|F|^n$ values->$|Y|*|F|*n$\nTotal number of parameters is linear in number of features Model is very simplistic\n Inference for Naïve Bayes\n Goal: compute posterior distribution over label variable Y \n • Step 1: get joint probability of label and evidence for each label\n      $P(Y,F_1, \\dots F_n)=\\\n      [P(y_1,f_1, \\dots f_n)\\\n      P(y_2,f_1, \\dots f_n)\\\n      \\dots\\\n      P(y_k,f_1, \\dots f_n)\n      ]$=\\\n      $[P(y_1)\\Pi_i P(f_i|y_1)\\\n      P(y_2)\\Pi_i P(f_i|y_2)\\\n      \\dots\\\n      P(y_k)\\Pi_i P(f_i|y_k)]$\\\n  \n  $P(Y,f_1, \\dots f_n)/P(f_1, \\dots f_n)=P(Y|f_1, \\dots f_n)$\\\n  • Step 2: sum to get probability of evidence\\\n• Step3:normalizebydividingStep1byStep2\\\nNaïve Bayes for Text:\nBag-of-words Naïve Bayes:\\\n• Features: Wi is the word at positon i\\\n• As before: predict label conditioned on feature variables (spam vs. ham) • As before: assume features are conditionally independent given label\\\n• New: each Wi is identically distributed\\\n“Tied” distributions and bag-of-words\\\n• Usually, each variable gets its own conditional probability distribution P(F|Y)\\\n • In a bag-of-words model\\\n• Each position is identically distributed\\\n• All positions share the same conditional probs P(W|Y)\\\n• Called “bag-of-words” because model is insensitive to word order or reordering\\\n\n\nImportant Concepts:\\\n\nData: labeled instances, e.g. emails marked spam/ham\\\n• Training set\n• Held out set\n• Test set\\\nFeatures: attribute-value pairs which characterize each x\\\nExperimentation cycle\\\n• Learn parameters (e.g. model probabilities) on training set\n• (Tune hyperparameters on held-out set)\n• Compute accuracy of test set\n• Very important: never “peek” at the test set!\\\nEvaluation\\\n• Accuracy: fraction of instances predicted correctly\\\nOverfitting and generalization\\\n• Want a classifier which does well on test data\n• Overfitting: fitting the training data very closely, but not generalizing well\n• We’ll investigate overfitting and generalization formally in a few lectures\\\n\nGeneralization and Overfitting:\nRelative frequency parameters will overfit the training data!\\\nTo generalize better: we need to smooth or regularize the estimates\\\n\nParameter Estimation:Estimating the distribution of a random variable\\\nEmpirically: use training data (learning!)\\\nThis is the estimate that maximizes the likelihood of the data:\n$P_{ML}(x)=count(x)/total samples$\\\nThis is the estimate that maximizes the likelihood of the data\\\n$L(x,\theta))=\\Pi_i P(x_i)$\nMaximum Likelihood: Relative frequencies are the maximum likelihood estimates\n Laplace Smoothing:Pretendyousaweveryoutcome once more than you actually did\\\n Laplace’s estimate (extended):Pretend you saw every outcome k extra times\nConfidences from a Classifier\\\nThe confidence of a probabilistic classifier: \n• Posterior over the top label:\n$confidence(x)=max_y P(y|x)$\n• Represent how sure the classifier is of the classification\n• Any probabilistic model will have confidences\n• No guarantee confidence is correct\nSummary\nBayes rule lets us do diagnostic queries with causal probabilities\\\nThe naïve Bayes assumption takes all features to be independent given the class label\\\nWe can build classifiers out of a naïve Bayes model using training data Smoothing estimates is important in real systems\\\nClassifier confidences are useful, when you can get them\\\n\nPerceptrons\\\nLinear Classifiers\nInputs are feature values; Each feature has a weight; Sum is the activation\\\nIf the activation is:\n• Positive, output +1 • Negative, output -1\nWeights:\nWeight Updates:Start with weights = 0\nFor each training instance:\n• Classify with current weights: y=+1 if $wf(x)>=0$ else y=-1\\\n• If correct (i.e., y=y*), no change!\\\n• If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1. $w=w+y^*f$\\\nMulticlass Decision Rule:  Prediction highest score wins: $y=argmax_y w_y f(x)$\n\nLearning: Multiclass Perceptron\nStart with all weights = 0\nPick up training examples one by one Predict with current weights\nIf wrong: lower score of wrong answer, raise score of right answer: $w_y=w_y+f(x)$, $w_{y^*}=w_{y^*}+f(x)$\\\n\nProperties of Perceptrons: Separability: true if some parameters get the training set perfectly correct\\\nConvergence: if the training is separable, perceptron will eventually converge (binary case)\\\nMistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separability\\\n\nProblems with the Perceptron:\\\nNoise: if the data isn’t separable, weights might thrash\n• Averaging weight vectors over time can help (averaged perceptron)\\\nMediocre generalization: finds a “barely” separating solution\\\nOvertraining: test / held-out accuracy usually rises, then falls\n• Overtraining is a kind of overfitting\\\n\nIdea: adjust the weight update to mitigate these effects\nMIRA*: choose an update size that fixes the current mistake... but, minimizes the change to w\\\n$w_y=w'_y+\taut f(x)$, $w'_{y^*}=w'_{y^*}+\taut f(x)$\\\nMinimum Correcting Update\nMaximum Step Size:You may not have enough features\n• Solution: cap the maximum possible value of t with some constant C\n$\taut^*=min((w'_y-w_{y^*}f+1)/2ff),C) $\\\nSupport Vector Machines: Maximizing the margin\\\nBasically, SVMs are MIRA where you optimize over all examples atonce\\\nClassification: Comparison\nNaïve Bayes\\\n• Builds a model training data\n• Gives prediction probabilities\n• Strong assumptions about feature independence • One pass through data (counting)\\\nPerceptrons / MIRA:\\\n• Makes less assumptions about data\n• Mistake-driven learning\n• Multiple passes through data (prediction) • Often more accurate\\\n\n \n\n## Module 12 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\"\n",
        "A generative model like naive Bayes makes use of this likelihood term, which expresses how to generate the features of a document if we knew it was of class c.\n",
        "By contrast a discriminative model in this text categorization scenario attempts to directly compute P(c|d). Perhaps it will learn to assign a high weight to document features that directly improve its ability to discriminate between possible classes, even if it couldn’t generate an example of one of the classes.\\\\\n",
        "\n",
        "\n",
        " Classification: the sigmoid:\n",
        "Logistic regression solves this task by learning, from a training set, a vector of weights and a bias term. \\\\\n",
        "z = w·x+b\\\\\n",
        "To create a probability, we’ll pass z through the sigmoid function, σ(z). The sigmoid function (named because it looks like an s) is also called the logistic func- tion, and gives logistic regression its name.\\\\\n",
        "σ(z)= 1 / 1+exp(−z)\\\\\n",
        "We call .5 the decision boundary\\\\\n",
        "For logistic regression and naive Bayes these combination features or feature interactions have to be designed by hand.\\\\\n",
        "For many tasks (especially when feature values can reference specific words) we’ll need large numbers of features. Often these are created automatically via fea- ture templates, abstract specifications of features.\\\\\n",
        "In order to avoid the extensive human effort of feature design, recent research in NLP has focused on representation learning: ways to learn features automatically in an unsupervised way from the input.\\\\\n",
        "Logistic regression has a number of advantages over naive Bayes. Naive Bayes has overly strong conditional independence assumptions. \\\\\n",
        "Furthermore, naive Bayes can work extremely well (some- times even better than logistic regression) on very small datasets (Ng and Jordan, 2002) or short documents (Wang and Manning, 2012). Furthermore, naive Bayes is easy to implement and very fast to train (there’s no optimization step).\\\\\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\" Learning in Logistic Regression\n",
        "the distance between the system output and the gold output, and we call this distance\n",
        "loss the loss function or the cost function. \\\\\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter5_3 = \"\"\" The cross-entropy loss function\\\\\n",
        "This is called conditional maximum likelihood estimation: we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x. The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.\\\\\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter5_4 = \"\"\" Gradient Descent:\\\\\n",
        "For logistic regression, this loss function is conveniently convex. A convex func- tion has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum.\\\\\n",
        "\n",
        "The magnitude of the amount to move in gradient descent is the value of the slope d L( f (x; w), y) weighted by a learning rate η . A higher (faster) learning rate means that we should move w more on each step.\\\\\n",
        "\n",
        "It is common to start with a higher learning rate and then slowly decrease it, so that it is a function of the iteration k of training; the notation ηk can be used to mean the value of the learning rate at iteration k.\\\\\n",
        "\n",
        "Mini-batch training\\\\\n",
        "For example in batch training we compute the gradient over the entire dataset. By seeing so many examples, batch training offers a superb estimate of which di- rection to move the weights, at the cost of spending a lot of time processing every single example in the training set to compute this perfect direction.\\\\\n",
        "The mini-batch gradient is the average of the individual gradients\\\\\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter5_5 = \"\"\" Regularization:\\\\A good model should be able to generalize well from the training data to the unseen test set, but a model that overfits will have poor generalization.\n",
        "To avoid overfitting, a new regularization term R(θ) is added to the objective function\\\\\n",
        "L2 regularization is a quadratic function of the weight values, named because it uses the (square of the) L2 norm of the weight values. The L2 norm, ||θ||2, is the same as the Euclidean distance of the vector θ from the origin. If θ consists of n weights,\\\\\n",
        "L1 regularization is a linear function of the weight values, named after the L1 norm ||W||1, the sum of the absolute values of the weights, or Manhattan distance\\\\\n",
        "These kinds of regularization come from statistics, where L1 regularization is called lasso regression (Tibshirani, 1996) and L2 regularization is called ridge regression, and both are commonly used in language processing.\\\\\n",
        " L2 regularization is easier to optimize because of its simple derivative (the derivative of θ2 is just 2θ), while L1 regularization is more complex (the derivative of |θ | is non-continuous at zero).\\\\\n",
        " But where L2 prefers weight vectors with many small weights, L1 prefers sparse solutions with some larger weights but many more weights set to zero. Thus L1 regularization leads to much sparser weight vectors, that is, far fewer features.\\\\\n",
        "Both L1 and L2 regularization have Bayesian interpretations as constraints on the prior of how weights should look. L1 regularization can be viewed as a Laplace prior on the weights. L2 regularization corresponds to assuming that weights are distributed according to a Gaussian distribution with mean μ = 0.\\\\\n",
        "\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter5_6 = \"\"\" Multinomial logistic regression:\n",
        "In such cases we use multinomial logistic regression, also called softmax re- gression (or, historically, the maxent classifier).\n",
        "\\\\\n",
        "For a vector z of dimensionality k, the softmax is defined as:\n",
        "\n",
        "$softmax(zi) =exp(zi)/ \\Sigma􏰂^k_{j=1} exp(zj) 1 ≤ i ≤ k$\n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\" Units： tanh : the tanh func- tion has the nice properties of being smoothly differentiable and mapping outlier(a problem called the vanishing gradient problem.)\\\\\n",
        "\n",
        "ReLU:y = max(z, 0),The rectifier function, on the other hand has nice properties that result from it being very close to linear.\\\\\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter7_2 = \"\"\" The XOR problem\\\\\n",
        "This example was first shown for the perceptron, which is a very simple neural unit that has a binary output and does not have a non-linear activation function\\\\\n",
        "It turns out, however, that it’s not possible to build a perceptron to compute logical XOR! \n",
        "The intuition behind this important result relies on understanding that a percep- tron is a linear classifier. \\\\\n",
        "We say that XOR is not a linearly separable function. \\\\\n",
        "The solution: neural networks:\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter7_3 = \"\"\" Feedforward Neural Networks:\\\\\n",
        "\n",
        "units in modern multilayer networks aren’t perceptrons (perceptrons are purely linear, but modern networks are made up of units with non-linearities like sigmoids\\\\\n",
        "\n",
        "\n",
        "The core of the neural network is the hidden layer h formed of hidden units hi, each of which is a neural unit as described in Section 7.1, taking a weighted sum of its inputs and then applying a non-linearity. \n",
        " \"\"\"\n",
        "slp_chapter7_4 = \"\"\" Feedforward networks for NLP: Classification:\\\\\n",
        "\n",
        "The idea of using word2vec or GloVe embeddings as our input representation — and more generally the idea of relying on another algorithm to have already learned an embedding representation for our input words — is called pretraining. \\\\\n",
        " Instead of using hand-built human-engineered features as the input to our classifier, we draw on deep learning’s ability to learn features from the data by representing words as word2vec or GloVe embeddings (Chapter 6). For a text with n input words/tokens w1,...,wn, the input vector will be the concatenated embeddings of the n words: [ew1 ; ...; ewn ].\\\\\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter7_5 = \"\"\" Feedforward Neural Language Modeling:\n",
        "Compared to n-gram models, neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction.\\\\\n",
        "\n",
        "Neural language models represent words in this prior context by their embed- dings, rather than just by their word identity as used in n-gram language models. Using embeddings allows neural language models to generalize better to unseen data. \\\\\n",
        " \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Training Neural Nets:\n",
        "How do we partial out the loss over all those intermediate layers? The answer is the algorithm called error backpropagation or backward differentiation.\\\\\n",
        "\n",
        "The cross-entropy loss that is used in neural networks is the same one we saw for logistic regression.\\\\\n",
        "\n",
        "the cross-entropy loss is simply the log of the output probability correspond- ing to the correct class, and we therefore also call this the negative log likelihood loss:\n",
        "LCE(yˆ,y) = −logyˆi, (where i is the correct class)\\\\\n",
        "\n",
        "The solution to computing this gradient is an algorithm called error backprop- agation or backprop (Rumelhart et al., 1986). While backprop was invented spe- cially for neural networks, it turns out to be the same as a more general procedure called backward differentiation, which depends on the notion of computation graphs. \\\\\n",
        "\n",
        "Backward differentiation on computation graphs\\\\\n",
        "Backwards differentiation makes use of the chain rule in calculus\\\\\n",
        "\n",
        "Various forms of regularization are used to prevent overfitting. One of the most important is dropout: randomly dropping some units and their connections from the network during training (Hinton et al. 2012, Srivastava et al. 2014). Tuning of hyperparameters is also important.\\\\\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter7_7 = \"\"\" \n",
        "Training the neural language model:\n",
        "\n",
        "Training the parameters to minimize loss will result both in an algorithm for language modeling (a word predictor) but also a new set of embeddings E that can be used as word representations for other tasks. \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\"Logistic Regression:\n",
        "Classifier components:\n",
        "1. A feature representation of the input.Foreachinputobservation x(i), this will be a vector of features [x1,x2,...,xn].\n",
        "2. A classification function that computes the estimated class 𝑦 via p(y|x).\n",
        "3. Annobjective function for learning, usually involving minimizing error on training examples.\n",
        "4. Analgorithm for optimizing the objective function.\\\\\n",
        "Sentiment classifier\\\\\n",
        "Decision boundary\\\\\n",
        "\n",
        "$\\hat y=1 \\ if\\  P(y=1|x)>0.5\\ else\\ 0$\\\\\n",
        "Extracting Features\\\\\n",
        "$z=\\Sigma_i w_ix_i+b$, $\\sigma(z)=1/(1+e^{-(wx+b)})$\\\\\n",
        "1. An objective function or loss function that tells us distance between the system output and the gold output. We will use cross-entropy loss.\\\\\n",
        "2. An algorithm for optimizing the objective function. We will use stochastic gradient descent to minimize the loss function.\n",
        "Loss functions: eg. mean squared error \n",
        "$L_{MSE}(\\hat y,y)=1/2(\\hat y-y)^2$\\\\\n",
        "Loss functions for probabilistic classification:cross entropy loss\\\\\n",
        "For one observation x, let’s maximize the probability of the correct label p(y|x).\n",
        "$p(y|x)=\\hat y^y(1-\\hat y)^{1-y}$\\\\\n",
        "Change to logs (still maximizing)\n",
        "$logp(y|x)=log[\\hat y^y(1-\\hat y)^{1-y}]=ylog\\hat y +(1-y)log(1-\\hat y)$\\\\\n",
        "Cross-entropy loss: The result is cross-entropy loss:$-logp(y|x)=-(ylog\\hat y +(1-y)log(1-\\hat y))$\\\\\n",
        "$-logp(y|x)=-(ylog(wx+b)) +(1-y)log(1-(wx+b))))$\\\\\n",
        "Gradient descent is a method that finds a minimum of a function by figuring out in which direction (in the space of the parameters θ) the function’s slope is rising the most steeply, and moving in the opposite direction.\\\\\n",
        " For logistic regression, this loss function is conveniently convex.\n",
        "A convex function has just one minimum, so there are no local minima\n",
        "to get stuck in.\n",
        "So gradient descent starting from any point is guaranteed to find the minimum.\\\\\n",
        " $w^{t+1}=w^t-\\eta d/dwf(x;w)$\\\\\n",
        " The loss for a batch of data or an entire dataset is just the average loss over the m examples\\\\\n",
        "\n",
        " $\\alpha Cost(w,b)/ \\alpha w_j=\\Sigma^m_{i=1}[\\sigma (wx^{(i)}+b)-y^{i}]x_j^{(i)}$\\\\\n",
        "Stochastic gradient descent chooses a single random example at a time and updates its weights on that example. As a result the updates can fluctuate.\\\\\n",
        "An alternate is batch training, which computes the gradient over the entire dataset. This gives a much better estimate of which direction to move the weights but takes a long time to compute.\\\\\n",
        "A commonly used compromise is mini-batch training, where we train on a small batch. \\\\\n",
        "Overfitting :In logistic regression, if a feature only occurs in one class then it will get a high weight. Sometimes we are just modelling noisy factors that just accidentally correlate with the class.\\\\\n",
        "Regularization: Lasso regression uses L1 regularization Ridge regression uses L2 regularization\\\\\n",
        "Multinomial logistic regression: To get a proper probability, we will use a generalization of the sigmoid function\n",
        "called the softmax function.$softmax(z_i) = e_{z_j}/ \\Sigma_{j=1}^k e^{z_j} 1 ≤ i ≤ k$\\\\\n",
        "Problem: a small change in the weights or bias of any single perceptron in the network can causes the output to completely flip from 0 to 1.\n",
        "Solution: sigmoid neuron\\\\\n",
        "Smoothness is crucial\\\\\n",
        "Activation Functions:neuron units\n",
        "apply a non-linear function f to z.\\\\\n",
        "The sigmoid function\n",
        "The tanh function\n",
        "The rectified linear unit ReLU\\\\\n",
        "The simplest kind of NN is the Feed-Forward Neural Network Multilayer network, all units are usually fully-connected, and no cycles.\\\\\n",
        "The computation has three steps:\\\\\n",
        "1. multiplying the weight matrix by the input vector x, 2. adding the bias vector b, and\n",
        "3. applying the activation function g (such as Sigmoid)\\\\\n",
        "Here are the final equations for a feedforward network with a single hidden layer, which takes an input vector x, outputs a probability distribution y, and is parameterized by weight matrices W and U and a bias vector b:\\\\\n",
        "h = σ(Wx+b)\\\\\n",
        "z = Uh\\\\\n",
        "y = softmax(z)\\\\\n",
        "Training Neural Nets:\n",
        "A loss function – cross-entropy loss\n",
        "An optimization algorithm – gradient descent\n",
        "A way of computing the gradient of the loss function – error propagation\\\\\n",
        "Although backpropagation was invented for neural nets, it is related to general procedure called backward differentiation, which depends on the notion of computation graphs.\\\\\n",
        "Chain rule:For a composite function f(x) = u(v(x)), the derivative of f(x) is:\n",
        "$df/dx=du/dv * dv/dx$\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "c02a29c4-4002-4e4c-85e4-6d9f6a79ef86"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\nA generative model like naive Bayes makes use of this likelihood term, which expresses how to generate the features of a document if we knew it was of class c.\nBy contrast a discriminative model in this text categorization scenario attempts to directly compute P(c|d). Perhaps it will learn to assign a high weight to document features that directly improve its ability to discriminate between possible classes, even if it couldn’t generate an example of one of the classes.\\\n\n\n Classification: the sigmoid:\nLogistic regression solves this task by learning, from a training set, a vector of weights and a bias term. \\\nz = w·x+b\\\nTo create a probability, we’ll pass z through the sigmoid function, σ(z). The sigmoid function (named because it looks like an s) is also called the logistic func- tion, and gives logistic regression its name.\\\nσ(z)= 1 / 1+exp(−z)\\\nWe call .5 the decision boundary\\\nFor logistic regression and naive Bayes these combination features or feature interactions have to be designed by hand.\\\nFor many tasks (especially when feature values can reference specific words) we’ll need large numbers of features. Often these are created automatically via fea- ture templates, abstract specifications of features.\\\nIn order to avoid the extensive human effort of feature design, recent research in NLP has focused on representation learning: ways to learn features automatically in an unsupervised way from the input.\\\nLogistic regression has a number of advantages over naive Bayes. Naive Bayes has overly strong conditional independence assumptions. \\\nFurthermore, naive Bayes can work extremely well (some- times even better than logistic regression) on very small datasets (Ng and Jordan, 2002) or short documents (Wang and Manning, 2012). Furthermore, naive Bayes is easy to implement and very fast to train (there’s no optimization step).\\\n\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Learning in Logistic Regression\nthe distance between the system output and the gold output, and we call this distance\nloss the loss function or the cost function. \\\n\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n The cross-entropy loss function\\This is called conditional maximum likelihood estimation: we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x. The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss. \n\n## SLP Chapter 5.4 Gradient Descent\n Gradient Descent:\\For logistic regression, this loss function is conveniently convex. A convex func- tion has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum.The magnitude of the amount to move in gradient descent is the value of the slope d L( f (x; w), y) weighted by a learning rate η . A higher (faster) learning rate means that we should move w more on each step.\\\n\nIt is common to start with a higher learning rate and then slowly decrease it, so that it is a function of the iteration k of training; the notation ηk can be used to mean the value of the learning rate at iteration k.\\\n\nMini-batch training\\\nFor example in batch training we compute the gradient over the entire dataset. By seeing so many examples, batch training offers a superb estimate of which di- rection to move the weights, at the cost of spending a lot of time processing every single example in the training set to compute this perfect direction.\\\nThe mini-batch gradient is the average of the individual gradients\\\n\n \n\n## SLP Chapter 5.5 Regularization\n Regularization:\\A good model should be able to generalize well from the training data to the unseen test set, but a model that overfits will have poor generalization.\nTo avoid overfitting, a new regularization term R(θ) is added to the objective function\\\nL2 regularization is a quadratic function of the weight values, named because it uses the (square of the) L2 norm of the weight values. The L2 norm, ||θ||2, is the same as the Euclidean distance of the vector θ from the origin. If θ consists of n weights,\\\nL1 regularization is a linear function of the weight values, named after the L1 norm ||W||1, the sum of the absolute values of the weights, or Manhattan distance\\\nThese kinds of regularization come from statistics, where L1 regularization is called lasso regression (Tibshirani, 1996) and L2 regularization is called ridge regression, and both are commonly used in language processing.\\\n L2 regularization is easier to optimize because of its simple derivative (the derivative of θ2 is just 2θ), while L1 regularization is more complex (the derivative of |θ | is non-continuous at zero).\\\n But where L2 prefers weight vectors with many small weights, L1 prefers sparse solutions with some larger weights but many more weights set to zero. Thus L1 regularization leads to much sparser weight vectors, that is, far fewer features.\\\nBoth L1 and L2 regularization have Bayesian interpretations as constraints on the prior of how weights should look. L1 regularization can be viewed as a Laplace prior on the weights. L2 regularization corresponds to assuming that weights are distributed according to a Gaussian distribution with mean μ = 0.\\\n\n\n \n\n## SLP Chapter 5.6 Multinomial logistic regression\n Multinomial logistic regression:\nIn such cases we use multinomial logistic regression, also called softmax re- gression (or, historically, the maxent classifier).\n\\\nFor a vector z of dimensionality k, the softmax is defined as:\n\n$softmax(zi) =exp(zi)/ \\Sigma􏰂^k_{j=1} exp(zj) 1 ≤ i ≤ k$\n\n\n## SLP Chapter 7.1 Units\n Units： tanh : the tanh func- tion has the nice properties of being smoothly differentiable and mapping outlier(a problem called the vanishing gradient problem.)ReLU:y = max(z, 0),The rectifier function, on the other hand has nice properties that result from it being very close to linear.\\\n\n \n\n## SLP Chapter 7.2 The XOR problem\n The XOR problem\\This example was first shown for the perceptron, which is a very simple neural unit that has a binary output and does not have a non-linear activation function\\It turns out, however, that it’s not possible to build a perceptron to compute logical XOR! \nThe intuition behind this important result relies on understanding that a percep- tron is a linear classifier. \\\nWe say that XOR is not a linearly separable function. \\\nThe solution: neural networks:\n\n\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Feedforward Neural Networks:units in modern multilayer networks aren’t perceptrons (perceptrons are purely linear, but modern networks are made up of units with non-linearities like sigmoids\\\n\n\nThe core of the neural network is the hidden layer h formed of hidden units hi, each of which is a neural unit as described in Section 7.1, taking a weighted sum of its inputs and then applying a non-linearity. \n \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Feedforward networks for NLP: Classification:The idea of using word2vec or GloVe embeddings as our input representation — and more generally the idea of relying on another algorithm to have already learned an embedding representation for our input words — is called pretraining. \\\n Instead of using hand-built human-engineered features as the input to our classifier, we draw on deep learning’s ability to learn features from the data by representing words as word2vec or GloVe embeddings (Chapter 6). For a text with n input words/tokens w1,...,wn, the input vector will be the concatenated embeddings of the n words: [ew1 ; ...; ewn ].\\\n\n\n\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Feedforward Neural Language Modeling:\nCompared to n-gram models, neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction.\\\n\nNeural language models represent words in this prior context by their embed- dings, rather than just by their word identity as used in n-gram language models. Using embeddings allows neural language models to generalize better to unseen data. \\\n \n\n## SLP Chapter 7.6 Training Neural Nets\n Training Neural Nets:\nHow do we partial out the loss over all those intermediate layers? The answer is the algorithm called error backpropagation or backward differentiation.\\\n\nThe cross-entropy loss that is used in neural networks is the same one we saw for logistic regression.\\\n\nthe cross-entropy loss is simply the log of the output probability correspond- ing to the correct class, and we therefore also call this the negative log likelihood loss:\nLCE(yˆ,y) = −logyˆi, (where i is the correct class)\\\n\nThe solution to computing this gradient is an algorithm called error backprop- agation or backprop (Rumelhart et al., 1986). While backprop was invented spe- cially for neural networks, it turns out to be the same as a more general procedure called backward differentiation, which depends on the notion of computation graphs. \\\n\nBackward differentiation on computation graphs\\\nBackwards differentiation makes use of the chain rule in calculus\\\n\nVarious forms of regularization are used to prevent overfitting. One of the most important is dropout: randomly dropping some units and their connections from the network during training (Hinton et al. 2012, Srivastava et al. 2014). Tuning of hyperparameters is also important.\\\n\n\n\n## SLP Chapter 7.7 Training the neural language model\n \nTraining the neural language model:\n\nTraining the parameters to minimize loss will result both in an algorithm for language modeling (a word predictor) but also a new set of embeddings E that can be used as word representations for other tasks. \n\n## Module 13 Lecture Notes\nLogistic Regression:\nClassifier components:\n1. A feature representation of the input.Foreachinputobservation x(i), this will be a vector of features [x1,x2,...,xn].\n2. A classification function that computes the estimated class 𝑦 via p(y|x).\n3. Annobjective function for learning, usually involving minimizing error on training examples.\n4. Analgorithm for optimizing the objective function.\\\nSentiment classifier\\\nDecision boundary\\\n\n$\\hat y=1 \\ if\\  P(y=1|x)>0.5\\ else\\ 0$\\\nExtracting Features\\\n$z=\\Sigma_i w_ix_i+b$, $\\sigma(z)=1/(1+e^{-(wx+b)})$\\\n1. An objective function or loss function that tells us distance between the system output and the gold output. We will use cross-entropy loss.\\\n2. An algorithm for optimizing the objective function. We will use stochastic gradient descent to minimize the loss function.\nLoss functions: eg. mean squared error \n$L_{MSE}(\\hat y,y)=1/2(\\hat y-y)^2$\\\nLoss functions for probabilistic classification:cross entropy loss\\\nFor one observation x, let’s maximize the probability of the correct label p(y|x).\n$p(y|x)=\\hat y^y(1-\\hat y)^{1-y}$\\\nChange to logs (still maximizing)\n$logp(y|x)=log[\\hat y^y(1-\\hat y)^{1-y}]=ylog\\hat y +(1-y)log(1-\\hat y)$\\\nCross-entropy loss: The result is cross-entropy loss:$-logp(y|x)=-(ylog\\hat y +(1-y)log(1-\\hat y))$\\\n$-logp(y|x)=-(ylog(wx+b)) +(1-y)log(1-(wx+b))))$\\\nGradient descent is a method that finds a minimum of a function by figuring out in which direction (in the space of the parameters θ) the function’s slope is rising the most steeply, and moving in the opposite direction.\\\n For logistic regression, this loss function is conveniently convex.\nA convex function has just one minimum, so there are no local minima\nto get stuck in.\nSo gradient descent starting from any point is guaranteed to find the minimum.\\\n $w^{t+1}=w^t-\\eta d/dwf(x;w)$\\\n The loss for a batch of data or an entire dataset is just the average loss over the m examples\\\n\n $\u0007lpha Cost(w,b)/ \u0007lpha w_j=\\Sigma^m_{i=1}[\\sigma (wx^{(i)}+b)-y^{i}]x_j^{(i)}$\\\nStochastic gradient descent chooses a single random example at a time and updates its weights on that example. As a result the updates can fluctuate.\\\nAn alternate is batch training, which computes the gradient over the entire dataset. This gives a much better estimate of which direction to move the weights but takes a long time to compute.\\\nA commonly used compromise is mini-batch training, where we train on a small batch. \\\nOverfitting :In logistic regression, if a feature only occurs in one class then it will get a high weight. Sometimes we are just modelling noisy factors that just accidentally correlate with the class.\\\nRegularization: Lasso regression uses L1 regularization Ridge regression uses L2 regularization\\\nMultinomial logistic regression: To get a proper probability, we will use a generalization of the sigmoid function\ncalled the softmax function.$softmax(z_i) = e_{z_j}/ \\Sigma_{j=1}^k e^{z_j} 1 ≤ i ≤ k$\\\nProblem: a small change in the weights or bias of any single perceptron in the network can causes the output to completely flip from 0 to 1.\nSolution: sigmoid neuron\\\nSmoothness is crucial\\\nActivation Functions:neuron units\napply a non-linear function f to z.\\\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\\\nThe simplest kind of NN is the Feed-Forward Neural Network Multilayer network, all units are usually fully-connected, and no cycles.\\\nThe computation has three steps:\\\n1. multiplying the weight matrix by the input vector x, 2. adding the bias vector b, and\n3. applying the activation function g (such as Sigmoid)\\\nHere are the final equations for a feedforward network with a single hidden layer, which takes an input vector x, outputs a probability distribution y, and is parameterized by weight matrices W and U and a bias vector b:\\\nh = σ(Wx+b)\\\nz = Uh\\\ny = softmax(z)\\\nTraining Neural Nets:\nA loss function – cross-entropy loss\nAn optimization algorithm – gradient descent\nA way of computing the gradient of the loss function – error propagation\\\nAlthough backpropagation was invented for neural nets, it is related to general procedure called backward differentiation, which depends on the notion of computation graphs.\\\nChain rule:For a composite function f(x) = u(v(x)), the derivative of f(x) is:\n$df/dx=du/dv * dv/dx$\n \n\n## Module 13 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Language Models Revisited\n",
        "We instantiate this intuition by using perplexity to measure the quality of a language model\\\\\n",
        "Intuitively, good models are those that assign higher probabilities to unseen data\\\\\n",
        "$pp_{\\theta}(w_{1:n})= p_{\\theta} w_{1:n}^{-1/n}$\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter9_2 = \"\"\" Recurrent Neural Networks:\n",
        "A simple recurrent neural network shown unrolled in time. Network layers are recalculated for each time step, while the weights U, V and W are shared in common across all time steps. \"\"\"\n",
        "slp_chapter9_3 = \"\"\" RNNs as Language Models \n",
        "Weight tying is a method that dispenses with this redundancy and uses a single set of embeddings at the input and softmax layers. \n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Lexical Semantics\\\\\n",
        "e.g\n",
        "mouse (N)\\\\\n",
        "      1.  any of numerous small rodents...\\\\\n",
        "      2.  a hand-operated device that controls a cursor...\\\\\n",
        "Here the form  is the lemma, also called the citation form.\\\\\n",
        "The specific forms sung or carpets or sing or duermes are called wordforms.\\\\\n",
        "A more formal definition of synonymy (between words rather than senses) is that two words are synonymous if they are substitutable for one another in any sentence without changing the truth conditions of the sentence, the situations in which the sentence would be true. We often say in this case that the two words have the same propositional meaning.\\\\\n",
        "Semantic fields are also related to topic models, like Latent Dirichlet Allocation, LDA, which apply unsupervised learning on large sets of texts to induce sets of associated words from text.\\\\\n",
        "A semantic frame is a set of words that denote perspectives or participants in a particular type of event. \\\\\n",
        "words have affective meanings or connotations\\\\\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter6_2 = \"\"\"Vector Semantics\n",
        "Vectors semantics is the standard way to represent word meaning in NLP\\\\\n",
        "Vectors for representing words are called embeddings\\\\\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter6_3 = \"\"\" \n",
        "Words and Vectors\n",
        "In a term-document matrix, each row represents a word in the vocabulary and each column represents a document from some collection of documents.\\\\\n",
        "Each cell in this matrix represents the number of times a particular word (defined by the row) occurs in a particular document (defined by the column). \\\\\n",
        "A vector space is a collection of vectors, characterized by their dimension.\\\\\n",
        "Term-document matrices were originally defined as a means of finding similar documents for the task of document information retrieval.\\\\\n",
        "Words as vectors: document dimensions: We do this by associating each word with a word vector— a row vector rather than a column vector, hence with different dimensions\\\\\n",
        "Words as vectors: word dimensions: An alternative to using the term-document matrix to represent words as vectors of document counts, is to use the term-term matrix, also called the word-word ma- trix or the term-context matrix, in which the columns are labeled by words rather than documents. \\\\\n",
        "This matrix is thus of dimensionality |V | × |V | and each cell records the number of times the row (target) word and the column (context) word co-occur in some context in some training corpus.\\\\\n",
        "Since most of these numbers are zero these are sparse vector representa- tions; there are efficient algorithms for storing and computing with sparse matrices.\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Cosine for measuring similarity\\\\\n",
        "The cosine—like most measures for vector similarity used in NLP—is based on the dot product operator from linear algebra, also called the inner product:\\\\\n",
        "dot product(v,w) = v·w = 􏰈viwi = v1w1 +v2w2 +...+vNwN\\\\\n",
        "This raw dot product, however, has a problem as a similarity metric: it favors long vectors. The vector length is defined as\\\\\n",
        "|v| = 􏰋􏰊􏰈v2i\\\\\n",
        "The dot product is higher if a vector is longer, with higher values in each dimension.\\\\\n",
        " This normalized dot product turns out to be the same as the cosine of the angle between the two vectors, following from the definition of the dot product between two vectors a and b::\n",
        " a·b = |a||b|cosθ\\\\\n",
        "a·b/|a||b| = cosθ\\\\\n",
        "The cosine similarity metric between two vectors v and w thus can be computed as:\n",
        "$\n",
        "cosine(v,w)=\\Sigma_{i=1}^N v_iw_i/(\\sqrt{ \\Sigma_{i=1}^N v_i^2}\\sqrt{ \\Sigma_{i=1}^N w_i^2})\n",
        "$\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter6_8 = \"\"\"Word2vec\n",
        " In the previous sections we saw how to represent a word as a sparse, long vector with dimensions corresponding to words in the vocabulary or documents in a collection. We now introduce a more powerful word representation: embeddings, short dense vectors. \\\\\n",
        "one method for computing embeddings: skip-gram with negative sampling, sometimes called SGNS.\\\\\n",
        "Word2vec embeddings are static em- beddings, meaning that the method learns one fixed embedding for each word in the vocabulary. \\\\\n",
        " The intuition of skip-gram is:\\\\\n",
        "1. Treat the target word and a neighboring context word as positive examples. \\\\\n",
        "2. Randomly sample other words in the lexicon to get negative samples.\\\\\n",
        "3. Use logistic regression to train a classifier to distinguish those two cases. (we rely on the intuition that two vectors are similar if they have a high dot product\\\\\n",
        "To turn the dot product into a probability, we’ll use the logistic or sigmoid function σ (x), the fundamental core of logistic regression\\\\\n",
        "Skip-gram makes the simplifying assumption that all context words are independent, allowing us to just multiply their probabilities:)\\\\\n",
        "4. Use the learned weights as the embeddings.\\\\\n",
        "In summary, skip-gram trains a probabilistic classifier that, given a test target word w and its context window of L words c1:L, assigns a probability based on how similar this context window is to the target word. The probability is based on applying the logistic (sigmoid) function to the dot product of the embeddings of the target word with each context word. To compute this probability, we just need embeddings for each target word and context word in the vocabulary.\\\\\n",
        "the goal of the learning algorithm is to adjust those embeddings to\\\\\n",
        "• Maximize the similarity of the target word, context word pairs (w,cpos) drawn from the positive examples\\\\\n",
        "• Minimize the similarity of the (w,cneg) pairs from the negative examples.\\\\\n",
        "Another very widely used static embedding model is GloVe\\\\\n",
        "\"\"\"\n",
        "slp_chapter6_10 = \"\"\" Semantic properties of embeddings :\n",
        "Different types of similarity or association: One parameter of vector semantic models that is relevant to both sparse tf-idf vectors and dense word2vec vectors is the size of the context window used to collect counts. \\\\\n",
        "The choice depends on the goals of the representation. Shorter context windows tend to lead to representations that are a bit more syntactic, since the information is coming from immediately nearby words. \\\\\n",
        "Analogy/Relational Similarity: Another semantic property of embeddings is their ability to capture relational meanings.\\\\\n",
        "parallelogram model for solving simple analogy problems of the form a is to b as a* is to what?\\\\\n",
        "There are some caveats. For example, the closest value returned by the paral-\n",
        "lelogram algorithm in word2vec or GloVe embedding spaces is usually not in fact b* but one of the 3 input words or their morphological variants (i.e., cherry:red :: potato:x returns potato or potatoes instead of brown), so these must be explicitly excluded. \\\\\n",
        "Furthermore while embedding spaces perform well if the task involves frequent words, small distances, and certain relations (like relating countries with their capitals or verbs/nouns with their inflected forms), the parallelogram method with embeddings doesn’t work as well for other relations \\\\\n",
        "\n",
        "Embeddings and Historical Semantics:\n",
        "Embeddings can also be a useful tool for studying how meaning changes over time, by computing multiple embedding spaces, each from texts written in a particular time period. \n",
        "\n",
        "\"\"\"\n",
        "slp_chapter6_11 = \"\"\" Bias and Embeddings\\\\\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Evaluating Vector Models:\n",
        "The semantic textual similarity task (Agirre et al. 2012, Agirre et al. 2015) evaluates the performance of sentence- level similarity algorithms, consisting of a set of pairs of sentences, each pair with human-labeled similarity scores.\\\\\n",
        "Another task used for evaluation is the analogy task, discussed on page 24, where thesystemhastosolveproblemsoftheformaistobasa*istob*,givena,b,anda*\n",
        "and having to find b* \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\" Language Modeling: Goal: Learn a function that returns the joint probability;\n",
        "Primary difficulty:\\\\\n",
        "1. There are too many parameters to accurately estimate. This is sometimes called the “curse of dimensionality”;\n",
        "2. Inn-gram-basedmodelswefailtogeneralizetorelatedwords/word sequences that we have observed.;\n",
        " Curse of dimensionality / sparse statistics\\\\\n",
        "What assumption do we make in n-gram LMs to simplify this?\\\\\n",
        "The probability of the next word only depends on the previous n-1 words.\\\\\n",
        "A small n makes it easier for us to get an estimate of the probability from data.\\\\\n",
        " Unseen sequences\\\\\n",
        " What happens when we have a new (unseen) combination of n words? 1. Back-off\n",
        "2. Smoothing/interpolation\n",
        "We are basically just stitching together short sequences of observed words.\\\\\n",
        " Alternate idea:Let’s try generalizing.\\\\\n",
        " A Neural Probabilistic LM\\\\\n",
        " A Neural Probabilistic LM\n",
        "Given:\n",
        "A training set w1 ... wt where wt ∈V\\\\\n",
        "Learn:\n",
        "f(w1 ... wt) = P(wt|w1 ... wt-1)\n",
        "Subject to giving a high probability to an unseen text/dev set (e.g. minimizing the perplexity)\\\\\n",
        "Constraint:\n",
        "Create a proper probability distribution (e.g. sums to 1) so that we can take the product of conditional probabilities to get the joint probability of a sentence\\\\\n",
        "Word embeddings\\\\\n",
        "NLP: Vector Semantics\\\\\n",
        "In N-gram LMs we represented words as a string of letters or as an index in a\n",
        "vocabulary list.\n",
        "Word Meaning: a meaning representation to encode:\n",
        "1. Synonyms – words that have similar meanings\n",
        "2. Antonyms – words that have opposite meanings\n",
        "3. Connotations – words that are positive or negative\n",
        "4. Semantic Roles – buy, sell, and pay are different parts of the same underlying purchasing event\n",
        "5. Support for entailment\\\\\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Vector Semantics part 2\\\\\n",
        "Dense embeddings you can download: Word2vec Fasttext Glove (Pennington, Socher, Manning) Magnitude (Patel and Sands)\\\\\n",
        "Skip-gram algorithm:\n",
        "1. Treat the target word and a neighboring context word as positive examples.\\\\\n",
        "2. Randomly sample other words in the lexicon to get negative samples\\\\\n",
        "3. Use logistic regression to train a classifier to distinguish those two cases\\\\\n",
        "4. Use the weights as the embeddings\\\\\n",
        "Skip-Gram Goal\n",
        "Given a tuple (t,c) = target, context • (apricot, jam)\n",
        "• (apricot, aardvark)\n",
        "Return probability that c is a real context word:\n",
        "P(+|t,c)\n",
        "P(−|t,c) = 1−P(+|t,c)\\\\\n",
        "plifying assumption that all context words are independent, allowin\n",
        "For all the context words:\n",
        "Assume all context words are independent\n",
        "Skip-Gram Training Data: Training data: input/output pairs centering on \"\" Assume a +/- 2 word window;\n",
        "Choosing noise words\n",
        "More common to chosen then according to $p_α(w)$ $P_\\alpha(w)=count(w)^\\alpha/\\Sigma_w count(w)^{\\alpha}$\\\\\n",
        "Learning the classifier: We’ll start with 0 or random weights\n",
        "Then adjust the word weights to\n",
        "• make the positive pairs more likely • and the negative pairs less likely\\\\\n",
        "Objective Criteria: We want to maximize...\n",
        "sum logP (+|t, c) + sum logP (-|t, c) \\\\\n",
        "How to learn word2vec (skip-gram) embeddings:Use logistic regression\\\\\n",
        "• Take a corpus and take pairs of words that co-occur as positive examples\\\\\n",
        "• Take pairs of words that don't co-occur as negative examples\\\\\n",
        "• Train the classifier to distinguish these by slowly adjusting all the embeddings to improve the classifier performance\\\\\n",
        "• Throw away the classifier code and keep the embeddings.\\\\\n",
        "Evaluating embeddings: Compare to human scores on word similarity-type tasks:\\\\\n",
        "Properties of embeddings Similarity depends on window size C\\\\\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "0c950b4b-92c9-4d3e-9c4d-b26a29b1df5e"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Language Models Revisited\nWe instantiate this intuition by using perplexity to measure the quality of a language model\\\nIntuitively, good models are those that assign higher probabilities to unseen data\\\n$pp_{\theta}(w_{1:n})= p_{\theta} w_{1:n}^{-1/n}$\n\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Recurrent Neural Networks:\nA simple recurrent neural network shown unrolled in time. Network layers are recalculated for each time step, while the weights U, V and W are shared in common across all time steps. \n\n## SLP Chapter 9.3 RNNs as Language Models\n RNNs as Language Models \nWeight tying is a method that dispenses with this redundancy and uses a single set of embeddings at the input and softmax layers. \n\n\n## SLP Chapter 6.1 Lexical Semantics\n Lexical Semantics\\e.g\nmouse (N)\\\n      1.  any of numerous small rodents...\\\n      2.  a hand-operated device that controls a cursor...\\\nHere the form  is the lemma, also called the citation form.\\\nThe specific forms sung or carpets or sing or duermes are called wordforms.\\\nA more formal definition of synonymy (between words rather than senses) is that two words are synonymous if they are substitutable for one another in any sentence without changing the truth conditions of the sentence, the situations in which the sentence would be true. We often say in this case that the two words have the same propositional meaning.\\\nSemantic fields are also related to topic models, like Latent Dirichlet Allocation, LDA, which apply unsupervised learning on large sets of texts to induce sets of associated words from text.\\\nA semantic frame is a set of words that denote perspectives or participants in a particular type of event. \\\nwords have affective meanings or connotations\\\n\n \n\n## SLP Chapter 6.2 Vector Semantics\nVector Semantics\nVectors semantics is the standard way to represent word meaning in NLP\\\nVectors for representing words are called embeddings\\\n\n\n\n## SLP Chapter 6.3 Words and Vectors\n \nWords and Vectors\nIn a term-document matrix, each row represents a word in the vocabulary and each column represents a document from some collection of documents.\\\nEach cell in this matrix represents the number of times a particular word (defined by the row) occurs in a particular document (defined by the column). \\\nA vector space is a collection of vectors, characterized by their dimension.\\\nTerm-document matrices were originally defined as a means of finding similar documents for the task of document information retrieval.\\\nWords as vectors: document dimensions: We do this by associating each word with a word vector— a row vector rather than a column vector, hence with different dimensions\\\nWords as vectors: word dimensions: An alternative to using the term-document matrix to represent words as vectors of document counts, is to use the term-term matrix, also called the word-word ma- trix or the term-context matrix, in which the columns are labeled by words rather than documents. \\\nThis matrix is thus of dimensionality |V | × |V | and each cell records the number of times the row (target) word and the column (context) word co-occur in some context in some training corpus.\\\nSince most of these numbers are zero these are sparse vector representa- tions; there are efficient algorithms for storing and computing with sparse matrices.\n\n \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Cosine for measuring similarity\\The cosine—like most measures for vector similarity used in NLP—is based on the dot product operator from linear algebra, also called the inner product:\\dot product(v,w) = v·w = 􏰈viwi = v1w1 +v2w2 +...+vNwN\\This raw dot product, however, has a problem as a similarity metric: it favors long vectors. The vector length is defined as\\|v| = 􏰋􏰊􏰈v2i\\The dot product is higher if a vector is longer, with higher values in each dimension.\\ This normalized dot product turns out to be the same as the cosine of the angle between the two vectors, following from the definition of the dot product between two vectors a and b::\n a·b = |a||b|cosθ\\\na·b/|a||b| = cosθ\\\nThe cosine similarity metric between two vectors v and w thus can be computed as:\n$\ncosine(v,w)=\\Sigma_{i=1}^N v_iw_i/(\\sqrt{ \\Sigma_{i=1}^N v_i^2}\\sqrt{ \\Sigma_{i=1}^N w_i^2})\n$\n\n \n\n## SLP Chapter 6.8 Word2vec\nWord2vec\n In the previous sections we saw how to represent a word as a sparse, long vector with dimensions corresponding to words in the vocabulary or documents in a collection. We now introduce a more powerful word representation: embeddings, short dense vectors. \\\none method for computing embeddings: skip-gram with negative sampling, sometimes called SGNS.\\\nWord2vec embeddings are static em- beddings, meaning that the method learns one fixed embedding for each word in the vocabulary. \\\n The intuition of skip-gram is:\\\n1. Treat the target word and a neighboring context word as positive examples. \\\n2. Randomly sample other words in the lexicon to get negative samples.\\\n3. Use logistic regression to train a classifier to distinguish those two cases. (we rely on the intuition that two vectors are similar if they have a high dot product\\\nTo turn the dot product into a probability, we’ll use the logistic or sigmoid function σ (x), the fundamental core of logistic regression\\\nSkip-gram makes the simplifying assumption that all context words are independent, allowing us to just multiply their probabilities:)\\\n4. Use the learned weights as the embeddings.\\\nIn summary, skip-gram trains a probabilistic classifier that, given a test target word w and its context window of L words c1:L, assigns a probability based on how similar this context window is to the target word. The probability is based on applying the logistic (sigmoid) function to the dot product of the embeddings of the target word with each context word. To compute this probability, we just need embeddings for each target word and context word in the vocabulary.\\\nthe goal of the learning algorithm is to adjust those embeddings to\\\n• Maximize the similarity of the target word, context word pairs (w,cpos) drawn from the positive examples\\\n• Minimize the similarity of the (w,cneg) pairs from the negative examples.\\\nAnother very widely used static embedding model is GloVe\\\n\n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Semantic properties of embeddings :\nDifferent types of similarity or association: One parameter of vector semantic models that is relevant to both sparse tf-idf vectors and dense word2vec vectors is the size of the context window used to collect counts. \\\nThe choice depends on the goals of the representation. Shorter context windows tend to lead to representations that are a bit more syntactic, since the information is coming from immediately nearby words. \\\nAnalogy/Relational Similarity: Another semantic property of embeddings is their ability to capture relational meanings.\\\nparallelogram model for solving simple analogy problems of the form a is to b as a* is to what?\\\nThere are some caveats. For example, the closest value returned by the paral-\nlelogram algorithm in word2vec or GloVe embedding spaces is usually not in fact b* but one of the 3 input words or their morphological variants (i.e., cherry:red :: potato:x returns potato or potatoes instead of brown), so these must be explicitly excluded. \\\nFurthermore while embedding spaces perform well if the task involves frequent words, small distances, and certain relations (like relating countries with their capitals or verbs/nouns with their inflected forms), the parallelogram method with embeddings doesn’t work as well for other relations \\\n\nEmbeddings and Historical Semantics:\nEmbeddings can also be a useful tool for studying how meaning changes over time, by computing multiple embedding spaces, each from texts written in a particular time period. \n\n\n\n## SLP Chapter 6.11 Bias and Embeddings\n Bias and Embeddings \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Evaluating Vector Models:\nThe semantic textual similarity task (Agirre et al. 2012, Agirre et al. 2015) evaluates the performance of sentence- level similarity algorithms, consisting of a set of pairs of sentences, each pair with human-labeled similarity scores.\\\nAnother task used for evaluation is the analogy task, discussed on page 24, where thesystemhastosolveproblemsoftheformaistobasa*istob*,givena,b,anda*\nand having to find b* \n\n## Module 13 Lecture Notes\nLogistic Regression:\nClassifier components:\n1. A feature representation of the input.Foreachinputobservation x(i), this will be a vector of features [x1,x2,...,xn].\n2. A classification function that computes the estimated class 𝑦 via p(y|x).\n3. Annobjective function for learning, usually involving minimizing error on training examples.\n4. Analgorithm for optimizing the objective function.\\\nSentiment classifier\\\nDecision boundary\\\n\n$\\hat y=1 \\ if\\  P(y=1|x)>0.5\\ else\\ 0$\\\nExtracting Features\\\n$z=\\Sigma_i w_ix_i+b$, $\\sigma(z)=1/(1+e^{-(wx+b)})$\\\n1. An objective function or loss function that tells us distance between the system output and the gold output. We will use cross-entropy loss.\\\n2. An algorithm for optimizing the objective function. We will use stochastic gradient descent to minimize the loss function.\nLoss functions: eg. mean squared error \n$L_{MSE}(\\hat y,y)=1/2(\\hat y-y)^2$\\\nLoss functions for probabilistic classification:cross entropy loss\\\nFor one observation x, let’s maximize the probability of the correct label p(y|x).\n$p(y|x)=\\hat y^y(1-\\hat y)^{1-y}$\\\nChange to logs (still maximizing)\n$logp(y|x)=log[\\hat y^y(1-\\hat y)^{1-y}]=ylog\\hat y +(1-y)log(1-\\hat y)$\\\nCross-entropy loss: The result is cross-entropy loss:$-logp(y|x)=-(ylog\\hat y +(1-y)log(1-\\hat y))$\\\n$-logp(y|x)=-(ylog(wx+b)) +(1-y)log(1-(wx+b))))$\\\nGradient descent is a method that finds a minimum of a function by figuring out in which direction (in the space of the parameters θ) the function’s slope is rising the most steeply, and moving in the opposite direction.\\\n For logistic regression, this loss function is conveniently convex.\nA convex function has just one minimum, so there are no local minima\nto get stuck in.\nSo gradient descent starting from any point is guaranteed to find the minimum.\\\n $w^{t+1}=w^t-\\eta d/dwf(x;w)$\\\n The loss for a batch of data or an entire dataset is just the average loss over the m examples\\\n\n $\u0007lpha Cost(w,b)/ \u0007lpha w_j=\\Sigma^m_{i=1}[\\sigma (wx^{(i)}+b)-y^{i}]x_j^{(i)}$\\\nStochastic gradient descent chooses a single random example at a time and updates its weights on that example. As a result the updates can fluctuate.\\\nAn alternate is batch training, which computes the gradient over the entire dataset. This gives a much better estimate of which direction to move the weights but takes a long time to compute.\\\nA commonly used compromise is mini-batch training, where we train on a small batch. \\\nOverfitting :In logistic regression, if a feature only occurs in one class then it will get a high weight. Sometimes we are just modelling noisy factors that just accidentally correlate with the class.\\\nRegularization: Lasso regression uses L1 regularization Ridge regression uses L2 regularization\\\nMultinomial logistic regression: To get a proper probability, we will use a generalization of the sigmoid function\ncalled the softmax function.$softmax(z_i) = e_{z_j}/ \\Sigma_{j=1}^k e^{z_j} 1 ≤ i ≤ k$\\\nProblem: a small change in the weights or bias of any single perceptron in the network can causes the output to completely flip from 0 to 1.\nSolution: sigmoid neuron\\\nSmoothness is crucial\\\nActivation Functions:neuron units\napply a non-linear function f to z.\\\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\\\nThe simplest kind of NN is the Feed-Forward Neural Network Multilayer network, all units are usually fully-connected, and no cycles.\\\nThe computation has three steps:\\\n1. multiplying the weight matrix by the input vector x, 2. adding the bias vector b, and\n3. applying the activation function g (such as Sigmoid)\\\nHere are the final equations for a feedforward network with a single hidden layer, which takes an input vector x, outputs a probability distribution y, and is parameterized by weight matrices W and U and a bias vector b:\\\nh = σ(Wx+b)\\\nz = Uh\\\ny = softmax(z)\\\nTraining Neural Nets:\nA loss function – cross-entropy loss\nAn optimization algorithm – gradient descent\nA way of computing the gradient of the loss function – error propagation\\\nAlthough backpropagation was invented for neural nets, it is related to general procedure called backward differentiation, which depends on the notion of computation graphs.\\\nChain rule:For a composite function f(x) = u(v(x)), the derivative of f(x) is:\n$df/dx=du/dv * dv/dx$\n \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "T3cm1WV542_a",
        "outputId": "fc98fdb3-3856-4ea0-8b98-65c1678c9f7f"
      },
      "source": [
        "#@title 默认标题文本\n",
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nThe MEU principle formalizes the general notion that an intelligent agent should “do the\nright thing,” but does not operationalize that advice.\\\n If an agent acts so as to maximize a utility function that correctly reflects the performance measure, then\nthe agent will achieve the highest possible performance score (averaged over all the possible\nenvironments).\\\n The performance measure gives a score for a history—a sequence of states.\\\n\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n  any reasonable preference relation to obey:\\ORDERABILITY:Given any two lotteries, a rational agent must either prefer one or\nelse rate them as equally preferable.\\\nTransitivity:Given any three lotteries, if an agent prefers A to B and prefers B to C,\nthen the agent must prefer A to C\\\nContinuity:\nA ≻ B ≻ C ⇒ ∃ p [p, A; 1 − p, C] ∼ B\\\nSubstitutability: If an agent is indifferent between two lotteries and , then the agent is indifferent between two more complex lotteries that are the same except that \nis substituted for in one of them.\nA ∼ B ⇒ [p, A; 1 − p, C] ∼ [p, B; 1 − p, C] .\\\nMonotonicity:\nA ≻ B ⇒ p > q ⇔ [p, A; 1 − p, B] ≻ [q, A; 1 − q, B]).\\\nDecomposability:\n[p, A; 1 − p, [q, B; 1 − q, C]] ∼ [p, A; (1 − p)q, B; (1 − p)(1 − q), C].\\\n\nEXISTENCE OF UTILITY FUNCTION:\nU(A) > U(B) ⇔ A ≻ B and U(A) = U(B) ⇔ A ∼ B.\\\nEXPECTED UTILITY OF A LOTTERY:\nU([p1, S1;…; pn, Sn]) = ∑i piU(Si).\\\n in a deterministic environment an agent needs only a preference ranking on states—the numbers don’t matter. This is called a value function or ordinal\nutility function.\\\n the existence of a utility function that describes an agent’s preference behavior does not necessarily mean that the agent is explicitly maximizing that\nutility function in its own deliberations.\n\n\n \n\n## AIMA Chapter 16.3 Utility Functions\nUtility functions map from lotteries to real numbers.\\We say that the agent exhibits a monotonic preference for more money. This does\nnot mean that money behaves as a utility function, because it says nothing about\npreferences between lotteries involving money.\\\n agents with curves of this shape are risk-averse: they prefer a sure thing with a\npayoff that is less than the expected monetary value of a gamble. On the other hand, in the\n“desperate” region at large negative wealth , the behavior is risk-seeking.\\\nThe value an agent will accept in lieu of a lottery is called the certainty equivalent of the\nlottery.\\\nThe difference between the EMV of a lottery and its certainty equivalent is called the\ninsurance premium.\\\n An agent that has a linear curve is said to be risk-neutral.\\\n The rational way to choose the best action, , is to maximize expected utility\\\n This tendency for the estimated expected utility of the best choice to be too high is called the optimizer’s curse\\\n Decision theory is a normative theory: it describes how a rational agent should act.\\\n  A descriptive theory, on the other hand, describes how actual agents—for example, humans—\nreally do act.\\\nOne explanation for the apparently irrational preferences is the certainty effect (Kahneman\nand Tversky, 1979): people are strongly attracted to gains that are certain. There are several\nreasons why this may be so.\\\nFirst, people may prefer to reduce their computational burden;Second, people may distrust the legitimacy of the stated probabilities.;Third, people may be accounting for their emotional state as well as their financial state.\\\npeople have ambiguity aversion\\\n framing effect: exact wording of a decision problem can have a big impact on the agent’s choices\\\nanchoring effect:People feel more comfortable making relative utility judgments rather than absolute ones.\n\n \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n  sequential decision problems, in which the agent’s utility depends on a sequence of decisions.\\ a sequential decision problem for a fully observable, stochastic environment\nwith a Markovian transition model and additive rewards is called a Markov decision\nprocess, or MDP, and consists of a set of states (with an initial state s0);\\\n a set ACTIONS(s) of\nactions in each state; \\\na transition model P(s′|s, a); \\\nand a reward function R(s, a, s′).\\\nA solution of this kind is called a policy.\\\ndynamic programming: simplifying a problem\nby recursively breaking it into smaller pieces and remembering the optimal solutions to the\npieces.\\\n1)The first question to answer is whether there is a finite horizon or an infinite horizon for\ndecision making: with a finite horizon, an optimal action in a given state may depend on how much time is left. A policy that\ndepends on the time is called nonstationary.\\\nWith no fixed time limit, on the other hand, there is no reason to behave differently in the same state at different times. Hence, an optimal action depends only on the current state,\nand the optimal policy is stationary.\\\n 2)The next question we must decide is how to calculate the utility of state sequences.\\\n additive discounted rewards: the utility of a history is Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + γR(s1, a1, s2) + γ2R(s2, a2, s3) + ⋯\\\nThere are several reasons why additive discounted rewards make sense.\\\n One is empirical:\\\n Another is economic:\\\nA third reason is uncertainty about the true rewards: they may never arrive for all sorts of\nreasons that are not taken into account in the transition model. Under certain assumptions,\na discount factor of gamma is equivalent to adding a probability 1 − γ of accidental\ntermination at every time step, independent of the action taken.\\\nA fourth justification arises from a natural property of preferences over histories.\\\nA final justification for discounted rewards is that it conveniently makes some nasty\ninfinities go away.:There are three solutions:1.1. With discounted rewards, the utility of an infinite sequence is finite.;2. If the environment contains terminal states and if the agent is guaranteed to get to one\neventually, then we will never need to compare infinite sequences.3. Infinite sequences can be compared in terms of the average reward obtained per\ntime step.\\\nthe utility of a state is the expected reward for the next\ntransition plus the discounted utility of the next state, assuming that the agent chooses the optimal\naction. \nThat is, the utility of a state is given by  π∗(s) = argmax a∈A(s) \nU(s) = max a∈A(s)∑s′ P(s′|s, a)[R(s, a, s′) + γU(s′)].\nThis is called the Bellman equation\\\nAnother important quantity is the action-utility function, or Q-function: is the\nexpected utility of taking a given action in a given state.\nQ(s, a) = ∑s′ P(s′|s, a)[R(s, a, s′) + γ U(s′)] = ∑s′ P(s′|s, a)[R(s, a, s′) + γ max a′ Q(s′, a′)]\n \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n algorithms for solving MDPs. \\The first three, value iteration, policy iteration, and linear programming, generate exact solutions offline. The fourth is a family of online approximate algorithms that includes Monte Carlo planning\\Value Iteration: Convergence of value iteration:The basic concept used in showing that value iteration converges is the notion of a\ncontraction.\nThe iteration step, called a Bellman update:\\\nUi+1(s) ← max ∑P(s′|s, a)[R(s, a, s′) + γ Ui(s′)] \\\nPOLICY EVALUATION: given a policy πi, calculate Ui = Uπi , the utility of each state if πi were to be executed.\nPOLICY IMPROVEMENT: Calculate a new MEU policy πi+1, using one-step look-ahead\nbased on Ui\nLinear programming or LP, which was mentioned briefly in, is a\ngeneral approach for formulating constrained optimization problems, and there are many\nindustrial-strength LP solvers available.\nHere we will consider online algorithms:The most straightforward approach is actually a simplification of the EXPECTIMINIMAX\nalgorithm for game trees with chance nodes: the EXPECTIMAX algorithm builds a tree of\nalternating max and chance nodes\n\n\n \n\n## Module 8 Lecture Notes\n Stochastic Transition Model\nThe transition function is sometimes written as T(s, a, s’), or explicitly as a probability: p( s’ | s, a )\\\nReward function: R( s, a, s’ )\\\nA Markov decision process or MDP is\\\n• a sequential decision problem\\\n• for a fully observable environment\\\n• with a stochastic transition model\\\n• that has additive rewards\\\nMDPs are non-deterministic search problems. One way of solving them is via expectimax search.\\\nthere are only two ways to define utilities: \\\n• Additive utility:  Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + R(s1, a1, s2) + 2R(s2, a2, s3) + ⋯\\\n• Discounted utility: Uh([s0, a0, s1, a1, s2,…]) = R(s0, a0, s1) + γR(s1, a1, s2) + γ^2R(s2, a2, s3) + ⋯\\\nInfinite Utilities:\\\nSolutions:\\\n1) Finite horizon: (similar to depth-limited search) \\\n   ▪ Terminate episodes after a fixed T steps (e.g. life)\\\n   ▪ Gives nonstationary policies (depends on time left)\\\n2) Discounting: use 0 < γ< 1\\\n  ▪ Smaller  means smaller “horizon” – shorter term focus\\\n3) Absorbing state: guarantee that for every policy, a terminal state will eventually be reached (like “overheated” for racing)\\\n\n\nValue Iteration:\nVi+1(s) ← max a ∑ s' T(s′|s, a)[R(s, a, s′) + γ Vi(s′)] \\\nProblem 1: It’s slow – O(S2A) per iteration\nProblem 2: The “max” at each state rarely changes\nProblem 3: The policy often converges long before the values\\\n\nPolicy Iteration: \nAlternative approach for optimal values:\\\n• Step 1: Policy evaluation: calculate utilities for some fixed policy (not optimal utilities!) until convergence\n• Step 2: Policy improvement: update policy using one-step look- ahead with resulting converged (but not optimal!) utilities as future values\n• Repeat steps until policy converges\\\nEvaluation: For fixed current policy , find values with policy evaluation:\\\nImprovement: For fixed values, get a better policy using policy extraction\\\nComparison\\\nIn value iteration:\n• Every iteration updates both the values and (implicitly) the policy\n• We don’t track the policy, but taking the max over actions implicitly recomputes it\nIn policy iteration:\n• We do several passes that update utilities with fixed policy (each pass is fast because we consider only one action, not all of them)\n• After the policy is evaluated, a new policy is chosen (slow like a value iteration pass)\n$V^\\pi_{k+1}(s)=\\Sigma_{s'}T(s,\\pi(s),s')[R(s,\\pi(s),s')+\\gamma V_k^\\pi(s')]$\n• The new policy will be better (or we’re done)\\\nUtilities are functions from outcomes (states of the world) to real numbers that describe an agent’s preferences\\\nMEU Principle\\\n\n\n\n## Module 8 Definitions\n* **Utility function** -  a single number to express the desirability of a state\n* **Expected utility** - average utility value of the outcomes, weighted by the probability that the outcome occurs\n* **Meu** - choose the action maximizes the agent’s expected utility\n* **Markovian** - the probability of reaching s′ from s depends only on s and not on the history of earlier states.\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\nbandit problems:\nEach arm Mi is a Markov reward process or MRP, that is, an MDP with only one\npossible action ai. It has states Si, transition model Pi(s′|s, ai), and reward Ri(s, ai, s′).\nThe arm defines a distribution over sequences of rewards Ri,0, Ri,1, Ri,2,…, where each \nRi,t is a random variable.\nThe overall bandit problem is an MDP: the state space is given by the Cartesian product \nS = S1 × ⋯ × Sn; the actions are a1,…, an; the transition model updates the state of\nwhichever arm Mi is selected, according to its specific transition model, leaving the\nother arms unchanged; and the discount factor is γ. \n\n## AIMA Chapter 22.1 Learning from Rewards\nAn alternative is reinforcement learning (RL), in which an agent interacts with the world\nand periodically receives rewards (or, in the terminology of psychology, reinforcements).\\\n\nModel-free reinforcement learning: two varieties:action-utility learning:1)most common form is Q-learning.2)Policy search: learns a policy π(s) that maps directly from states to\nactions.\\\npassive reinforcement learning, where the agent’s policy is\nfixed and the task is to learn the utilities of states (or of state–action pairs); this could also\ninvolve learning a model of the environment.\\\nactive reinforcement learning, where the agent must also figure out what to do. The\nprincipal issue is exploration: an agent must experience as much as possible of its\nenvironment in order to learn how to behave in it.\\\n\n \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n  Passive Reinforcement Learning: We start with the simple case of a fully observable environment with a small number of\nactions and states, in which an agent already has a fixed policy that determines its\nactions.\\\nThe agent is trying to learn the utility function —the expected total discounted\nreward if policy is executed beginning in state $U^\\pi(s)$.\nThe passive learning task is similar to the policy evaluation task\\\n\n 1)direct utility estimation: direct utility estimation is that the utility of a state is defined as the expected\ntotal reward from that state onward (called the expected reward-to-go): we can view direct utility estimation\nas searching for in a hypothesis space that is much larger than it needs to be, in that it\nincludes many functions that violate the Bellman equations. For this reason, the algorithm\noften converges very slowly.\\\n2)adaptive dynamic programming (or ADP):  For a passive\nlearning agent, this means plugging the learned transition model P(s′|s,π(s)) and the\nobserved rewards R(s,π(s),s′) into Equation (22.2):\nUi(s) = ∑s′ P(s′∣∣∣∣s,πi(s))[R(s,πi(s),s′) + γUi(s′)].\n to calculate the utilities of the states.\nmodified policy iteration: using a simplified value iteration process to update the utility estimates after each change to the\nlearned model. \\\nThe transition model \nP(s′|s,a) is represented as a table and it is estimated directly from the counts that are\naccumulated in Ns′|sa.The counts record how often state s′\n is reached when executing a in s.\\\ntemporal-difference (TD): this update rule uses the difference in utilities between successive states (and thus successive times).\nUπ(s) ← Uπ(s) + α[R(s,π(s),s′) + γUπ(s′) − Uπ(s)].\nNotice that TD does not need a transition model to perform its updates. The\nenvironment itself supplies the connection between neighboring states in the form of\nobserved transitions.\n The prioritized\nsweeping heuristic prefers to make adjustments to states whose likely successors have just\nundergone a large adjustment in their own utility estimates.\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Active Reinforcement Learning\n, an agent must make a tradeoff between exploitation of\nthe current best action to maximize its short-term reward and exploration of previously\nunknown states to gain information that can lead to a change in policy (and to greater\nrewards in the future).\\\nExploration:\\\n“greedy in the limit of infinite exploration,” or GLIE\nGLIE schemes: 1) choose a random\naction at time step t with probability 1/t and to follow the greedy policy otherwise. While\nthis does eventually converge to an optimal policy, it can be slow. 2)better:\ngive some weight to actions that the agent has not tried very often, while tending to avoid\nactions that are believed to be of low utility \\\n\n\nHere, f is the exploration function. The function determines how greed (preference\nfor high values of the utility ) is traded off against curiosity (preference for actions that\nhave not been tried often and have a low count ).\\\nU +(s) ← maxa f (∑s′ P(s′|s,a)[R(s,a,s′) + γU +(s′)], N(s,a)) .\nf(u,n) =  R+ (if n < Ne)/ u (otherwise)  R+ is an optimistic estimate of the best possible reward obtainable in any state\\\n\nSafe exploration\\\nThe first approach, Bayesian reinforcement learning.  Let \nbe the expected utility, averaged over all possible start states, obtained by executing policy \nin model h . Then we have\nπ∗ = argmax π ∑h P(h|e)Uπh .\\\n\nThe second approach, derived from robust control theory, allows for a set of possible\nmodels without assigning probabilities to them, and defines an optimal robust policy as\none that gives the best outcome in the worst case over :\nπ∗ = argmax π minh Uπh.\\\n\n\n Temporal-difference Q-learning\n  a TD Q-learning agent does\nnot need a transition model, either for learning or for action selection.\nQ(s,a) ← Q(s,a) + α[R(s,a,s′) + γ max a′ Q(s′,a′) − Q(s,a)].\\\nAn exploratory Q-learning agent. It is an active learner that learns the value Q(s,a) of each action in each\nsituation. It uses the same exploration function f as the exploratory ADP agent, but avoids having to\nlearn the transition model.\\\n\nQ-learning has a close relative called SARSA (for state, action, reward, state, action).\ndifference: When exploration is happening, however, they differ: if the\nexploration yields a negative reward, SARSA penalizes the action, while Q-learning does\nnot.\\\nQ-learning is an off-policy learning algorithm, ” SARSA is an on-policy algorithm; Q-learning is more flexible than SARSA, a Q-learning agent can\nlearn how to behave well when under the control of a wide variety of exploration policies.\nOn the other hand, SARSA is appropriate if the overall policy is even partly controlled by\nother agents or programs, in which case it is better to learn a Q-function for what will\nactually happen rather than what would happen if the agent got to pick estimated best\nactions.\n \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n  Generalization in Reinforcement Learning\u0001) Approximating direct utility estimation\n The state and the sum of rewards received constitute a training example for a\nsupervised learning algorithm.\nθi ← θi − α = θi + α[uj(s) − Uˆθ(s)] . ∂Ej(s) ∂θi ∂Uˆθ(s) ∂θi\nThis is called the Widrow–Hoff rule, or the delta rule, for online least-squares.\n2)Approximating temporal-difference learning\nθi ← θi + α[R(s,a,s′) + γUˆθ(s′) − Uˆθ(s)] ∂Uˆθ(s) ∂θi\nfor utilities and\nθi ← θi + α[R(s,a,s′) + γ max a′ Qˆθ(s′,a′) − Qˆθ(s,a)] ∂Qˆθ(s,a) ∂θi\nfor Q-values.\n3)Deep reinforcement learning\n4)Reward shaping: R′(s,a,s′) = R(s,a,s′) + γΦ(s′) − Φ(s).The potential fuction Φ can be constructed to reflect any desirable aspects of the state, such\nas achievement of subgoals or distance to a desired terminal state.\\\n5)Hierarchical reinforcement learning:Another way to cope with very long action sequences is to break them up into a few smaller\npieces, and then break those into smaller pieces still, and so on until the action sequences\nare short enough to make learning easy.\n\n\n## Module 9 Lecture Notes\n Important ideas in reinforcement learning that came up\nExploration: you have to try unknown actions to get information\\\n Exploitation: eventually, you have to use what you know\\\nRegret: even if you learn intelligently, you make mistakes \\\nSampling: because of chance, you have to try things repeatedly \\\nDifficulty: learning can be much harder than solving a known MDP\\\n\nModel-Based Idea:\\\nLearn an approximate model based on experiences Solve for values as if the learned model were correct\nStep 1: Learn empirical MDP model\nCount outcomes s’ for each s, a\nNormalize to give an estimate of T(s,a,s′)\nDiscover each R(s,a,s′)  when we experience (s, a, s’)\nStep 2: Solve the learned MDP\nFor example, use value iteration, as before\\\n\n\n\nModel-Free Learning:\\\nPassive Reinforcement Learning：\\\nSimplified task: policy evaluation\nInput: a fixed policy π(s)\nYou don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’)\\\n Goal: learn the state values\\\nLearner is “along for the ride”\nNo choice about what actions to take\nJust execute the policy and learn from experience\nThis is NOT offline planning! You actually take actions in the world.\nDirect Evaluation:\\\nGoal: Compute values for each state under π\\\nIdea: Average together observed sample values\\\nAct according to π\nEvery time you visit a state, write down what the sum of discounted rewards turned out to be\nAverage those samples\\\nWhat’s good about direct evaluation?\nIt’s easy to understand\nIt doesn’t require any knowledge of T, R\nIt eventually computes the correct average values, using just sample transitions\\\nWhat bad about it?\nIt wastes information about state connections Each state must be learned separately\nSo, it takes a long time to learn\\\nSimplified Bellman updates calculate V for a fixed policy:\nEach round, replace V with a one-step-look-ahead layer over V\\\n$V^\\pi_{k+1}(s)=\\Sigma_{s'}T(s,\\pi(s),s')[R(s,\\pi(s),s')+\\gamma V_k^\\pi(s')]$\nThis approach fully exploited the connections between the states Unfortunately, we need T and R to do it!\\\nIdea: Take samples of outcomes s’ (by doing the action!) and average\\\n$V^\\pi_{k+1}(s)=n/1\\Sigma_isample_i$\n\\\nTemporal Difference Learning\\\nBig idea: learn from every experience!\nUpdate V(s) each time we experience a transition (s, a, s’, r) Likely outcomes s’ will contribute updates more often\\\nTemporal difference learning of values\\\n  Policy still fixed, still doing evaluation!\\\nMove values toward value of whatever successor occurs: running average\\\nSample of V(s): R(s,a,s′) + γVπ(s′)\\\nUpdate to V(s):Vπ(s) ← Vπ(s) + α[sample− Vπ(s)].\n\\\nExponential moving average\nThe running interpolation update: Makes recent samples more important,Forgets about the past (distant past values were wrong anyway)\n\\\nDecreasing learning rate (alpha) can give converging averages\\\nProblems with TD Value Learning\\\nTD value leaning is a model-free way to do policy evaluation, mimicking Bellman updates with running sample averages\nHowever, if we want to turn values into a (new) policy, we’re sunk:Idea: learn Q-values, not values\nMakes action selection model-free too!\\\n\n\nActive Reinforcement Learning\\\nFull reinforcement learning: optimal policies (like value iteration)\nYou don’t know the transitions T(s,a,s’) You don’t know the rewards R(s,a,s’) You choose the actions now\\\nGoal: learn the optimal policy / values\\\nLearner makes choices!\\\nFundamental tradeoff: exploration vs. exploitation\\\nThis is NOT offline planning! You actually take actions in the world and find out what happens...\\\n\nDetour: Q-Value Iteration\nQ-Learning:sample-based Q-value iteration\n$Q_{k+1}(s,a) =\\Sigma_{s'} T(s,a,s′)[R(s,a,s′) + \\gamma max_{a′} Q_k(s′,a′) ].$\\\nExploration vs. Exploitation\\\nSeveral schemes for forcing exploration\nSimplest: random actions (ε-greedy) Every time step, flip a coin\\\nWith (small) probability ε, act randomly\\\nWith (large) probability 1-ε, act on current policy\\\nAnother solution: exploration functions\\\nTakes a value estimate u and a visit count n, and returns an optimistic utility\ne.g. f(u,n)=u+k/n\\\nRegular Q-Update:\n$ Q_{k+1}(s,a) =\u0007lpha R(s,a,s′) + \\gamma max_{a'}Q(s',a') $\\\nModified Q-Update:\n$Q_{k+1}(s,a) =\u0007lpha R(s,a,s′) +\\gamma max_{a'}f(Q(s',a'),N(s',a')) $\\\nRegret is a measure of your total mistake cost: the difference between your (expected) rewards, including youthful suboptimality, and optimal (expected) rewards\nMinimizing regret goes beyond learning to be optimal – it requires optimally learning to be optimal\\\nApproximate Q-Learning:\nIdeal function: returns the actual minimax value of the position In practice: typically weighted linear sum of features:\\\n$Eval(s)=w_1f_(s)+w_2f_2(s)+\\dots$\n Advantage: our experience is summed up in a few powerful numbers Disadvantage: states may share features but actually be very different in value!\n\\\n\n\n\n## Module 9 Definitions\n* **Term** - your definition here\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Agents in the real world need to handle uncertainty, whether due to partial observability,\nnondeterminism, or adversaries. An agent may never know for sure what state it is in now\nor where it will end up after a sequence of actions.\nThe right thing to do—the rational\ndecision—therefore depends on both the relative importance of various goals and the likelihood that,\nand degree to which, they will be achieved.\\\nuse logic to cope with a domain like\nmedical diagnosis thus fails for three main reasons:\nLAZINESS: It is too much work to list the complete set of antecedents or consequents\nneeded to ensure an exceptionless rule and too hard to use such rules.\nTHEORETICAL IGNORANCE: Medical science has no complete theory for the domain.\nPRACTICAL IGNORANCE: Even if we know all the rules, we might be uncertain about\na particular patient because not all the necessary tests have been or can be run.\\\nThe agent’s\nknowledge can at best provide only a degree of belief in the relevant sentences. Our main\ntool for dealing with degrees of belief is probability theory.\n In the terminology, the ontological commitments of logic and probability theory are the same—that the\nworld is composed of facts that do or do not hold in any particular case—but the\nepistemological commitments are different: a logical agent believes each sentence to be\ntrue or false or has no opinion, whereas a probabilistic agent may have a numerical degree\nof belief between 0 (for sentences that are certainly false) and 1 (certainly true).\nThe theory of probability provides a way of summarizing the uncertainty that comes from our\nlaziness and ignorance, thereby solving the qualification problem.\\\nPreferences, as expressed by utilities, are combined with probabilities in the general theory\nof rational decisions called decision theory: Decision theory = probability theory + utility theory.\\\nThe fundamental idea of decision theory is that an agent is rational if and only if it chooses the\naction that yields the highest expected utility, averaged over all the possible outcomes of the action.\nThis is called the principle of maximum expected utility (MEU).\\\n The primary difference\nis that the decision-theoretic agent’s belief state represents not just the possibilities for world\nstates but also their probabilities.\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n Basic Probability Notation\n In probability\ntheory, the set of all possible worlds is called the sample space. The possible worlds are\nmutually exclusive and exhaustive—two possible worlds cannot both be the case, and one\npossible world must be the case.\\\nA fully specified probability model associates a numerical probability P(ω) with each\npossible world.1\n\n unconditional or prior\nprobabilities (and sometimes just “priors” for short): refer to degrees of belief in\npropositions in the absence of any other information.\n\n\n \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Inference Using Full Joint Distributions:\n Let E be the list of\nevidence variables (just Toothache in the example), let e be the list of observed values for\nthem, and let Y be the remaining unobserved variables (just Catch in the example). The\nquery is P(X | e) and can be evaluated as\n P(X | e) = α P(X, e) = α∑P(X, e, y),\n\n## AIMA Chapter 12.4 Independence\n Independence:\nP(a | b) = P(a) or P(b | a) = P(b) or P(a ∧ b) = P(a)P(b).\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Bayes’ Rule and Its Use:\nP(Y | X) = α P(X | Y )P(Y ),\nwhere is the normalization constant needed to make the entries in sum to 1.\nUsing Bayes’ rule: Combining evidence\nThe general definition of conditional independence of two variables X and Y , given a third\nvariable Z, is\nP(X, Y |Z) = P(X |Z)P(Y |Z).\n As with absolute\nindependence in Equation (12.11) , the equivalent forms：\nP(X | Y ,Z) = P(X |Z) and P(Y | X,Z) = P(Y |Z)\nIn general, for symptoms that are all conditionally independent given \"\" , the size of\nthe representation grows as O(n) instead of O(2n). That means that conditional independence\nassertions can allow probabilistic systems to scale up; moreover, they are much more commonly\navailable than absolute independence assertions.\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n Naive Bayes Models: \nP(Cause, Effect1,…, Effectn) = P(Cause)∏P(Effecti | Cause).\nSuch a probability distribution is called a naive Bayes model—“naive” because it is often\nused (as a simplifying assumption) in cases where the “effect” variables are not strictly\nindependent given the cause variable.\nThe naive Bayes model is sometimes called a\nBayesian classifier\\\n Call the observed effects E = e, while the remaining\neffect variables Y are unobserved. Then the standard method for inference from the joint\ndistribution (Equation (12.9) ) can be applied:\nP(Cause | e) = α∑y P(Cause, e, y).\\\nP(Cause | e) = α∑y P (Cause) P (y|Cause) (∏j P (ej ∣ Cause)) \\\n= α P (Cause) (∏j P (ej∣ Cause))∑y P (y| Cause)\\\n = α P (Cause)∏j P (ej ∣∣ Cause)\n\n\n  Here the “cause”\nis the Category variable, and the “effect” variables are the presence or absence of certain key\nwords, HasWordi.\nThe naive Bayes model assumes that words occur independently in documents, with\nfrequencies determined by the document category.\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n The Wumpus World Revisited \nP(P1,3|known, b) = α ∑frontier P(b|known, P1,3,frontier) ∑other P(P1,3)P(known)P(frontier)P(other)\n = αP(known)P(P1,3) ∑frontier P(b| known, P1,3,frontier)P(frontier) ∑other P(other) \n = α′P(P1,3) ∑frontier P(b| known, P1,3,frontier)P(frontier),\n\nSummary:\nUncertainty arises because of both laziness and ignorance. It is inescapable in complex,\nnondeterministic, or partially observable environments.\\\nProbabilities express the agent’s inability to reach a definite decision regarding the truth\nof a sentence. Probabilities summarize the agent’s beliefs relative to the evidence.\\\nDecision theory combines the agent’s beliefs and desires, defining the best action as the\none that maximizes expected utility.\\\nBasic probability statements include prior or unconditional probabilities and posterior\nor conditional probabilities over simple and complex propositions.\\\nThe axioms of probability constrain the probabilities of logically related propositions.\nAn agent that violates the axioms must behave irrationally in some cases.\\\nThe full joint probability distribution specifies the probability of each complete\nassignment of values to random variables. It is usually too large to create or use in its\nexplicit form, but when it is available it can be used to answer queries simply by adding\nup entries for the possible worlds corresponding to the query propositions.\\\nAbsolute independence between subsets of random variables allows the full joint\ndistribution to be factored into smaller joint distributions, greatly reducing its\ncomplexity.\\\nBayes’ rule allows unknown probabilities to be computed from known conditional\nprobabilities, usually in the causal direction. Applying Bayes’ rule with many pieces of\nevidence runs into the same scaling problems as does the full joint distribution.\\\nConditional independence brought about by direct causal relationships in the domain\nallows the full joint distribution to be factored into smaller, conditional distributions.\nThe naive Bayes model assumes the conditional independence of all effect variables,\ngiven a single cause variable; its size grows linearly with the number of effects.\\\nA wumpus-world agent can calculate probabilities for unobserved aspects of the world,\nthereby improving on the decisions of a purely logical agent. Conditional independence\nmakes these calculations tractable.\n\n\n## SLP Chapter 3.1 N-Grams\n N-Grams:\nhow can we compute probabilities of entire sequences like P(w1 , w2 , ..., wn )? One thing we can do is decompose this probability using the chain rule of proba- bility:\nP(X1...Xn) = P(X1)P(X2|X1)P(X3|X1:2)...P(Xn|X1:n−1) n\n= 􏰃P(Xk|X1:k−1)\\\nApplying the chain rule to words, we get\nP(w1:n) = P(w1)P(w2|w1)P(w3|w1:2)...P(wn|w1:n−1) n\n= 􏰃P(wk|w1:k−1)\\\nThe intuition of the n-gram model is that instead of computing the probability of a word given its entire history, we can approximate the history by just the last few words.\\\nHow do we estimate these bigram or n-gram probabilities? An intuitive way to\nestimate probabilities is called maximum likelihood estimation or MLE. \\\nEquation 3.12 (like Eq. 3.11) estimates the n-gram probability by dividing the\nobserved frequency of a particular sequence by the observed frequency of a prefix.\nThis ratio is called a relative frequency. \\\n\n \n\n## SLP Chapter 3.2 Evaluating Language Models\n Evaluating Language Models\nThe best way to evaluate the performance of a language model is to embed it in an application and measure how much the application improves. Such end-to-end evaluation is called extrinsic evaluation.\\\nAn intrinsic evaluation metric is one that mea- sures the quality of a model independent of any application.\\\nThe perplexity (sometimes called PP for short) of a language model on a test set is the inverse probability of the test set, normalized by the number of words. \\\nthe higher the conditional probabil- ity of the word sequence, the lower the perplexity.\\\n\n \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Sampling sentences from a language model\\Sampling from a distribution means to choose random points\\Thus sampling from a language model—which rep- resents a distribution over sentences—means to generate some sentences, choosing each sentence according to its likelihood as defined by the model. \\ \n\n## SLP Chapter 3.4 Generalization and Zeros\n Generalization and Zeros:\nIt is equally important to get training data in the appropriate dialect or variety, especially when processing social media posts or spoken transcripts.\\\nMatching genres and dialects is still not sufficient. Our models may still be subject to the problem of sparsity. \\\nUnknown Words:\nwe have to deal with words we haven’t seen before, which we’ll\nOOV call unknown words, or out of vocabulary (OOV) words\\\n\nThere are two common ways to train the probabilities of the unknown word model <UNK>. The first one is to turn the problem back into a closed vocabulary one by choosing a fixed vocabulary in advance:\\\n1. Choose a vocabulary (word list) that is fixed in advance.\\\n2. Convert in the training set any word that is not in this set (any OOV word) to\nthe unknown word token <UNK> in a text normalization step.\\\n3. Estimatetheprobabilitiesfor<UNK>fromitscountsjustlikeanyotherregular\nword in the training set.\n \n\n## SLP Chapter 3.5 Smoothing\n Smoothing:\nThis modification is called smoothing or discounting. In this section and the fol- lowing ones we’ll introduce a variety of ways to do smoothing: Laplace (add-one) smoothing, add-k smoothing, stupid backoff, and Kneser-Ney smoothing.\\\nLaplace Smoothing:\nLaplace smoothing merely adds one to each count (hence its alternate name add- one smoothing). \\\nA related way to view smoothing is as discounting (lowering) some non-zero counts in order to get the probability mass that will be assigned to the zero counts. Thus, instead of referring to the discounted counts c∗, we might describe a smooth- ing algorithm in terms of a relative discount dc, the ratio of the discounted counts to the original counts:\ndc = c∗ c\\\nOne alternative to add-one smoothing is to move a bit less of the probability mass from the seen to the unseen events. \\\nIn other words, sometimes using less context is a good thing, helping to general- ize more for contexts that the model hasn’t learned much about. There are two ways to use this n-gram “hierarchy”. In backoff, we use the trigram if the evidence is sufficient, otherwise we use the bigram, otherwise the unigram. In other words, we only “back off” to a lower-order n-gram if we have zero evidence for a higher-order n-gram. By contrast, in interpolation, we always mix the probability estimates from all the n-gram estimators, weighing and combining the trigram, bigram, and unigram counts.\\\nThis kind of backoff with discounting is also called Katz backoff. In Katz back- off we rely on a discounted probability P∗ if we’ve seen this n-gram before (i.e., if we have non-zero counts). Otherwise, we recursively back off to the Katz probabil- ity for the shorter-history (N-1)-gram.\n \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Huge Language Models and Stupid Backoff\\Another option is to build approximate language models using techniques like Bloom filters\\Although with these toolkits it is possible to build web-scale language models using full Kneser-Ney smoothing, Brants et al. (2007) show that with very large lan- guage models a much simpler algorithm may be sufficient. The algorithm is called stupid backoff.\n\n## Module 10 Lecture Notes\n\n Probability Distributions:\\\nMarginal Distributions: $P(X_1=x_1)=\\Sigma_{x_2} P(X_1=x_1,X_2=x_2)$\\\nConditional Probabilities: $P(a|b)=P(a,b)/p(b)$\\\nProbabilistic Inference\\\nGeneral case:\n• Evidencevariables:  $E_1 \\dots E_k$\\\n• Query*variable: Q\\\n• Hidden variables: $H_1 \\dots H_k$\\\nStep 1: Select the entries consistent with the evidence\\\nStep 2: Sum out H to get joint of Query and evidence\\\nStep 3: Normalize\\\nThe Product Rule:\nP(y)P(x|y)=P(x,y)\nThe Chain Rule:\n$P(x_1,x_2,x_3)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)$\\\n$P(x_1,x_2,\\dots ,x_n)=P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)$\\\nBayes’ Rule:\n$P(x,y)=P(x|y)P(y)=P(x|y)P(y|x)P(x)$\\\n$P(x,y)=P(x|y)=(P(y|x)P(x))/P(y)$\\\nProbabilistic Language Models:Probabilistic Language Models\nOne goal: assign a probability to a sentence\\\nJoint probability of words in sentence:\n$P(w_1,w_2,\\dots ,w_n)=\\Pi_i P(w_i|w_1w_2\\dots)$\\\nHow to estimate these probabilities:\\\nSimplifying Assumption = Markov Assumption:\nSimplest case: Unigram model: $P(w_1,w_2,\\dots ,w_n)=\\Pi_i $\\\nBigram model:$P(w_i|w_1,w_2,\\dots ,w_i-1)=\\Pi_i P(w_i|w_{i-1})$\\\nN-gram models: In general this is an insufficient model of language\n• because language has long-distance dependencies\\\nLanguage Modeling:Estimating N-gram Probabilities\\\nEstimating bigram probabilities\n\n \n\n## Module 10 Definitions\n* **Conditional probabilities** - P(a | b) = P(a ∧ b)/ P(b)\n* **Product rule** - P(a ∧ b) = P(a | b)P(b) and P(a ∧ b) = P(b | a)P(a).\n* **Full joint probability distribution** - it follows that a probability model is completely determined by the joint distribution for all of the random variables\n* **Inclusion–exclusion principle** - P(a ∨ b) = P(a) + P(b) − P(a ∧ b)\n* **Marginalization, or summing out** - P (Y) = ∑P (Y, Z = z)\n* **Conditioning** - P(Y) = ∑Z P(Y | Z)P(Z)\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n A Bayesian network is a directed graph in which each node is annotated with quantitative\nprobability information. The full specification is as follows:\n1. Each node corresponds to a random variable, which may be discrete or continuous.\\\n2. Directed links or arrows connect pairs of nodes. If there is an arrow from node X to\nnode Y , X is said to be a parent of Y . The graph has no directed cycles and hence is\na directed acyclic graph, or DAG.\\\n3. Each node Xi has associated probability information θ(Xi|Parents(Xi)) that\nquantifies the effect of the parents on the node using a finite number of parameters.\\\nThe local probability information attached to each node in Figure takes the form of a\nconditional probability table (CPT).\\\nEach row in a CPT contains the conditional probability of each node value for a\nconditioning case.\n \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n The Semantics of Bayesian Networks\n The semantics of Bayes nets defines each entry in the joint distribution as follows:\n P(x1,…,xn) = n∏i=1 θ(xi|parents(Xi)),\\\n P(x1,…,xn) = n∏i=1 P(xi|parents(Xi)).\\\n  product rule: P(x1,…,xn) = P(xn|xn−1,…,x1)P(xn−1,…,x1).\\\n   Chain rule:\n  P(x1,…,xn) = P(xn|xn−1,…,x1)P(xn−1|xn−2,…,x1) ⋯ P(x2|x1)P(x1)  = n∏i=1 P(xi|xi−1,…,x1). \\\n  P(Xi|Xi−1,…,X1) = P(Xi|Parents(Xi))\\\n\n\n  Compactness and node ordering\\\n  The most general conditional independence question one might ask in a Bayes net is\nwhether a set of nodes X is conditionally independent of another set Y, given a third set Z.\nThis can be determined efficiently by examining the Bayes net to see whether Z d-separates\nX and Y. The process works as follows:\\\n1. Consider just the ancestral subgraph consisting of X, Y, Z, and their ancestors.\\\n2. Add links between any unlinked pair of nodes that share a common child; now we\nhave the so-called moral graph.\\\n3. Replace all directed links by undirected links.\\\n4. If Z blocks all paths between X and Y in the resulting graph, then Z d-separates X\nand Y. In that case, X is conditionally independent of Y, given Z. Otherwise, the\noriginal Bayes net does not require conditional independence.\\\nIn brief, then, d-separation means separation in the undirected, moralized, ancestral\nsubgraph.\\\n\n In fact, this is a worst-case scenario in which the relationship between\nthe parents and the child is completely arbitrary. Usually, such relationships are describable\nby a canonical distribution that fits some standard pattern.\\\nAnother important pattern that occurs often in practice is context-specific independence or\nCSI.\n\\\nBayesian nets with continuous variables:\n1)One way to handle continuous variables is with discretization—that is, dividing up\nthe possible values into a fixed set of intervals.\\\n2)another solution—sometimes called a nonparametric representation—is to define the\nconditional distribution implicitly with a collection of instances, each containing specific\nvalues of the parent and child variables.\\\n  hybrid Bayesian network.: The most common choice is the linear–\nGaussian conditional distribution, in which the child has a Gaussian distribution whose\nmean μ varies linearly with the value of the parent and whose standard deviation σ is fixed.\\\nEach variable is conditionally independent of its non-descendants, given its parents.\\\n When discrete variables are added as parents (not as children) of\ncontinuous variables, the network defines a conditional Gaussian, or CG, distribution:\ngiven any assignment to the discrete variables, the distribution over the continuous\nvariables is a multivariate Gaussian.\\\nthe distributions for discrete variables with continuous parents:1)   One way to make soft thresholds is to use the\nintegral of the standard normal distribution\\\n2)An alternative to the probit model is the expit or inverse logit model.\n \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Exact Inference in Bayesian Networks\\  Inference by enumeration\\ a query can be answered using a Bayes net by computing sums of products of conditional\nprobabilities from the network.\nthere will be$O(2^n)$ terms in the sum, each a product of $O(n)$probability values. A naive implementation\nwould therefore have complexity $O(n2^n)$\\\nThis can be reduced to by taking advantage of the nested structure of the\ncomputation.\\\nThe ENUMERATION-ASK algorithm: you will see that it contains\nrepeated subexpressions. $O(2^n)$\\\n 2)The variable elimination algorithm:  do the calculation\nonce and save the results for later use.,This is a form of dynamic programming.\\\n e.g. variable elimination algorithm,：Variable ordering and variable relevance： Every choice of ordering yields a valid algorithm, but different\norderings cause different intermediate factors to be generated during the calculation.\\\nIn general, the time and space requirements of variable elimination are dominated by the\nsize of the largest factor constructed during the operation of the algorithm. This in turn is\ndetermined by the order of elimination of variables and by the structure of the network.\\\n 3)The complexity of exact inference\nThe complexity of exact inference in Bayes nets depends strongly on the structure of the\nnetwork.\\\n These are called singly connected networks or polytrees, and\nthey have a particularly nice property: The time and space complexity of exact inference in\npolytrees is linear in the size of the network.\\\nAs well as reducing satisfiability problems to Bayes net inference, we can reduce Bayes net\ninference to satisfiability, which allows us to take advantage of the powerful machinery\ndeveloped for SAT-solving (see Chapter 7 ). In this case, the reduction is to a particular\nform of SAT solving called weighted model counting (WMC).\\\n4)Clustering algorithms: Using clustering algorithms (also\nknown as join tree algorithms), the time can be reduced to O(n).\\\nThe two Boolean nodes are replaced by a meganode that takes on four possible\nvalues: tt, tf, ft, and ff. The meganode has only one parent.\n\\\n \n\n## Module 11 Lecture Notes\n \n Independence:\n $P(x,y)=P(x)P(y)$ or $P(x|y)=P(x)$\\\n Conditional Independence:\n $P(x|z,y)=P(x)P(z)$ or $P(x,y|z)=P(x|z)P(x=y|z)$\\\n Bayes’ Nets(graphical models): a technique for describing complex joint distributions (models) using simple, local distributions (conditional probabilities)\\\nA Bayes net = Topology (graph) + Local Conditional Probabilities\\\nProbabilities in BNs: $P(x_1,x_2,\\dots ,x_n)=\\Pi_{i=1}^{n} P(x_i|Parents(x_{i}))$\\\nCausal Chains: x->y->z $P(x,y,z)=P(x)P(y|x)P(z|y)$ Not Guaranteed X independent of Z ; Guaranteed X independent of Z given Y\\\nCommon Cause: x<-y->z $P(x,y,z)=P(y)P(x|y)P(z|y)$ Not Guaranteed X independent of Z ; Guaranteed X independent of Z given Y\\\nCommon Effect(v-structures): x->z<-y $P(x,y,z)=P(y)P(x|y)P(z|y)$ Guaranteed X independent of Z ; Not  X and Y independent given Z\\\nReachability\\\nActive Triples\n\n\nConsider all (undirected) paths from X to Y: if No active paths = independence\\\nA path is active if each triple is active\\\n1)Causal chain A -> B -> C where B is unobserved (either direction)\n2)Common cause A<-B -> C where B is unobserved\n3) Common effect (aka v-structure)\nA -> B <- C where B or one of its descendents is observed\\\n\n1)If one or more paths is active, then independence not guaranteed \\\n2)if all paths are inactive then independence is guaranteed\\\n3)A path is active if each triple is active\\\nInference: calculating some useful quantity from a joint probability distribution\\\nInference by Enumeration:\\\nWe want $P(Q|e_1,e_2\\dots)$\\\nStep 1: Select the entries consistent with the evidence\\\nStep 2: Sum out H to get joint of Query and evidence: $P(Q,e_1,e_2\\dots, e_k)=\\Sigma_{h_1\\dots h_r}P(Q,h_1\\dots h_r,e_1,e_2\\dots, e_k)$\\\nStep 3: Normalize\\\n $P(Q|e_1,e_2\\dots, e_k)=1/Z*P(Q,e_1,e_2\\dots, e_k)$\\\n Inference by Enumeration:slow->You join up the whole joint distribution before you sum out the hidden variables\\\n  Advanced technique: Variable Elimination\n§ Interleave joining and marginalizing\n§ Still NP-hard, but usually much faster than inference by enumeration\n \n\n## Module 11 Definitions\n* **Independence** - $P(x,y)=P(x)P(y)$ or $P(x|y)=P(x)$\n* **Conditional independence** - $P(x|z,y)=P(x)P(z)$ or $P(x,y|z)=P(x|z)P(x=y|z)$\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Forms of Learning：\nWhen the output is one of a finite set of values (such as sunny/cloudy/rainy or true/false),\nthe learning problem is called classification. When it is a number (such as tomorrow’s\ntemperature, measured either as an integer or a real number), the learning problem has the\n(admittedly obscure1) name regression.\\\nThere are three types of feedback that can accompany the inputs, and that determine the\nthree main types of learning:\\\nsupervised learning\\\nunsupervised learning: The most common unsupervised learning task is clustering\\\n reinforcement learning\n\n \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Supervised Learning:\nThe function h is called a hypothesis about the world. It is drawn from a hypothesis space\nH of possible functions.\\\nWith alternative vocabulary, we can say that h is a model of the data, drawn from a model\nclass H, or we can say a function drawn from a function class. We call the output yi the\nground truth—the true answer we are asking our model to predict.\\\n we can perform exploratory data analysis:examining the data with statistical tests and visualizations—histograms, scatter plots, box\nplots—to get a feel for the data, and some insight into what hypothesis space might be\nappropriate.\\\n How do we choose a good hypothesis from within the hypothesis space? We could hope for\na consistent hypothesis: and h such that each xi in the training set has h(xi) = yi. With\ncontinuous-valued outputs we can’t expect an exact match to the ground truth; instead we\nlook for a best-fit function for which each h(xi) is close to yi\\\nOne way to analyze hypothesis spaces is by the bias they impose (regardless of the training\ndata set) and the variance they produce (from one training set to another).\\\nOne way to analyze hypothesis spaces is by the bias they impose (regardless of the training\ndata set) and the variance they produce (from one training set to another).\\\n the small difference in the data set translates into a small difference in the\nhypothesis. We call that low variance.\\\nOften there is a bias–variance tradeoff: a choice between more complex, low-bias\nhypotheses that fit the training data well and simpler, low-variance hypotheses that may\ngeneralize better.\\\n\n\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Learning Decision Trees\nA decision tree is a representation of a function that maps a vector of attribute values to a\nsingle output value—a “decision.”\\\n  Learning decision trees from examples:\nLEARN-DECISION-TREE algorithm adopts a greedy divide-and-conquer strategy: always test the\nmost important attribute first, then recursively solve the smaller subproblems that are\ndefined by the possible results of the test.\\\n There are\nfour cases to consider for these recursive subproblems:\\\n1. If the remaining examples are all positive (or all negative), then we are done: we\ncan answer Yes or No.\\\n2. If there are some positive and some negative examples, then choose the best\nattribute to split them.\\\n3. If there are no examples left, it means that no example has been observed for this\ncombination of attribute values, and we return the most common output value from\nthe set of examples that were used in constructing the node’s parent.\\\n4. If there are no attributes left, but both positive and negative examples, it means that\nthese examples have exactly the same description, but different classifications. This\ncan happen because there is an error or noise in the data\\\nWe can evaluate the performance of a learning algorithm with a learning curve,\\\n Choosing attribute tests:The decision tree learning algorithm chooses the attribute with the highest IMPORTANCE. We\nwill now show how to measure importance, using the notion of information gain, which is\ndefined in terms of entropy\\\nEntropy: H(V ) = −∑k P(vk)log2 P(vk).\\\n define as the entropy of a Boolean random variable that is true with\nprobability:B(q) = −(q log2 q + (1 − q)log2(1 − q)).\\\n If a training\nset contains positive examples and negative examples:\nH(Output) = B(p/(p+n) ).\nRemainder(A) = d∑k=1  pk + nk/ p + n B(pk/ pk + nk ). \\\nGain(A) = B( p/p + n) − Remainder(A).\\\n Generalization and overfitting: For decision trees, a technique called decision tree pruning combats overfitting.:\n  how large a gain should we require in order to\nsplit on a particular attribute?using a statistical significance test.\\\n combine them using an approach called early stopping\\\n Broadening the applicability of decision trees:\n 1, MISSING DATA;2, CONTINUOUS AND MULTIVALUED INPUT ATTRIBUTES:\\\n CONTINUOUS-VALUED OUTPUT ATTRIBUTE:The name CART, standing for Classification And Regression Trees, is used to cover\nboth classes.\n \n\n## AIMA Chapter 5.5 Stochastic Games\n Model Selection and Optimization:\nFirst we will make the assumption that the future examples will be like the past. We call this\nthe stationarity assumption;;\\\n a training set to create the hypothesis, and a test set to evaluate it.;\\\n we might want to compare two completely\ndifferent machine learning models: We call these “knobs” hyperparameters—parameters\nof the model class, not of the individual model.\\\n1. A training set to train candidate models.\n2. A validation set, also known as a development set or dev set, to evaluate the\ncandidate models and choose the best one.\\\nWhat if we don’t have enough data to make all three of these data sets? We can squeeze\nmore out of the data using a technique called k-fold cross-validation\\\nThe extreme is k = n, also known as leave-one-out cross-validation or LOOCV. Even with\ncross-validation, we still need a separate test set.\\\n We can think of the task of finding a good hypothesis as two\nsubtasks: model selection4 chooses a good hypothesis space, and optimization (also called\ntraining) finds the best hypothesis within that space.\\\n\nModel selection\\\nWe say that a model that exactly fits all the training data has interpolated the data\n\n\n Regularization\\\nAnother way to simplify models is to reduce the dimensions that the models work with. A\nprocess of feature selection\\\n\nHyperparameter tuning:The simplest approach to hyperparameter tuning is hand-tuning: guess some parameter\nvalues based on past experience, train a model, measure its performance on the validation\ndata, analyze the results, and use your intuition to suggest new parameter values.\\\na more systematic approach called grid search is appropriate:\n Bayesian optimization treats the task of choosing good hyperparameter values as\na machine learning problem in itself.\\\nAn alternative to Bayesian optimization is population-based training (PBT). PBT starts by\nusing random search to train (in parallel) a population of models, each with different\nhyperparameter values.\n\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\nThe Theory of Learning:\nany hypothesis that is consistent with a sufficiently large set of training examples is\nunlikely to be seriously wrong: that is, it must be probably approximately correct (PAC).\\\n One can think of an\napproximately correct hypothesis as being “close” to the true function in hypothesis space: it\nlies inside what is called the ϵ-ball around the true function f. The hypothesis space outside\nthis ball is called H bad.']\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n 19.6 Linear Regression and Classification\\ Univariate linear regression\\ The task of\nfinding the hw that best fits these data is called linear regression.\\\nWe see that the loss function is convex, this is\ntrue for every linear regression problem with an loss function, and implies that there are\nno local minima.\\\n \n Gradient descent\\\n The parameter which we called the step size in Section 4.2 , is usually called the\nlearning rate when we are trying to minimize loss in a learning problem.\\\nFor univariate regression, the loss is quadratic, so the partial derivative will be linear. (The\nonly calculus you need to know is the chain rule\\\nThese updates constitute the batch gradient descent learning rule for univariate linear\nregression (also called deterministic gradient descent).\\\nA faster variant is called stochastic gradient descent or SGD: it randomly selects a small\nnumber of training examples at each step, and updates\\\nA step that covers all the training examples is called an epoch.\\\n\nMultivariable linear regression\\\nWe call the expression (X⊤X)−1X⊤\n the pseudoinverse of the data matrix, w∗ = (X⊤X)−1X⊤y. is called the normal equation.\\\nWith univariate linear regression we didn’t have to worry about overfitting. But with\nmultivariable linear regression in high-dimensional spaces it is possible that some\ndimension that is actually irrelevant appears by chance to be useful, resulting in overfitting.\\\n$Complexity(h_w) = \\Sigma |w_i|^q. $\nAs with loss functions, with q = 1 we have L1 regularization9, which minimizes the sum of\nthe absolute values; with q = 2, L2 regularization minimizes the sum of squares. Which\nregularization function should you pick? That depends on the specific problem, but L1\nregularization has an important advantage: it tends to produce a sparse model. That is, it\noften sets many weights to zero, effectively declaring the corresponding attributes to be\ncompletely irrelevant\\\n\n Linear classifiers with a hard threshold\\\nA decision boundary is a line (or a surface, in higher dimensions) that separates the two\nclasses.   A linear decision\nboundary is called a linear separator and data that admit such a separator are called\nlinearly separable. \\\n\nAlternatively, we can think of h as the result of passing the linear function w ⋅ x through a\nthreshold function\\\na simple weight update rule that converges to a solution—that is, to a\nlinear separator that classifies the data perfectly—provided the data are linearly separable.\nFor a single example (x, y), we have\nwi ← wi + α (y − hw(x)) × xi\\\nThis\nrule is called the perceptron learning rule\\\n\n Linear classification with logistic regression\\\n We have seen that passing the output of a linear function through the threshold function\ncreates a linear classifier; yet the hard nature of the threshold causes some problems\\\nAll of these issues can be resolved to a large extent by softening the threshold function—\napproximating the hard threshold with a continuous, differentiable function.\\\n\n  \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n  \nNaive Bayes Classifiers\n multinomial naive Bayes classifier:\nThe intuition of the classifier is shown in Fig. 4.1. We represent a text document\nas if it were a bag-of-words, that is, an unordered set of words with their position ignored, keeping only their frequency in the document.\\\n\nNaive Bayes is a probabilistic classifier, meaning that for a document d, out of all classes c ∈ C the classifier returns the class cˆ which has the maximum posterior ˆ probability given the document. \\\ncˆ = argmaxP(c|d)\\\ncˆ = argmaxP(c|d) = argmaxP(d|c)P(c)\\\n\nNaive Bayes classifiers therefore make two simplifying assumptions:The first is the bag of words assumption discussed intuitively above,; The second is commonly called the naive Bayes assumption: this is the condi- tional independence assumption \n$c_{NB} = argmax P(c) 􏰄\\Pi P(f|c)$\ncNB = argmax P(c) 􏰄 P(wi |c)\n Classifiers that use a linear combination of the inputs to make a classification decision —like naive Bayes and also logistic regression— are called linear classifiers.\n \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Training the Naive Bayes Classifier \nNote once again that it is crucial that the vocabulary V consists of the union of all the word types in all classes, not just the words in one class c (try to convince yourself why this must be true\\\nWhat do we do about words that occur in our test data but are not in our vocab- ulary at all because they did not occur in any training document in any class? The solution for such unknown words is to ignore them—remove them from the test document and not include any probability for them at all.\\\nFinally, some systems choose to completely ignore another class of words: stop words\\\n\n\n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Optimizing for Sentiment Analysis：First, for sentiment classification and a number of other text classification tasks, whether a word occurs or not seems to matter more than its frequency.、、\n This variant is called binary\nmultinomial naive Bayes or binary NB\n \n \n \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Naive Bayes for other text classification tasks：\nthe most effective naive Bayes features are not words at all, but character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’, ‘thei’), or, even simpler byte n-grams,；\n \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Naive Bayes as a Language Model:\\Specifically, a naive Bayes model can be viewed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. \\Since the likelihood features from the naive Bayes model assign a probability to each word P(word|c), the model also assigns a probability to each sentence:\n$P(s|c) = 􏰄 \\Pi P(w_i|c)  i∈ positions$\nonce we multiply in the prior a full naive Bayes model might well make a different classification decision.\\\n\n\n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Evaluation: Precision, Recall, F-measure;\refer to these human labels as the gold labels. \\To evaluate any system for detecting things, we start by building a confusion matrix\\e.g.true positives are documents that are indeed spam (indicated by human-created gold labels) that our system correctly said were spam. False negatives are documents that are indeed spam but our system incorrectly labeled as non-spam.\\That’s why instead of accuracy we generally turn to two other metrics shown in Fig. 4.4: precision and recall. Precision measures the percentage of the items that the system detected (i.e., the system labeled as positive) that are in fact positive (i.e., are positive according to the human gold labels). Precision is defined as\nPrecision =\n true positives/\ntrue positives + false positives\\\nRecall measures the percentage of items actually present in the input that were correctly identified by the system. Recall is defined as\\\nRecall =true positives/\ntrue positives + false negatives\\\nThe simplest of these combinations is the F-measure F1 = 2PR/(P+R)\\\n In macroaveraging, we compute the performance for each class, and then average over classes. In microaveraging, we collect the de- cisions for all classes into a single confusion matrix, and then compute precision and recall from that table. \\\n\n\n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Test sets and Cross-validation\nwe use the training set to train the model, then use the development test set (also called a devset) to perhaps tune some parameters\\\nin order to save lots of data for training, the test set (or devset) might not be large enough to be representative. Wouldn’t it be better if we could somehow use all our data for training and still use\nall our data for test? We can do this by cross-validation.\\\nIn cross-validation, we choose a number k, and partition our data into k disjoint\nsubsets called folds. \n \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Statistical Significance Testing:\\In the paradigm of statistical hypothesis testing, we test this by formalizing two hypotheses.\nH0 :δ(x)≤0\nH1 : δ(x)>0 (4.20)\nThe hypothesis H0, called the null hypothesis, supposes that δ(x) is actually nega- tive or zero, meaning that A is not better than B. We would like to know if we can confidently rule out this hypothesis, and instead support H1, that A is better.\n\n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Avoiding Harms in Classification\\ \n\n## Module 12 Lecture Notes\n\nNaïve Bayes: Assume all features are independent effects of the label\\\nA general Naive Bayes model:$P(Y,F_1,\\dots F_n)=P(Y)\\Pi_i P(F_i|Y)$\\\n$|Y|*|F|^n$ values->$|Y|*|F|*n$\nTotal number of parameters is linear in number of features Model is very simplistic\n Inference for Naïve Bayes\n Goal: compute posterior distribution over label variable Y \n • Step 1: get joint probability of label and evidence for each label\n      $P(Y,F_1, \\dots F_n)=\\\n      [P(y_1,f_1, \\dots f_n)\\\n      P(y_2,f_1, \\dots f_n)\\\n      \\dots\\\n      P(y_k,f_1, \\dots f_n)\n      ]$=\\\n      $[P(y_1)\\Pi_i P(f_i|y_1)\\\n      P(y_2)\\Pi_i P(f_i|y_2)\\\n      \\dots\\\n      P(y_k)\\Pi_i P(f_i|y_k)]$\\\n  \n  $P(Y,f_1, \\dots f_n)/P(f_1, \\dots f_n)=P(Y|f_1, \\dots f_n)$\\\n  • Step 2: sum to get probability of evidence\\\n• Step3:normalizebydividingStep1byStep2\\\nNaïve Bayes for Text:\nBag-of-words Naïve Bayes:\\\n• Features: Wi is the word at positon i\\\n• As before: predict label conditioned on feature variables (spam vs. ham) • As before: assume features are conditionally independent given label\\\n• New: each Wi is identically distributed\\\n“Tied” distributions and bag-of-words\\\n• Usually, each variable gets its own conditional probability distribution P(F|Y)\\\n • In a bag-of-words model\\\n• Each position is identically distributed\\\n• All positions share the same conditional probs P(W|Y)\\\n• Called “bag-of-words” because model is insensitive to word order or reordering\\\n\n\nImportant Concepts:\\\n\nData: labeled instances, e.g. emails marked spam/ham\\\n• Training set\n• Held out set\n• Test set\\\nFeatures: attribute-value pairs which characterize each x\\\nExperimentation cycle\\\n• Learn parameters (e.g. model probabilities) on training set\n• (Tune hyperparameters on held-out set)\n• Compute accuracy of test set\n• Very important: never “peek” at the test set!\\\nEvaluation\\\n• Accuracy: fraction of instances predicted correctly\\\nOverfitting and generalization\\\n• Want a classifier which does well on test data\n• Overfitting: fitting the training data very closely, but not generalizing well\n• We’ll investigate overfitting and generalization formally in a few lectures\\\n\nGeneralization and Overfitting:\nRelative frequency parameters will overfit the training data!\\\nTo generalize better: we need to smooth or regularize the estimates\\\n\nParameter Estimation:Estimating the distribution of a random variable\\\nEmpirically: use training data (learning!)\\\nThis is the estimate that maximizes the likelihood of the data:\n$P_{ML}(x)=count(x)/total samples$\\\nThis is the estimate that maximizes the likelihood of the data\\\n$L(x,\theta))=\\Pi_i P(x_i)$\nMaximum Likelihood: Relative frequencies are the maximum likelihood estimates\n Laplace Smoothing:Pretendyousaweveryoutcome once more than you actually did\\\n Laplace’s estimate (extended):Pretend you saw every outcome k extra times\nConfidences from a Classifier\\\nThe confidence of a probabilistic classifier: \n• Posterior over the top label:\n$confidence(x)=max_y P(y|x)$\n• Represent how sure the classifier is of the classification\n• Any probabilistic model will have confidences\n• No guarantee confidence is correct\nSummary\nBayes rule lets us do diagnostic queries with causal probabilities\\\nThe naïve Bayes assumption takes all features to be independent given the class label\\\nWe can build classifiers out of a naïve Bayes model using training data Smoothing estimates is important in real systems\\\nClassifier confidences are useful, when you can get them\\\n\nPerceptrons\\\nLinear Classifiers\nInputs are feature values; Each feature has a weight; Sum is the activation\\\nIf the activation is:\n• Positive, output +1 • Negative, output -1\nWeights:\nWeight Updates:Start with weights = 0\nFor each training instance:\n• Classify with current weights: y=+1 if $wf(x)>=0$ else y=-1\\\n• If correct (i.e., y=y*), no change!\\\n• If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1. $w=w+y^*f$\\\nMulticlass Decision Rule:  Prediction highest score wins: $y=argmax_y w_y f(x)$\n\nLearning: Multiclass Perceptron\nStart with all weights = 0\nPick up training examples one by one Predict with current weights\nIf wrong: lower score of wrong answer, raise score of right answer: $w_y=w_y+f(x)$, $w_{y^*}=w_{y^*}+f(x)$\\\n\nProperties of Perceptrons: Separability: true if some parameters get the training set perfectly correct\\\nConvergence: if the training is separable, perceptron will eventually converge (binary case)\\\nMistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separability\\\n\nProblems with the Perceptron:\\\nNoise: if the data isn’t separable, weights might thrash\n• Averaging weight vectors over time can help (averaged perceptron)\\\nMediocre generalization: finds a “barely” separating solution\\\nOvertraining: test / held-out accuracy usually rises, then falls\n• Overtraining is a kind of overfitting\\\n\nIdea: adjust the weight update to mitigate these effects\nMIRA*: choose an update size that fixes the current mistake... but, minimizes the change to w\\\n$w_y=w'_y+\taut f(x)$, $w'_{y^*}=w'_{y^*}+\taut f(x)$\\\nMinimum Correcting Update\nMaximum Step Size:You may not have enough features\n• Solution: cap the maximum possible value of t with some constant C\n$\taut^*=min((w'_y-w_{y^*}f+1)/2ff),C) $\\\nSupport Vector Machines: Maximizing the margin\\\nBasically, SVMs are MIRA where you optimize over all examples atonce\\\nClassification: Comparison\nNaïve Bayes\\\n• Builds a model training data\n• Gives prediction probabilities\n• Strong assumptions about feature independence • One pass through data (counting)\\\nPerceptrons / MIRA:\\\n• Makes less assumptions about data\n• Mistake-driven learning\n• Multiple passes through data (prediction) • Often more accurate\\\n\n \n\n## Module 12 Definitions\n* **Concept** - definition\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\nA generative model like naive Bayes makes use of this likelihood term, which expresses how to generate the features of a document if we knew it was of class c.\nBy contrast a discriminative model in this text categorization scenario attempts to directly compute P(c|d). Perhaps it will learn to assign a high weight to document features that directly improve its ability to discriminate between possible classes, even if it couldn’t generate an example of one of the classes.\\\n\n\n Classification: the sigmoid:\nLogistic regression solves this task by learning, from a training set, a vector of weights and a bias term. \\\nz = w·x+b\\\nTo create a probability, we’ll pass z through the sigmoid function, σ(z). The sigmoid function (named because it looks like an s) is also called the logistic func- tion, and gives logistic regression its name.\\\nσ(z)= 1 / 1+exp(−z)\\\nWe call .5 the decision boundary\\\nFor logistic regression and naive Bayes these combination features or feature interactions have to be designed by hand.\\\nFor many tasks (especially when feature values can reference specific words) we’ll need large numbers of features. Often these are created automatically via fea- ture templates, abstract specifications of features.\\\nIn order to avoid the extensive human effort of feature design, recent research in NLP has focused on representation learning: ways to learn features automatically in an unsupervised way from the input.\\\nLogistic regression has a number of advantages over naive Bayes. Naive Bayes has overly strong conditional independence assumptions. \\\nFurthermore, naive Bayes can work extremely well (some- times even better than logistic regression) on very small datasets (Ng and Jordan, 2002) or short documents (Wang and Manning, 2012). Furthermore, naive Bayes is easy to implement and very fast to train (there’s no optimization step).\\\n\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Learning in Logistic Regression\nthe distance between the system output and the gold output, and we call this distance\nloss the loss function or the cost function. \\\n\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n The cross-entropy loss function\\This is called conditional maximum likelihood estimation: we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x. The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss. \n\n## SLP Chapter 5.4 Gradient Descent\n Gradient Descent:\\For logistic regression, this loss function is conveniently convex. A convex func- tion has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum.The magnitude of the amount to move in gradient descent is the value of the slope d L( f (x; w), y) weighted by a learning rate η . A higher (faster) learning rate means that we should move w more on each step.\\\n\nIt is common to start with a higher learning rate and then slowly decrease it, so that it is a function of the iteration k of training; the notation ηk can be used to mean the value of the learning rate at iteration k.\\\n\nMini-batch training\\\nFor example in batch training we compute the gradient over the entire dataset. By seeing so many examples, batch training offers a superb estimate of which di- rection to move the weights, at the cost of spending a lot of time processing every single example in the training set to compute this perfect direction.\\\nThe mini-batch gradient is the average of the individual gradients\\\n\n \n\n## SLP Chapter 5.5 Regularization\n Regularization:\\A good model should be able to generalize well from the training data to the unseen test set, but a model that overfits will have poor generalization.\nTo avoid overfitting, a new regularization term R(θ) is added to the objective function\\\nL2 regularization is a quadratic function of the weight values, named because it uses the (square of the) L2 norm of the weight values. The L2 norm, ||θ||2, is the same as the Euclidean distance of the vector θ from the origin. If θ consists of n weights,\\\nL1 regularization is a linear function of the weight values, named after the L1 norm ||W||1, the sum of the absolute values of the weights, or Manhattan distance\\\nThese kinds of regularization come from statistics, where L1 regularization is called lasso regression (Tibshirani, 1996) and L2 regularization is called ridge regression, and both are commonly used in language processing.\\\n L2 regularization is easier to optimize because of its simple derivative (the derivative of θ2 is just 2θ), while L1 regularization is more complex (the derivative of |θ | is non-continuous at zero).\\\n But where L2 prefers weight vectors with many small weights, L1 prefers sparse solutions with some larger weights but many more weights set to zero. Thus L1 regularization leads to much sparser weight vectors, that is, far fewer features.\\\nBoth L1 and L2 regularization have Bayesian interpretations as constraints on the prior of how weights should look. L1 regularization can be viewed as a Laplace prior on the weights. L2 regularization corresponds to assuming that weights are distributed according to a Gaussian distribution with mean μ = 0.\\\n\n\n \n\n## SLP Chapter 5.6 Multinomial logistic regression\n Multinomial logistic regression:\nIn such cases we use multinomial logistic regression, also called softmax re- gression (or, historically, the maxent classifier).\n\\\nFor a vector z of dimensionality k, the softmax is defined as:\n\n$softmax(zi) =exp(zi)/ \\Sigma􏰂^k_{j=1} exp(zj) 1 ≤ i ≤ k$\n\n\n## SLP Chapter 7.1 Units\n Units： tanh : the tanh func- tion has the nice properties of being smoothly differentiable and mapping outlier(a problem called the vanishing gradient problem.)ReLU:y = max(z, 0),The rectifier function, on the other hand has nice properties that result from it being very close to linear.\\\n\n \n\n## SLP Chapter 7.2 The XOR problem\n The XOR problem\\This example was first shown for the perceptron, which is a very simple neural unit that has a binary output and does not have a non-linear activation function\\It turns out, however, that it’s not possible to build a perceptron to compute logical XOR! \nThe intuition behind this important result relies on understanding that a percep- tron is a linear classifier. \\\nWe say that XOR is not a linearly separable function. \\\nThe solution: neural networks:\n\n\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Feedforward Neural Networks:units in modern multilayer networks aren’t perceptrons (perceptrons are purely linear, but modern networks are made up of units with non-linearities like sigmoids\\\n\n\nThe core of the neural network is the hidden layer h formed of hidden units hi, each of which is a neural unit as described in Section 7.1, taking a weighted sum of its inputs and then applying a non-linearity. \n \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Feedforward networks for NLP: Classification:The idea of using word2vec or GloVe embeddings as our input representation — and more generally the idea of relying on another algorithm to have already learned an embedding representation for our input words — is called pretraining. \\\n Instead of using hand-built human-engineered features as the input to our classifier, we draw on deep learning’s ability to learn features from the data by representing words as word2vec or GloVe embeddings (Chapter 6). For a text with n input words/tokens w1,...,wn, the input vector will be the concatenated embeddings of the n words: [ew1 ; ...; ewn ].\\\n\n\n\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Feedforward Neural Language Modeling:\nCompared to n-gram models, neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction.\\\n\nNeural language models represent words in this prior context by their embed- dings, rather than just by their word identity as used in n-gram language models. Using embeddings allows neural language models to generalize better to unseen data. \\\n \n\n## SLP Chapter 7.6 Training Neural Nets\n Training Neural Nets:\nHow do we partial out the loss over all those intermediate layers? The answer is the algorithm called error backpropagation or backward differentiation.\\\n\nThe cross-entropy loss that is used in neural networks is the same one we saw for logistic regression.\\\n\nthe cross-entropy loss is simply the log of the output probability correspond- ing to the correct class, and we therefore also call this the negative log likelihood loss:\nLCE(yˆ,y) = −logyˆi, (where i is the correct class)\\\n\nThe solution to computing this gradient is an algorithm called error backprop- agation or backprop (Rumelhart et al., 1986). While backprop was invented spe- cially for neural networks, it turns out to be the same as a more general procedure called backward differentiation, which depends on the notion of computation graphs. \\\n\nBackward differentiation on computation graphs\\\nBackwards differentiation makes use of the chain rule in calculus\\\n\nVarious forms of regularization are used to prevent overfitting. One of the most important is dropout: randomly dropping some units and their connections from the network during training (Hinton et al. 2012, Srivastava et al. 2014). Tuning of hyperparameters is also important.\\\n\n\n\n## SLP Chapter 7.7 Training the neural language model\n \nTraining the neural language model:\n\nTraining the parameters to minimize loss will result both in an algorithm for language modeling (a word predictor) but also a new set of embeddings E that can be used as word representations for other tasks. \n\n## Module 13 Lecture Notes\nLogistic Regression:\nClassifier components:\n1. A feature representation of the input.Foreachinputobservation x(i), this will be a vector of features [x1,x2,...,xn].\n2. A classification function that computes the estimated class 𝑦 via p(y|x).\n3. Annobjective function for learning, usually involving minimizing error on training examples.\n4. Analgorithm for optimizing the objective function.\\\nSentiment classifier\\\nDecision boundary\\\n\n$\\hat y=1 \\ if\\  P(y=1|x)>0.5\\ else\\ 0$\\\nExtracting Features\\\n$z=\\Sigma_i w_ix_i+b$, $\\sigma(z)=1/(1+e^{-(wx+b)})$\\\n1. An objective function or loss function that tells us distance between the system output and the gold output. We will use cross-entropy loss.\\\n2. An algorithm for optimizing the objective function. We will use stochastic gradient descent to minimize the loss function.\nLoss functions: eg. mean squared error \n$L_{MSE}(\\hat y,y)=1/2(\\hat y-y)^2$\\\nLoss functions for probabilistic classification:cross entropy loss\\\nFor one observation x, let’s maximize the probability of the correct label p(y|x).\n$p(y|x)=\\hat y^y(1-\\hat y)^{1-y}$\\\nChange to logs (still maximizing)\n$logp(y|x)=log[\\hat y^y(1-\\hat y)^{1-y}]=ylog\\hat y +(1-y)log(1-\\hat y)$\\\nCross-entropy loss: The result is cross-entropy loss:$-logp(y|x)=-(ylog\\hat y +(1-y)log(1-\\hat y))$\\\n$-logp(y|x)=-(ylog(wx+b)) +(1-y)log(1-(wx+b))))$\\\nGradient descent is a method that finds a minimum of a function by figuring out in which direction (in the space of the parameters θ) the function’s slope is rising the most steeply, and moving in the opposite direction.\\\n For logistic regression, this loss function is conveniently convex.\nA convex function has just one minimum, so there are no local minima\nto get stuck in.\nSo gradient descent starting from any point is guaranteed to find the minimum.\\\n $w^{t+1}=w^t-\\eta d/dwf(x;w)$\\\n The loss for a batch of data or an entire dataset is just the average loss over the m examples\\\n\n $\u0007lpha Cost(w,b)/ \u0007lpha w_j=\\Sigma^m_{i=1}[\\sigma (wx^{(i)}+b)-y^{i}]x_j^{(i)}$\\\nStochastic gradient descent chooses a single random example at a time and updates its weights on that example. As a result the updates can fluctuate.\\\nAn alternate is batch training, which computes the gradient over the entire dataset. This gives a much better estimate of which direction to move the weights but takes a long time to compute.\\\nA commonly used compromise is mini-batch training, where we train on a small batch. \\\nOverfitting :In logistic regression, if a feature only occurs in one class then it will get a high weight. Sometimes we are just modelling noisy factors that just accidentally correlate with the class.\\\nRegularization: Lasso regression uses L1 regularization Ridge regression uses L2 regularization\\\nMultinomial logistic regression: To get a proper probability, we will use a generalization of the sigmoid function\ncalled the softmax function.$softmax(z_i) = e_{z_j}/ \\Sigma_{j=1}^k e^{z_j} 1 ≤ i ≤ k$\\\nProblem: a small change in the weights or bias of any single perceptron in the network can causes the output to completely flip from 0 to 1.\nSolution: sigmoid neuron\\\nSmoothness is crucial\\\nActivation Functions:neuron units\napply a non-linear function f to z.\\\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\\\nThe simplest kind of NN is the Feed-Forward Neural Network Multilayer network, all units are usually fully-connected, and no cycles.\\\nThe computation has three steps:\\\n1. multiplying the weight matrix by the input vector x, 2. adding the bias vector b, and\n3. applying the activation function g (such as Sigmoid)\\\nHere are the final equations for a feedforward network with a single hidden layer, which takes an input vector x, outputs a probability distribution y, and is parameterized by weight matrices W and U and a bias vector b:\\\nh = σ(Wx+b)\\\nz = Uh\\\ny = softmax(z)\\\nTraining Neural Nets:\nA loss function – cross-entropy loss\nAn optimization algorithm – gradient descent\nA way of computing the gradient of the loss function – error propagation\\\nAlthough backpropagation was invented for neural nets, it is related to general procedure called backward differentiation, which depends on the notion of computation graphs.\\\nChain rule:For a composite function f(x) = u(v(x)), the derivative of f(x) is:\n$df/dx=du/dv * dv/dx$\n \n\n## Module 13 Definitions\n* **Concept** - definition\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Language Models Revisited\nWe instantiate this intuition by using perplexity to measure the quality of a language model\\\nIntuitively, good models are those that assign higher probabilities to unseen data\\\n$pp_{\theta}(w_{1:n})= p_{\theta} w_{1:n}^{-1/n}$\n\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Recurrent Neural Networks:\nA simple recurrent neural network shown unrolled in time. Network layers are recalculated for each time step, while the weights U, V and W are shared in common across all time steps. \n\n## SLP Chapter 9.3 RNNs as Language Models\n RNNs as Language Models \nWeight tying is a method that dispenses with this redundancy and uses a single set of embeddings at the input and softmax layers. \n\n\n## SLP Chapter 6.1 Lexical Semantics\n Lexical Semantics\\e.g\nmouse (N)\\\n      1.  any of numerous small rodents...\\\n      2.  a hand-operated device that controls a cursor...\\\nHere the form  is the lemma, also called the citation form.\\\nThe specific forms sung or carpets or sing or duermes are called wordforms.\\\nA more formal definition of synonymy (between words rather than senses) is that two words are synonymous if they are substitutable for one another in any sentence without changing the truth conditions of the sentence, the situations in which the sentence would be true. We often say in this case that the two words have the same propositional meaning.\\\nSemantic fields are also related to topic models, like Latent Dirichlet Allocation, LDA, which apply unsupervised learning on large sets of texts to induce sets of associated words from text.\\\nA semantic frame is a set of words that denote perspectives or participants in a particular type of event. \\\nwords have affective meanings or connotations\\\n\n \n\n## SLP Chapter 6.2 Vector Semantics\nVector Semantics\nVectors semantics is the standard way to represent word meaning in NLP\\\nVectors for representing words are called embeddings\\\n\n\n\n## SLP Chapter 6.3 Words and Vectors\n \nWords and Vectors\nIn a term-document matrix, each row represents a word in the vocabulary and each column represents a document from some collection of documents.\\\nEach cell in this matrix represents the number of times a particular word (defined by the row) occurs in a particular document (defined by the column). \\\nA vector space is a collection of vectors, characterized by their dimension.\\\nTerm-document matrices were originally defined as a means of finding similar documents for the task of document information retrieval.\\\nWords as vectors: document dimensions: We do this by associating each word with a word vector— a row vector rather than a column vector, hence with different dimensions\\\nWords as vectors: word dimensions: An alternative to using the term-document matrix to represent words as vectors of document counts, is to use the term-term matrix, also called the word-word ma- trix or the term-context matrix, in which the columns are labeled by words rather than documents. \\\nThis matrix is thus of dimensionality |V | × |V | and each cell records the number of times the row (target) word and the column (context) word co-occur in some context in some training corpus.\\\nSince most of these numbers are zero these are sparse vector representa- tions; there are efficient algorithms for storing and computing with sparse matrices.\n\n \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Cosine for measuring similarity\\The cosine—like most measures for vector similarity used in NLP—is based on the dot product operator from linear algebra, also called the inner product:\\dot product(v,w) = v·w = 􏰈viwi = v1w1 +v2w2 +...+vNwN\\This raw dot product, however, has a problem as a similarity metric: it favors long vectors. The vector length is defined as\\|v| = 􏰋􏰊􏰈v2i\\The dot product is higher if a vector is longer, with higher values in each dimension.\\ This normalized dot product turns out to be the same as the cosine of the angle between the two vectors, following from the definition of the dot product between two vectors a and b::\n a·b = |a||b|cosθ\\\na·b/|a||b| = cosθ\\\nThe cosine similarity metric between two vectors v and w thus can be computed as:\n$\ncosine(v,w)=\\Sigma_{i=1}^N v_iw_i/(\\sqrt{ \\Sigma_{i=1}^N v_i^2}\\sqrt{ \\Sigma_{i=1}^N w_i^2})\n$\n\n \n\n## SLP Chapter 6.8 Word2vec\nWord2vec\n In the previous sections we saw how to represent a word as a sparse, long vector with dimensions corresponding to words in the vocabulary or documents in a collection. We now introduce a more powerful word representation: embeddings, short dense vectors. \\\none method for computing embeddings: skip-gram with negative sampling, sometimes called SGNS.\\\nWord2vec embeddings are static em- beddings, meaning that the method learns one fixed embedding for each word in the vocabulary. \\\n The intuition of skip-gram is:\\\n1. Treat the target word and a neighboring context word as positive examples. \\\n2. Randomly sample other words in the lexicon to get negative samples.\\\n3. Use logistic regression to train a classifier to distinguish those two cases. (we rely on the intuition that two vectors are similar if they have a high dot product\\\nTo turn the dot product into a probability, we’ll use the logistic or sigmoid function σ (x), the fundamental core of logistic regression\\\nSkip-gram makes the simplifying assumption that all context words are independent, allowing us to just multiply their probabilities:)\\\n4. Use the learned weights as the embeddings.\\\nIn summary, skip-gram trains a probabilistic classifier that, given a test target word w and its context window of L words c1:L, assigns a probability based on how similar this context window is to the target word. The probability is based on applying the logistic (sigmoid) function to the dot product of the embeddings of the target word with each context word. To compute this probability, we just need embeddings for each target word and context word in the vocabulary.\\\nthe goal of the learning algorithm is to adjust those embeddings to\\\n• Maximize the similarity of the target word, context word pairs (w,cpos) drawn from the positive examples\\\n• Minimize the similarity of the (w,cneg) pairs from the negative examples.\\\nAnother very widely used static embedding model is GloVe\\\n\n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Semantic properties of embeddings :\nDifferent types of similarity or association: One parameter of vector semantic models that is relevant to both sparse tf-idf vectors and dense word2vec vectors is the size of the context window used to collect counts. \\\nThe choice depends on the goals of the representation. Shorter context windows tend to lead to representations that are a bit more syntactic, since the information is coming from immediately nearby words. \\\nAnalogy/Relational Similarity: Another semantic property of embeddings is their ability to capture relational meanings.\\\nparallelogram model for solving simple analogy problems of the form a is to b as a* is to what?\\\nThere are some caveats. For example, the closest value returned by the paral-\nlelogram algorithm in word2vec or GloVe embedding spaces is usually not in fact b* but one of the 3 input words or their morphological variants (i.e., cherry:red :: potato:x returns potato or potatoes instead of brown), so these must be explicitly excluded. \\\nFurthermore while embedding spaces perform well if the task involves frequent words, small distances, and certain relations (like relating countries with their capitals or verbs/nouns with their inflected forms), the parallelogram method with embeddings doesn’t work as well for other relations \\\n\nEmbeddings and Historical Semantics:\nEmbeddings can also be a useful tool for studying how meaning changes over time, by computing multiple embedding spaces, each from texts written in a particular time period. \n\n\n\n## SLP Chapter 6.11 Bias and Embeddings\n Bias and Embeddings \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Evaluating Vector Models:\nThe semantic textual similarity task (Agirre et al. 2012, Agirre et al. 2015) evaluates the performance of sentence- level similarity algorithms, consisting of a set of pairs of sentences, each pair with human-labeled similarity scores.\\\nAnother task used for evaluation is the analogy task, discussed on page 24, where thesystemhastosolveproblemsoftheformaistobasa*istob*,givena,b,anda*\nand having to find b* \n\n## Module 13 Lecture Notes\nLogistic Regression:\nClassifier components:\n1. A feature representation of the input.Foreachinputobservation x(i), this will be a vector of features [x1,x2,...,xn].\n2. A classification function that computes the estimated class 𝑦 via p(y|x).\n3. Annobjective function for learning, usually involving minimizing error on training examples.\n4. Analgorithm for optimizing the objective function.\\\nSentiment classifier\\\nDecision boundary\\\n\n$\\hat y=1 \\ if\\  P(y=1|x)>0.5\\ else\\ 0$\\\nExtracting Features\\\n$z=\\Sigma_i w_ix_i+b$, $\\sigma(z)=1/(1+e^{-(wx+b)})$\\\n1. An objective function or loss function that tells us distance between the system output and the gold output. We will use cross-entropy loss.\\\n2. An algorithm for optimizing the objective function. We will use stochastic gradient descent to minimize the loss function.\nLoss functions: eg. mean squared error \n$L_{MSE}(\\hat y,y)=1/2(\\hat y-y)^2$\\\nLoss functions for probabilistic classification:cross entropy loss\\\nFor one observation x, let’s maximize the probability of the correct label p(y|x).\n$p(y|x)=\\hat y^y(1-\\hat y)^{1-y}$\\\nChange to logs (still maximizing)\n$logp(y|x)=log[\\hat y^y(1-\\hat y)^{1-y}]=ylog\\hat y +(1-y)log(1-\\hat y)$\\\nCross-entropy loss: The result is cross-entropy loss:$-logp(y|x)=-(ylog\\hat y +(1-y)log(1-\\hat y))$\\\n$-logp(y|x)=-(ylog(wx+b)) +(1-y)log(1-(wx+b))))$\\\nGradient descent is a method that finds a minimum of a function by figuring out in which direction (in the space of the parameters θ) the function’s slope is rising the most steeply, and moving in the opposite direction.\\\n For logistic regression, this loss function is conveniently convex.\nA convex function has just one minimum, so there are no local minima\nto get stuck in.\nSo gradient descent starting from any point is guaranteed to find the minimum.\\\n $w^{t+1}=w^t-\\eta d/dwf(x;w)$\\\n The loss for a batch of data or an entire dataset is just the average loss over the m examples\\\n\n $\u0007lpha Cost(w,b)/ \u0007lpha w_j=\\Sigma^m_{i=1}[\\sigma (wx^{(i)}+b)-y^{i}]x_j^{(i)}$\\\nStochastic gradient descent chooses a single random example at a time and updates its weights on that example. As a result the updates can fluctuate.\\\nAn alternate is batch training, which computes the gradient over the entire dataset. This gives a much better estimate of which direction to move the weights but takes a long time to compute.\\\nA commonly used compromise is mini-batch training, where we train on a small batch. \\\nOverfitting :In logistic regression, if a feature only occurs in one class then it will get a high weight. Sometimes we are just modelling noisy factors that just accidentally correlate with the class.\\\nRegularization: Lasso regression uses L1 regularization Ridge regression uses L2 regularization\\\nMultinomial logistic regression: To get a proper probability, we will use a generalization of the sigmoid function\ncalled the softmax function.$softmax(z_i) = e_{z_j}/ \\Sigma_{j=1}^k e^{z_j} 1 ≤ i ≤ k$\\\nProblem: a small change in the weights or bias of any single perceptron in the network can causes the output to completely flip from 0 to 1.\nSolution: sigmoid neuron\\\nSmoothness is crucial\\\nActivation Functions:neuron units\napply a non-linear function f to z.\\\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\\\nThe simplest kind of NN is the Feed-Forward Neural Network Multilayer network, all units are usually fully-connected, and no cycles.\\\nThe computation has three steps:\\\n1. multiplying the weight matrix by the input vector x, 2. adding the bias vector b, and\n3. applying the activation function g (such as Sigmoid)\\\nHere are the final equations for a feedforward network with a single hidden layer, which takes an input vector x, outputs a probability distribution y, and is parameterized by weight matrices W and U and a bias vector b:\\\nh = σ(Wx+b)\\\nz = Uh\\\ny = softmax(z)\\\nTraining Neural Nets:\nA loss function – cross-entropy loss\nAn optimization algorithm – gradient descent\nA way of computing the gradient of the loss function – error propagation\\\nAlthough backpropagation was invented for neural nets, it is related to general procedure called backward differentiation, which depends on the notion of computation graphs.\\\nChain rule:For a composite function f(x) = u(v(x)), the derivative of f(x) is:\n$df/dx=du/dv * dv/dx$\n \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}