{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3ab48099-f74f-452e-eeb6-62f0cf6d79b7"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "33913882-425c-42f7-858c-d2d31f7ccc63"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = True"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" The utility is a human defined function to evaluate the state and thus it cannot be completely correct. But it could still obtain an optimal policy as long as it describes the state properly. \"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\" Your summary here \"\"\"\n",
        "chapter16_3 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\" MDP: A solution or policy must specify what the agent should do for all states that it can reach. An MDP is defined for a fully observable, stochastic environment. Discounted reward is not absolutely necessary, proper policy might not always exist. For certain state, it may exists more than one optimal policies.\n",
        "\"an optimal action depends only on the current state, and the optimal is stationary\"\"\"\n",
        "chapter17_2 =  \"\"\"The basic concept used in showing that value iteration converges is the notion of a con-\n",
        "Contraction traction. Roughly speaking, a contraction is a function of one argument that, when applied to\n",
        "two different inputs in turn, produces two output values that are “closer together,” by at least\n",
        "some constant factor, than the original inputs. For example, the function “divide by two” is\n",
        "a contraction, because, after we divide any two numbers by two, their difference is halved.\n",
        "Notice that the “divide by two” function has a fixed point, namely zero, that is unchanged by\n",
        "the application of the function \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\" MPDs are non-deterministic search problems. One way of solving them is vial expectimax search. The stationary distribution of a Markov chain describes the distribution of Xt after a sufficiently long time that the distribution of Xt does not change any longer. MDP Components: 1. A set of states s S 2. A set of actions a A. A transition fundtion T(s,a,s'): Probability that executing action a in s will lead to s'P(s'|s,a). The probability is called the model. 3. A reward function R(s,a,S') Sometimes just R(s) or R(s') 4. An intial state S0 5.Optionally, one ormore terminal states. A policy maps from a state onto the action to take if the agent is in that state pi(S) = a (Policy pi telss the agent what action to take at state S). Even if the same policy is executed multiple times by the agent, this may lead to different sequence of states and actions(environment history), and thus a different score under the reward function. Therefore we need to compute the expected utility of all the possible paths generated by a policy\n",
        "If final reward is 1 and -1, r (penalty to for a step) < -1.65: best policy is exit asap (even it means fall into a pit). -0.03<r<0: bouncing against the wall by facing opposite of -1. r > 0: avoid any terminal states so that I can bounce around and keep collecting rewards. Discout: each time we descend a level, we multiply in the discount once. Sooner rewards probably do have higher utility than later rewards. It also helps our algorithms converge. eg. discount of 0.5 U([1,2,3] = 1 + 0.5*2 + 0.5*0.5*3). U([1,2,3]) < U([3,2,1])\n",
        "If we assume stationary preferences, then there are only two ways to define utilities. Additive utility: U([r0, r1, r2...]) = r0 + r1 + r2 + ... Discounted utility: U([r0, r1, r2...]) = r0 + d*r1 + d^2*r2 + ...\n",
        "Solution for infinite utilities: 1. finite horizon(similar to depth-limited search): gives nonstationary policies(pi depends on time left) 2. Discounting use 0<gamma<1, smaller gamma means smaller horizion - shorter term focus 3. absorbing state: guarantee that for every policy, a terminal state will eventually be reached(like \"overheated\" for racing) \n",
        "Value iteration: 1. start with V0(s) = 0. No time steps left means an expected reward sum of zero. 2. Given vector of Vk(s) values, do one ply of expectimax from each state3. repeat until convergence. 4. Complexity of each iteration: O(S^2 *A). Value iteration will converge to unique optimal values. Basic idea: approximations get refined towards optimal values. Policy may converge long before values do\n",
        "Fixed policy evaluation: idea 1: turn recursive bellman equations into updates(like value iteration) O(S^3) per iteration. Idea 2: Without the maxes, the Bellman equations are just a linear system\n",
        "actions are easier to select from q-values than values. Ti(s) = arg max_a Q(s, a). Problem with value iteration: 1. it's slow 2. the max at each state rarely changes\n",
        "Policy iteration: step 1: policy evaluation: calculate utilities for some fixed policy (not optimal policies) until convergence. Step 2: Policy improvement: update policy using one step look-ahead with resulting converged(but not optimal!) utilities as future values. Repeat steps until policy converges. Policy iteration is still optimal. It can converge (much) faster under some conditions\n",
        "Policy iteration and value iteration: Both compute the same thing (all optimal values). Both are dynamic programing. In value iteration, every iteration updates both the values and (implicitly) the policy. We don't track the policy, but taking the max over actions implicityly recomputes it. In policy iteration: we do several passes that update utilities with fixed policy (each pass is fast because we consider only one action, not all of them). After the policy is evaluated, a new policy is chosen(slow like a value iteration pass). The new policy will be better (or we are done)\n",
        "For worst-case minimax reasoning, terminal function scale doesn't matter. We just want better states to have higher evaluations (get the ordering right). We call this insensitivity to monotonic transformations. For average-case expectimax reasoning, we need magnitudes to be meaningful\n",
        "Any rational preferences can be summarized as a utility function. we can just specify what the utilities are and not the behaviors. We can say what the utilities of certain things are and then our agents will behave in order to maximize their expected utilities. Rational preferences imply behavior describable as maximization of expected utility.\n",
        "Maximum exptected utility (MEU) principle: Choose the action that maximizes expected utility. Note. and agent can be entirely rational (consistent with MEU) without ever representing or manipulating utilities and probabilities. eg. a lookup table for perfect tic tac toe, a reflex vacuum cleaner\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long. \n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"which assigns a single number to express the desirability of a state\",\n",
        "\"expected utility\" : \"of an action given the evidence, is the average utility of the outcomes, weighted by the probability that the outcome occurs\",\n",
        "\"Markov decision process\": \"is a sequential decision problem for a fully observable environment with a stochastic trasnistion model that has additive rewards. \",\n",
        "\"rewards\" : \"The performance of an agent in an MDP is the sum of the rewards for the transitinos it takes\",\n",
        "\"nonstionary\": \"A policy that depends on the time is called nonstationary\",\n",
        "\"optimal policy\": \"A policy that yields the highest expected utility\"\n",
        "}\n",
        "\n"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d1976af0-8657-416a-fbdd-a5445d2d772a"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n The utility is a human defined function to evaluate the state and thus it cannot be completely correct. But it could still obtain an optimal policy as long as it describes the state properly.  $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## AIMA Chapter 16.3 Utility Functions\n Your summary here \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n MDP: A solution or policy must specify what the agent should do for all states that it can reach. An MDP is defined for a fully observable, stochastic environment. Discounted reward is not absolutely necessary, proper policy might not always exist. For certain state, it may exists more than one optimal policies.\n\"an optimal action depends only on the current state, and the optimal is stationary\n\n## AIMA Chapter 17.2 Algorithms for MDPs\nThe basic concept used in showing that value iteration converges is the notion of a con-\nContraction traction. Roughly speaking, a contraction is a function of one argument that, when applied to\ntwo different inputs in turn, produces two output values that are “closer together,” by at least\nsome constant factor, than the original inputs. For example, the function “divide by two” is\na contraction, because, after we divide any two numbers by two, their difference is halved.\nNotice that the “divide by two” function has a fixed point, namely zero, that is unchanged by\nthe application of the function \n\n## Module 8 Lecture Notes\n MPDs are non-deterministic search problems. One way of solving them is vial expectimax search. The stationary distribution of a Markov chain describes the distribution of Xt after a sufficiently long time that the distribution of Xt does not change any longer. MDP Components: 1. A set of states s S 2. A set of actions a A. A transition fundtion T(s,a,s'): Probability that executing action a in s will lead to s'P(s'|s,a). The probability is called the model. 3. A reward function R(s,a,S') Sometimes just R(s) or R(s') 4. An intial state S0 5.Optionally, one ormore terminal states. A policy maps from a state onto the action to take if the agent is in that state pi(S) = a (Policy pi telss the agent what action to take at state S). Even if the same policy is executed multiple times by the agent, this may lead to different sequence of states and actions(environment history), and thus a different score under the reward function. Therefore we need to compute the expected utility of all the possible paths generated by a policy\nIf final reward is 1 and -1, r (penalty to for a step) < -1.65: best policy is exit asap (even it means fall into a pit). -0.03<r<0: bouncing against the wall by facing opposite of -1. r > 0: avoid any terminal states so that I can bounce around and keep collecting rewards. Discout: each time we descend a level, we multiply in the discount once. Sooner rewards probably do have higher utility than later rewards. It also helps our algorithms converge. eg. discount of 0.5 U([1,2,3] = 1 + 0.5*2 + 0.5*0.5*3). U([1,2,3]) < U([3,2,1])\nIf we assume stationary preferences, then there are only two ways to define utilities. Additive utility: U([r0, r1, r2...]) = r0 + r1 + r2 + ... Discounted utility: U([r0, r1, r2...]) = r0 + d*r1 + d^2*r2 + ...\nSolution for infinite utilities: 1. finite horizon(similar to depth-limited search): gives nonstationary policies(pi depends on time left) 2. Discounting use 0<gamma<1, smaller gamma means smaller horizion - shorter term focus 3. absorbing state: guarantee that for every policy, a terminal state will eventually be reached(like \"overheated\" for racing) \nValue iteration: 1. start with V0(s) = 0. No time steps left means an expected reward sum of zero. 2. Given vector of Vk(s) values, do one ply of expectimax from each state3. repeat until convergence. 4. Complexity of each iteration: O(S^2 *A). Value iteration will converge to unique optimal values. Basic idea: approximations get refined towards optimal values. Policy may converge long before values do\nFixed policy evaluation: idea 1: turn recursive bellman equations into updates(like value iteration) O(S^3) per iteration. Idea 2: Without the maxes, the Bellman equations are just a linear system\nactions are easier to select from q-values than values. Ti(s) = arg max_a Q(s, a). Problem with value iteration: 1. it's slow 2. the max at each state rarely changes\nPolicy iteration: step 1: policy evaluation: calculate utilities for some fixed policy (not optimal policies) until convergence. Step 2: Policy improvement: update policy using one step look-ahead with resulting converged(but not optimal!) utilities as future values. Repeat steps until policy converges. Policy iteration is still optimal. It can converge (much) faster under some conditions\nPolicy iteration and value iteration: Both compute the same thing (all optimal values). Both are dynamic programing. In value iteration, every iteration updates both the values and (implicitly) the policy. We don't track the policy, but taking the max over actions implicityly recomputes it. In policy iteration: we do several passes that update utilities with fixed policy (each pass is fast because we consider only one action, not all of them). After the policy is evaluated, a new policy is chosen(slow like a value iteration pass). The new policy will be better (or we are done)\nFor worst-case minimax reasoning, terminal function scale doesn't matter. We just want better states to have higher evaluations (get the ordering right). We call this insensitivity to monotonic transformations. For average-case expectimax reasoning, we need magnitudes to be meaningful\nAny rational preferences can be summarized as a utility function. we can just specify what the utilities are and not the behaviors. We can say what the utilities of certain things are and then our agents will behave in order to maximize their expected utilities. Rational preferences imply behavior describable as maximization of expected utility.\nMaximum exptected utility (MEU) principle: Choose the action that maximizes expected utility. Note. and agent can be entirely rational (consistent with MEU) without ever representing or manipulating utilities and probabilities. eg. a lookup table for perfect tic tac toe, a reflex vacuum cleaner\n\n## Module 8 Definitions\n* **Utility function** - which assigns a single number to express the desirability of a state\n* **Expected utility** - of an action given the evidence, is the average utility of the outcomes, weighted by the probability that the outcome occurs\n* **Markov decision process** - is a sequential decision problem for a fully observable environment with a stochastic trasnistion model that has additive rewards. \n* **Rewards** - The performance of an agent in an MDP is the sum of the rewards for the transitinos it takes\n* **Nonstionary** - A policy that depends on the time is called nonstationary\n* **Optimal policy** - A policy that yields the highest expected utility\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_2 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_3 =  \"\"\" In the worst case, the agent enters an absorbing state where no actions have any effect and no rewards are received. Q-learning is an off-policy learning algorithm, because it learns Q-values that answer the O\u000b-policy\n",
        "question “What would this action be worth in this state, assuming that I stop using whatever\n",
        "policy I am using now, and start acting according to a policy that chooses the best action\n",
        "(according to my estimates)?”\"\"\"\n",
        "chapter22_4 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_5 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" reinforcement learning: we don't know probablity distribution over states. We don't know rewards either. Reinforcement Learning: Still assume a Markov decision process (MDP):\n",
        "▪ A set of states s ∈ S\n",
        "▪ A set of actions (per state) A\n",
        "▪ A model T(s,a,s’)\n",
        "▪ A reward function R(s,a,s’)\n",
        "▪ Still looking for a policy π(s)\n",
        "▪ New twist: don’t know T or R\n",
        "▪ I.e. we don’t know which states are good or what the actions do\n",
        "▪ Must actually try actions and states out to learn\n",
        "学习方法1：Model based learning: Model-Based Idea: ▪ Learn an approximate model based on experiences ▪ Solve for values as if the learned model were correct\n",
        "▪ Step 1: Learn empirical MDP model ▪ Count outcomes s’ for each s, a ▪ Normalize to give an estimate of ▪ Discover each when we experience (s, a, s’)\n",
        "▪ Step 2: Solve the learned MDP ▪ For example, use value iteration, as before\n",
        "学习方法2: Model-Free Learning/Passive Reinforcement Learning Simplified task: policy evaluation\n",
        "▪ Input: a fixed policy π(s) ▪ You don’t know the transitions T(s,a,s’) ▪ You don’t know the rewards R(s,a,s’) ▪ Goal: learn the state values\n",
        "▪ In this case:\n",
        "▪ Learner is “along for the ride” ▪ No choice about what actions to take ▪ Just execute the policy and learn from experience ▪ This is NOT offline planning! You actually take actions in the world.▪ Goal: Compute values for each state under π\n",
        "▪ Idea: Average together observed sample values\n",
        "▪ Act according to π\n",
        "▪ Every time you visit a state, write down what the\n",
        "sum of discounted rewards turned out to be\n",
        "▪ Average those samples\n",
        "▪ This is called direct evaluation\n",
        "What’s good about direct evaluation?\n",
        "▪ It’s easy to understand\n",
        "▪ It doesn’t require any knowledge of T, R\n",
        "▪ It eventually computes the correct average values,\n",
        "using just sample transitions\n",
        "▪ What bad about it?\n",
        "▪ It wastes information about state connections\n",
        "▪ Each state must be learned separately\n",
        "▪ So, it takes a long time to learn\n",
        "学习方法3 : Temporal Difference learning. Fixed Policy. Update V(s) each time there's a transition (s,a,s',r) Sample = R(s, pi(s), s') + lambda*V_pi(s') Problems with TD Value Learning\n",
        "▪ TD value leaning is a model-free way to do policy evaluation, mimicking\n",
        "Bellman updates with running sample averages\n",
        "▪ It can't turn values into a (new) policy, :\n",
        "▪ Idea: learn Q-values, not values\n",
        "▪ Makes action selection model-free too!\n",
        "学习方法4： Q learning: Even though we're acting sub-optimally, we converge towards the optimal policy This is called off-policy learning\n",
        "▪ Caveats:\n",
        "▪ You have to explore enough\n",
        "▪ You have to eventually make the learning rate\n",
        "small enough\n",
        "▪ … but not decrease it too quickly\n",
        "▪ Basically, in the limit, it doesn’t matter how you select actions (!)\"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"bandit problems\" : \"Each arm Mi is a Markov reward process or MRP, that is, an MDP with only one Markov reward process possible action ai. \",\n",
        "}"
      ],
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 857
        },
        "cellView": "form",
        "outputId": "e2da2914-4b11-4a54-9fbe-d4d46b4c6048"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Your summary here \n\n## AIMA Chapter 22.1 Learning from Rewards\n Your summary here \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n In the worst case, the agent enters an absorbing state where no actions have any effect and no rewards are received. Q-learning is an off-policy learning algorithm, because it learns Q-values that answer the O\n-policy\nquestion “What would this action be worth in this state, assuming that I stop using whatever\npolicy I am using now, and start acting according to a policy that chooses the best action\n(according to my estimates)?”\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n reinforcement learning: we don't know probablity distribution over states. We don't know rewards either. Reinforcement Learning: Still assume a Markov decision process (MDP):\n▪ A set of states s ∈ S\n▪ A set of actions (per state) A\n▪ A model T(s,a,s’)\n▪ A reward function R(s,a,s’)\n▪ Still looking for a policy π(s)\n▪ New twist: don’t know T or R\n▪ I.e. we don’t know which states are good or what the actions do\n▪ Must actually try actions and states out to learn\n学习方法1：Model based learning: Model-Based Idea: ▪ Learn an approximate model based on experiences ▪ Solve for values as if the learned model were correct\n▪ Step 1: Learn empirical MDP model ▪ Count outcomes s’ for each s, a ▪ Normalize to give an estimate of ▪ Discover each when we experience (s, a, s’)\n▪ Step 2: Solve the learned MDP ▪ For example, use value iteration, as before\n学习方法2: Model-Free Learning/Passive Reinforcement Learning Simplified task: policy evaluation\n▪ Input: a fixed policy π(s) ▪ You don’t know the transitions T(s,a,s’) ▪ You don’t know the rewards R(s,a,s’) ▪ Goal: learn the state values\n▪ In this case:\n▪ Learner is “along for the ride” ▪ No choice about what actions to take ▪ Just execute the policy and learn from experience ▪ This is NOT offline planning! You actually take actions in the world.▪ Goal: Compute values for each state under π\n▪ Idea: Average together observed sample values\n▪ Act according to π\n▪ Every time you visit a state, write down what the\nsum of discounted rewards turned out to be\n▪ Average those samples\n▪ This is called direct evaluation\nWhat’s good about direct evaluation?\n▪ It’s easy to understand\n▪ It doesn’t require any knowledge of T, R\n▪ It eventually computes the correct average values,\nusing just sample transitions\n▪ What bad about it?\n▪ It wastes information about state connections\n▪ Each state must be learned separately\n▪ So, it takes a long time to learn\n学习方法3 : Temporal Difference learning. Fixed Policy. Update V(s) each time there's a transition (s,a,s',r) Sample = R(s, pi(s), s') + lambda*V_pi(s') Problems with TD Value Learning\n▪ TD value leaning is a model-free way to do policy evaluation, mimicking\nBellman updates with running sample averages\n▪ It can't turn values into a (new) policy, :\n▪ Idea: learn Q-values, not values\n▪ Makes action selection model-free too!\n学习方法4： Q learning: Even though we're acting sub-optimally, we converge towards the optimal policy This is called off-policy learning\n▪ Caveats:\n▪ You have to explore enough\n▪ You have to eventually make the learning rate\nsmall enough\n▪ … but not decrease it too quickly\n▪ Basically, in the limit, it doesn’t matter how you select actions (!)\n\n## Module 9 Definitions\n* **Bandit problems** - Each arm Mi is a Markov reward process or MRP, that is, an MDP with only one Markov reward process possible action ai. \n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_6 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" Probabilistic Language Models\n",
        "One goal: assign a probability to a sentence\n",
        "• Autocomplete for texting\n",
        "• Machine Translation\n",
        "• Spelling Correction\n",
        "• Speech Recognition\n",
        "Other Natural Language Generation tasks: summarization, question-answering, dialog\n",
        "systems \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"term\" : \"your definition here\",\n",
        "}"
      ],
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "f550637f-e3ed-4680-8fb9-4747842b0744"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Your summary here \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Probabilistic Language Models\nOne goal: assign a probability to a sentence\n• Autocomplete for texting\n• Machine Translation\n• Spelling Correction\n• Speech Recognition\nOther Natural Language Generation tasks: summarization, question-answering, dialog\nsystems \n\n## Module 10 Definitions\n* **Term** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" Bayesian networks can represent\n",
        "essentially any full joint probability distribution and in many cases can do so very concisely. \n",
        "If two variables (nodes) X and Y in a Bayesian network do not share a path, X and Y are conditionally independent.\"\"\"\n",
        "chapter13_2 = \"\"\" (a) A node X is conditionally independent of its non-descendants (e.g., the Zi js)\n",
        "given its parents (the Uis shown in the lavender area). (b) A node X is conditionally independent\n",
        "of all other nodes in the network given its Markov blanket (the lavender area). The compactness of Bayesian networks\n",
        "is an example of a general property of locally structured (also called sparse) systems. In a Locally structured\n",
        "locally structured system, each subcomponent interacts directly with only a bounded number Sparse\n",
        "of other components, regardless of the total number of components. Local structure is usually\n",
        "associated with linear rather than exponential growth in complexity.the specification of the joint distribution is equivalent to the\n",
        "general assertion that, for every variable Xi in the network,\n",
        "P(Xi jXi􀀀1; : : : ;X1) = P(Xi jParents(Xi)) ; (13.3)\n",
        "provided that Parents(Xi) \u0012 fXi􀀀1; : : : ;X1g. This last condition is satisfied by numbering the\n",
        "Topological ordering nodes in topological order—that is, in any order consistent with the directed graph structure.\n",
        "What Equation (13.3) says is that the Bayesian network is a correct representation of the\n",
        "domain only if each node is conditionally independent of its other predecessors in the node\n",
        "ordering, given its parents. We can satisfy this condition with this methodology:\n",
        "1. Nodes: First determine the set of variables that are required to model the domain. Now\n",
        "order them, fX1; : : : ;Xng. Any order will work, but the resulting network will be more\n",
        "compact if the variables are ordered such that causes precede effects.\n",
        "2. Links: For i = 1 to n do:\n",
        "• Choose a minimal set of parents for Xi from X1; : : : ;Xi􀀀1, such that Equation (13.3)\n",
        "is satisfied.\n",
        "• For each parent insert a link from the parent to Xi.\n",
        "• CPTs: Write down the conditional probability table, P(XijParents(Xi)).\"\"\"\n",
        "chapter13_3 = \"\"\" The basic task for any probabilistic inference system is to compute the posterior probability\n",
        "distribution for a set of query variables, given some observed event—usually, some assign- Event\n",
        "ment of values to a set of evidence variables.5 there is at most one undirected path (i.e., ignoring the direction of the arrows) between any\n",
        "two nodes in the network. These are called singly connected networks or polytrees, and Singly connected\n",
        "they have a particularly nice property: The time and space complexity of exact inference in Polytree polytrees is linear in the size of the network.For multiply connected networks, such as the insurance network in Figure 13.9, variable Multiply connected\n",
        "elimination can have exponential time and space complexity in the worst case, even when the\n",
        "number of parents per node is bounded. in a polytree network, one would need to issue O(n) queries\n",
        "Clustering costing O(n) each, for a total of O(n2) time. Using clustering algorithms (also known as\n",
        "Join tree join tree algorithms), the time can be reduced to O(n). For this reason, these algorithms are\n",
        "widely used in commercial Bayes net tools.\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" N-gram models work well for classifying newspaper sections, as well as for other classification\n",
        "tasks such as spam detection (distinguishing spam email from non-spam), sentiment Spam detection\n",
        "analysis (classifying a movie or product review as positive or negative) and author attribu- Sentiment analysis\n",
        "tion (Hemingway has a different style and vocabulary than Faulkner or Shakespeare). Other n-gram models: character level model: well suited for the task of language identification. It's also good at decide unseen words' classification. eg “dextroamphetamine” is a drug name, “Kallenberger” is a person name,\n",
        "and “Plattsburg” is a city name. Question: Are X and Y conditionally independent given evidence variables {Z}?\n",
        "▪ Yes, if X and Y “d-separated” by Z ▪ Consider all (undirected) paths from X to Y ▪ No active paths = independence! A path is active if each triple is active:\n",
        "▪ Causal chain A → B → C where B is unobserved (either direction) ▪ Common cause A ← B → C where B is unobserved ▪ Common effect (aka v-structure)\n",
        "A → B ← C where B or one of its descendants is observed. All it takes to block a path is a single inactive segment. The whole path is active if every triple along that path is active.if any of them is inactive, then that whole path is inactive. Inference: 1. posterior probability P(Q|E1 = e1,...Ek = ek) 2. most likely explanations: argmax_q P(Q = q| E1 = e1...)\n",
        "For inference by enumeration, the joint probability table is exponential with respect to the non observed variables. \n",
        "Every variable in a Bayes net is independent of all of its descendants given its parent.\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"X is conditionally independent of Y given Z\" : \"if and only if 对所有 x,y,z: P(x,y|z) = P(x|z)P(y|z) or equivalenty if and only if P(x|z,y) = P(x|z)\",    \n",
        "\n",
        "}\n"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 636
        },
        "cellView": "form",
        "outputId": "dea5dd92-7732-43c0-9788-b71491edbd9c"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n The basic task for any probabilistic inference system is to compute the posterior probability\ndistribution for a set of query variables, given some observed event—usually, some assign- Event\nment of values to a set of evidence variables.5 there is at most one undirected path (i.e., ignoring the direction of the arrows) between any\ntwo nodes in the network. These are called singly connected networks or polytrees, and Singly connected\nthey have a particularly nice property: The time and space complexity of exact inference in Polytree polytrees is linear in the size of the network.For multiply connected networks, such as the insurance network in Figure 13.9, variable Multiply connected\nelimination can have exponential time and space complexity in the worst case, even when the\nnumber of parents per node is bounded. in a polytree network, one would need to issue O(n) queries\nClustering costing O(n) each, for a total of O(n2) time. Using clustering algorithms (also known as\nJoin tree join tree algorithms), the time can be reduced to O(n). For this reason, these algorithms are\nwidely used in commercial Bayes net tools.\n\n## Module 11 Lecture Notes\n N-gram models work well for classifying newspaper sections, as well as for other classification\ntasks such as spam detection (distinguishing spam email from non-spam), sentiment Spam detection\nanalysis (classifying a movie or product review as positive or negative) and author attribu- Sentiment analysis\ntion (Hemingway has a different style and vocabulary than Faulkner or Shakespeare). Other n-gram models: character level model: well suited for the task of language identification. It's also good at decide unseen words' classification. eg “dextroamphetamine” is a drug name, “Kallenberger” is a person name,\nand “Plattsburg” is a city name. Question: Are X and Y conditionally independent given evidence variables {Z}?\n▪ Yes, if X and Y “d-separated” by Z ▪ Consider all (undirected) paths from X to Y ▪ No active paths = independence! A path is active if each triple is active:\n▪ Causal chain A → B → C where B is unobserved (either direction) ▪ Common cause A ← B → C where B is unobserved ▪ Common effect (aka v-structure)\nA → B ← C where B or one of its descendants is observed. All it takes to block a path is a single inactive segment. The whole path is active if every triple along that path is active.if any of them is inactive, then that whole path is inactive\n\n## Module 11 Definitions\n* **X is conditionally independent of y given z** - if and only if 对所有 x,y,z: P(x,y|z) = P(x|z)P(y|z) or equivalenty if and only if P(x|z,y) = P(x|z)\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_6 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_7 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_9 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_10 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\" Classification: given inputs x, predict labels y\n",
        "▪ Examples: ▪ Spam detection (input: document,\n",
        "classes: spam / ham) ▪ OCR (input: images, classes: characters) ▪ Medical diagnosis (input: symptoms,\n",
        "classes: diseases) ▪ Automatic essay grading (input: document,\n",
        "classes: grades) ▪ Fraud detection (input: account activity,\n",
        "classes: fraud / no fraud) ▪ Customer service email routing and more\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"classification task\" : \"spam classification, digit recognition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "c1f86f8b-21b9-4352-b184-860f0f1ffcd4"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Your summary here \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n Your summary here \n\n## Module 12 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_6 = \"\"\" For more than two classes (eg 3 way sentiment classification: positive, negative, or neutral), we use multinomial logistic regression, also called softmax regression.In multinomial logistic regression\n",
        "the target y is a variable that ranges over more than two classes; we want to know the probability of y being in each potential class c 属于 C, p(y = c|x). The multinomial logistic classifier uses a generalization of the sigmoid, called\n",
        "softmax the softmax function, to compute the probability p(y = c|x). The softmax function takes a vector z = [z1; z2; :::; zk] of k arbitrary values and maps them to a probability\n",
        "distribution, with each value in the range (0,1), and all the values summing to 1.\n",
        "Like the sigmoid, it is an exponential function. The denominator sum(i=1, k) exp(zi) is used to normalize all the values into probabilities.\n",
        "like the sigmoid, the input to the softmax will be the dot product between\n",
        "a weight vector w and an input vector x (plus a bias). But now we’ll need separate\n",
        "weight vectors (and bias) for each of the K classes. Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.\n",
        "Thus if one of the inputs is larger than the others, it will tend to push its probability\n",
        "toward 1, and suppress the probabilities of the smaller inputs.\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\" There are 3 commonly used non-linear functions used for f:\n",
        "The sigmoid function\n",
        "The tanh function\n",
        "The rectified linear unit ReLU\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"Activation values\" : \"instead of directly ouputting z = w*x+b, which is a linear function of x, neuron units apply a non-linear function f to z, The output of this function is called the activation value for the unit\",    \n",
        "}\n"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "4d7f86eb-a681-4ee3-ee6b-a9ac7a29ff8d"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "## WARNING: Your definition of 'Activation values' is too long!\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Your summary here \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Your summary here \n\n## SLP Chapter 5.3 The cross-entropy loss function\n Your summary here \n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n For more than two classes (eg 3 way sentiment classification: positive, negative, or neutral), we use multinomial logistic regression, also called softmax regression.In multinomial logistic regression\nthe target y is a variable that ranges over more than two classes; we want to know the probability of y being in each potential class c 属于 C, p(y = c|x). The multinomial logistic classifier uses a generalization of the sigmoid, called\nsoftmax the softmax function, to compute the probability p(y = c|x). The softmax function takes a vector z = [z1; z2; :::; zk] of k arbitrary values and maps them to a probability\ndistribution, with each value in the range (0,1), and all the values summing to 1.\nLike the sigmoid, it is an exponential function. The denominator sum(i=1, k) exp(zi) is used to normalize all the values into probabilities.\nlike the sigmoid, the input to the softmax will be the dot product between\na weight vector w and an input vector x (plus a bias). But now we’ll need separate\nweight vectors (and bias) for each of the K classes. Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.\nThus if one of the inputs is larger than the others, it will tend to push its probability\ntoward 1, and suppress the probabilities of the smaller inputs.\n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n There are 3 commonly used non-linear functions used for f:\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\n\n## Module 13 Definitions\n* **Activation values** - instead of directly ouputting z = w*x+b, which is a linear function of x, neuron units apply a non-linear function f to z, The output of this function is called the activation value for the unit\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1d628778-e699-4a97-8bac-944f359f33b9"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_10 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_11 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\"Term-Document Matrix: Each column vector represents a document in the document collection. Each row represents a word. The value in a cell is how often that term occurred in that\n",
        "document. Document vectors can be sparse. We can measure how similar two documents are by comparing their column vectors. By comparing two row vectors tells how similar two words are. Words that occur in very similar sets of documents would therefore be very similar. \n",
        "Term document matrix can 1. retrieve similar documents 2. plagiarism detection\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "length = len(module14_lecture_notes.split(' '))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))\n"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 84 words long, so you could write an additional 416 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 14 Lecture Notes\", module14_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "T3cm1WV542_a",
        "outputId": "6d20dc76-6ada-4f15-b0c1-61ae4d6b5e6c"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n The utility is a human defined function to evaluate the state and thus it cannot be completely correct. But it could still obtain an optimal policy as long as it describes the state properly.  $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## AIMA Chapter 16.3 Utility Functions\n Your summary here \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n MDP: A solution or policy must specify what the agent should do for all states that it can reach. An MDP is defined for a fully observable, stochastic environment. Discounted reward is not absolutely necessary, proper policy might not always exist. For certain state, it may exists more than one optimal policies.\n\"an optimal action depends only on the current state, and the optimal is stationary\n\n## AIMA Chapter 17.2 Algorithms for MDPs\nThe basic concept used in showing that value iteration converges is the notion of a con-\nContraction traction. Roughly speaking, a contraction is a function of one argument that, when applied to\ntwo different inputs in turn, produces two output values that are “closer together,” by at least\nsome constant factor, than the original inputs. For example, the function “divide by two” is\na contraction, because, after we divide any two numbers by two, their difference is halved.\nNotice that the “divide by two” function has a fixed point, namely zero, that is unchanged by\nthe application of the function \n\n## Module 8 Lecture Notes\n MPDs are non-deterministic search problems. One way of solving them is vial expectimax search. The stationary distribution of a Markov chain describes the distribution of Xt after a sufficiently long time that the distribution of Xt does not change any longer. MDP Components: 1. A set of states s S 2. A set of actions a A. A transition fundtion T(s,a,s'): Probability that executing action a in s will lead to s'P(s'|s,a). The probability is called the model. 3. A reward function R(s,a,S') Sometimes just R(s) or R(s') 4. An intial state S0 5.Optionally, one ormore terminal states. A policy maps from a state onto the action to take if the agent is in that state pi(S) = a (Policy pi telss the agent what action to take at state S). Even if the same policy is executed multiple times by the agent, this may lead to different sequence of states and actions(environment history), and thus a different score under the reward function. Therefore we need to compute the expected utility of all the possible paths generated by a policy\nIf final reward is 1 and -1, r (penalty to for a step) < -1.65: best policy is exit asap (even it means fall into a pit). -0.03<r<0: bouncing against the wall by facing opposite of -1. r > 0: avoid any terminal states so that I can bounce around and keep collecting rewards. Discout: each time we descend a level, we multiply in the discount once. Sooner rewards probably do have higher utility than later rewards. It also helps our algorithms converge. eg. discount of 0.5 U([1,2,3] = 1 + 0.5*2 + 0.5*0.5*3). U([1,2,3]) < U([3,2,1])\nIf we assume stationary preferences, then there are only two ways to define utilities. Additive utility: U([r0, r1, r2...]) = r0 + r1 + r2 + ... Discounted utility: U([r0, r1, r2...]) = r0 + d*r1 + d^2*r2 + ...\nSolution for infinite utilities: 1. finite horizon(similar to depth-limited search): gives nonstationary policies(pi depends on time left) 2. Discounting use 0<gamma<1, smaller gamma means smaller horizion - shorter term focus 3. absorbing state: guarantee that for every policy, a terminal state will eventually be reached(like \"overheated\" for racing) \nValue iteration: 1. start with V0(s) = 0. No time steps left means an expected reward sum of zero. 2. Given vector of Vk(s) values, do one ply of expectimax from each state3. repeat until convergence. 4. Complexity of each iteration: O(S^2 *A). Value iteration will converge to unique optimal values. Basic idea: approximations get refined towards optimal values. Policy may converge long before values do\nFixed policy evaluation: idea 1: turn recursive bellman equations into updates(like value iteration) O(S^3) per iteration. Idea 2: Without the maxes, the Bellman equations are just a linear system\nactions are easier to select from q-values than values. Ti(s) = arg max_a Q(s, a). Problem with value iteration: 1. it's slow 2. the max at each state rarely changes\nPolicy iteration: step 1: policy evaluation: calculate utilities for some fixed policy (not optimal policies) until convergence. Step 2: Policy improvement: update policy using one step look-ahead with resulting converged(but not optimal!) utilities as future values. Repeat steps until policy converges. Policy iteration is still optimal. It can converge (much) faster under some conditions\nPolicy iteration and value iteration: Both compute the same thing (all optimal values). Both are dynamic programing. In value iteration, every iteration updates both the values and (implicitly) the policy. We don't track the policy, but taking the max over actions implicityly recomputes it. In policy iteration: we do several passes that update utilities with fixed policy (each pass is fast because we consider only one action, not all of them). After the policy is evaluated, a new policy is chosen(slow like a value iteration pass). The new policy will be better (or we are done)\nFor worst-case minimax reasoning, terminal function scale doesn't matter. We just want better states to have higher evaluations (get the ordering right). We call this insensitivity to monotonic transformations. For average-case expectimax reasoning, we need magnitudes to be meaningful\nAny rational preferences can be summarized as a utility function. we can just specify what the utilities are and not the behaviors. We can say what the utilities of certain things are and then our agents will behave in order to maximize their expected utilities. Rational preferences imply behavior describable as maximization of expected utility.\nMaximum exptected utility (MEU) principle: Choose the action that maximizes expected utility. Note. and agent can be entirely rational (consistent with MEU) without ever representing or manipulating utilities and probabilities. eg. a lookup table for perfect tic tac toe, a reflex vacuum cleaner\n\n## Module 8 Definitions\n* **Utility function** - which assigns a single number to express the desirability of a state\n* **Expected utility** - of an action given the evidence, is the average utility of the outcomes, weighted by the probability that the outcome occurs\n* **Markov decision process** - is a sequential decision problem for a fully observable environment with a stochastic trasnistion model that has additive rewards. \n* **Rewards** - The performance of an agent in an MDP is the sum of the rewards for the transitinos it takes\n* **Nonstionary** - A policy that depends on the time is called nonstationary\n* **Optimal policy** - A policy that yields the highest expected utility\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Your summary here \n\n## AIMA Chapter 22.1 Learning from Rewards\n Your summary here \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **Term** - your definition here\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Your summary here \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Term** - your definition here\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Your summary here \n\n## Module 11 Definitions\n* **X is conditionally independent of y given z** - if and only if 对所有 x,y,z: P(x,y|z) = P(x|z)P(y|z) or equivalenty if and only if P(x|z,y) = P(x|z)\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Your summary here \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n Your summary here \n\n## Module 12 Definitions\n* **Concept** - definition\n\n\n## WARNING: Your definition of 'Activation values' is too long!\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Your summary here \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Your summary here \n\n## SLP Chapter 5.3 The cross-entropy loss function\n Your summary here \n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n For more than two classes (eg 3 way sentiment classification: positive, negative, or neutral), we use multinomial logistic regression, also called softmax regression.In multinomial logistic regression\nthe target y is a variable that ranges over more than two classes; we want to know the probability of y being in each potential class c 属于 C, p(y = c|x). The multinomial logistic classifier uses a generalization of the sigmoid, called\nsoftmax the softmax function, to compute the probability p(y = c|x). The softmax function takes a vector z = [z1; z2; :::; zk] of k arbitrary values and maps them to a probability\ndistribution, with each value in the range (0,1), and all the values summing to 1.\nLike the sigmoid, it is an exponential function. The denominator sum(i=1, k) exp(zi) is used to normalize all the values into probabilities.\nlike the sigmoid, the input to the softmax will be the dot product between\na weight vector w and an input vector x (plus a bias). But now we’ll need separate\nweight vectors (and bias) for each of the K classes. Like the sigmoid, the softmax has the property of squashing values toward 0 or 1.\nThus if one of the inputs is larger than the others, it will tend to push its probability\ntoward 1, and suppress the probabilities of the smaller inputs.\n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n There are 3 commonly used non-linear functions used for f:\nThe sigmoid function\nThe tanh function\nThe rectified linear unit ReLU\n\n## Module 13 Definitions\n* **Activation values** - instead of directly ouputting z = w*x+b, which is a linear function of x, neuron units apply a non-linear function f to z, The output of this function is called the activation value for the unit\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 14 Lecture Notes\nTerm-Document Matrix: Each column vector represents a document in the document collection. Each row represents a word. The value in a cell is how often that term occurred in that\ndocument. Document vectors can be sparse. We can measure how similar two documents are by comparing their column vectors. By comparing two row vectors tells how similar two words are. Words that occur in very similar sets of documents would therefore be very similar. \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}