{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "M_Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "af5e3ab7-0115-44c5-af59-6604f085892a"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "2c077128-ce55-44ac-ce58-a577d1ef5f4e"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = True"
      ],
      "execution_count": 114,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" Chapter 16.1 summary. \"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\" Your summary here \"\"\"\n",
        "chapter16_3 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\" Markovian Transition: The probability of reaching s′ from s depends only on s and not on the history of earlier states.\n",
        "A sequential decision problem: 1. fully observable, stochastic environment 2. Markovian transition model 3. additive rewards - Markov decision process:\n",
        "1. a set of states (with an initial state s0);<br>\n",
        "2. a set ACTIONS(s) of actions in each state; <br>\n",
        "3. a transition model P(s′|s,a); <br>\n",
        "4. and a reward function R(s,a,s′). <br>\n",
        "Optimal policy: A policy that yields the highest expected utility. <br>\n",
        "Finite horizon: an optimal action in a given state may depend on how much time is left. <br>\n",
        "Nonstationary Policy: Depends on the time is called nonstationary.<br>\n",
        "Infinite Horizon: With no fixed time limit, on the other hand, there is no reason to behave differently in the same state at different times. <br>\n",
        "An optimal action in this case depends only on the current state and the optimal policy is stationary.<br>\n",
        "There can be finite state sequences in an infinite-horizon MDP that contains a terminal state. <br>\n",
        "Utility function: additive discounted rewards <br>\n",
        "Why discount? Sooner rewards probably do have higher utility than later rewards, Also helps our algorithms converge <br>\n",
        "Difficulty with infinite horizon f the environment does not contain a terminal state, or if the agent never reaches one, then all environment histories will be infinitely long, and utilities with additive undiscounted rewards will generally be infinite. <br>\n",
        "Solutions:<br>\n",
        "Discounted utilities with infinite horizons: the optimal policy is independent of the starting state. <br>\n",
        "action-utility function[Q-function: Q(s,a)]: Expected utility of taking a given action in a given state. <br>\n",
        "\n",
        "\"\"\"\n",
        "chapter17_2 =  \"\"\" \n",
        "1. Value Iteration: <br>\n",
        "If we have n states, then we have n Bellman equations that need to be updated in every iteration. <br>\n",
        "Start with arbitrary values, and then update the equation to get updated utilities/values for each state. <br>\n",
        "Problem 1: It’s slow – O(S2A) per iteration <br>\n",
        "Problem 2: The “max” at each state rarely changes <br>\n",
        "Problem 3: The policy often converges long before the values<br>\n",
        "Theorem: will converge to unique optimal values- Basic idea: approximations get refined towards optimal values, Policy may converge long before values do <br>\n",
        "2. Policy Iteration: <br> Evaluate the policy Calculate Utility {U_i} of the states under given policy [max of q values] <br>Improve the policy: Calculate a new policy  using one-step look-ahead based on {U_i}  <br>\n",
        "3. Online Learning: expectimax search <br> \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"Your definition here\",\n",
        "\"expected utility\" : \"your definition here\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": 115,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1024
        },
        "outputId": "588ee0eb-0d00-47e3-ad7d-0bff83222426"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 17.1 Sequential Decision Problems\n Markovian Transition: The probability of reaching s′ from s depends only on s and not on the history of earlier states.\nA sequential decision problem: 1. fully observable, stochastic environment 2. Markovian transition model 3. additive rewards - Markov decision process:\n1. a set of states (with an initial state s0);<br>\n2. a set ACTIONS(s) of actions in each state; <br>\n3. a transition model P(s′|s,a); <br>\n4. and a reward function R(s,a,s′). <br>\nOptimal policy: A policy that yields the highest expected utility. <br>\nFinite horizon: an optimal action in a given state may depend on how much time is left. <br>\nNonstationary Policy: Depends on the time is called nonstationary.<br>\nInfinite Horizon: With no fixed time limit, on the other hand, there is no reason to behave differently in the same state at different times. <br>\nAn optimal action in this case depends only on the current state and the optimal policy is stationary.<br>\nThere can be finite state sequences in an infinite-horizon MDP that contains a terminal state. <br>\nUtility function: additive discounted rewards <br>\nWhy discount? Sooner rewards probably do have higher utility than later rewards, Also helps our algorithms converge <br>\nDifficulty with infinite horizon f the environment does not contain a terminal state, or if the agent never reaches one, then all environment histories will be infinitely long, and utilities with additive undiscounted rewards will generally be infinite. <br>\nSolutions:<br>\nDiscounted utilities with infinite horizons: the optimal policy is independent of the starting state. <br>\naction-utility function[Q-function: Q(s,a)]: Expected utility of taking a given action in a given state. <br>\n\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n \n1. Value Iteration: <br>\nIf we have n states, then we have n Bellman equations that need to be updated in every iteration. <br>\nStart with arbitrary values, and then update the equation to get updated utilities/values for each state. <br>\nProblem 1: It’s slow – O(S2A) per iteration <br>\nProblem 2: The “max” at each state rarely changes <br>\nProblem 3: The policy often converges long before the values<br>\nTheorem: will converge to unique optimal values- Basic idea: approximations get refined towards optimal values, Policy may converge long before values do <br>\n2. Policy Iteration: <br> Evaluate the policy Calculate Utility {U_i} of the states under given policy [max of q values] <br>Improve the policy: Calculate a new policy  using one-step look-ahead based on {U_i}  <br>\n3. Online Learning: expectimax search <br> \n\n## Module 8 Lecture Notes\n Your summary here \n\n## Module 8 Definitions\n* **Utility function** - Your definition here\n* **Expected utility** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\" RL differs from MDPs as agents are in the MDPs and do not know the reward or the transition function and act in order to learn more <br>\n",
        "1. Model based RL: Use the transition model of the environment to learn the rewards and to make decisions on what actions to take. e.g. in Chess, we know what actions can be taken at a given state (even though we don't know which action is the best) <br>\n",
        "Systems often learn a utility function n terms of the sum of rewards <br>\n",
        "2. Model free RL: The agent does not know, nor learns the transition model <br>\n",
        "  2.1 The agent learns the action-utility mapping or quality function that is a function of the sum of rewards if an action is taken at a state s <br>\n",
        "  2.2 Policy search: The agent learns a policy which maps to an action for a state (Reflex Agent) <br>\n",
        "Passive RL: Policy is fixed, agent learns utility of the states under that policy (need not learn the transition model) <br>\n",
        "Active RL: Agent tries actions to explore the environment (figure out what to do) <br>\n",
        "Apprenticeship learning: Learn using demonstrations instead of rewards <br>\"\"\"\n",
        "chapter22_2 =  \"\"\" Passive learning agent learns the utility function (expected sum of rewards for a given state) under a fixed policy. This is different than the policy evaluation (in MDP) as in this case the agent does now know the transition and the reward functions of the environments. <br>\n",
        "1. Direct utility estimation (DUE) is that the utility of a state is defined as the expected total reward from that state onward (called the expected reward-to-go), and that each trial provides a sample of this quantity for each state visited. <br>\n",
        "DUE ignores that the utility of a state is determined by the reward and the expected utility of the successor states. In this sense it misses chances to learn and converges slowly. <br>\n",
        "It’s easy to understand,It doesn’t require any knowledge of T, R,It eventually computes the correct average values, using just sample transitions <br>\n",
        "What bad about it? It wastes information about state connections Each state must be learned separately So, it takes a long time to learn <br>\n",
        "2. Adaptive Dynamic Programming: Takes advantage of the constraints among the utilities of states by learning the transition model that connects them and solving the corresponding Markov decision process using dynamic programming. <br>\n",
        "modified policy iteration: Update the utility function based on change in the learned model, and because the model only slightly changes after every observation, this is simplified. <br>\n",
        "In terms of how quickly its value estimates improve, the ADP agent is limited only by its ability to learn the transition model.<br>\n",
        "3. Temporal-difference learning: Use the observed transitions to adjust the utilities of the observed states so that they agree with the constraint equations. <br>\n",
        "TD does not need a transition model to perform its updates. <br> TD value leaning is a model-free way to do policy evaluation, mimicking\n",
        "Bellman updates with running sample averages\n",
        "However, if we want to turn values into a (new) policy, we’re sunk <br>\n",
        "TD adjusts a state to agree with its observed successor, whereas ADP adjusts the state to agree with all of the successors that might occur, weighted by their probabilities. \"\"\"\n",
        "chapter22_3 =  \"\"\" Absorbing State: Where no actions have any effect and no rewards are received. <br>\n",
        "Bayesian reinforcement learning: assumes a prior probability P(h) over hypotheses h about what the true model is; the posterior probability P(h | e) is obtained in the usual way by Bayes’ rule given the observations to date. <br>\n",
        "Temporal-difference Q-learning: Q-learning converges to optimal policy -- even if you’re acting suboptimally! This is called off-policy learning\n",
        "Have to explore enough, have to eventually make the learning rate small enough but not decrease it too quickly\n",
        "Basically, in the limit, it doesn’t matter how you select actions <br>\n",
        "\"\"\"\n",
        "chapter22_4 =  \"\"\" In real world, can keep track of all Q values (too many states) -> instead learn for a small number of states through experience and generalize the experience to new similar states. <br>\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"term\" : \"your definition here\",\n",
        "}"
      ],
      "execution_count": 117,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1370
        },
        "outputId": "e4902ca5-9672-47a4-ce33-f343c45682c7"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Your summary here \n\n## AIMA Chapter 22.1 Learning from Rewards\n RL differs from MDPs as agents are in the MDPs and do not know the reward or the transition function and act in order to learn more <br>\n1. Model based RL: Use the transition model of the environment to learn the rewards and to make decisions on what actions to take. e.g. in Chess, we know what actions can be taken at a given state (even though we don't know which action is the best) <br>\nSystems often learn a utility function n terms of the sum of rewards <br>\n2. Model free RL: The agent does not know, nor learns the transition model <br>\n  2.1 The agent learns the action-utility mapping or quality function that is a function of the sum of rewards if an action is taken at a state s <br>\n  2.2 Policy search: The agent learns a policy which maps to an action for a state (Reflex Agent) <br>\nPassive RL: Policy is fixed, agent learns utility of the states under that policy (need not learn the transition model) <br>\nActive RL: Agent tries actions to explore the environment (figure out what to do) <br>\nApprenticeship learning: Learn using demonstrations instead of rewards <br>\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Passive learning agent learns the utility function (expected sum of rewards for a given state) under a fixed policy. This is different than the policy evaluation (in MDP) as in this case the agent does now know the transition and the reward functions of the environments. <br>\n1. Direct utility estimation (DUE) is that the utility of a state is defined as the expected total reward from that state onward (called the expected reward-to-go), and that each trial provides a sample of this quantity for each state visited. <br>\nDUE ignores that the utility of a state is determined by the reward and the expected utility of the successor states. In this sense it misses chances to learn and converges slowly. <br>\nIt’s easy to understand,It doesn’t require any knowledge of T, R,It eventually computes the correct average values, using just sample transitions <br>\nWhat bad about it? It wastes information about state connections Each state must be learned separately So, it takes a long time to learn <br>\n2. Adaptive Dynamic Programming: Takes advantage of the constraints among the utilities of states by learning the transition model that connects them and solving the corresponding Markov decision process using dynamic programming. <br>\nmodified policy iteration: Update the utility function based on change in the learned model, and because the model only slightly changes after every observation, this is simplified. <br>\nIn terms of how quickly its value estimates improve, the ADP agent is limited only by its ability to learn the transition model.<br>\n3. Temporal-difference learning: Use the observed transitions to adjust the utilities of the observed states so that they agree with the constraint equations. <br>\nTD does not need a transition model to perform its updates. <br> TD value leaning is a model-free way to do policy evaluation, mimicking\nBellman updates with running sample averages\nHowever, if we want to turn values into a (new) policy, we’re sunk <br>\nTD adjusts a state to agree with its observed successor, whereas ADP adjusts the state to agree with all of the successors that might occur, weighted by their probabilities. \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Absorbing State: Where no actions have any effect and no rewards are received. <br>\nBayesian reinforcement learning: assumes a prior probability P(h) over hypotheses h about what the true model is; the posterior probability P(h | e) is obtained in the usual way by Bayes’ rule given the observations to date. <br>\nTemporal-difference Q-learning: Q-learning converges to optimal policy -- even if you’re acting suboptimally! This is called off-policy learning\nHave to explore enough, have to eventually make the learning rate small enough but not decrease it too quickly\nBasically, in the limit, it doesn’t matter how you select actions <br>\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n In real world, can keep track of all Q values (too many states) -> instead learn for a small number of states through experience and generalize the experience to new similar states. <br>\n \n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **Term** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\"Observed variables (evidence): Agent knows certain things about the state of the world (e.g.,sensor readings or symptoms) <br>\n",
        "Unobserved variables (states): Agent needs to reason about other aspects (e.g. where an object is or what disease is present)\n",
        "Model: Agent knows something about how the known variables relate to the unknown variables <br>\n",
        "The fundamental idea of decision theory is that an agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all the possible outcomes of the action. This is called the principle of maximum expected utility (MEU). <br>\n",
        "\"\"\"\n",
        "chapter12_2 = \"\"\" y, the possible worlds are called the sample space, and they mutually exclusive and exhaustive. <br>\n",
        " \"\"\"\n",
        "chapter12_3 = \"\"\" Summing up a row: marginalization, or summing out—because we sum up the probabil- ities for each possible value of the other variables, thereby taking them out of the equation. <br>\n",
        "Marginal distributions are sub-tables which eliminate variables, Combine collapsed rows by adding <br>\n",
        "\"\"\"\n",
        "chapter12_6 = \"\"\" The naive Bayes model assumes the conditional independence of all effect variables, given a single cause variable; its size grows linearly with the number of effects.\n",
        " \"\"\"\n",
        "chapter12_7 = \"\"\" A wumpus-world agent can calculate probabilities for unobserved aspects of the world, thereby improving on the decisions of a purely logical agent. Conditional independence makes these calculations tractable. \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" Unigram: Use no words to estimate the next words <br>\n",
        "Bigram: one word <br>\n",
        "An intuitive way to estimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood estimation the MLE estimate for the parameters of an n-gram model by getting counts from a\n",
        "corpus, and normalizing the counts so that they lie between 0 and 1. <br>\n",
        "By dividing the observed frequency of a particular sequence by the observed frequency of a prefix. This ratio is called a relative frequency <br>\n",
        "\"\"\"\n",
        "SLP_chapter3_2 = \"\"\" Extrinsic evaluation: The best way to evaluate the performance of a language model is to embed it in an application and measure how much the application improves. <br>\n",
        "An intrinsic evaluation metric is one that mea- intrinsic evaluation sures the quality of a model independent of any application. <br>\n",
        "Training on the test set introduces a bias that makes the probabilities all look too high, and causes huge inaccuracies in perplexity <br>\n",
        "Perplexity: weighted average branching factor of a language <br>\n",
        " \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Sampling from a distribution means to choose random points according to their likelihood. Thus sampling from a language model—which represents a distribution over sentences—means to generate some sentences, choosing\n",
        "each sentence according to its likelihood as defined by the model. Thus we are more likely to generate sentences that the model thinks have a high probability and less likely to generate sentences that the model thinks have a low probability. <br>\n",
        "\"\"\"\n",
        "SLP_chapter3_4 = \"\"\"  Statistical models are likely to be pretty useless as predictors if the training sets and the test sets are way different <br>\n",
        "Solution: To be sure to use a training corpus that has a similar genre and daialect to whatever task we are trying to accomplish. <br>\n",
        "Our models may still be subject to the problem of sparsity. <br>\n",
        "zero probability n-grams: By definition, perplexity is based on the inverse probability of thetest set. Thus if some words have zero probability, we can’t compute perplexity at\n",
        "all, since we can’t divide by 0 <br>\n",
        "Unknown words, or out of vocabulary (OOV) words:  by adding a pseudo-word called <UNK><br>\n",
        "1. This can be handled by making the vocab closed and treating UNK word as any other word. <br>\n",
        "2. When we don’t have a prior vocabulary in advance, is to create such a vocabulary implicitly, replacing words in the training data by <UNK> based on their frequency. For example we can replace by <UNK> all words that occur fewer than n times in the training set, where n is some small number, or equivalently select a vocabulary size V in advance (say 50,000) and choose the top V words by frequency and replace the rest by UNK. <br>\n",
        "\"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Problem: words that are in our vocabulary (they are not unknown words) but appear in a test set in an unseen context <br>\n",
        "1. Laplace Smoothing: add one to all the n-gram counts, before we normalize them into probabilities.<br>\n",
        "Discounting: discounting (lowering) some non-zero counts in order to get the probability mass that will be assigned to the zero counts. <br>\n",
        "2.  Add-k smoothing: move a bit less of the probability mass from the seen to the unseen events. Instead of adding 1 to each count, we add a fracadd-k tional count k <br>\n",
        "3. Backoff and Interpolation:\n",
        "Backoff: we use the trigram if the evidence is sufficient, otherwise we use the bigram, otherwise the unigram. <br>\n",
        "Interpolation: we always mix the probability estimates from all the n-gram estimators, weighing and combining the trigram, bigram, and unigram counts. <br>\n",
        "Learning weights for interpolation:  λs are learned from a held-out corpus. A held-out corpus is an additional training corpus that we use to set hyperparameters <br>\n",
        "Backoff with discounting is also called Katz backoff: \n",
        "\"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Stupid backoff gives up the idea of trying to make the language\n",
        "model a true probability distribution. There is no discounting of the higher-order\n",
        "probabilities. If a higher-order n-gram has a zero count, we simply backoff to a\n",
        "lower order n-gram, weighed by a fixed (context-independent) weight. \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"Perplexity\" : \"of a language model on a test set is the inverse probability of the test set, normalized by the number of words\",\n",
        "}"
      ],
      "execution_count": 119,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1995
        },
        "outputId": "b0a08427-7d80-48de-ee25-1a082dcb1c2a"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 120,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\nObserved variables (evidence): Agent knows certain things about the state of the world (e.g.,sensor readings or symptoms) <br>\nUnobserved variables (states): Agent needs to reason about other aspects (e.g. where an object is or what disease is present)\nModel: Agent knows something about how the known variables relate to the unknown variables <br>\nThe fundamental idea of decision theory is that an agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all the possible outcomes of the action. This is called the principle of maximum expected utility (MEU). <br>\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n y, the possible worlds are called the sample space, and they mutually exclusive and exhaustive. <br>\n \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Summing up a row: marginalization, or summing out—because we sum up the probabil- ities for each possible value of the other variables, thereby taking them out of the equation. <br>\nMarginal distributions are sub-tables which eliminate variables, Combine collapsed rows by adding <br>\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n The naive Bayes model assumes the conditional independence of all effect variables, given a single cause variable; its size grows linearly with the number of effects.\n \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n A wumpus-world agent can calculate probabilities for unobserved aspects of the world, thereby improving on the decisions of a purely logical agent. Conditional independence makes these calculations tractable. \n\n## SLP Chapter 3.1 N-Grams\n Unigram: Use no words to estimate the next words <br>\nBigram: one word <br>\nAn intuitive way to estimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood estimation the MLE estimate for the parameters of an n-gram model by getting counts from a\ncorpus, and normalizing the counts so that they lie between 0 and 1. <br>\nBy dividing the observed frequency of a particular sequence by the observed frequency of a prefix. This ratio is called a relative frequency <br>\n\n\n## SLP Chapter 3.2 Evaluating Language Models\n Extrinsic evaluation: The best way to evaluate the performance of a language model is to embed it in an application and measure how much the application improves. <br>\nAn intrinsic evaluation metric is one that mea- intrinsic evaluation sures the quality of a model independent of any application. <br>\nTraining on the test set introduces a bias that makes the probabilities all look too high, and causes huge inaccuracies in perplexity <br>\nPerplexity: weighted average branching factor of a language <br>\n \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Sampling from a distribution means to choose random points according to their likelihood. Thus sampling from a language model—which represents a distribution over sentences—means to generate some sentences, choosing\neach sentence according to its likelihood as defined by the model. Thus we are more likely to generate sentences that the model thinks have a high probability and less likely to generate sentences that the model thinks have a low probability. <br>\n\n\n## SLP Chapter 3.4 Generalization and Zeros\n  Statistical models are likely to be pretty useless as predictors if the training sets and the test sets are way different <br>\nSolution: To be sure to use a training corpus that has a similar genre and daialect to whatever task we are trying to accomplish. <br>\nOur models may still be subject to the problem of sparsity. <br>\nzero probability n-grams: By definition, perplexity is based on the inverse probability of thetest set. Thus if some words have zero probability, we can’t compute perplexity at\nall, since we can’t divide by 0 <br>\nUnknown words, or out of vocabulary (OOV) words:  by adding a pseudo-word called <UNK><br>\n1. This can be handled by making the vocab closed and treating UNK word as any other word. <br>\n2. When we don’t have a prior vocabulary in advance, is to create such a vocabulary implicitly, replacing words in the training data by <UNK> based on their frequency. For example we can replace by <UNK> all words that occur fewer than n times in the training set, where n is some small number, or equivalently select a vocabulary size V in advance (say 50,000) and choose the top V words by frequency and replace the rest by UNK. <br>\n\n\n## SLP Chapter 3.5 Smoothing\n Problem: words that are in our vocabulary (they are not unknown words) but appear in a test set in an unseen context <br>\n1. Laplace Smoothing: add one to all the n-gram counts, before we normalize them into probabilities.<br>\nDiscounting: discounting (lowering) some non-zero counts in order to get the probability mass that will be assigned to the zero counts. <br>\n2.  Add-k smoothing: move a bit less of the probability mass from the seen to the unseen events. Instead of adding 1 to each count, we add a fracadd-k tional count k <br>\n3. Backoff and Interpolation:\nBackoff: we use the trigram if the evidence is sufficient, otherwise we use the bigram, otherwise the unigram. <br>\nInterpolation: we always mix the probability estimates from all the n-gram estimators, weighing and combining the trigram, bigram, and unigram counts. <br>\nLearning weights for interpolation:  λs are learned from a held-out corpus. A held-out corpus is an additional training corpus that we use to set hyperparameters <br>\nBackoff with discounting is also called Katz backoff: \n\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Stupid backoff gives up the idea of trying to make the language\nmodel a true probability distribution. There is no discounting of the higher-order\nprobabilities. If a higher-order n-gram has a zero count, we simply backoff to a\nlower order n-gram, weighed by a fixed (context-independent) weight. \n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Perplexity** - of a language model on a test set is the inverse probability of the test set, normalized by the number of words\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\"Bayesian networks can represent essentially any full joint probability distribution and in many cases can do so very concisely. <br>\n",
        " \"\"\"\n",
        "chapter13_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter13_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 121,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177
        },
        "outputId": "410dda5e-37a1-4864-a463-423fc3fcdb91"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  \n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## Module 11 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" transfer learning, in which knowl- edge from one domain is transferred to a new domain, so that learning can proceed faster with less data. We do assume, however, that the designer of the system chooses a model framework that can lead to effective learning.<br>\n",
        "Going from a specific set of observations to a general rule is called induction; from the observations that the sun rose every day in the past, we induce that the sun will come up tomorrow. This differs from the deduction because the inductive conclusions may be incorrect, whereas deductive conclusions are guaranteed to be correct if the premises are correct. <br>\n",
        " \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\" Naive Bayes Classification: <br>\n",
        "Assumptions: \n",
        "1. Bag of words: context does not matter <br>\n",
        "2. Naive Bayes Assumption: All features/words are independent of the class. <br>\n",
        "Done in log space to avoid underflow and increase speed <br>\n",
        "Log Space: computes the predicted class as a linear function of input features. Classifiers that use a linear combination of the inputs\n",
        "to make a classification decision —like naive Bayes and also logistic regression—\n",
        "are called linear classifiers. <br>\"\"\"\n",
        "slp_chapter4_2 = \"\"\" P(c) = count of documents of category C/ Total number of documents <br>\n",
        "P(w_i|c) = Count of occurence of w_i in documents from category c / total word count in category c documents <br>\n",
        "Stop Words: Top 100 occuring words can be removed from both train and test set, hasn't proved useful in improving accuracy <br>\n",
        "Unknown words: Remove from test set <br>\n",
        " \"\"\"\n",
        "slp_chapter4_4 = \"\"\"For sentiment analysis, occurrence of the word matters and not the frequency, therefore clip the count to 1 - binary naive Bayes <br>\n",
        "Step1: For each document when creating bag of words, clip down the frequency to 1. Count multiple if occurs in multiple documents <br>\n",
        "Step2: Handling negation - Add prefix NOT_ to all tokens <br>\n",
        "If the training data is in sufficient:  derive the positive and negative word features using sentiment lexicons. Add new features like feature that is counted whenever a word from that lexicon occurs. Thus we might add a feature called ‘this word occurs in the positive lexicon’, and treat all instances of words in the lexicon as counts for that one feature, instead of counting each word separately <br>\n",
        "\"\"\"\n",
        "slp_chapter4_5 = \"\"\" Spam Detection: rather than using all the words as individual features, is to predefine likely sets of words or phrases as features, combined with features that are not purely linguistic.<br>\n",
        "Language ID: Use character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,‘thei’), or, even simpler byte n-grams<br>\n",
        "\"\"\"\n",
        "slp_chapter4_9 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_10 = \"\"\" Your summary here \"\"\"\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 123,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l6550Q-qUPDP"
      },
      "source": [
        "![Screen Shot 2021-12-04 at 8.55.08 PM.png](data:image/png;base64,[encoded data, 419976 characters])"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1194
        },
        "outputId": "9b058584-7f99-436f-ad1f-bf40d5de8cf6"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 19.1 Forms of Learning\", chapter19_1),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 19.1 Forms of Learning\n transfer learning, in which knowl- edge from one domain is transferred to a new domain, so that learning can proceed faster with less data. We do assume, however, that the designer of the system chooses a model framework that can lead to effective learning.<br>\nGoing from a specific set of observations to a general rule is called induction; from the observations that the sun rose every day in the past, we induce that the sun will come up tomorrow. This differs from the deduction because the inductive conclusions may be incorrect, whereas deductive conclusions are guaranteed to be correct if the premises are correct. <br>\n \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Naive Bayes Classification: <br>\nAssumptions: \n1. Bag of words: context does not matter <br>\n2. Naive Bayes Assumption: All features/words are independent of the class. <br>\nDone in log space to avoid underflow and increase speed <br>\nLog Space: computes the predicted class as a linear function of input features. Classifiers that use a linear combination of the inputs\nto make a classification decision —like naive Bayes and also logistic regression—\nare called linear classifiers. <br>\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n P(c) = count of documents of category C/ Total number of documents <br>\nP(w_i|c) = Count of occurence of w_i in documents from category c / total word count in category c documents <br>\nStop Words: Top 100 occuring words can be removed from both train and test set, hasn't proved useful in improving accuracy <br>\nUnknown words: Remove from test set <br>\n \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\nFor sentiment analysis, occurrence of the word matters and not the frequency, therefore clip the count to 1 - binary naive Bayes <br>\nStep1: For each document when creating bag of words, clip down the frequency to 1. Count multiple if occurs in multiple documents <br>\nStep2: Handling negation - Add prefix NOT_ to all tokens <br>\nIf the training data is in sufficient:  derive the positive and negative word features using sentiment lexicons. Add new features like feature that is counted whenever a word from that lexicon occurs. Thus we might add a feature called ‘this word occurs in the positive lexicon’, and treat all instances of words in the lexicon as counts for that one feature, instead of counting each word separately <br>\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Spam Detection: rather than using all the words as individual features, is to predefine likely sets of words or phrases as features, combined with features that are not purely linguistic.<br>\nLanguage ID: Use character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,‘thei’), or, even simpler byte n-grams<br>\n\n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n Your summary here \n\n## Module 12 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\"Logistic regression has a number of advantages over naive\n",
        "Bayes. Naive Bayes has overly strong conditional independence assumptions. Consider two features which are strongly correlated; in fact, imagine that we just add the\n",
        "same feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate, multiplying them both in, overestimating the evidence. By contrast, logistic\n",
        "regression is much more robust to correlated features; if two features f1 and f2 are perfectly correlated, regression will simply assign part of the weight to w1 and part to w2. Thus when there are many correlated features, logistic regression will assign a more accurate probability than naive Bayes. So logistic regression generally works better on larger documents or datasets and is a common default.\n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\" This is called conditional maximum likelihood estimation: we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x. The resulting loss\n",
        "function is the negative log likelihood loss, generally called the cross-entropy loss <br>\n",
        "\"\"\"\n",
        "slp_chapter5_4 = \"\"\" For logistic regression, this loss function is conveniently convex. A convex function has just one minimum; there are no local minima to get stuck in, so gradient\n",
        "descent starting from any point is guaranteed to find the minimum. (By contrast, the loss for multi-layer neural networks is non-convex, and gradient descent may get stuck in local minima for neural network training and never find the global optimum.) <br>\"\"\"\n",
        "slp_chapter5_5 = \"\"\" For example in batch training we compute the gradient over the entire dataset. By seeing so many examples, batch training offers a superb estimate of which direction to move the weights, at the cost of spending a lot of time processing every single example in the training set to compute this perfect direction.\n",
        "mini-batch A compromise is mini-batch training: we train on a group of m examples \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\" Neural nets: better than logistic regression as do not require feature handling and learn features on its own <br>\n",
        "Activation Functions (make nn non linear): \n",
        "1. Sigmoid - maps to [0,1] and is differentiable <br>\n",
        "2. tanh: better than sigmoid <br>\n",
        "3. ReLU: max(0,z) most commonly used <br>\n",
        "Rectifiers are better activation functions as they do not saturate the higher values of z. tanh and sigmoid value is close to 1 for large values of z and gradient is close to 0. Causes problem in learning: Vanishing Gradient <br>\n",
        "\"\"\"\n",
        "slp_chapter7_2 = \"\"\" A single perceptron cannot solve the XOR problem as a perceptron is a linear classifier (does not have a non linear activation) while XOR is not linearly seperable. <br>\n",
        "Neural networks: With different layers, the hidden layers represent a form of the input. \"\"\"\n",
        "slp_chapter7_3 = \"\"\" Softmax function: To normalize output of a network to probability distribution. <br>\n",
        "We use non-linear activation functions for each layer in a neural network is that if we did not, the resulting network is exactly equivalent to a single-layer network. \"\"\"\n",
        "slp_chapter7_4 = \"\"\" The idea of using word2vec or GloVe embeddings as our input representation and more generally the idea of relying on another algorithm to have already learned pretraining an embedding representation for our input words — is called pretraining \"\"\"\n",
        "slp_chapter7_5 = \"\"\" Compared to n-gram models, neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction. On the other hand, neural net language models are much more complex, slower to train, and less interpretable than n-gram models, so for many (especially smaller) tasks an n-gram language model is still the right tool. \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Various forms of regularization are used to prevent overfitting. One of the most dropout important is dropout: randomly dropping some units and their connections from the network during training <br>\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 125,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qwsKRev2lEk5"
      },
      "source": [
        "![Screen Shot 2021-12-06 at 11.28.40 AM.png](data:image/png;base64,[encoded data, 65336 characters])"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1498
        },
        "outputId": "d2d7e13a-428e-483f-d525-820c7fb1cb38"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 126,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\nLogistic regression has a number of advantages over naive\nBayes. Naive Bayes has overly strong conditional independence assumptions. Consider two features which are strongly correlated; in fact, imagine that we just add the\nsame feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate, multiplying them both in, overestimating the evidence. By contrast, logistic\nregression is much more robust to correlated features; if two features f1 and f2 are perfectly correlated, regression will simply assign part of the weight to w1 and part to w2. Thus when there are many correlated features, logistic regression will assign a more accurate probability than naive Bayes. So logistic regression generally works better on larger documents or datasets and is a common default.\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n This is called conditional maximum likelihood estimation: we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x. The resulting loss\nfunction is the negative log likelihood loss, generally called the cross-entropy loss <br>\n\n\n## SLP Chapter 5.4 Gradient Descent\n For logistic regression, this loss function is conveniently convex. A convex function has just one minimum; there are no local minima to get stuck in, so gradient\ndescent starting from any point is guaranteed to find the minimum. (By contrast, the loss for multi-layer neural networks is non-convex, and gradient descent may get stuck in local minima for neural network training and never find the global optimum.) <br>\n\n## SLP Chapter 5.5 Regularization\n For example in batch training we compute the gradient over the entire dataset. By seeing so many examples, batch training offers a superb estimate of which direction to move the weights, at the cost of spending a lot of time processing every single example in the training set to compute this perfect direction.\nmini-batch A compromise is mini-batch training: we train on a group of m examples \n\n## SLP Chapter 7.1 Units\n Neural nets: better than logistic regression as do not require feature handling and learn features on its own <br>\nActivation Functions (make nn non linear): \n1. Sigmoid - maps to [0,1] and is differentiable <br>\n2. tanh: better than sigmoid <br>\n3. ReLU: max(0,z) most commonly used <br>\nRectifiers are better activation functions as they do not saturate the higher values of z. tanh and sigmoid value is close to 1 for large values of z and gradient is close to 0. Causes problem in learning: Vanishing Gradient <br>\n\n\n## SLP Chapter 7.2 The XOR problem\n A single perceptron cannot solve the XOR problem as a perceptron is a linear classifier (does not have a non linear activation) while XOR is not linearly seperable. <br>\nNeural networks: With different layers, the hidden layers represent a form of the input. \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Softmax function: To normalize output of a network to probability distribution. <br>\nWe use non-linear activation functions for each layer in a neural network is that if we did not, the resulting network is exactly equivalent to a single-layer network. \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n The idea of using word2vec or GloVe embeddings as our input representation and more generally the idea of relying on another algorithm to have already learned pretraining an embedding representation for our input words — is called pretraining \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Compared to n-gram models, neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction. On the other hand, neural net language models are much more complex, slower to train, and less interpretable than n-gram models, so for many (especially smaller) tasks an n-gram language model is still the right tool. \n\n## SLP Chapter 7.6 Training Neural Nets\n Various forms of regularization are used to prevent overfitting. One of the most dropout important is dropout: randomly dropping some units and their connections from the network during training <br>\n\n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 13 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" perplexity to measure the quality of a language model. The perplexity (PP) of a model on an unseen test set is the inverse probability that assigns to the test set, normalized by\n",
        "the test set length <br>\"\"\"\n",
        "slp_chapter9_2 = \"\"\" The hidden layer from the previous time step provides a form of memory, or\n",
        "context, that encodes earlier processing and informs the decisions to be made at\n",
        "later points in time. <br>\"\"\"\n",
        "\n",
        "slp_chapter9_3 = \"\"\"  RNNs don’t have the limited context problem that n-gram models have, since the hidden state can in principle represent information about all of the\n",
        "preceding words all the way back to the beginning of the sequence. <br>\n",
        "we always give the model the correct history sequence to predict the next word (rather than feeding the model its best case from the previousteacher forcing time step) is called teacher forcing.\n",
        "<br> The weights in the network are adjusted to minimize the average CE loss over\n",
        "the training sequence via gradient descent <br>\n",
        ". The rows of E represent the word embeddings for each word in the vocabulary learned during the training process with the goal that words that have similar meaning and function will have similar embeddings. And, since the length of these embeddings corresponds to\n",
        "the size of the hidden layer dh, the shape of the embedding matrix E is |V| ×dh. The final layer matrix V provides a way to score the likelihood of each word in the vocabulary given the evidence present in the final hidden layer of the network through the calculation of Vh. This entails that it also has the dimensionality |V| × dh. That is, the rows of V provide a second set of learned word embeddings\n",
        "that capture relevant aspects of word meaning and function. This leads to an obWeight tying vious question – is it even necessary to have both? Weight tying is a method that dispenses with this redundancy and uses a single set of embeddings at the input and softmax layers. That is, E = V. To do this, we set the dimensionality of the final hidden layer to be the same dh, (or add an additional projection layer to do the same thing), and simply use the same matrix for both layers. In addition to providing improved perplexity results, this approach significantly reduces the number of\n",
        "parameters required for the model.\n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Distributional Hypothesis: Words that occur in similar contexts tend to have similar meanings - Similarity in how words are distributed and similarity in what they mean <br>\n",
        "Lemma or citation form: , wordforms: different forms of the same lemma, word sense: different meanings of the same word <br>\n",
        "principle of contrast: t a difference in linguistic form is always associated with some difference in meaning. <br>\n",
        "Relatedness or association: belong to the same semantic field<br>\n",
        "\"\"\"\n",
        "slp_chapter6_3 = \"\"\" Information retrieval (IR) is the task of finding the document d from the D\n",
        "documents in some collection that best matches a query q. <br>\n",
        "1. Document Dimension: Each word is a row vector <br>\n",
        " \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Word2Vec: Learn the probability of a word occuring close to another word, self-supervision, avoids the need for\n",
        "any sort of hand-labeled supervision signal <br>\n",
        "Easier to do as this is not about predicting the next word. <br>\n",
        "Word2vec simplifies the architecture (training a logistic regression classifier instead of a multi-layer neural network with hidden layers). <br>\n",
        "Skip-gram:\n",
        "1. Treat the target word and a neighboring context word as positive examples.<br>\n",
        "2. Randomly sample other words in the lexicon to get negative samples.<br>\n",
        "3. Use logistic regression to train a classifier to distinguish those two cases.<br>\n",
        "4. Use the learned weights as the embeddings.<br>\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 130,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1297
        },
        "outputId": "c0ca654f-3757-4b7e-a7af-f566211334e0"
      },
      "source": [
        " #@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 128,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n perplexity to measure the quality of a language model. The perplexity (PP) of a model on an unseen test set is the inverse probability that assigns to the test set, normalized by\nthe test set length <br>\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n The hidden layer from the previous time step provides a form of memory, or\ncontext, that encodes earlier processing and informs the decisions to be made at\nlater points in time. <br>\n \n\n## SLP Chapter 9.3 RNNs as Language Models\n  RNNs don’t have the limited context problem that n-gram models have, since the hidden state can in principle represent information about all of the\npreceding words all the way back to the beginning of the sequence. <br>\nwe always give the model the correct history sequence to predict the next word (rather than feeding the model its best case from the previousteacher forcing time step) is called teacher forcing.\n<br> The weights in the network are adjusted to minimize the average CE loss over\nthe training sequence via gradient descent <br>\n. The rows of E represent the word embeddings for each word in the vocabulary learned during the training process with the goal that words that have similar meaning and function will have similar embeddings. And, since the length of these embeddings corresponds to\nthe size of the hidden layer dh, the shape of the embedding matrix E is |V| ×dh. The final layer matrix V provides a way to score the likelihood of each word in the vocabulary given the evidence present in the final hidden layer of the network through the calculation of Vh. This entails that it also has the dimensionality |V| × dh. That is, the rows of V provide a second set of learned word embeddings\nthat capture relevant aspects of word meaning and function. This leads to an obWeight tying vious question – is it even necessary to have both? Weight tying is a method that dispenses with this redundancy and uses a single set of embeddings at the input and softmax layers. That is, E = V. To do this, we set the dimensionality of the final hidden layer to be the same dh, (or add an additional projection layer to do the same thing), and simply use the same matrix for both layers. In addition to providing improved perplexity results, this approach significantly reduces the number of\nparameters required for the model.\n\n\n## SLP Chapter 6.1 Lexical Semantics\n Distributional Hypothesis: Words that occur in similar contexts tend to have similar meanings - Similarity in how words are distributed and similarity in what they mean <br>\nLemma or citation form: , wordforms: different forms of the same lemma, word sense: different meanings of the same word <br>\nprinciple of contrast: t a difference in linguistic form is always associated with some difference in meaning. <br>\nRelatedness or association: belong to the same semantic field<br>\n\n\n## SLP Chapter 6.3 Words and Vectors\n Information retrieval (IR) is the task of finding the document d from the D\ndocuments in some collection that best matches a query q. <br>\n1. Document Dimension: Each word is a row vector <br>\n \n\n## SLP Chapter 6.8 Word2vec\n Word2Vec: Learn the probability of a word occuring close to another word, self-supervision, avoids the need for\nany sort of hand-labeled supervision signal <br>\nEasier to do as this is not about predicting the next word. <br>\nWord2vec simplifies the architecture (training a logistic regression classifier instead of a multi-layer neural network with hidden layers). <br>\nSkip-gram:\n1. Treat the target word and a neighboring context word as positive examples.<br>\n2. Randomly sample other words in the lexicon to get negative samples.<br>\n3. Use logistic regression to train a classifier to distinguish those two cases.<br>\n4. Use the learned weights as the embeddings.<br>\n\n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 8422
        },
        "id": "T3cm1WV542_a",
        "outputId": "ac0f212e-e87a-488b-a256-48dd1ab0f2ef"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 131,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 17.1 Sequential Decision Problems\n Markovian Transition: The probability of reaching s′ from s depends only on s and not on the history of earlier states.\nA sequential decision problem: 1. fully observable, stochastic environment 2. Markovian transition model 3. additive rewards - Markov decision process:\n1. a set of states (with an initial state s0);<br>\n2. a set ACTIONS(s) of actions in each state; <br>\n3. a transition model P(s′|s,a); <br>\n4. and a reward function R(s,a,s′). <br>\nOptimal policy: A policy that yields the highest expected utility. <br>\nFinite horizon: an optimal action in a given state may depend on how much time is left. <br>\nNonstationary Policy: Depends on the time is called nonstationary.<br>\nInfinite Horizon: With no fixed time limit, on the other hand, there is no reason to behave differently in the same state at different times. <br>\nAn optimal action in this case depends only on the current state and the optimal policy is stationary.<br>\nThere can be finite state sequences in an infinite-horizon MDP that contains a terminal state. <br>\nUtility function: additive discounted rewards <br>\nWhy discount? Sooner rewards probably do have higher utility than later rewards, Also helps our algorithms converge <br>\nDifficulty with infinite horizon f the environment does not contain a terminal state, or if the agent never reaches one, then all environment histories will be infinitely long, and utilities with additive undiscounted rewards will generally be infinite. <br>\nSolutions:<br>\nDiscounted utilities with infinite horizons: the optimal policy is independent of the starting state. <br>\naction-utility function[Q-function: Q(s,a)]: Expected utility of taking a given action in a given state. <br>\n\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n \n1. Value Iteration: <br>\nIf we have n states, then we have n Bellman equations that need to be updated in every iteration. <br>\nStart with arbitrary values, and then update the equation to get updated utilities/values for each state. <br>\nProblem 1: It’s slow – O(S2A) per iteration <br>\nProblem 2: The “max” at each state rarely changes <br>\nProblem 3: The policy often converges long before the values<br>\nTheorem: will converge to unique optimal values- Basic idea: approximations get refined towards optimal values, Policy may converge long before values do <br>\n2. Policy Iteration: <br> Evaluate the policy Calculate Utility {U_i} of the states under given policy [max of q values] <br>Improve the policy: Calculate a new policy  using one-step look-ahead based on {U_i}  <br>\n3. Online Learning: expectimax search <br> \n\n## Module 8 Lecture Notes\n Your summary here \n\n## Module 8 Definitions\n* **Utility function** - Your definition here\n* **Expected utility** - your definition here\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Your summary here \n\n## AIMA Chapter 22.1 Learning from Rewards\n RL differs from MDPs as agents are in the MDPs and do not know the reward or the transition function and act in order to learn more <br>\n1. Model based RL: Use the transition model of the environment to learn the rewards and to make decisions on what actions to take. e.g. in Chess, we know what actions can be taken at a given state (even though we don't know which action is the best) <br>\nSystems often learn a utility function n terms of the sum of rewards <br>\n2. Model free RL: The agent does not know, nor learns the transition model <br>\n  2.1 The agent learns the action-utility mapping or quality function that is a function of the sum of rewards if an action is taken at a state s <br>\n  2.2 Policy search: The agent learns a policy which maps to an action for a state (Reflex Agent) <br>\nPassive RL: Policy is fixed, agent learns utility of the states under that policy (need not learn the transition model) <br>\nActive RL: Agent tries actions to explore the environment (figure out what to do) <br>\nApprenticeship learning: Learn using demonstrations instead of rewards <br>\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Passive learning agent learns the utility function (expected sum of rewards for a given state) under a fixed policy. This is different than the policy evaluation (in MDP) as in this case the agent does now know the transition and the reward functions of the environments. <br>\n1. Direct utility estimation (DUE) is that the utility of a state is defined as the expected total reward from that state onward (called the expected reward-to-go), and that each trial provides a sample of this quantity for each state visited. <br>\nDUE ignores that the utility of a state is determined by the reward and the expected utility of the successor states. In this sense it misses chances to learn and converges slowly. <br>\nIt’s easy to understand,It doesn’t require any knowledge of T, R,It eventually computes the correct average values, using just sample transitions <br>\nWhat bad about it? It wastes information about state connections Each state must be learned separately So, it takes a long time to learn <br>\n2. Adaptive Dynamic Programming: Takes advantage of the constraints among the utilities of states by learning the transition model that connects them and solving the corresponding Markov decision process using dynamic programming. <br>\nmodified policy iteration: Update the utility function based on change in the learned model, and because the model only slightly changes after every observation, this is simplified. <br>\nIn terms of how quickly its value estimates improve, the ADP agent is limited only by its ability to learn the transition model.<br>\n3. Temporal-difference learning: Use the observed transitions to adjust the utilities of the observed states so that they agree with the constraint equations. <br>\nTD does not need a transition model to perform its updates. <br> TD value leaning is a model-free way to do policy evaluation, mimicking\nBellman updates with running sample averages\nHowever, if we want to turn values into a (new) policy, we’re sunk <br>\nTD adjusts a state to agree with its observed successor, whereas ADP adjusts the state to agree with all of the successors that might occur, weighted by their probabilities. \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Absorbing State: Where no actions have any effect and no rewards are received. <br>\nBayesian reinforcement learning: assumes a prior probability P(h) over hypotheses h about what the true model is; the posterior probability P(h | e) is obtained in the usual way by Bayes’ rule given the observations to date. <br>\nTemporal-difference Q-learning: Q-learning converges to optimal policy -- even if you’re acting suboptimally! This is called off-policy learning\nHave to explore enough, have to eventually make the learning rate small enough but not decrease it too quickly\nBasically, in the limit, it doesn’t matter how you select actions <br>\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n In real world, can keep track of all Q values (too many states) -> instead learn for a small number of states through experience and generalize the experience to new similar states. <br>\n \n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **Term** - your definition here\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\nObserved variables (evidence): Agent knows certain things about the state of the world (e.g.,sensor readings or symptoms) <br>\nUnobserved variables (states): Agent needs to reason about other aspects (e.g. where an object is or what disease is present)\nModel: Agent knows something about how the known variables relate to the unknown variables <br>\nThe fundamental idea of decision theory is that an agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all the possible outcomes of the action. This is called the principle of maximum expected utility (MEU). <br>\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n y, the possible worlds are called the sample space, and they mutually exclusive and exhaustive. <br>\n \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Summing up a row: marginalization, or summing out—because we sum up the probabil- ities for each possible value of the other variables, thereby taking them out of the equation. <br>\nMarginal distributions are sub-tables which eliminate variables, Combine collapsed rows by adding <br>\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n The naive Bayes model assumes the conditional independence of all effect variables, given a single cause variable; its size grows linearly with the number of effects.\n \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n A wumpus-world agent can calculate probabilities for unobserved aspects of the world, thereby improving on the decisions of a purely logical agent. Conditional independence makes these calculations tractable. \n\n## SLP Chapter 3.1 N-Grams\n Unigram: Use no words to estimate the next words <br>\nBigram: one word <br>\nAn intuitive way to estimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood estimation the MLE estimate for the parameters of an n-gram model by getting counts from a\ncorpus, and normalizing the counts so that they lie between 0 and 1. <br>\nBy dividing the observed frequency of a particular sequence by the observed frequency of a prefix. This ratio is called a relative frequency <br>\n\n\n## SLP Chapter 3.2 Evaluating Language Models\n Extrinsic evaluation: The best way to evaluate the performance of a language model is to embed it in an application and measure how much the application improves. <br>\nAn intrinsic evaluation metric is one that mea- intrinsic evaluation sures the quality of a model independent of any application. <br>\nTraining on the test set introduces a bias that makes the probabilities all look too high, and causes huge inaccuracies in perplexity <br>\nPerplexity: weighted average branching factor of a language <br>\n \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Sampling from a distribution means to choose random points according to their likelihood. Thus sampling from a language model—which represents a distribution over sentences—means to generate some sentences, choosing\neach sentence according to its likelihood as defined by the model. Thus we are more likely to generate sentences that the model thinks have a high probability and less likely to generate sentences that the model thinks have a low probability. <br>\n\n\n## SLP Chapter 3.4 Generalization and Zeros\n  Statistical models are likely to be pretty useless as predictors if the training sets and the test sets are way different <br>\nSolution: To be sure to use a training corpus that has a similar genre and daialect to whatever task we are trying to accomplish. <br>\nOur models may still be subject to the problem of sparsity. <br>\nzero probability n-grams: By definition, perplexity is based on the inverse probability of thetest set. Thus if some words have zero probability, we can’t compute perplexity at\nall, since we can’t divide by 0 <br>\nUnknown words, or out of vocabulary (OOV) words:  by adding a pseudo-word called <UNK><br>\n1. This can be handled by making the vocab closed and treating UNK word as any other word. <br>\n2. When we don’t have a prior vocabulary in advance, is to create such a vocabulary implicitly, replacing words in the training data by <UNK> based on their frequency. For example we can replace by <UNK> all words that occur fewer than n times in the training set, where n is some small number, or equivalently select a vocabulary size V in advance (say 50,000) and choose the top V words by frequency and replace the rest by UNK. <br>\n\n\n## SLP Chapter 3.5 Smoothing\n Problem: words that are in our vocabulary (they are not unknown words) but appear in a test set in an unseen context <br>\n1. Laplace Smoothing: add one to all the n-gram counts, before we normalize them into probabilities.<br>\nDiscounting: discounting (lowering) some non-zero counts in order to get the probability mass that will be assigned to the zero counts. <br>\n2.  Add-k smoothing: move a bit less of the probability mass from the seen to the unseen events. Instead of adding 1 to each count, we add a fracadd-k tional count k <br>\n3. Backoff and Interpolation:\nBackoff: we use the trigram if the evidence is sufficient, otherwise we use the bigram, otherwise the unigram. <br>\nInterpolation: we always mix the probability estimates from all the n-gram estimators, weighing and combining the trigram, bigram, and unigram counts. <br>\nLearning weights for interpolation:  λs are learned from a held-out corpus. A held-out corpus is an additional training corpus that we use to set hyperparameters <br>\nBackoff with discounting is also called Katz backoff: \n\n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Stupid backoff gives up the idea of trying to make the language\nmodel a true probability distribution. There is no discounting of the higher-order\nprobabilities. If a higher-order n-gram has a zero count, we simply backoff to a\nlower order n-gram, weighed by a fixed (context-independent) weight. \n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Perplexity** - of a language model on a test set is the inverse probability of the test set, normalized by the number of words\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## Module 11 Definitions\n* **Concept** - definition\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 19.1 Forms of Learning\n transfer learning, in which knowl- edge from one domain is transferred to a new domain, so that learning can proceed faster with less data. We do assume, however, that the designer of the system chooses a model framework that can lead to effective learning.<br>\nGoing from a specific set of observations to a general rule is called induction; from the observations that the sun rose every day in the past, we induce that the sun will come up tomorrow. This differs from the deduction because the inductive conclusions may be incorrect, whereas deductive conclusions are guaranteed to be correct if the premises are correct. <br>\n \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Naive Bayes Classification: <br>\nAssumptions: \n1. Bag of words: context does not matter <br>\n2. Naive Bayes Assumption: All features/words are independent of the class. <br>\nDone in log space to avoid underflow and increase speed <br>\nLog Space: computes the predicted class as a linear function of input features. Classifiers that use a linear combination of the inputs\nto make a classification decision —like naive Bayes and also logistic regression—\nare called linear classifiers. <br>\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n P(c) = count of documents of category C/ Total number of documents <br>\nP(w_i|c) = Count of occurence of w_i in documents from category c / total word count in category c documents <br>\nStop Words: Top 100 occuring words can be removed from both train and test set, hasn't proved useful in improving accuracy <br>\nUnknown words: Remove from test set <br>\n \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\nFor sentiment analysis, occurrence of the word matters and not the frequency, therefore clip the count to 1 - binary naive Bayes <br>\nStep1: For each document when creating bag of words, clip down the frequency to 1. Count multiple if occurs in multiple documents <br>\nStep2: Handling negation - Add prefix NOT_ to all tokens <br>\nIf the training data is in sufficient:  derive the positive and negative word features using sentiment lexicons. Add new features like feature that is counted whenever a word from that lexicon occurs. Thus we might add a feature called ‘this word occurs in the positive lexicon’, and treat all instances of words in the lexicon as counts for that one feature, instead of counting each word separately <br>\n\n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Spam Detection: rather than using all the words as individual features, is to predefine likely sets of words or phrases as features, combined with features that are not purely linguistic.<br>\nLanguage ID: Use character n-grams, 2-grams (‘zw’) 3-grams (‘nya’, ‘ Vo’), or 4-grams (‘ie z’,‘thei’), or, even simpler byte n-grams<br>\n\n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n Your summary here \n\n## Module 12 Definitions\n* **Concept** - definition\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\nLogistic regression has a number of advantages over naive\nBayes. Naive Bayes has overly strong conditional independence assumptions. Consider two features which are strongly correlated; in fact, imagine that we just add the\nsame feature f1 twice. Naive Bayes will treat both copies of f1 as if they were separate, multiplying them both in, overestimating the evidence. By contrast, logistic\nregression is much more robust to correlated features; if two features f1 and f2 are perfectly correlated, regression will simply assign part of the weight to w1 and part to w2. Thus when there are many correlated features, logistic regression will assign a more accurate probability than naive Bayes. So logistic regression generally works better on larger documents or datasets and is a common default.\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n This is called conditional maximum likelihood estimation: we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x. The resulting loss\nfunction is the negative log likelihood loss, generally called the cross-entropy loss <br>\n\n\n## SLP Chapter 5.4 Gradient Descent\n For logistic regression, this loss function is conveniently convex. A convex function has just one minimum; there are no local minima to get stuck in, so gradient\ndescent starting from any point is guaranteed to find the minimum. (By contrast, the loss for multi-layer neural networks is non-convex, and gradient descent may get stuck in local minima for neural network training and never find the global optimum.) <br>\n\n## SLP Chapter 5.5 Regularization\n For example in batch training we compute the gradient over the entire dataset. By seeing so many examples, batch training offers a superb estimate of which direction to move the weights, at the cost of spending a lot of time processing every single example in the training set to compute this perfect direction.\nmini-batch A compromise is mini-batch training: we train on a group of m examples \n\n## SLP Chapter 7.1 Units\n Neural nets: better than logistic regression as do not require feature handling and learn features on its own <br>\nActivation Functions (make nn non linear): \n1. Sigmoid - maps to [0,1] and is differentiable <br>\n2. tanh: better than sigmoid <br>\n3. ReLU: max(0,z) most commonly used <br>\nRectifiers are better activation functions as they do not saturate the higher values of z. tanh and sigmoid value is close to 1 for large values of z and gradient is close to 0. Causes problem in learning: Vanishing Gradient <br>\n\n\n## SLP Chapter 7.2 The XOR problem\n A single perceptron cannot solve the XOR problem as a perceptron is a linear classifier (does not have a non linear activation) while XOR is not linearly seperable. <br>\nNeural networks: With different layers, the hidden layers represent a form of the input. \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Softmax function: To normalize output of a network to probability distribution. <br>\nWe use non-linear activation functions for each layer in a neural network is that if we did not, the resulting network is exactly equivalent to a single-layer network. \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n The idea of using word2vec or GloVe embeddings as our input representation and more generally the idea of relying on another algorithm to have already learned pretraining an embedding representation for our input words — is called pretraining \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Compared to n-gram models, neural language models can handle much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction. On the other hand, neural net language models are much more complex, slower to train, and less interpretable than n-gram models, so for many (especially smaller) tasks an n-gram language model is still the right tool. \n\n## SLP Chapter 7.6 Training Neural Nets\n Various forms of regularization are used to prevent overfitting. One of the most dropout important is dropout: randomly dropping some units and their connections from the network during training <br>\n\n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 13 Definitions\n* **Concept** - definition\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n perplexity to measure the quality of a language model. The perplexity (PP) of a model on an unseen test set is the inverse probability that assigns to the test set, normalized by\nthe test set length <br>\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n The hidden layer from the previous time step provides a form of memory, or\ncontext, that encodes earlier processing and informs the decisions to be made at\nlater points in time. <br>\n \n\n## SLP Chapter 9.3 RNNs as Language Models\n  RNNs don’t have the limited context problem that n-gram models have, since the hidden state can in principle represent information about all of the\npreceding words all the way back to the beginning of the sequence. <br>\nwe always give the model the correct history sequence to predict the next word (rather than feeding the model its best case from the previousteacher forcing time step) is called teacher forcing.\n<br> The weights in the network are adjusted to minimize the average CE loss over\nthe training sequence via gradient descent <br>\n. The rows of E represent the word embeddings for each word in the vocabulary learned during the training process with the goal that words that have similar meaning and function will have similar embeddings. And, since the length of these embeddings corresponds to\nthe size of the hidden layer dh, the shape of the embedding matrix E is |V| ×dh. The final layer matrix V provides a way to score the likelihood of each word in the vocabulary given the evidence present in the final hidden layer of the network through the calculation of Vh. This entails that it also has the dimensionality |V| × dh. That is, the rows of V provide a second set of learned word embeddings\nthat capture relevant aspects of word meaning and function. This leads to an obWeight tying vious question – is it even necessary to have both? Weight tying is a method that dispenses with this redundancy and uses a single set of embeddings at the input and softmax layers. That is, E = V. To do this, we set the dimensionality of the final hidden layer to be the same dh, (or add an additional projection layer to do the same thing), and simply use the same matrix for both layers. In addition to providing improved perplexity results, this approach significantly reduces the number of\nparameters required for the model.\n\n\n## SLP Chapter 6.1 Lexical Semantics\n Distributional Hypothesis: Words that occur in similar contexts tend to have similar meanings - Similarity in how words are distributed and similarity in what they mean <br>\nLemma or citation form: , wordforms: different forms of the same lemma, word sense: different meanings of the same word <br>\nprinciple of contrast: t a difference in linguistic form is always associated with some difference in meaning. <br>\nRelatedness or association: belong to the same semantic field<br>\n\n\n## SLP Chapter 6.3 Words and Vectors\n Information retrieval (IR) is the task of finding the document d from the D\ndocuments in some collection that best matches a query q. <br>\n1. Document Dimension: Each word is a row vector <br>\n \n\n## SLP Chapter 6.8 Word2vec\n Word2Vec: Learn the probability of a word occuring close to another word, self-supervision, avoids the need for\nany sort of hand-labeled supervision signal <br>\nEasier to do as this is not about predicting the next word. <br>\nWord2vec simplifies the architecture (training a logistic regression classifier instead of a multi-layer neural network with hidden layers). <br>\nSkip-gram:\n1. Treat the target word and a neighboring context word as positive examples.<br>\n2. Randomly sample other words in the lexicon to get negative samples.<br>\n3. Use logistic regression to train a classifier to distinguish those two cases.<br>\n4. Use the learned weights as the embeddings.<br>\n\n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}