{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9ebc63bf-62da-48a0-8048-5b5b14dc52dc"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "79fa5424-598b-4b4b-8385-7c7ec4c0d998"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = True"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" \n",
        "- MEU\n",
        "  - action = argmax_（下面是a）EU(a)\n",
        "\"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\"\n",
        "- notation that describe an agent's preferences\n",
        "  - A > B (>是波浪形的) the agent prefers A over B\n",
        "  - A ~ B the agent is indifferent between A and B\n",
        "  - A >~ B (> 是波浪形的，~在下面) the agent prefers A over B or is indifferent between\n",
        "    - Orderability\n",
        "      - Given any two lotteries, a rational agent must either prefer one or else rate them as equally preferable. That is, the agent cannot avoid deciding\n",
        "    - Transitivity\n",
        "      - Given any three lotteries, if an agent prefers to and prefers to , then the agent must prefer to \n",
        "    - CONTINUITY\n",
        "      - If some lottery is between and in preference, then there is some probability for which the rational agent will be indifferent between getting for sure and the lottery that yields with probability and with probability .\n",
        "\n",
        "\"\"\"\n",
        "chapter16_3 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter17_2 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\"\n",
        "- MDP offline solution\n",
        "- RL online learning\n",
        "- Value Iteration\n",
        "  - Propagating information through the state space by means of local updates\n",
        "  - The Bellman equation is the basis of the value iteration algorithm for solving MDPs.\n",
        "  - n possible states means n possible bellman equations, 1 for each state. \n",
        "  - n equations conatins n unknowns, the state utilities. \n",
        "  - Equations are non-linear due to max. So cannot solved by linear algebra.\n",
        "  - We start with arbitrary initial values for the utilities, calculate the right-hand side of the equation, and plug it into the left-hand side—thereby updating the utility of each state from the utilities of its neighbors. We repeat this until we reach an equilibrium.\n",
        "  - Ui(s) is utility value for state s at ith iteration. It is Bellman Update.\n",
        "  - The update is assumed to be applied simultaneously to all the states at each iteration.\n",
        "  - If applied infinitely, guaranteed to reach equilibrium. \n",
        "  - Contraction. A contraction has only one fixed point; if there were two fixed points they would not get closer together when the function was applied, so it would not be a contraction. When the function is applied to any argument, the value must get closer to the fixed point, so repeated application reaches fixed point in the limit.\n",
        "  - Measure distances between utility vectors by max norm.\n",
        "  - Bellman update function beocmes Ui+1 = BUi with Bellman update operator B applied to all states.\n",
        "  - Can use contraction property to analyze rate of convergence. \n",
        "  - Terminal condition is |Ui+1 - U| < e\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"assigns a single number to express the desirability of a state\",\n",
        "\"expected utility of an action given the evidence EU(a)\" : \"the average utility value of the outcomes, weighted by the probability that the outcome occurs\",\n",
        "\"maximum expected utility (MEU)\": \"a rational agent should choose the action that maximizes the agent’s expected utility\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "47dd1ead-0a76-4eac-e7d7-977dc1afaa0e"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n \n- MEU\n  - action = argmax_（下面是a）EU(a)\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\n- notation that describe an agent's preferences\n  - A > B (>是波浪形的) the agent prefers A over B\n  - A ~ B the agent is indifferent between A and B\n  - A >~ B (> 是波浪形的，~在下面) the agent prefers A over B or is indifferent between\n    - Orderability\n      - Given any two lotteries, a rational agent must either prefer one or else rate them as equally preferable. That is, the agent cannot avoid deciding\n    - Transitivity\n      - Given any three lotteries, if an agent prefers to and prefers to , then the agent must prefer to \n    - CONTINUITY\n      - If some lottery is between and in preference, then there is some probability for which the rational agent will be indifferent between getting for sure and the lottery that yields with probability and with probability .\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n Your summary here \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n Your summary here \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Your summary here \n\n## Module 8 Lecture Notes\n\n- MDP offline solution\n- RL online learning\n- Value Iteration\n  - Propagating information through the state space by means of local updates\n  - The Bellman equation is the basis of the value iteration algorithm for solving MDPs.\n  - n possible states means n possible bellman equations, 1 for each state. \n  - n equations conatins n unknowns, the state utilities. \n  - Equations are non-linear due to max. So cannot solved by linear algebra.\n  - We start with arbitrary initial values for the utilities, calculate the right-hand side of the equation, and plug it into the left-hand side—thereby updating the utility of each state from the utilities of its neighbors. We repeat this until we reach an equilibrium.\n  - Ui(s) is utility value for state s at ith iteration. It is Bellman Update.\n  - The update is assumed to be applied simultaneously to all the states at each iteration.\n  - If applied infinitely, guaranteed to reach equilibrium. \n  - Contraction. A contraction has only one fixed point; if there were two fixed points they would not get closer together when the function was applied, so it would not be a contraction. When the function is applied to any argument, the value must get closer to the fixed point, so repeated application reaches fixed point in the limit.\n  - Measure distances between utility vectors by max norm.\n  - Bellman update function beocmes Ui+1 = BUi with Bellman update operator B applied to all states.\n  - Can use contraction property to analyze rate of convergence. \n  - Terminal condition is |Ui+1 - U| < e\n\n\n## Module 8 Definitions\n* **Utility function** - assigns a single number to express the desirability of a state\n* **Expected utility of an action given the evidence eu(a)** - the average utility value of the outcomes, weighted by the probability that the outcome occurs\n* **Maximum expected utility (meu)** - a rational agent should choose the action that maximizes the agent’s expected utility\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\"\n",
        "- Q-learning, \n",
        "  - where the agent learns a Q-function, or quality-function, , denoting the sum of rewards from state onward if action is taken. \n",
        "\"\"\"\n",
        "chapter22_2 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_3 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_4 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_5 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"term\" : \"your definition here\",\n",
        "}"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 568
        },
        "cellView": "form",
        "outputId": "a3ecba7a-42fa-4845-ca64-b162ee0c79d7"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Your summary here \n\n## AIMA Chapter 22.1 Learning from Rewards\n\n- Q-learning, \n  - where the agent learns a Q-function, or quality-function, , denoting the sum of rewards from state onward if action is taken. \n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **Term** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_6 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"term\" : \"your definition here\",\n",
        "}"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "7b20db3c-a752-4e85-8688-b551aee094a4"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Your summary here \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Term** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter13_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter13_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 411
        },
        "cellView": "form",
        "outputId": "3c8c0ab3-1283-49b5-ffb9-7937fd824a64"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Your summary here \n\n## Module 11 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\"\n",
        "- bias\n",
        "  - bias we mean (loosely) the tendency of a predictive hypothesis to deviate from the expected value when averaged over different training sets. \n",
        "- variance\n",
        "  - we mean the amount of change in the hypothesis due to fluctuation in the training data\n",
        "- \n",
        "\"\"\"\n",
        "chapter19_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_6 = \"\"\" \n",
        "- decision boundary\n",
        "  - is a line (or a surface, in higher dimensions) that separates the two classes. \n",
        "  - A linear decision boundary is called a linear separator and data that admit such a separator are called linearly separable. \n",
        "- perceptron learning rule\n",
        "  - If the output is correct (i.e. y = hw(x) ) then the weights are not changed.\n",
        "  - If y is 1 but hw(x) is 0, then wi is increased when the corresponding input xi is positive and decreased when xi is negative. This makes sense, because we want to make\n",
        "  sure w*x bigger so that hw(x) outputs a 1.\n",
        "  - If y is 0 but hw(x) is 1, then wi is decreased when the corresponding input xi is positive and increased when xi is negative. This makes sense, because we want to make \n",
        "  w*x smaller so that hw(x) outputs a 0.\n",
        "\"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\"\n",
        "- Naive Bayes is a probabilistic classifier, meaning that for a document d, out of all classes c ∈ C the classifier returns \n",
        "the class cˆ which has the maximum posterior ˆ probability given the document. we use the hat notation ˆ to mean “our\n",
        "estimate of the correct class”\n",
        "    - c^ = argmaxP(c|d) c ∈ C\n",
        "- Bayesian inference\n",
        "  - This idea of Bayesian inference was first applied to text classification.\n",
        "   The intuition of Bayesian classification is to use Bayes’ rule to transform into other probabilities that have some useful properties.\n",
        "   it gives us a way to break down any conditional probability P(x|y) into three other probabilities:\n",
        "  - P(x|y) = P(y|x)P(x)/ P(y)\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter4_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_7 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_9 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_10 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\" \n",
        "- Supervised learning is a machine learning approach that’s defined by its use of labeled datasets. These datasets are designed to train or “supervise” algorithms into classifying data or predicting outcomes accurately\n",
        "  - Classification problems use an algorithm to accurately assign test data into specific categories, such as separating apples from oranges. Or, in the real world, supervised learning algorithms can be used to classify spam in a separate folder from your inbox. Linear classifiers, support vector machines, decision trees and random forest are all common types of classification algorithms. \n",
        "  - Regression is another type of supervised learning method that uses an algorithm to understand the relationship between dependent and independent variables. Regression models are helpful for predicting numerical values based on different data points, such as sales revenue projections for a given business. Some popular regression algorithms are linear regression, logistic regression and polynomial regression.\n",
        "- Unsupervised learning uses machine learning algorithms to analyze and cluster unlabeled data sets. These algorithms discover hidden patterns in data without the need for human intervention (hence, they are “unsupervised”).\n",
        "  - Clustering is a data mining technique for grouping unlabeled data based on their similarities or differences. For example, K-means clustering algorithms assign similar data points into groups, where the K value represents the size of the grouping and granularity. This technique is helpful for market segmentation, image compression, etc.\n",
        "  - Association is another type of unsupervised learning method that uses different rules to find relationships between variables in a given dataset. These methods are frequently used for market basket analysis and recommendation engines, along the lines of “Customers Who Bought This Item Also Bought” recommendations.\n",
        "  - Dimensionality reduction is a learning technique used when the number of features  (or dimensions) in a given dataset is too high. It reduces the number of data inputs to a manageable size while also preserving the data integrity. Often, this technique is used in the preprocessing data stage, such as when autoencoders remove noise from visual data to improve picture quality.\n",
        "- Reinforcement Learning\n",
        "  - the agent learns from a series of reinforcements: rewards (like win in a game) and punishments (like lost)\n",
        "  - up to the agent to decide which of the actions prior to the reinforcement were most responsible for it, and to alter its actions to aim towards more rewards in the future.\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "8c91968d-b75f-4fb6-ea56-574c53395c5d"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\n- bias\n  - bias we mean (loosely) the tendency of a predictive hypothesis to deviate from the expected value when averaged over different training sets. \n- variance\n  - we mean the amount of change in the hypothesis due to fluctuation in the training data\n- \n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n \n- decision boundary\n  - is a line (or a surface, in higher dimensions) that separates the two classes. \n  - A linear decision boundary is called a linear separator and data that admit such a separator are called linearly separable. \n- perceptron learning rule\n  - If the output is correct (i.e. y = hw(x) ) then the weights are not changed.\n  - If y is 1 but hw(x) is 0, then wi is increased when the corresponding input xi is positive and decreased when xi is negative. This makes sense, because we want to make\n  sure w*x bigger so that hw(x) outputs a 1.\n  - If y is 0 but hw(x) is 1, then wi is decreased when the corresponding input xi is positive and increased when xi is negative. This makes sense, because we want to make \n  w*x smaller so that hw(x) outputs a 0.\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\n- Naive Bayes is a probabilistic classifier, meaning that for a document d, out of all classes c ∈ C the classifier returns \nthe class cˆ which has the maximum posterior ˆ probability given the document. we use the hat notation ˆ to mean “our\nestimate of the correct class”\n    - c^ = argmaxP(c|d) c ∈ C\n- Bayesian inference\n  - This idea of Bayesian inference was first applied to text classification.\n   The intuition of Bayesian classification is to use Bayes’ rule to transform into other probabilities that have some useful properties.\n   it gives us a way to break down any conditional probability P(x|y) into three other probabilities:\n  - P(x|y) = P(y|x)P(x)/ P(y)\n\n\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n \n- Supervised learning is a machine learning approach that’s defined by its use of labeled datasets. These datasets are designed to train or “supervise” algorithms into classifying data or predicting outcomes accurately\n  - Classification problems use an algorithm to accurately assign test data into specific categories, such as separating apples from oranges. Or, in the real world, supervised learning algorithms can be used to classify spam in a separate folder from your inbox. Linear classifiers, support vector machines, decision trees and random forest are all common types of classification algorithms. \n  - Regression is another type of supervised learning method that uses an algorithm to understand the relationship between dependent and independent variables. Regression models are helpful for predicting numerical values based on different data points, such as sales revenue projections for a given business. Some popular regression algorithms are linear regression, logistic regression and polynomial regression.\n- Unsupervised learning uses machine learning algorithms to analyze and cluster unlabeled data sets. These algorithms discover hidden patterns in data without the need for human intervention (hence, they are “unsupervised”).\n  - Clustering is a data mining technique for grouping unlabeled data based on their similarities or differences. For example, K-means clustering algorithms assign similar data points into groups, where the K value represents the size of the grouping and granularity. This technique is helpful for market segmentation, image compression, etc.\n  - Association is another type of unsupervised learning method that uses different rules to find relationships between variables in a given dataset. These methods are frequently used for market basket analysis and recommendation engines, along the lines of “Customers Who Bought This Item Also Bought” recommendations.\n  - Dimensionality reduction is a learning technique used when the number of features  (or dimensions) in a given dataset is too high. It reduces the number of data inputs to a manageable size while also preserving the data integrity. Often, this technique is used in the preprocessing data stage, such as when autoencoders remove noise from visual data to improve picture quality.\n- Reinforcement Learning\n  - the agent learns from a series of reinforcements: rewards (like win in a game) and punishments (like lost)\n  - up to the agent to decide which of the actions prior to the reinforcement were most responsible for it, and to alter its actions to aim towards more rewards in the future.\n\n\n\n\n## Module 12 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\"\n",
        "- generative model\n",
        "  - understanding what dogs look like\n",
        "and what cats look like. You might literally ask such\n",
        "a model to generate\n",
        "  - generative model like naive Bayes makes use of this likelihood term,\n",
        "-  discriminative model\n",
        "  - trying to learn to distinguish the classes \n",
        "  - attempts to directly compute P(c|d). \n",
        "- A machine learning system for classification has\n",
        "  - A feature representation of the input.\n",
        "  - A classification function that computes yˆ, the estimated class, via p(y|x). (sigmod, softmax)\n",
        "  - An objective function for learning, usually involving minimizing error on\n",
        "training examples (like cross-entropy loss function)\n",
        "  - An algorithm for optimizing the objective function. (like stochastic gradient descent algorithm)\n",
        "- Logistic regression has 2 phases\n",
        "  - training\n",
        "    - a vector of weights and a bias term (also called intercept)\n",
        "      - The weight wi represents how important that input feature is to the classification decision\n",
        "  - test\n",
        "- goal of bianry logistic regression\n",
        "  - train a classifier that can make a binary decision about the class of a new input observation\n",
        "- sigmod function (logistic function)\n",
        "  - named because it looks like an s\n",
        "  - σ(z)= 1 = 1 / 1+(e)^−z = 1 / 1 + exp(−z)\n",
        "    - where z = w·x+b, w is the weight vector, x is the input feature vector and b is the bias term\n",
        "  - takes a real-valued number and maps it into the range [0, 1], which is just what we want for a probability\n",
        "  - sigmod property, 1−σ(x) = σ(−x), so we could also have expressed P(y = 0) as σ (−(w · x + b)).\n",
        "- Advantage of logistic regression over Naive Bayes\n",
        "  - Naive Bayes has overly strong conditional independence assumptions\n",
        "  - logistic regression is much more robust to correlated features; if two features f1 and f2 are perfectly correlated, regression will simply assign part of the weight to w1 and part to w2.\n",
        "    - more correlated features\n",
        "      - logistic assigns more accurate probability\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\" \n",
        "- supervised classification\n",
        "- loss/cost function\n",
        "  - opposite of similarity\n",
        "    - a metric for how close the current label (yˆ) is to the true gold label y. \n",
        "  - we measure the distance between the system output and the gold output\n",
        "- cross-enropy loss\n",
        "  - commonly used for logistic regression and also for neural networks\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter5_3 = \"\"\"\n",
        "- loss function expresses\n",
        "  - for an observation x, how close the classifier output (yˆ = σ (w · x + b)) is to the correct output (y, which is 0 or 1)\n",
        "  - L(yˆ,y) = How much yˆ differs from the true y\n",
        "  - conditional maximum likelihood estimation\n",
        "    - we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x\n",
        "    - The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter5_4 = \"\"\"\n",
        "- convex\n",
        "  - A convex function has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum.\n",
        "- Stochastic gradient descent \n",
        "  - an online algorithm (process input example by example rather than waiting till the entire)\n",
        "  - minimizes the loss function by computing its gradient after each training example, and nudging θ in the right direction \n",
        "- Learning rate  η \n",
        "  - a hyperparameter that must be adjusted\n",
        "  - high, steps too large, overshoot the minimum of the loss function\n",
        "  - low, too small, too long to get to the minimum\n",
        "  - chosen by the algorithm designer to affect how algorithm works\n",
        "\"\"\"\n",
        "slp_chapter5_5 = \"\"\"  \n",
        "- regularization\n",
        "  - to avoid overfitting\n",
        "  - used to penalize large weights\n",
        "  - L2 regularization\n",
        "    - a quadratic function of the weight values, named because it uses the (square of the) L2 norm of the weight values. \n",
        "    - The L2 norm, ||θ||2, is the same as the Euclidean distance of the vector θ from the origin.\n",
        "    - θˆ = argmax 􏰺logP(y(i)|x(i)) −α􏰺θj2\n",
        "    - ridge regression\n",
        "    - is easier to optimize because of its simple derivative (the derivative of θ2 is just 2θ\n",
        "    - prefers weight vectors with many small weights\n",
        "    - corresponds to assuming that weights are distributed according to a Gaussian distribution with mean μ = 0.\n",
        "  - L1 regularization\n",
        "    - a linear function of the weight values\n",
        "    - named after the L1 norm ||W||1, the sum of the absolute values of the weights, or Manhattan distance\n",
        "    - lasso regression\n",
        "    - is more complex (the derivative of |θ | is non-continuous at zero).\n",
        "    - prefers sparse solutions with some larger weights but many more weights set to zero\n",
        "    - far fewer features\n",
        "    -  Laplace prior on the weights.\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter5_6 = \"\"\"\n",
        "-  multinomial logistic regression / softmax regression / maxent classifier\n",
        "  - more than 2 classes\n",
        "- features in multinomial logistic regression\n",
        "  - separate weight vectors (and biases) for each of the K classes. \n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\"\n",
        "- Neural Network\n",
        "  - a network of small computing units\n",
        "    - each of which takes a vector of input values and produces a single output value\n",
        "  - Instead of building rich kinds of feature templates like logistic regression\n",
        "    - take raw words as inputs and learn to induce features as part of the process of learning to classify.\n",
        "  -  The power of neural networks comes from the ability of early layers to learn representations that can be utilized by later layers in the network.\n",
        "- Deep learning\n",
        "  -  The use of modern neural nets because modern networks are often deep\n",
        "- Unit\n",
        "  - building block of a neural network - single computational unit\n",
        "  - takes a set of real valued numbers as input, performs some computation, and produces an output\n",
        "  - activation\n",
        "    - neural units apply a non-linear function f to z. We will refer to the output of this function as the activation value for the unit\n",
        "      - y=a= f(z)\n",
        "      - sigmod\n",
        "        - y = σ(w·x+b) = 1 / 1 + exp(-(w*x+b))\n",
        "        - very high values of z result in values of y that are saturated, i.e., extremely close to 1, and have derivatives very close to 0. Zero derivatives cause problems for learning (and for tanh)\n",
        "      - tanh\n",
        "        - a variant of the sigmoid that ranges from -1 to +1\n",
        "          - y = e^z - e^-z / e^z + e^−z\n",
        "          - being smoothly differentiable and mapping outlier values toward the mean. \n",
        "      - rectified linear unit ReLU \n",
        "        - simplest and most commonly used\n",
        "          - y = max(z, 0)\n",
        "          - result from it being very close to linear\n",
        "\"\"\"\n",
        "slp_chapter7_2 = \"\"\"\n",
        "- perceptron\n",
        "  - a simple neural unit\n",
        "    - has binary output\n",
        "    - does not have a non-linear activation function\n",
        "- XOR: not linearly separable\n",
        "- Non-linear activation functions\n",
        "  - a network formed by many layers of purely linear units can always be reduce to a single layer of linear units with appropriate weights\n",
        "\"\"\"\n",
        "slp_chapter7_3 = \"\"\" \n",
        "- feedforward network (multi-layer perceptrons (or MLPs);)\n",
        "  - a multilayer network in which the units are connected with no cycles;\n",
        "  - the outputs from units in each layer are passed to units in the next higher layer\n",
        "  - no outputs are passed back to lower layers\n",
        "  - but actually the unit in modern multilayer networks are not perceptrons\n",
        "    - perceptrons are purely linear\n",
        "    - modern networks are made up of units with non-linear functions\n",
        "  - 3 kinds of nodes\n",
        "    - input units\n",
        "      - The input layer x is a vector of simple scalar values\n",
        "    - hidden units\n",
        "      - The core of the neural network is the hidden layer h formed of hidden units hi,\n",
        "        - taking a weighted sum of its inputs and then applying a non-linearity. \n",
        "        - each layer is fully connected\n",
        "        - represents the parameters for the entire hidden layer by combining the weight vector and bias for each unit i into a single weight matrix W and a single bias vector b for the whole layer \n",
        "          - advantage of using a single matrix W for the weights of the entire layer\n",
        "            - the hidden layer computation for a feedforward network can be done very efficiently with simple matrix operations\n",
        "              - multiplying the weight matrix by the input vector x\n",
        "              - adding the bias vector b\n",
        "              - applying the activation function g\n",
        "      - output of hidden layer, the vector h\n",
        "        - h = σ(Wx+b)\n",
        "        - forms a representation of the input.\n",
        "    - output units\n",
        "      - take this new representation h and compute a final output\n",
        "      - output layer is thus a vector y that gives a probability distribution across the output nodes\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter7_4 = \"\"\"\n",
        "- pretraining\n",
        "  - Using pretrained embedding representations, whether simple static word embeddings like word2vec or the more powerful contextual embeddings is one of the central ideas of deep learning.\n",
        "\"\"\"\n",
        "slp_chapter7_5 = \"\"\"  \n",
        "- Neural language models' advantages over n-gram\n",
        "  - hanlde longer histories\n",
        "  - generalize better over contexts of similar words\n",
        "  - more accurate at word-prediction\n",
        "  - more complex\n",
        "  - slow to train\n",
        "  - less interpretable than n-gram models\n",
        "- feedforward neural LM\n",
        "  - takes as input at time t a representation of some number of previous words (wt−1,wt−2, etc.) \n",
        "  - outputs a probability distribution over possible next words\n",
        "  - represent words in this prior context by their embed- dings, rather than just by their word identity \n",
        "- forward inference / decoding\n",
        "  - the task, given an input, of running a forward pass on the network to produce a probability distribution over possible outputs, in this case next words.\n",
        "  -  first represent each of the N previous words as a one-hot vector of length |V |, i.e., with one dimension for each word in the vocabulary. A one-hot vector is a vector that has one element equal to 1—in the dimension corresponding to that word’s index in the vocabulary— while all the other elements are set to zero. \n",
        "    - Select three embeddings from E: \n",
        "      - e = [Ext−3;Ext−2;Ext−1]\n",
        "      - we formed the embedding layer e by concatenating the 3 embeddings for the three context vectors\n",
        "    - Multiply by W\n",
        "      - h = σ(We+b)\n",
        "    - Multiply by U\n",
        "      - z = Uh\n",
        "    - Apply softmax\n",
        "      - yˆ = softmax(z)\n",
        "\"\"\"\n",
        "slp_chapter7_6 = \"\"\" \n",
        "- Training neural nets\n",
        "  - loss function to model the distance between the system output and the gold output\n",
        "  - find parameters (using gradient descent optimization algorithm) that minimize the loss function\n",
        "  - gradient descent requires knowing the gradient of the loss function (hard for neural networks with millions of parameters in many layers)\n",
        "    - error backpropagation / backward differentiation\n",
        "- loss function\n",
        "  - binary classifier\n",
        "    - same as logistic regression\n",
        "  - multinomial classifier\n",
        "    - the log of the output probability correspond- ing to the correct class, and we therefore also call this the negative log likelihood loss\n",
        "- computing the gradient\n",
        "  - error backpropagation / backprop / backward differentiation\n",
        "    - depends on the notion of computation graphs\n",
        "      - a representation of the process of computing a mathematical expression, in which the computation is broken down into separate operations, each of which is modeled as a node in a graph.\n",
        "      - chain rule\n",
        "        - computing the derivative of a composite function f (x) = u(v(w(x))), the derivative of f (x) is:\n",
        "          - df/dx = du / dv · dv / dw· dw / dx\n",
        "- regularization\n",
        "  - dropout\n",
        "    - randomly dropping some units and their connections from the network during training\n",
        "  - Tuning of hyperparameters\n",
        "    - include the learning rate η, the mini-batch size, the model architecture (the number of layers, the number of hidden nodes per layer, the choice of activation functions), how to regularize,\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "slp_chapter7_7 = \"\"\"\n",
        "- training the neural language model\n",
        "  - the parameters θ = E, W, U, b.\n",
        "    - it's okay to freeze the embedding layer E with initial word2vec values sometimes\n",
        "      - don’t update E during language model training.\n",
        "    - However, often we’d like to learn the embeddings simultaneously with training the network. This is useful when the task the network is designed for \n",
        "      - (sentiment classification, or translation, or parsing) places strong constraints on what makes a good representation for words.\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"logistic regression\" : \"discovering the link between features or cues and some particular outcome\",  \n",
        "\"neural network\":\"a series of logistic regression classifiers stacked on top of each other\", \n",
        "\"decision boundary\": \"a real value to make a decision\",\n",
        "\"representation learning\": \"ways to learn features automatically in an unsupervised way from the input\",\n",
        "\"gradient descent\" : \"an optimization algorithm for iteratively updating the weights so as to minimize this loss function\"\n",
        "}\n"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "fbe5ea9a-3631-4ced-df49-b1cbd704df29"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\n- generative model\n  - understanding what dogs look like\nand what cats look like. You might literally ask such\na model to generate\n  - generative model like naive Bayes makes use of this likelihood term,\n-  discriminative model\n  - trying to learn to distinguish the classes \n  - attempts to directly compute P(c|d). \n- A machine learning system for classification has\n  - A feature representation of the input.\n  - A classification function that computes yˆ, the estimated class, via p(y|x). (sigmod, softmax)\n  - An objective function for learning, usually involving minimizing error on\ntraining examples (like cross-entropy loss function)\n  - An algorithm for optimizing the objective function. (like stochastic gradient descent algorithm)\n- Logistic regression has 2 phases\n  - training\n    - a vector of weights and a bias term (also called intercept)\n      - The weight wi represents how important that input feature is to the classification decision\n  - test\n- goal of bianry logistic regression\n  - train a classifier that can make a binary decision about the class of a new input observation\n- sigmod function (logistic function)\n  - named because it looks like an s\n  - σ(z)= 1 = 1 / 1+(e)^−z = 1 / 1 + exp(−z)\n    - where z = w·x+b, w is the weight vector, x is the input feature vector and b is the bias term\n  - takes a real-valued number and maps it into the range [0, 1], which is just what we want for a probability\n  - sigmod property, 1−σ(x) = σ(−x), so we could also have expressed P(y = 0) as σ (−(w · x + b)).\n- Advantage of logistic regression over Naive Bayes\n  - Naive Bayes has overly strong conditional independence assumptions\n  - logistic regression is much more robust to correlated features; if two features f1 and f2 are perfectly correlated, regression will simply assign part of the weight to w1 and part to w2.\n    - more correlated features\n      - logistic assigns more accurate probability\n\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n \n- supervised classification\n- loss/cost function\n  - opposite of similarity\n    - a metric for how close the current label (yˆ) is to the true gold label y. \n  - we measure the distance between the system output and the gold output\n- cross-enropy loss\n  - commonly used for logistic regression and also for neural networks\n\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\n- loss function expresses\n  - for an observation x, how close the classifier output (yˆ = σ (w · x + b)) is to the correct output (y, which is 0 or 1)\n  - L(yˆ,y) = How much yˆ differs from the true y\n  - conditional maximum likelihood estimation\n    - we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x\n    - The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.\n\n\n\n## SLP Chapter 5.4 Gradient Descent\n\n- convex\n  - A convex function has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum.\n- Stochastic gradient descent \n  - an online algorithm (process input example by example rather than waiting till the entire)\n  - minimizes the loss function by computing its gradient after each training example, and nudging θ in the right direction \n- Learning rate  η \n  - a hyperparameter that must be adjusted\n  - high, steps too large, overshoot the minimum of the loss function\n  - low, too small, too long to get to the minimum\n  - chosen by the algorithm designer to affect how algorithm works\n\n\n## SLP Chapter 5.5 Regularization\n  \n- regularization\n  - to avoid overfitting\n  - used to penalize large weights\n  - L2 regularization\n    - a quadratic function of the weight values, named because it uses the (square of the) L2 norm of the weight values. \n    - The L2 norm, ||θ||2, is the same as the Euclidean distance of the vector θ from the origin.\n    - θˆ = argmax 􏰺logP(y(i)|x(i)) −α􏰺θj2\n    - ridge regression\n    - is easier to optimize because of its simple derivative (the derivative of θ2 is just 2θ\n    - prefers weight vectors with many small weights\n    - corresponds to assuming that weights are distributed according to a Gaussian distribution with mean μ = 0.\n  - L1 regularization\n    - a linear function of the weight values\n    - named after the L1 norm ||W||1, the sum of the absolute values of the weights, or Manhattan distance\n    - lasso regression\n    - is more complex (the derivative of |θ | is non-continuous at zero).\n    - prefers sparse solutions with some larger weights but many more weights set to zero\n    - far fewer features\n    -  Laplace prior on the weights.\n\n\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n\n-  multinomial logistic regression / softmax regression / maxent classifier\n  - more than 2 classes\n- features in multinomial logistic regression\n  - separate weight vectors (and biases) for each of the K classes. \n\n\n## SLP Chapter 7.1 Units\n\n- Neural Network\n  - a network of small computing units\n    - each of which takes a vector of input values and produces a single output value\n  - Instead of building rich kinds of feature templates like logistic regression\n    - take raw words as inputs and learn to induce features as part of the process of learning to classify.\n  -  The power of neural networks comes from the ability of early layers to learn representations that can be utilized by later layers in the network.\n- Deep learning\n  -  The use of modern neural nets because modern networks are often deep\n- Unit\n  - building block of a neural network - single computational unit\n  - takes a set of real valued numbers as input, performs some computation, and produces an output\n  - activation\n    - neural units apply a non-linear function f to z. We will refer to the output of this function as the activation value for the unit\n      - y=a= f(z)\n      - sigmod\n        - y = σ(w·x+b) = 1 / 1 + exp(-(w*x+b))\n        - very high values of z result in values of y that are saturated, i.e., extremely close to 1, and have derivatives very close to 0. Zero derivatives cause problems for learning (and for tanh)\n      - tanh\n        - a variant of the sigmoid that ranges from -1 to +1\n          - y = e^z - e^-z / e^z + e^−z\n          - being smoothly differentiable and mapping outlier values toward the mean. \n      - rectified linear unit ReLU \n        - simplest and most commonly used\n          - y = max(z, 0)\n          - result from it being very close to linear\n\n\n## SLP Chapter 7.2 The XOR problem\n\n- perceptron\n  - a simple neural unit\n    - has binary output\n    - does not have a non-linear activation function\n- XOR: not linearly separable\n- Non-linear activation functions\n  - a network formed by many layers of purely linear units can always be reduce to a single layer of linear units with appropriate weights\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n \n- feedforward network (multi-layer perceptrons (or MLPs);)\n  - a multilayer network in which the units are connected with no cycles;\n  - the outputs from units in each layer are passed to units in the next higher layer\n  - no outputs are passed back to lower layers\n  - but actually the unit in modern multilayer networks are not perceptrons\n    - perceptrons are purely linear\n    - modern networks are made up of units with non-linear functions\n  - 3 kinds of nodes\n    - input units\n      - The input layer x is a vector of simple scalar values\n    - hidden units\n      - The core of the neural network is the hidden layer h formed of hidden units hi,\n        - taking a weighted sum of its inputs and then applying a non-linearity. \n        - each layer is fully connected\n        - represents the parameters for the entire hidden layer by combining the weight vector and bias for each unit i into a single weight matrix W and a single bias vector b for the whole layer \n          - advantage of using a single matrix W for the weights of the entire layer\n            - the hidden layer computation for a feedforward network can be done very efficiently with simple matrix operations\n              - multiplying the weight matrix by the input vector x\n              - adding the bias vector b\n              - applying the activation function g\n      - output of hidden layer, the vector h\n        - h = σ(Wx+b)\n        - forms a representation of the input.\n    - output units\n      - take this new representation h and compute a final output\n      - output layer is thus a vector y that gives a probability distribution across the output nodes\n\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n\n- pretraining\n  - Using pretrained embedding representations, whether simple static word embeddings like word2vec or the more powerful contextual embeddings is one of the central ideas of deep learning.\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n  \n- Neural language models' advantages over n-gram\n  - hanlde longer histories\n  - generalize better over contexts of similar words\n  - more accurate at word-prediction\n  - more complex\n  - slow to train\n  - less interpretable than n-gram models\n- feedforward neural LM\n  - takes as input at time t a representation of some number of previous words (wt−1,wt−2, etc.) \n  - outputs a probability distribution over possible next words\n  - represent words in this prior context by their embed- dings, rather than just by their word identity \n- forward inference / decoding\n  - the task, given an input, of running a forward pass on the network to produce a probability distribution over possible outputs, in this case next words.\n  -  first represent each of the N previous words as a one-hot vector of length |V |, i.e., with one dimension for each word in the vocabulary. A one-hot vector is a vector that has one element equal to 1—in the dimension corresponding to that word’s index in the vocabulary— while all the other elements are set to zero. \n    - Select three embeddings from E: \n      - e = [Ext−3;Ext−2;Ext−1]\n      - we formed the embedding layer e by concatenating the 3 embeddings for the three context vectors\n    - Multiply by W\n      - h = σ(We+b)\n    - Multiply by U\n      - z = Uh\n    - Apply softmax\n      - yˆ = softmax(z)\n\n\n## SLP Chapter 7.6 Training Neural Nets\n \n- Training neural nets\n  - loss function to model the distance between the system output and the gold output\n  - find parameters (using gradient descent optimization algorithm) that minimize the loss function\n  - gradient descent requires knowing the gradient of the loss function (hard for neural networks with millions of parameters in many layers)\n    - error backpropagation / backward differentiation\n- loss function\n  - binary classifier\n    - same as logistic regression\n  - multinomial classifier\n    - the log of the output probability correspond- ing to the correct class, and we therefore also call this the negative log likelihood loss\n- computing the gradient\n  - error backpropagation / backprop / backward differentiation\n    - depends on the notion of computation graphs\n      - a representation of the process of computing a mathematical expression, in which the computation is broken down into separate operations, each of which is modeled as a node in a graph.\n      - chain rule\n        - computing the derivative of a composite function f (x) = u(v(w(x))), the derivative of f (x) is:\n          - df/dx = du / dv · dv / dw· dw / dx\n- regularization\n  - dropout\n    - randomly dropping some units and their connections from the network during training\n  - Tuning of hyperparameters\n    - include the learning rate η, the mini-batch size, the model architecture (the number of layers, the number of hidden nodes per layer, the choice of activation functions), how to regularize,\n\n\n\n\n\n## SLP Chapter 7.7 Training the neural language model\n\n- training the neural language model\n  - the parameters θ = E, W, U, b.\n    - it's okay to freeze the embedding layer E with initial word2vec values sometimes\n      - don’t update E during language model training.\n    - However, often we’d like to learn the embeddings simultaneously with training the network. This is useful when the task the network is designed for \n      - (sentiment classification, or translation, or parsing) places strong constraints on what makes a good representation for words.\n\n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 13 Definitions\n* **Logistic regression** - discovering the link between features or cues and some particular outcome\n* **Neural network** - a series of logistic regression classifiers stacked on top of each other\n* **Decision boundary** - a real value to make a decision\n* **Representation learning** - ways to learn features automatically in an unsupervised way from the input\n* **Gradient descent** - an optimization algorithm for iteratively updating the weights so as to minimize this loss function\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_10 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_11 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\"\n",
        "Neural Language models have several advantages over n-gram LMs: 1. They don’t need smoothing\n",
        "2. They can handle much longer histories.\n",
        "3. They can generalize over contexts of similar words.\n",
        "4. Neural LMs tend to have much higher predictive accuracy than n-gram LMs. Disadvantage: slower to train than traditional n-gram LMs\n",
        "\n",
        "Better results. They achieve better preplexity scores than SOTA n-gram LMs. Larger N. NN LMs can scale to much larger orders of n. This is achievable because\n",
        "parameters are associated only with individual words, and not with n-grams.\n",
        "They generalize across contexts. For example, by observing that the words blue, green, red, black, etc. appear in similar contexts, the model will be able to assign a reasonable score to the green car even though it never observed in training, because it did observe blue car and red car.\n",
        "A by-product of training are word embeddings\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 945
        },
        "outputId": "a9626cc5-6834-4c4f-8912-ef2aa4acee0e"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "T3cm1WV542_a",
        "outputId": "43ade8b5-c692-4b3f-f69e-f90665e19630"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n \n- MEU\n  - action = argmax_（下面是a）EU(a)\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\n- notation that describe an agent's preferences\n  - A > B (>是波浪形的) the agent prefers A over B\n  - A ~ B the agent is indifferent between A and B\n  - A >~ B (> 是波浪形的，~在下面) the agent prefers A over B or is indifferent between\n    - Orderability\n      - Given any two lotteries, a rational agent must either prefer one or else rate them as equally preferable. That is, the agent cannot avoid deciding\n    - Transitivity\n      - Given any three lotteries, if an agent prefers to and prefers to , then the agent must prefer to \n    - CONTINUITY\n      - If some lottery is between and in preference, then there is some probability for which the rational agent will be indifferent between getting for sure and the lottery that yields with probability and with probability .\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n Your summary here \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n Your summary here \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Your summary here \n\n## Module 8 Lecture Notes\n\n- MDP offline solution\n- RL online learning\n- Value Iteration\n  - Propagating information through the state space by means of local updates\n  - The Bellman equation is the basis of the value iteration algorithm for solving MDPs.\n  - n possible states means n possible bellman equations, 1 for each state. \n  - n equations conatins n unknowns, the state utilities. \n  - Equations are non-linear due to max. So cannot solved by linear algebra.\n  - We start with arbitrary initial values for the utilities, calculate the right-hand side of the equation, and plug it into the left-hand side—thereby updating the utility of each state from the utilities of its neighbors. We repeat this until we reach an equilibrium.\n  - Ui(s) is utility value for state s at ith iteration. It is Bellman Update.\n  - The update is assumed to be applied simultaneously to all the states at each iteration.\n  - If applied infinitely, guaranteed to reach equilibrium. \n  - Contraction. A contraction has only one fixed point; if there were two fixed points they would not get closer together when the function was applied, so it would not be a contraction. When the function is applied to any argument, the value must get closer to the fixed point, so repeated application reaches fixed point in the limit.\n  - Measure distances between utility vectors by max norm.\n  - Bellman update function beocmes Ui+1 = BUi with Bellman update operator B applied to all states.\n  - Can use contraction property to analyze rate of convergence. \n  - Terminal condition is |Ui+1 - U| < e\n\n\n## Module 8 Definitions\n* **Utility function** - assigns a single number to express the desirability of a state\n* **Expected utility of an action given the evidence eu(a)** - the average utility value of the outcomes, weighted by the probability that the outcome occurs\n* **Maximum expected utility (meu)** - a rational agent should choose the action that maximizes the agent’s expected utility\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Your summary here \n\n## AIMA Chapter 22.1 Learning from Rewards\n\n- Q-learning, \n  - where the agent learns a Q-function, or quality-function, , denoting the sum of rewards from state onward if action is taken. \n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **Term** - your definition here\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Your summary here \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Term** - your definition here\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Your summary here \n\n## Module 11 Definitions\n* **Concept** - definition\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\n- bias\n  - bias we mean (loosely) the tendency of a predictive hypothesis to deviate from the expected value when averaged over different training sets. \n- variance\n  - we mean the amount of change in the hypothesis due to fluctuation in the training data\n- \n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n \n- decision boundary\n  - is a line (or a surface, in higher dimensions) that separates the two classes. \n  - A linear decision boundary is called a linear separator and data that admit such a separator are called linearly separable. \n- perceptron learning rule\n  - If the output is correct (i.e. y = hw(x) ) then the weights are not changed.\n  - If y is 1 but hw(x) is 0, then wi is increased when the corresponding input xi is positive and decreased when xi is negative. This makes sense, because we want to make\n  sure w*x bigger so that hw(x) outputs a 1.\n  - If y is 0 but hw(x) is 1, then wi is decreased when the corresponding input xi is positive and increased when xi is negative. This makes sense, because we want to make \n  w*x smaller so that hw(x) outputs a 0.\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\n- Naive Bayes is a probabilistic classifier, meaning that for a document d, out of all classes c ∈ C the classifier returns \nthe class cˆ which has the maximum posterior ˆ probability given the document. we use the hat notation ˆ to mean “our\nestimate of the correct class”\n    - c^ = argmaxP(c|d) c ∈ C\n- Bayesian inference\n  - This idea of Bayesian inference was first applied to text classification.\n   The intuition of Bayesian classification is to use Bayes’ rule to transform into other probabilities that have some useful properties.\n   it gives us a way to break down any conditional probability P(x|y) into three other probabilities:\n  - P(x|y) = P(y|x)P(x)/ P(y)\n\n\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n \n- Supervised learning is a machine learning approach that’s defined by its use of labeled datasets. These datasets are designed to train or “supervise” algorithms into classifying data or predicting outcomes accurately\n  - Classification problems use an algorithm to accurately assign test data into specific categories, such as separating apples from oranges. Or, in the real world, supervised learning algorithms can be used to classify spam in a separate folder from your inbox. Linear classifiers, support vector machines, decision trees and random forest are all common types of classification algorithms. \n  - Regression is another type of supervised learning method that uses an algorithm to understand the relationship between dependent and independent variables. Regression models are helpful for predicting numerical values based on different data points, such as sales revenue projections for a given business. Some popular regression algorithms are linear regression, logistic regression and polynomial regression.\n- Unsupervised learning uses machine learning algorithms to analyze and cluster unlabeled data sets. These algorithms discover hidden patterns in data without the need for human intervention (hence, they are “unsupervised”).\n  - Clustering is a data mining technique for grouping unlabeled data based on their similarities or differences. For example, K-means clustering algorithms assign similar data points into groups, where the K value represents the size of the grouping and granularity. This technique is helpful for market segmentation, image compression, etc.\n  - Association is another type of unsupervised learning method that uses different rules to find relationships between variables in a given dataset. These methods are frequently used for market basket analysis and recommendation engines, along the lines of “Customers Who Bought This Item Also Bought” recommendations.\n  - Dimensionality reduction is a learning technique used when the number of features  (or dimensions) in a given dataset is too high. It reduces the number of data inputs to a manageable size while also preserving the data integrity. Often, this technique is used in the preprocessing data stage, such as when autoencoders remove noise from visual data to improve picture quality.\n- Reinforcement Learning\n  - the agent learns from a series of reinforcements: rewards (like win in a game) and punishments (like lost)\n  - up to the agent to decide which of the actions prior to the reinforcement were most responsible for it, and to alter its actions to aim towards more rewards in the future.\n\n\n\n\n## Module 12 Definitions\n* **Concept** - definition\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\n- generative model\n  - understanding what dogs look like\nand what cats look like. You might literally ask such\na model to generate\n  - generative model like naive Bayes makes use of this likelihood term,\n-  discriminative model\n  - trying to learn to distinguish the classes \n  - attempts to directly compute P(c|d). \n- A machine learning system for classification has\n  - A feature representation of the input.\n  - A classification function that computes yˆ, the estimated class, via p(y|x). (sigmod, softmax)\n  - An objective function for learning, usually involving minimizing error on\ntraining examples (like cross-entropy loss function)\n  - An algorithm for optimizing the objective function. (like stochastic gradient descent algorithm)\n- Logistic regression has 2 phases\n  - training\n    - a vector of weights and a bias term (also called intercept)\n      - The weight wi represents how important that input feature is to the classification decision\n  - test\n- goal of bianry logistic regression\n  - train a classifier that can make a binary decision about the class of a new input observation\n- sigmod function (logistic function)\n  - named because it looks like an s\n  - σ(z)= 1 = 1 / 1+(e)^−z = 1 / 1 + exp(−z)\n    - where z = w·x+b, w is the weight vector, x is the input feature vector and b is the bias term\n  - takes a real-valued number and maps it into the range [0, 1], which is just what we want for a probability\n  - sigmod property, 1−σ(x) = σ(−x), so we could also have expressed P(y = 0) as σ (−(w · x + b)).\n- Advantage of logistic regression over Naive Bayes\n  - Naive Bayes has overly strong conditional independence assumptions\n  - logistic regression is much more robust to correlated features; if two features f1 and f2 are perfectly correlated, regression will simply assign part of the weight to w1 and part to w2.\n    - more correlated features\n      - logistic assigns more accurate probability\n\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n \n- supervised classification\n- loss/cost function\n  - opposite of similarity\n    - a metric for how close the current label (yˆ) is to the true gold label y. \n  - we measure the distance between the system output and the gold output\n- cross-enropy loss\n  - commonly used for logistic regression and also for neural networks\n\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\n- loss function expresses\n  - for an observation x, how close the classifier output (yˆ = σ (w · x + b)) is to the correct output (y, which is 0 or 1)\n  - L(yˆ,y) = How much yˆ differs from the true y\n  - conditional maximum likelihood estimation\n    - we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x\n    - The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.\n\n\n\n## SLP Chapter 5.4 Gradient Descent\n\n- convex\n  - A convex function has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum.\n- Stochastic gradient descent \n  - an online algorithm (process input example by example rather than waiting till the entire)\n  - minimizes the loss function by computing its gradient after each training example, and nudging θ in the right direction \n- Learning rate  η \n  - a hyperparameter that must be adjusted\n  - high, steps too large, overshoot the minimum of the loss function\n  - low, too small, too long to get to the minimum\n  - chosen by the algorithm designer to affect how algorithm works\n\n\n## SLP Chapter 5.5 Regularization\n  \n- regularization\n  - to avoid overfitting\n  - used to penalize large weights\n  - L2 regularization\n    - a quadratic function of the weight values, named because it uses the (square of the) L2 norm of the weight values. \n    - The L2 norm, ||θ||2, is the same as the Euclidean distance of the vector θ from the origin.\n    - θˆ = argmax 􏰺logP(y(i)|x(i)) −α􏰺θj2\n    - ridge regression\n    - is easier to optimize because of its simple derivative (the derivative of θ2 is just 2θ\n    - prefers weight vectors with many small weights\n    - corresponds to assuming that weights are distributed according to a Gaussian distribution with mean μ = 0.\n  - L1 regularization\n    - a linear function of the weight values\n    - named after the L1 norm ||W||1, the sum of the absolute values of the weights, or Manhattan distance\n    - lasso regression\n    - is more complex (the derivative of |θ | is non-continuous at zero).\n    - prefers sparse solutions with some larger weights but many more weights set to zero\n    - far fewer features\n    -  Laplace prior on the weights.\n\n\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n\n-  multinomial logistic regression / softmax regression / maxent classifier\n  - more than 2 classes\n- features in multinomial logistic regression\n  - separate weight vectors (and biases) for each of the K classes. \n\n\n## SLP Chapter 7.1 Units\n\n- Neural Network\n  - a network of small computing units\n    - each of which takes a vector of input values and produces a single output value\n  - Instead of building rich kinds of feature templates like logistic regression\n    - take raw words as inputs and learn to induce features as part of the process of learning to classify.\n  -  The power of neural networks comes from the ability of early layers to learn representations that can be utilized by later layers in the network.\n- Deep learning\n  -  The use of modern neural nets because modern networks are often deep\n- Unit\n  - building block of a neural network - single computational unit\n  - takes a set of real valued numbers as input, performs some computation, and produces an output\n  - activation\n    - neural units apply a non-linear function f to z. We will refer to the output of this function as the activation value for the unit\n      - y=a= f(z)\n      - sigmod\n        - y = σ(w·x+b) = 1 / 1 + exp(-(w*x+b))\n        - very high values of z result in values of y that are saturated, i.e., extremely close to 1, and have derivatives very close to 0. Zero derivatives cause problems for learning (and for tanh)\n      - tanh\n        - a variant of the sigmoid that ranges from -1 to +1\n          - y = e^z - e^-z / e^z + e^−z\n          - being smoothly differentiable and mapping outlier values toward the mean. \n      - rectified linear unit ReLU \n        - simplest and most commonly used\n          - y = max(z, 0)\n          - result from it being very close to linear\n\n\n## SLP Chapter 7.2 The XOR problem\n\n- perceptron\n  - a simple neural unit\n    - has binary output\n    - does not have a non-linear activation function\n- XOR: not linearly separable\n- Non-linear activation functions\n  - a network formed by many layers of purely linear units can always be reduce to a single layer of linear units with appropriate weights\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n \n- feedforward network (multi-layer perceptrons (or MLPs);)\n  - a multilayer network in which the units are connected with no cycles;\n  - the outputs from units in each layer are passed to units in the next higher layer\n  - no outputs are passed back to lower layers\n  - but actually the unit in modern multilayer networks are not perceptrons\n    - perceptrons are purely linear\n    - modern networks are made up of units with non-linear functions\n  - 3 kinds of nodes\n    - input units\n      - The input layer x is a vector of simple scalar values\n    - hidden units\n      - The core of the neural network is the hidden layer h formed of hidden units hi,\n        - taking a weighted sum of its inputs and then applying a non-linearity. \n        - each layer is fully connected\n        - represents the parameters for the entire hidden layer by combining the weight vector and bias for each unit i into a single weight matrix W and a single bias vector b for the whole layer \n          - advantage of using a single matrix W for the weights of the entire layer\n            - the hidden layer computation for a feedforward network can be done very efficiently with simple matrix operations\n              - multiplying the weight matrix by the input vector x\n              - adding the bias vector b\n              - applying the activation function g\n      - output of hidden layer, the vector h\n        - h = σ(Wx+b)\n        - forms a representation of the input.\n    - output units\n      - take this new representation h and compute a final output\n      - output layer is thus a vector y that gives a probability distribution across the output nodes\n\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n\n- pretraining\n  - Using pretrained embedding representations, whether simple static word embeddings like word2vec or the more powerful contextual embeddings is one of the central ideas of deep learning.\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n  \n- Neural language models' advantages over n-gram\n  - hanlde longer histories\n  - generalize better over contexts of similar words\n  - more accurate at word-prediction\n  - more complex\n  - slow to train\n  - less interpretable than n-gram models\n- feedforward neural LM\n  - takes as input at time t a representation of some number of previous words (wt−1,wt−2, etc.) \n  - outputs a probability distribution over possible next words\n  - represent words in this prior context by their embed- dings, rather than just by their word identity \n- forward inference / decoding\n  - the task, given an input, of running a forward pass on the network to produce a probability distribution over possible outputs, in this case next words.\n  -  first represent each of the N previous words as a one-hot vector of length |V |, i.e., with one dimension for each word in the vocabulary. A one-hot vector is a vector that has one element equal to 1—in the dimension corresponding to that word’s index in the vocabulary— while all the other elements are set to zero. \n    - Select three embeddings from E: \n      - e = [Ext−3;Ext−2;Ext−1]\n      - we formed the embedding layer e by concatenating the 3 embeddings for the three context vectors\n    - Multiply by W\n      - h = σ(We+b)\n    - Multiply by U\n      - z = Uh\n    - Apply softmax\n      - yˆ = softmax(z)\n\n\n## SLP Chapter 7.6 Training Neural Nets\n \n- Training neural nets\n  - loss function to model the distance between the system output and the gold output\n  - find parameters (using gradient descent optimization algorithm) that minimize the loss function\n  - gradient descent requires knowing the gradient of the loss function (hard for neural networks with millions of parameters in many layers)\n    - error backpropagation / backward differentiation\n- loss function\n  - binary classifier\n    - same as logistic regression\n  - multinomial classifier\n    - the log of the output probability correspond- ing to the correct class, and we therefore also call this the negative log likelihood loss\n- computing the gradient\n  - error backpropagation / backprop / backward differentiation\n    - depends on the notion of computation graphs\n      - a representation of the process of computing a mathematical expression, in which the computation is broken down into separate operations, each of which is modeled as a node in a graph.\n      - chain rule\n        - computing the derivative of a composite function f (x) = u(v(w(x))), the derivative of f (x) is:\n          - df/dx = du / dv · dv / dw· dw / dx\n- regularization\n  - dropout\n    - randomly dropping some units and their connections from the network during training\n  - Tuning of hyperparameters\n    - include the learning rate η, the mini-batch size, the model architecture (the number of layers, the number of hidden nodes per layer, the choice of activation functions), how to regularize,\n\n\n\n\n\n## SLP Chapter 7.7 Training the neural language model\n\n- training the neural language model\n  - the parameters θ = E, W, U, b.\n    - it's okay to freeze the embedding layer E with initial word2vec values sometimes\n      - don’t update E during language model training.\n    - However, often we’d like to learn the embeddings simultaneously with training the network. This is useful when the task the network is designed for \n      - (sentiment classification, or translation, or parsing) places strong constraints on what makes a good representation for words.\n\n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 13 Definitions\n* **Logistic regression** - discovering the link between features or cues and some particular outcome\n* **Neural network** - a series of logistic regression classifiers stacked on top of each other\n* **Decision boundary** - a real value to make a decision\n* **Representation learning** - ways to learn features automatically in an unsupervised way from the input\n* **Gradient descent** - an optimization algorithm for iteratively updating the weights so as to minimize this loss function\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}