{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CIS521_Midterm2_Notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3bec8c29-5d58-4453-ed15-9dd40cbe3fb5"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "37def0a1-075a-4b47-846e-1f32fa7b171e"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = False"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\"\n",
        "Principle of maximum expected utility - an agent should choose the action that gives the highest expected utility.\n",
        "\"\"\" \n",
        "chapter16_2 =  \"\"\"\n",
        "List of constraints we expect a reasonable preference relation to obey (Axioms of utility theory):\n",
        " - Orderability (Given two lotteries, either one is preferred or they are equally preferable)\n",
        " - Transitivity (If A is preferred to B, and B to C, then it follows A is preferred to C)\n",
        " - Continuity (For a lottery B between A and C in preference, there exists a probability p in which the agent is indifferent to B, A with probability p, \\\n",
        " and C with probability 1-p)\n",
        " - Substitutability (If the agent is indifferent between A and B, one can be substituted for the other with no change in the preference relation)\n",
        " - Monotonicity (For two lotteries with the same outcomes A and B, if outcome A is preferred, the lottery with higher probability of A must also be preferred)\n",
        " - Decomposability (Laws of probability can be used to reduce complex lotteries into simpler ones, aka \"no fun in gambling rule\")\n",
        "\"\"\"\n",
        "chapter16_3 =  \"\"\"\n",
        "Certainty effect - people like gains that are certain to be realized\n",
        "Ambiguity aversion - people like known probabilities rather than unknown ones\n",
        "Framing effect - wording makes a difference in perception\n",
        "Anchoring effect - sometimes the actual utility doesnt matter as much as the utility in relation to other choices\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\"\n",
        "Markov Decision Process: \n",
        " - Fully observable/stochastic environment, Markovian transition model, additive rewards\n",
        " - Contains set of states, actions, transition model, and reward function\n",
        " - Solution to this problem is known as a policy, which determines the best action to take at a given state\n",
        " - The optimal policy is the one that gives the highest expected utility \n",
        "\n",
        "Finite Horizon: Game ends after a certain amount of turns/time, optimal policy is nonstationary\n",
        "Infinite Horizon: Game can go on forever, optimal policy is stationary\n",
        "\n",
        "Additive Rewards: Total reward increases by adding the reward for reaching consecutive states with no penalty\n",
        "Discounted Rewards: Total reward increases, but rewards for reaching later states are reduced, aka immediate rewards are worth more than future rewards\n",
        "\n",
        "Proper policy: Policy guaranteed to reach a terminal state\n",
        "\"\"\"\n",
        "chapter17_2 =  \"\"\"\n",
        "Value Iteration: Iteratively update the values of each state using Bellman equation\n",
        "\n",
        "Policy Iteration: Alternates between two steps\n",
        " - Policy Evaluation (Given current policy calculate utility of each state)\n",
        " - Policy Improvement (Calculate a new policy based on the new utilities)\n",
        "\n",
        "Modified Policy Iteration: More efficient normally than both regular policy iteration and value iteration, get next utility estimate by repeating bellman update k times\n",
        "Asynchronous Policy Iteration: Instead of updating all states at once, modify a subset of states at a time, guaranteed to converge to optimal policy under certain conditions\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\"\n",
        "Expectimax node: outcome is uncertain, expected utility is calculated\n",
        "\n",
        "Value Iteration:\n",
        " - Time complexity is O(S^2A)\n",
        " - Converges to unique optimal values\n",
        " - Policy may converge many steps before values do\n",
        "\n",
        "Policy Iteration: \n",
        " - Time complexity is O(S^2)\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"assigns a value determining how good a state is\",\n",
        "\"expected utility\" : \"average utility value of all outcomes given a chosen action\",\n",
        "\"lottery\": \"set of outcomes with associated probabilites for a given action\",\n",
        "\"ordinal utility function\": \"function that gives a preference ranking to a set of states\",\n",
        "\"preference elicitation\": \"determining an agent's utility function through presenting choices and observing preferences\",\n",
        "\"optimizer's curse\": \"expected utility is often overestimated\",\n",
        "\"sequential decision problems\": \"utility is determined by a sequence of actions\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "0fb2f849-424b-47c8-8f56-f0f98535c1db"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nPrinciple of maximum expected utility - an agent should choose the action that gives the highest expected utility.\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nList of constraints we expect a reasonable preference relation to obey (Axioms of utility theory):\n - Orderability (Given two lotteries, either one is preferred or they are equally preferable)\n - Transitivity (If A is preferred to B, and B to C, then it follows A is preferred to C)\n - Continuity (For a lottery B between A and C in preference, there exists a probability p in which the agent is indifferent to B, A with probability p,  and C with probability 1-p)\n - Substitutability (If the agent is indifferent between A and B, one can be substituted for the other with no change in the preference relation)\n - Monotonicity (For two lotteries with the same outcomes A and B, if outcome A is preferred, the lottery with higher probability of A must also be preferred)\n - Decomposability (Laws of probability can be used to reduce complex lotteries into simpler ones, aka \"no fun in gambling rule\")\n\n\n## AIMA Chapter 16.3 Utility Functions\n\nCertainty effect - people like gains that are certain to be realized\nAmbiguity aversion - people like known probabilities rather than unknown ones\nFraming effect - wording makes a difference in perception\nAnchoring effect - sometimes the actual utility doesnt matter as much as the utility in relation to other choices\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\nMarkov Decision Process: \n - Fully observable/stochastic environment, Markovian transition model, additive rewards\n - Contains set of states, actions, transition model, and reward function\n - Solution to this problem is known as a policy, which determines the best action to take at a given state\n - The optimal policy is the one that gives the highest expected utility \n\nFinite Horizon: Game ends after a certain amount of turns/time, optimal policy is nonstationary\nInfinite Horizon: Game can go on forever, optimal policy is stationary\n\nAdditive Rewards: Total reward increases by adding the reward for reaching consecutive states with no penalty\nDiscounted Rewards: Total reward increases, but rewards for reaching later states are reduced, aka immediate rewards are worth more than future rewards\n\nProper policy: Policy guaranteed to reach a terminal state\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\nValue Iteration: Iteratively update the values of each state using Bellman equation\n\nPolicy Iteration: Alternates between two steps\n - Policy Evaluation (Given current policy calculate utility of each state)\n - Policy Improvement (Calculate a new policy based on the new utilities)\n\nModified Policy Iteration: More efficient normally than both regular policy iteration and value iteration, get next utility estimate by repeating bellman update k times\nAsynchronous Policy Iteration: Instead of updating all states at once, modify a subset of states at a time, guaranteed to converge to optimal policy under certain conditions\n\n\n## Module 8 Lecture Notes\n\nExpectimax node: outcome is uncertain, expected utility is calculated\n\nValue Iteration:\n - Time complexity is O(S^2A)\n - Converges to unique optimal values\n - Policy may converge many steps before values do\n\nPolicy Iteration: \n - Time complexity is O(S^2)\n\n\n## Module 8 Definitions\n* **Utility function** - assigns a value determining how good a state is\n* **Expected utility** - average utility value of all outcomes given a chosen action\n* **Lottery** - set of outcomes with associated probabilites for a given action\n* **Ordinal utility function** - function that gives a preference ranking to a set of states\n* **Preference elicitation** - determining an agent's utility function through presenting choices and observing preferences\n* **Optimizer's curse** - expected utility is often overestimated\n* **Sequential decision problems** - utility is determined by a sequence of actions\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\"\n",
        "Bandit Problem: Given a set of possible options, which one do you take - the one that has given the best rewards so far, or one of the ones that has not been tried?\n",
        "\"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\"\n",
        "Reinforcement Learning: Goal is to learn an optimal policy based on observed rewards, for many games, state spaces can be far too large for regular MDP computation, \\\n",
        "so reinforcement learning methods must be used\n",
        "\n",
        "Utility-based agent: Learns a utility function that is used to select actions for a given state based on maximum expected utility\n",
        "Q-learning agent: Learns an action-utility function (Q-function), which given a state and action returns the expected utility\n",
        "Reflex agent: For a given state, it has a fixed given action\n",
        "\"\"\"\n",
        "chapter22_2 =  \"\"\"\n",
        "Passive Reinforcement Learning: Agent operates using a fixed policy, goal is to learn state utilities/a model of the environment. Similar to policy evaluation, \\\n",
        "but the transition model as well as the reward function are not known\n",
        "\"\"\"\n",
        "chapter22_3 =  \"\"\"\n",
        "Active Reinforcement Learning: Agent must learn about the environment, and then also learn what actions to take\n",
        "\n",
        "Exploration/Exploitation Tradeoff: Tradeoff an agent must make between choosing the best path it knows at a given moment and exploring it further, and choosing \\\n",
        "a path that isn't optimal according to it's current policy, but may actually end up being better after exploring it. \n",
        "\n",
        "Q-Learning:\n",
        " - Model-free method (does not need a transition model for learning or action selection)\n",
        " - Assigns a values, known as the q-value, to state-action pairs\n",
        " - Off-policy learning algorithm\n",
        "\n",
        "SARSA Learning:\n",
        " - Q-learning goes off of the best Q-value from the reached state, whereas SARSA waits until an action is taken and then uses the Q-value for that action\n",
        " - Identical with greedy agents, different when exploration is involved\n",
        " - On-policy learning algorithm \n",
        " - More realistic, sometimes it is better to learn through what happens rather than the best thing that can happen\n",
        "\"\"\"\n",
        "chapter22_4 =  \"\"\"\n",
        "Function Approximation: A way to help a reinforcement learning algorithm generalize by using a representation other than a lookup table. Also helps as some \\\n",
        "games have far too many states to store in a lookup table. Works by creating a compressed representation, which can also help the model choose actions for states \\\n",
        "it has not seen before based on information it has gotten from similar states\n",
        "\n",
        "Best to use an online learning algorithm that updates the model every time a new trial is completed\n",
        "\"\"\"\n",
        "chapter22_5 =  \"\"\"\n",
        "Policy Search: Continuously adjust the policy until performance stops improving\n",
        "\n",
        "Policy Value: Expected reward-to-go for the policy \n",
        "\n",
        "Policy Gradient: Used to optimize and update the policy\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\"\n",
        "Important reinforcement learning ideas:\n",
        " - Exploration (Must explore the unknown to get information)\n",
        " - Exploitation (Want to use the knowledge to choose the best actions)\n",
        " - Regret (Even with intelligent learning, you will make mistakes)\n",
        " - Sampling (Due to stochastic events, must try things repeatedly)\n",
        " - Difficulty (Learning can be much harder than solving MDPs)\n",
        "\n",
        "Model-based learning:\n",
        " - Build a model based on experiences\n",
        " - Step 1, learn an empirical MDP model (count outcomes, normalize, estimate transition model and reward model)\n",
        " - Step 2, solve learned MDP\n",
        "\n",
        "Q-learning will converge to the optimal policy, even if you aren't acting optimally\n",
        "\n",
        "Regret measures total mistake cost, calculated as the difference between expected rewards and optimal rewards\n",
        "\n",
        "Approximate Q-Learning:\n",
        " - Feature representation allows us to write a q function as the product of feature vector and weight vector\n",
        " - Advantage, experience can be summed up in a few numbers\n",
        " - Disadvantage, states with similar features may actually be very different in terms of value\n",
        "\"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"reward-to-go\" : \"expected total reward from a state and the consecutive states past it\",\n",
        "\"greedy agent\" : \"takes the best path at every step as learned by the current model, does not explore\",\n",
        "\"exploration function\" : \"decides how the trade off between exploration and exploitation is determined\",\n",
        "}"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "1ec31057-e6d0-46a0-bd99-62b1a6804e5a"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n\nBandit Problem: Given a set of possible options, which one do you take - the one that has given the best rewards so far, or one of the ones that has not been tried?\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n\nReinforcement Learning: Goal is to learn an optimal policy based on observed rewards, for many games, state spaces can be far too large for regular MDP computation, so reinforcement learning methods must be used\n\nUtility-based agent: Learns a utility function that is used to select actions for a given state based on maximum expected utility\nQ-learning agent: Learns an action-utility function (Q-function), which given a state and action returns the expected utility\nReflex agent: For a given state, it has a fixed given action\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\nPassive Reinforcement Learning: Agent operates using a fixed policy, goal is to learn state utilities/a model of the environment. Similar to policy evaluation, but the transition model as well as the reward function are not known\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\nActive Reinforcement Learning: Agent must learn about the environment, and then also learn what actions to take\n\nExploration/Exploitation Tradeoff: Tradeoff an agent must make between choosing the best path it knows at a given moment and exploring it further, and choosing a path that isn't optimal according to it's current policy, but may actually end up being better after exploring it. \n\nQ-Learning:\n - Model-free method (does not need a transition model for learning or action selection)\n - Assigns a values, known as the q-value, to state-action pairs\n - Off-policy learning algorithm\n\nSARSA Learning:\n - Q-learning goes off of the best Q-value from the reached state, whereas SARSA waits until an action is taken and then uses the Q-value for that action\n - Identical with greedy agents, different when exploration is involved\n - On-policy learning algorithm \n - More realistic, sometimes it is better to learn through what happens rather than the best thing that can happen\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\nFunction Approximation: A way to help a reinforcement learning algorithm generalize by using a representation other than a lookup table. Also helps as some games have far too many states to store in a lookup table. Works by creating a compressed representation, which can also help the model choose actions for states it has not seen before based on information it has gotten from similar states\n\nBest to use an online learning algorithm that updates the model every time a new trial is completed\n\n\n## Module 9 Lecture Notes\n\nImportant reinforcement learning ideas:\n - Exploration (Must explore the unknown to get information)\n - Exploitation (Want to use the knowledge to choose the best actions)\n - Regret (Even with intelligent learning, you will make mistakes)\n - Sampling (Due to stochastic events, must try things repeatedly)\n - Difficulty (Learning can be much harder than solving MDPs)\n\nModel-based learning:\n - Build a model based on experiences\n - Step 1, learn an empirical MDP model (count outcomes, normalize, estimate transition model and reward model)\n - Step 2, solve learned MDP\n\nQ-learning will converge to the optimal policy, even if you aren't acting optimally\n\nRegret measures total mistake cost, calculated as the difference between expected rewards and optimal rewards\n\nApproximate Q-Learning:\n - Feature representation allows us to write a q function as the product of feature vector and weight vector\n - Advantage, experience can be summed up in a few numbers\n - Disadvantage, states with similar features may actually be very different in terms of value\n\n\n## Module 9 Definitions\n* **Reward-to-go** - expected total reward from a state and the consecutive states past it\n* **Greedy agent** - takes the best path at every step as learned by the current model, does not explore\n* **Exploration function** - decides how the trade off between exploration and exploitation is determined\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\"\n",
        "Belief state: Representation of the possible states an agent may be in\n",
        "Issues with this:\n",
        " - When given partial information, the agent must consider all possible explanation, even unlikely ones. This can lead to very large and too complex to handle belief-state spaces\n",
        " - Handling every possible eventuality means considering the unlikely, and can again get too large to handle\n",
        " - There may be no solution, and there may be no way to compare which plan to take given that none of them reach the solution\n",
        "\n",
        "Qualification Problem: if there are possible things that can go wrong that cannot be known if they will occur, success of a plan cannot be determined\n",
        "\n",
        "Rational Decision: Pick the best action, determined based on the relative importance of possible goal states as well as the probability and degree that each one will be achieved\n",
        "\n",
        "Probability Theory: Way to summarize and deal with uncertainty\n",
        "\n",
        "Utility Theory: Way to assign value to certain states, helping us decide which ones are more desirable than others\n",
        "\n",
        "Decision Theory = Probability Theory + Utility Theory\n",
        "\"\"\"\n",
        "chapter12_2 = \"\"\"\n",
        "Probability Overview\n",
        "\"\"\"\n",
        "chapter12_3 = \"\"\"\n",
        "Frequentist view - Numbers come from experiments\n",
        "Objectivist view - Probabilities are universal aspects that can be observed (possibly through experiments)\n",
        "Subjectivist view - Probabilities quantify an agent's beliefs\n",
        "\n",
        "Marginalization - Probability of variable value given by summing over other variables and their probabilities\n",
        "Conditioning - Variant of marginalization involving conditional probability\n",
        "Normalization - Ensures probabilities add up to 1\n",
        "\"\"\"\n",
        "chapter12_4 = \"\"\"\n",
        "Independence in probability\n",
        "\"\"\"\n",
        "chapter12_5 = \"\"\"\n",
        "Bayes' Theorem: Deals with conditional probability\n",
        "\n",
        "Naive Bayes Model assumes that a single cause can lead to a multitude of conditionally independent effects. Can often be used in cases where the \\\n",
        "effects are not actually conditionally independent.\n",
        "\"\"\"\n",
        "chapter12_6 = \"\"\"\n",
        "Wumpus World\n",
        "\"\"\"\n",
        "chapter12_7 = \"\"\"\n",
        "Summary\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\"\n",
        "Problems with n-gram model MLE:\n",
        " - Out of vocabulary items (handled with unknown token)\n",
        " - Fixed vocab \n",
        " - Any word not in vocab replaced with unknown token\n",
        " - Zeros avoided with smoothing (Laplace?)\n",
        " - Use log space to avoid numerical underflow\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"sample space\" : \"set of possible outcomes\",\n",
        "\"probability model\" : \"assigns a probability to each possible outcome\",\n",
        "\"prior probability\" : \"probability with no information\",\n",
        "\"conditional/posterior probability\" : \"probability given some information about an outcome that has already occured\",\n",
        "}"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "967ed709-d628-484f-eb42-74d9e67ee590"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\nBelief state: Representation of the possible states an agent may be in\nIssues with this:\n - When given partial information, the agent must consider all possible explanation, even unlikely ones. This can lead to very large and too complex to handle belief-state spaces\n - Handling every possible eventuality means considering the unlikely, and can again get too large to handle\n - There may be no solution, and there may be no way to compare which plan to take given that none of them reach the solution\n\nQualification Problem: if there are possible things that can go wrong that cannot be known if they will occur, success of a plan cannot be determined\n\nRational Decision: Pick the best action, determined based on the relative importance of possible goal states as well as the probability and degree that each one will be achieved\n\nProbability Theory: Way to summarize and deal with uncertainty\n\nUtility Theory: Way to assign value to certain states, helping us decide which ones are more desirable than others\n\nDecision Theory = Probability Theory + Utility Theory\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n\nProbability Overview\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n\nFrequentist view - Numbers come from experiments\nObjectivist view - Probabilities are universal aspects that can be observed (possibly through experiments)\nSubjectivist view - Probabilities quantify an agent's beliefs\n\nMarginalization - Probability of variable value given by summing over other variables and their probabilities\nConditioning - Variant of marginalization involving conditional probability\nNormalization - Ensures probabilities add up to 1\n\n\n## AIMA Chapter 12.4 Independence\n\nIndependence in probability\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\nBayes' Theorem: Deals with conditional probability\n\nNaive Bayes Model assumes that a single cause can lead to a multitude of conditionally independent effects. Can often be used in cases where the effects are not actually conditionally independent.\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\nWumpus World\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n\nSummary\n\n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n\nProblems with n-gram model MLE:\n - Out of vocabulary items (handled with unknown token)\n - Fixed vocab \n - Any word not in vocab replaced with unknown token\n - Zeros avoided with smoothing (Laplace?)\n - Use log space to avoid numerical underflow\n\n\n## Module 10 Definitions\n* **Sample space** - set of possible outcomes\n* **Probability model** - assigns a probability to each possible outcome\n* **Prior probability** - probability with no information\n* **Conditional/posterior probability** - probability given some information about an outcome that has already occured\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\"\n",
        "Bayesian Network: \n",
        " - Directed graph where each node contains probabilities\n",
        " - Nodes represent random variables (can be discrete or continuous)\n",
        " - Directed acyclic graph\n",
        " - Each node contains a conditional probability distribution (P(Xi|Parents(Xi)))\n",
        "\n",
        "Conditioning Case: Possible combination of values the parent nodes could take on\n",
        "\"\"\"\n",
        "chapter13_2 = \"\"\"\n",
        "Bayesian Network Semantics\n",
        "\"\"\"\n",
        "chapter13_3 = \"\"\"\n",
        "Representation of Bayesian Networks\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\"\n",
        "Bayes' nets are simpler and easier to think about when they reflect true causal patterns. However, they don't need to actually be causal, especially if \\\n",
        "variables are missing. When no causality exists, arrows reflect correlation. Topology may encode causal structure, but what it is actually encoding is \\\n",
        "conditional independence\n",
        "\n",
        "Joint distribution over N boolean variables has size 2^N\n",
        "An N-node net with nodes having up to k parents has O(N * 2^(k+1))\n",
        "\n",
        "Three probability relationships:\n",
        " - Causal chains\n",
        " - Common cause\n",
        " - Common effect\n",
        "\n",
        "Check all paths between two nodes, if one or more is active then independence is not guaranteed. If all paths inactive then independence guaranteed\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"inference\" : \"calculating some value from a joint probability distribution\",    \n",
        "}\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 654
        },
        "cellView": "form",
        "outputId": "0f81f24a-f0b6-4b15-9e51-9f8a74f3df79"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n\nBayesian Network: \n - Directed graph where each node contains probabilities\n - Nodes represent random variables (can be discrete or continuous)\n - Directed acyclic graph\n - Each node contains a conditional probability distribution (P(Xi|Parents(Xi)))\n\nConditioning Case: Possible combination of values the parent nodes could take on\n\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n\nBayesian Network Semantics\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n\nRepresentation of Bayesian Networks\n\n\n## Module 11 Lecture Notes\n\nBayes' nets are simpler and easier to think about when they reflect true causal patterns. However, they don't need to actually be causal, especially if variables are missing. When no causality exists, arrows reflect correlation. Topology may encode causal structure, but what it is actually encoding is conditional independence\n\nJoint distribution over N boolean variables has size 2^N\nAn N-node net with nodes having up to k parents has O(N * 2^(k+1))\n\nThree probability relationships:\n - Causal chains\n - Common cause\n - Common effect\n\nCheck all paths between two nodes, if one or more is active then independence is not guaranteed. If all paths inactive then independence guaranteed\n\n\n## Module 11 Definitions\n* **Inference** - calculating some value from a joint probability distribution\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\"\n",
        "Agent Improvement depends on:\n",
        " - Component of the agent that is being improved\n",
        " - Prior knowledge the agent has\n",
        " - How the data and component are represented\n",
        " - Feedback the agent can learn from \n",
        "\n",
        "Possible Agent Components:\n",
        " - Mapping from current state to actions\n",
        " - Ability to infer information about the world given its percept sequence\n",
        " - Information on how the world changes as well as results of taking certain actions\n",
        " - Utility function that gives a value to states\n",
        " - Action-value information that gives a value to certain actions\n",
        " - Goal states that provide some sort of utility when achieved\n",
        "\n",
        "Inductive Learning: Learn a general function from a series of input-output pairs \n",
        "Deductive Learning: Take a known general rule, and modify it to a new rule using logic\n",
        "\n",
        "Unsupervised Learning: Learning from unlabeled data \n",
        "Reinforcement Learning: Learning through feedback\n",
        "Supervised Learning: Learning from labeled data \n",
        "Semi-supervised Learning: Some data is labeled, much of it is not\n",
        "\"\"\"\n",
        "chapter19_2 = \"\"\"\n",
        "Supervised Learning:\n",
        " - How to choose between multiple consistent hypotheses?\n",
        " - Tradeoff between complex models that fit the training data, and simpler models that generalize better\n",
        " - Tradeoff between hypothesis space expressiveness and time complexity for finding a good hypothesis within the space\n",
        "\"\"\"\n",
        "chapter19_3 = \"\"\"\n",
        "Add decision tree pseudocode\n",
        "\n",
        "Learning Curve: Illustrates the tradeoff between training size and test set accuracy\n",
        "\n",
        "Entropy measures the uncertainty within a random variable. Random variables with more predictable outcomes have lower entropy, and vice versa. Information \\\n",
        "gain is the expected reduction in entropy\n",
        "\n",
        "Decision Tree pruning is done to combat overfitting, works by getting rid of nodes at the bottom of a tree that seem to be irrelevant. Early stopping \\\n",
        "also helps with overfitting\n",
        "\"\"\"\n",
        "chapter19_4 = \"\"\"\n",
        "Holdout cross validation - Training set and test set\n",
        "K-fold cross validation - Split data into k sets, each one is used once as a test set while all others are used as training, increases computation time k fold\n",
        "\n",
        "Regularization: Used to penalize complex functions\n",
        "\"\"\"\n",
        "chapter19_5 = \"\"\"\n",
        "PAC: Probably approximately correct if it holds for a large enough sample size\n",
        "\"\"\"\n",
        "chapter19_6 = \"\"\"\n",
        "Linear Models\n",
        "\"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\"\n",
        "\"\"\"\n",
        "slp_chapter4_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_7 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_9 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_10 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\"\n",
        "Perceptron properties:\n",
        " - Separability (true if model is perfect on training set)\n",
        " - Convergence (will converge eventually if training data is linearly seperable)\n",
        " - Mistakes bounded (binary case maximum mistakes is less than k/(margin^2))\n",
        "\n",
        "Support vector machines:\n",
        " - Maximize margin\n",
        " - Only support vectors matter, can ignore other training examples\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"hypothesis space\" : \"set of possible functions\",    \n",
        "\"consistent\" : \"fits all of the data points exactly\",  \n",
        "\"occam's razor\" : \"best solution is usually the simplest one\",  \n",
        "\"realizable\" : \"true function is contained within the hypothesis space\",  \n",
        "}\n",
        "\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "31d6f6e1-ecf3-426d-ab28-7658a67f7026"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\nAgent Improvement depends on:\n - Component of the agent that is being improved\n - Prior knowledge the agent has\n - How the data and component are represented\n - Feedback the agent can learn from \n\nPossible Agent Components:\n - Mapping from current state to actions\n - Ability to infer information about the world given its percept sequence\n - Information on how the world changes as well as results of taking certain actions\n - Utility function that gives a value to states\n - Action-value information that gives a value to certain actions\n - Goal states that provide some sort of utility when achieved\n\nInductive Learning: Learn a general function from a series of input-output pairs \nDeductive Learning: Take a known general rule, and modify it to a new rule using logic\n\nUnsupervised Learning: Learning from unlabeled data \nReinforcement Learning: Learning through feedback\nSupervised Learning: Learning from labeled data \nSemi-supervised Learning: Some data is labeled, much of it is not\n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n\nSupervised Learning:\n - How to choose between multiple consistent hypotheses?\n - Tradeoff between complex models that fit the training data, and simpler models that generalize better\n - Tradeoff between hypothesis space expressiveness and time complexity for finding a good hypothesis within the space\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n\nAdd decision tree pseudocode\n\nLearning Curve: Illustrates the tradeoff between training size and test set accuracy\n\nEntropy measures the uncertainty within a random variable. Random variables with more predictable outcomes have lower entropy, and vice versa. Information gain is the expected reduction in entropy\n\nDecision Tree pruning is done to combat overfitting, works by getting rid of nodes at the bottom of a tree that seem to be irrelevant. Early stopping also helps with overfitting\n\n\n## AIMA Chapter 5.5 Stochastic Games\n\nHoldout cross validation - Training set and test set\nK-fold cross validation - Split data into k sets, each one is used once as a test set while all others are used as training, increases computation time k fold\n\nRegularization: Used to penalize complex functions\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nPAC: Probably approximately correct if it holds for a large enough sample size\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nLinear Models\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n\nPerceptron properties:\n - Separability (true if model is perfect on training set)\n - Convergence (will converge eventually if training data is linearly seperable)\n - Mistakes bounded (binary case maximum mistakes is less than k/(margin^2))\n\nSupport vector machines:\n - Maximize margin\n - Only support vectors matter, can ignore other training examples\n\n\n## Module 12 Definitions\n* **Hypothesis space** - set of possible functions\n* **Consistent** - fits all of the data points exactly\n* **Occam's razor** - best solution is usually the simplest one\n* **Realizable** - true function is contained within the hypothesis space\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\"\n",
        "Generative Model: Attempts to find a model that learns about or can generate individual classes\n",
        "Discriminative Model: Attempts to find a boundary to distinguish classes, doesn't necessarily need to learn about the classes themselves\n",
        "\n",
        "Logistic Regression:\n",
        " - Discriminative \n",
        " - Often works better on larger datasets\n",
        "\n",
        "Naive Bayes:\n",
        " - Generative\n",
        " - Can work very well on small datasets\n",
        " - Easy to implement and can be trained quickly\n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\"\n",
        "Learning requires a cost function and an optimization algorithm\n",
        "\"\"\"\n",
        "slp_chapter5_3 = \"\"\"\n",
        "Loss functions should return small values for mostly correct guesses, and large values for mostly incorrect guesses\n",
        "\"\"\"\n",
        "slp_chapter5_4 = \"\"\"\n",
        "Gradient: vector showing the direction in which the loss is increasing the most \n",
        "Gradient descent: Move in the direction opposite the gradient\n",
        "Learning rate: How fast to adjust the weight vector\n",
        "\n",
        "Stochastic training: Compute gradient after each data point (fast, inaccurate)\n",
        "Batch training: Compute gradient over all data points, then update (slow, accurate)\n",
        "Mini-batch training: Compute gradient over subset of data points, then update (compromise)\n",
        "\"\"\"\n",
        "slp_chapter5_5 = \"\"\"\n",
        "Regularization: used to penalize large weights and prevent overfitting. Means a model with many high value weights that fits the training data \\\n",
        "very well will be penalized more than one that doesnt fit the training data as well but has fewer weights, each with smaller values\n",
        "\n",
        "L1 Regularization: \n",
        " - Manhattan distance\n",
        " - Lasso regression\n",
        " - Harder to optimize (derivative non-continuous at zero)\n",
        " - Prefers sparse solutions where some weights can be large but many are 0\n",
        " - Leads to fewer features\n",
        "\n",
        "L2 Regularization: \n",
        " - Euclidean distance\n",
        " - Ridge regression\n",
        " - Easier to optimize (simple derivative)\n",
        " - Prefers weight vectors where most of the weights are very small\n",
        " - Leads to more features\n",
        "\"\"\"\n",
        "slp_chapter5_6 = \"\"\"\n",
        "Softmax function: Generalization of the sigmoid function to multiple classes\n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\"\n",
        "Activation Functions provide nonlinearity to neural networks\n",
        "\n",
        "Sigmoid:\n",
        " - Not commonly used\n",
        " - Range [0, 1]\n",
        "\n",
        "Tanh:\n",
        " - Similar but almost always better than sigmoid\n",
        " - Range [-1, 1]\n",
        " - Smoothly differentiable\n",
        " - Maps outliers towards the mean\n",
        "\n",
        "ReLU:\n",
        " - Simple, commonly used\n",
        " - Range [0, 1]\n",
        " - Nearly linear\n",
        " - No vanishing gradient problem (derivative is 1 for all high values of z)\n",
        "\n",
        "Vanishing Gradient Problem: When values of z get too high, activation functions such as sigmoid and tanh tend to output values very close to 1, \\\n",
        "and thus derivatives very close to 0. When these derivatives are multiplied together, the value reaches near 0, and the gradient becomes too small \\\n",
        "for learning to continue. \n",
        "\"\"\"\n",
        "slp_chapter7_2 = \"\"\"\n",
        "Neural networks can solve problems (such as XOR) that are not linearly separable when they have nonlinear activation functions\n",
        "\"\"\"\n",
        "slp_chapter7_3 = \"\"\"\n",
        "Feedforward networks consist of an input layer, hidden layer(s), and an output layer\n",
        "\"\"\"\n",
        "slp_chapter7_4 = \"\"\"\n",
        "Pretraining: Using another algorithm to learn an embedding representation for the input, which can then be passed into the next algorithm\n",
        "\"\"\"\n",
        "slp_chapter7_5 = \"\"\"\n",
        "Neural Language Model compared to n-gram models:\n",
        " - Can handle longer word histories\n",
        " - Generalize better\n",
        " - More accurate word predictions\n",
        " - Much more complex\n",
        " - Slower\n",
        " - Less interpretable\n",
        " - Generally better for larger tasks, whereas n-gram are generally better for smaller tasks\n",
        "\"\"\"\n",
        "slp_chapter7_6 = \"\"\"\n",
        "Hyperparameters:\n",
        " - Learning Rate\n",
        " - Mini-batch size\n",
        " - Model architecture\n",
        " - Loss function\n",
        " - Optimizer\n",
        "\"\"\"\n",
        "slp_chapter7_7 = \"\"\"\n",
        "Training neural language model\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\"\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "b8c096a3-0a5c-4649-e58f-a008d5ba3e9e"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\nGenerative Model: Attempts to find a model that learns about or can generate individual classes\nDiscriminative Model: Attempts to find a boundary to distinguish classes, doesn't necessarily need to learn about the classes themselves\n\nLogistic Regression:\n - Discriminative \n - Often works better on larger datasets\n\nNaive Bayes:\n - Generative\n - Can work very well on small datasets\n - Easy to implement and can be trained quickly\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n\nLearning requires a cost function and an optimization algorithm\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\nLoss functions should return small values for mostly correct guesses, and large values for mostly incorrect guesses\n\n\n## SLP Chapter 5.4 Gradient Descent\n\nGradient: vector showing the direction in which the loss is increasing the most \nGradient descent: Move in the direction opposite the gradient\nLearning rate: How fast to adjust the weight vector\n\nStochastic training: Compute gradient after each data point (fast, inaccurate)\nBatch training: Compute gradient over all data points, then update (slow, accurate)\nMini-batch training: Compute gradient over subset of data points, then update (compromise)\n\n\n## SLP Chapter 5.5 Regularization\n\nRegularization: used to penalize large weights and prevent overfitting. Means a model with many high value weights that fits the training data very well will be penalized more than one that doesnt fit the training data as well but has fewer weights, each with smaller values\n\nL1 Regularization: \n - Manhattan distance\n - Lasso regression\n - Harder to optimize (derivative non-continuous at zero)\n - Prefers sparse solutions where some weights can be large but many are 0\n - Leads to fewer features\n\nL2 Regularization: \n - Euclidean distance\n - Ridge regression\n - Easier to optimize (simple derivative)\n - Prefers weight vectors where most of the weights are very small\n - Leads to more features\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n\nSoftmax function: Generalization of the sigmoid function to multiple classes\n\n\n## SLP Chapter 7.1 Units\n\nActivation Functions provide nonlinearity to neural networks\n\nSigmoid:\n - Not commonly used\n - Range [0, 1]\n\nTanh:\n - Similar but almost always better than sigmoid\n - Range [-1, 1]\n - Smoothly differentiable\n - Maps outliers towards the mean\n\nReLU:\n - Simple, commonly used\n - Range [0, 1]\n - Nearly linear\n - No vanishing gradient problem (derivative is 1 for all high values of z)\n\nVanishing Gradient Problem: When values of z get too high, activation functions such as sigmoid and tanh tend to output values very close to 1, and thus derivatives very close to 0. When these derivatives are multiplied together, the value reaches near 0, and the gradient becomes too small for learning to continue. \n\n\n## SLP Chapter 7.2 The XOR problem\n\nNeural networks can solve problems (such as XOR) that are not linearly separable when they have nonlinear activation functions\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n\nFeedforward networks consist of an input layer, hidden layer(s), and an output layer\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n\nPretraining: Using another algorithm to learn an embedding representation for the input, which can then be passed into the next algorithm\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n\nNeural Language Model compared to n-gram models:\n - Can handle longer word histories\n - Generalize better\n - More accurate word predictions\n - Much more complex\n - Slower\n - Less interpretable\n - Generally better for larger tasks, whereas n-gram are generally better for smaller tasks\n\n\n## SLP Chapter 7.6 Training Neural Nets\n\nHyperparameters:\n - Learning Rate\n - Mini-batch size\n - Model architecture\n - Loss function\n - Optimizer\n\n\n## SLP Chapter 7.7 Training the neural language model\n\nTraining neural language model\n\n\n## Module 13 Lecture Notes\n\n\n\n## Module 13 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\"\n",
        "Perplexity: Intrinisic measure for language models, calculated as the inverse probability of the test set, normalized by the length of the test set\n",
        "\"\"\"\n",
        "slp_chapter9_2 = \"\"\"\n",
        "Recurrent Neural Network: Neural network that contains some sort of cycle within the connected layers, i.e. the input to a layer is directly or indirectly \\\n",
        "affected by the layer's own earlier outputs\n",
        "\"\"\"\n",
        "slp_chapter9_3 = \"\"\"\n",
        "RNNs as language models:\n",
        " - Process input sequence a single word at a time\n",
        " - Predicts next word based on current word and previous hidden state\n",
        " - No limited context problem (such problem occurs in n-gram models) since hidden state can extend all the way to the start\n",
        "\"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\"\n",
        "Lexical semantics\n",
        "\"\"\"\n",
        "slp_chapter6_2 = \"\"\"\n",
        "Vector semantics\n",
        "\"\"\"\n",
        "slp_chapter6_3 = \"\"\"\n",
        "Term-document matrix: rows of words, columns of documents, with each cell representing how many times a given word occurs in a given document\n",
        "\n",
        "Term-term matrix: rows of words, columns of words, with each cell representing how often the row (target) word appears in some context with the column (context) word \\\n",
        "within some training corpus\n",
        "\"\"\"\n",
        "slp_chapter6_4 = \"\"\"\n",
        "Cosine similarity:\n",
        " - Calculates how similar two words are\n",
        " - Values between 0 and 1\n",
        " - Adjusts for word frequency\n",
        "\"\"\"\n",
        "slp_chapter6_8 = \"\"\"\n",
        "Skip-gram with negative sampling (SGNS, word2vec) learns two embeddings for each word, one with the word as the target and another with the word as \\\n",
        "the context. The learned weights are used as the embeddings that are then passed into other language models\n",
        "\"\"\"\n",
        "slp_chapter6_10 = \"\"\"\n",
        "First-order co-occurence (syntagmic association): Words that are typically near each other\n",
        "Second-order co-occurence (paradigmatic association): Words that have similar neighbors\n",
        "\"\"\"\n",
        "slp_chapter6_11 = \"\"\"\n",
        "Embeddings can often amplify bias\n",
        "\"\"\"\n",
        "slp_chapter6_12 = \"\"\"\n",
        "Similarity: Intrinsic evaluation method, compare word similarity output to human created ratings\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\"\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "48c85235-e4ac-4e73-b458-1cd45804ddec"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n\nPerplexity: Intrinisic measure for language models, calculated as the inverse probability of the test set, normalized by the length of the test set\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n\nRecurrent Neural Network: Neural network that contains some sort of cycle within the connected layers, i.e. the input to a layer is directly or indirectly affected by the layer's own earlier outputs\n\n\n## SLP Chapter 9.3 RNNs as Language Models\n\nRNNs as language models:\n - Process input sequence a single word at a time\n - Predicts next word based on current word and previous hidden state\n - No limited context problem (such problem occurs in n-gram models) since hidden state can extend all the way to the start\n\n\n## SLP Chapter 6.1 Lexical Semantics\n\nLexical semantics\n\n\n## SLP Chapter 6.2 Vector Semantics\n\nVector semantics\n\n\n## SLP Chapter 6.3 Words and Vectors\n\nTerm-document matrix: rows of words, columns of documents, with each cell representing how many times a given word occurs in a given document\n\nTerm-term matrix: rows of words, columns of words, with each cell representing how often the row (target) word appears in some context with the column (context) word within some training corpus\n\n\n## SLP Chapter 6.4 Cosine for measuring similarity\n\nCosine similarity:\n - Calculates how similar two words are\n - Values between 0 and 1\n - Adjusts for word frequency\n\n\n## SLP Chapter 6.8 Word2vec\n\nSkip-gram with negative sampling (SGNS, word2vec) learns two embeddings for each word, one with the word as the target and another with the word as the context. The learned weights are used as the embeddings that are then passed into other language models\n\n\n## SLP Chapter 6.10 Semantic properties of embeddings\n\nFirst-order co-occurence (syntagmic association): Words that are typically near each other\nSecond-order co-occurence (paradigmatic association): Words that have similar neighbors\n\n\n## SLP Chapter 6.11 Bias and Embeddings\n\nEmbeddings can often amplify bias\n\n\n## SLP Chapter 6.12 Evaluating Vector Models\n\nSimilarity: Intrinsic evaluation method, compare word similarity output to human created ratings\n\n\n## Module 13 Lecture Notes\n\n\n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "T3cm1WV542_a",
        "outputId": "5345d8d2-459c-4a4c-f167-d4ff44c33633"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nPrinciple of maximum expected utility - an agent should choose the action that gives the highest expected utility.\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nList of constraints we expect a reasonable preference relation to obey (Axioms of utility theory):\n - Orderability (Given two lotteries, either one is preferred or they are equally preferable)\n - Transitivity (If A is preferred to B, and B to C, then it follows A is preferred to C)\n - Continuity (For a lottery B between A and C in preference, there exists a probability p in which the agent is indifferent to B, A with probability p,  and C with probability 1-p)\n - Substitutability (If the agent is indifferent between A and B, one can be substituted for the other with no change in the preference relation)\n - Monotonicity (For two lotteries with the same outcomes A and B, if outcome A is preferred, the lottery with higher probability of A must also be preferred)\n - Decomposability (Laws of probability can be used to reduce complex lotteries into simpler ones, aka \"no fun in gambling rule\")\n\n\n## AIMA Chapter 16.3 Utility Functions\n\nCertainty effect - people like gains that are certain to be realized\nAmbiguity aversion - people like known probabilities rather than unknown ones\nFraming effect - wording makes a difference in perception\nAnchoring effect - sometimes the actual utility doesnt matter as much as the utility in relation to other choices\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\nMarkov Decision Process: \n - Fully observable/stochastic environment, Markovian transition model, additive rewards\n - Contains set of states, actions, transition model, and reward function\n - Solution to this problem is known as a policy, which determines the best action to take at a given state\n - The optimal policy is the one that gives the highest expected utility \n\nFinite Horizon: Game ends after a certain amount of turns/time, optimal policy is nonstationary\nInfinite Horizon: Game can go on forever, optimal policy is stationary\n\nAdditive Rewards: Total reward increases by adding the reward for reaching consecutive states with no penalty\nDiscounted Rewards: Total reward increases, but rewards for reaching later states are reduced, aka immediate rewards are worth more than future rewards\n\nProper policy: Policy guaranteed to reach a terminal state\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\nValue Iteration: Iteratively update the values of each state using Bellman equation\n\nPolicy Iteration: Alternates between two steps\n - Policy Evaluation (Given current policy calculate utility of each state)\n - Policy Improvement (Calculate a new policy based on the new utilities)\n\nModified Policy Iteration: More efficient normally than both regular policy iteration and value iteration, get next utility estimate by repeating bellman update k times\nAsynchronous Policy Iteration: Instead of updating all states at once, modify a subset of states at a time, guaranteed to converge to optimal policy under certain conditions\n\n\n## Module 8 Lecture Notes\n\nExpectimax node: outcome is uncertain, expected utility is calculated\n\nValue Iteration:\n - Time complexity is O(S^2A)\n - Converges to unique optimal values\n - Policy may converge many steps before values do\n\nPolicy Iteration: \n - Time complexity is O(S^2)\n\n\n## Module 8 Definitions\n* **Utility function** - assigns a value determining how good a state is\n* **Expected utility** - average utility value of all outcomes given a chosen action\n* **Lottery** - set of outcomes with associated probabilites for a given action\n* **Ordinal utility function** - function that gives a preference ranking to a set of states\n* **Preference elicitation** - determining an agent's utility function through presenting choices and observing preferences\n* **Optimizer's curse** - expected utility is often overestimated\n* **Sequential decision problems** - utility is determined by a sequence of actions\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n\nBandit Problem: Given a set of possible options, which one do you take - the one that has given the best rewards so far, or one of the ones that has not been tried?\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n\nReinforcement Learning: Goal is to learn an optimal policy based on observed rewards, for many games, state spaces can be far too large for regular MDP computation, so reinforcement learning methods must be used\n\nUtility-based agent: Learns a utility function that is used to select actions for a given state based on maximum expected utility\nQ-learning agent: Learns an action-utility function (Q-function), which given a state and action returns the expected utility\nReflex agent: For a given state, it has a fixed given action\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\nPassive Reinforcement Learning: Agent operates using a fixed policy, goal is to learn state utilities/a model of the environment. Similar to policy evaluation, but the transition model as well as the reward function are not known\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\nActive Reinforcement Learning: Agent must learn about the environment, and then also learn what actions to take\n\nExploration/Exploitation Tradeoff: Tradeoff an agent must make between choosing the best path it knows at a given moment and exploring it further, and choosing a path that isn't optimal according to it's current policy, but may actually end up being better after exploring it. \n\nQ-Learning:\n - Model-free method (does not need a transition model for learning or action selection)\n - Assigns a values, known as the q-value, to state-action pairs\n - Off-policy learning algorithm\n\nSARSA Learning:\n - Q-learning goes off of the best Q-value from the reached state, whereas SARSA waits until an action is taken and then uses the Q-value for that action\n - Identical with greedy agents, different when exploration is involved\n - On-policy learning algorithm \n - More realistic, sometimes it is better to learn through what happens rather than the best thing that can happen\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\nFunction Approximation: A way to help a reinforcement learning algorithm generalize by using a representation other than a lookup table. Also helps as some games have far too many states to store in a lookup table. Works by creating a compressed representation, which can also help the model choose actions for states it has not seen before based on information it has gotten from similar states\n\nBest to use an online learning algorithm that updates the model every time a new trial is completed\n\n\n## Module 9 Lecture Notes\n\nImportant reinforcement learning ideas:\n - Exploration (Must explore the unknown to get information)\n - Exploitation (Want to use the knowledge to choose the best actions)\n - Regret (Even with intelligent learning, you will make mistakes)\n - Sampling (Due to stochastic events, must try things repeatedly)\n - Difficulty (Learning can be much harder than solving MDPs)\n\nModel-based learning:\n - Build a model based on experiences\n - Step 1, learn an empirical MDP model (count outcomes, normalize, estimate transition model and reward model)\n - Step 2, solve learned MDP\n\nQ-learning will converge to the optimal policy, even if you aren't acting optimally\n\nRegret measures total mistake cost, calculated as the difference between expected rewards and optimal rewards\n\nApproximate Q-Learning:\n - Feature representation allows us to write a q function as the product of feature vector and weight vector\n - Advantage, experience can be summed up in a few numbers\n - Disadvantage, states with similar features may actually be very different in terms of value\n\n\n## Module 9 Definitions\n* **Reward-to-go** - expected total reward from a state and the consecutive states past it\n* **Greedy agent** - takes the best path at every step as learned by the current model, does not explore\n* **Exploration function** - decides how the trade off between exploration and exploitation is determined\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\nBelief state: Representation of the possible states an agent may be in\nIssues with this:\n - When given partial information, the agent must consider all possible explanation, even unlikely ones. This can lead to very large and too complex to handle belief-state spaces\n - Handling every possible eventuality means considering the unlikely, and can again get too large to handle\n - There may be no solution, and there may be no way to compare which plan to take given that none of them reach the solution\n\nQualification Problem: if there are possible things that can go wrong that cannot be known if they will occur, success of a plan cannot be determined\n\nRational Decision: Pick the best action, determined based on the relative importance of possible goal states as well as the probability and degree that each one will be achieved\n\nProbability Theory: Way to summarize and deal with uncertainty\n\nUtility Theory: Way to assign value to certain states, helping us decide which ones are more desirable than others\n\nDecision Theory = Probability Theory + Utility Theory\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n\nProbability Overview\n\n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n\nFrequentist view - Numbers come from experiments\nObjectivist view - Probabilities are universal aspects that can be observed (possibly through experiments)\nSubjectivist view - Probabilities quantify an agent's beliefs\n\nMarginalization - Probability of variable value given by summing over other variables and their probabilities\nConditioning - Variant of marginalization involving conditional probability\nNormalization - Ensures probabilities add up to 1\n\n\n## AIMA Chapter 12.4 Independence\n\nIndependence in probability\n\n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\nBayes' Theorem: Deals with conditional probability\n\nNaive Bayes Model assumes that a single cause can lead to a multitude of conditionally independent effects. Can often be used in cases where the effects are not actually conditionally independent.\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\nWumpus World\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n\nSummary\n\n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n\nProblems with n-gram model MLE:\n - Out of vocabulary items (handled with unknown token)\n - Fixed vocab \n - Any word not in vocab replaced with unknown token\n - Zeros avoided with smoothing (Laplace?)\n - Use log space to avoid numerical underflow\n\n\n## Module 10 Definitions\n* **Sample space** - set of possible outcomes\n* **Probability model** - assigns a probability to each possible outcome\n* **Prior probability** - probability with no information\n* **Conditional/posterior probability** - probability given some information about an outcome that has already occured\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n\nBayesian Network: \n - Directed graph where each node contains probabilities\n - Nodes represent random variables (can be discrete or continuous)\n - Directed acyclic graph\n - Each node contains a conditional probability distribution (P(Xi|Parents(Xi)))\n\nConditioning Case: Possible combination of values the parent nodes could take on\n\n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n\nBayesian Network Semantics\n\n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n\nRepresentation of Bayesian Networks\n\n\n## Module 11 Lecture Notes\n\nBayes' nets are simpler and easier to think about when they reflect true causal patterns. However, they don't need to actually be causal, especially if variables are missing. When no causality exists, arrows reflect correlation. Topology may encode causal structure, but what it is actually encoding is conditional independence\n\nJoint distribution over N boolean variables has size 2^N\nAn N-node net with nodes having up to k parents has O(N * 2^(k+1))\n\nThree probability relationships:\n - Causal chains\n - Common cause\n - Common effect\n\nCheck all paths between two nodes, if one or more is active then independence is not guaranteed. If all paths inactive then independence guaranteed\n\n\n## Module 11 Definitions\n* **Inference** - calculating some value from a joint probability distribution\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n\nAgent Improvement depends on:\n - Component of the agent that is being improved\n - Prior knowledge the agent has\n - How the data and component are represented\n - Feedback the agent can learn from \n\nPossible Agent Components:\n - Mapping from current state to actions\n - Ability to infer information about the world given its percept sequence\n - Information on how the world changes as well as results of taking certain actions\n - Utility function that gives a value to states\n - Action-value information that gives a value to certain actions\n - Goal states that provide some sort of utility when achieved\n\nInductive Learning: Learn a general function from a series of input-output pairs \nDeductive Learning: Take a known general rule, and modify it to a new rule using logic\n\nUnsupervised Learning: Learning from unlabeled data \nReinforcement Learning: Learning through feedback\nSupervised Learning: Learning from labeled data \nSemi-supervised Learning: Some data is labeled, much of it is not\n\n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n\nSupervised Learning:\n - How to choose between multiple consistent hypotheses?\n - Tradeoff between complex models that fit the training data, and simpler models that generalize better\n - Tradeoff between hypothesis space expressiveness and time complexity for finding a good hypothesis within the space\n\n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n\nAdd decision tree pseudocode\n\nLearning Curve: Illustrates the tradeoff between training size and test set accuracy\n\nEntropy measures the uncertainty within a random variable. Random variables with more predictable outcomes have lower entropy, and vice versa. Information gain is the expected reduction in entropy\n\nDecision Tree pruning is done to combat overfitting, works by getting rid of nodes at the bottom of a tree that seem to be irrelevant. Early stopping also helps with overfitting\n\n\n## AIMA Chapter 5.5 Stochastic Games\n\nHoldout cross validation - Training set and test set\nK-fold cross validation - Split data into k sets, each one is used once as a test set while all others are used as training, increases computation time k fold\n\nRegularization: Used to penalize complex functions\n\n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n\nPAC: Probably approximately correct if it holds for a large enough sample size\n\n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n\nLinear Models\n\n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n\n\n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n\nPerceptron properties:\n - Separability (true if model is perfect on training set)\n - Convergence (will converge eventually if training data is linearly seperable)\n - Mistakes bounded (binary case maximum mistakes is less than k/(margin^2))\n\nSupport vector machines:\n - Maximize margin\n - Only support vectors matter, can ignore other training examples\n\n\n## Module 12 Definitions\n* **Hypothesis space** - set of possible functions\n* **Consistent** - fits all of the data points exactly\n* **Occam's razor** - best solution is usually the simplest one\n* **Realizable** - true function is contained within the hypothesis space\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n\nGenerative Model: Attempts to find a model that learns about or can generate individual classes\nDiscriminative Model: Attempts to find a boundary to distinguish classes, doesn't necessarily need to learn about the classes themselves\n\nLogistic Regression:\n - Discriminative \n - Often works better on larger datasets\n\nNaive Bayes:\n - Generative\n - Can work very well on small datasets\n - Easy to implement and can be trained quickly\n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n\nLearning requires a cost function and an optimization algorithm\n\n\n## SLP Chapter 5.3 The cross-entropy loss function\n\nLoss functions should return small values for mostly correct guesses, and large values for mostly incorrect guesses\n\n\n## SLP Chapter 5.4 Gradient Descent\n\nGradient: vector showing the direction in which the loss is increasing the most \nGradient descent: Move in the direction opposite the gradient\nLearning rate: How fast to adjust the weight vector\n\nStochastic training: Compute gradient after each data point (fast, inaccurate)\nBatch training: Compute gradient over all data points, then update (slow, accurate)\nMini-batch training: Compute gradient over subset of data points, then update (compromise)\n\n\n## SLP Chapter 5.5 Regularization\n\nRegularization: used to penalize large weights and prevent overfitting. Means a model with many high value weights that fits the training data very well will be penalized more than one that doesnt fit the training data as well but has fewer weights, each with smaller values\n\nL1 Regularization: \n - Manhattan distance\n - Lasso regression\n - Harder to optimize (derivative non-continuous at zero)\n - Prefers sparse solutions where some weights can be large but many are 0\n - Leads to fewer features\n\nL2 Regularization: \n - Euclidean distance\n - Ridge regression\n - Easier to optimize (simple derivative)\n - Prefers weight vectors where most of the weights are very small\n - Leads to more features\n\n\n## SLP Chapter 5.6 Multinomial logistic regression\n\nSoftmax function: Generalization of the sigmoid function to multiple classes\n\n\n## SLP Chapter 7.1 Units\n\nActivation Functions provide nonlinearity to neural networks\n\nSigmoid:\n - Not commonly used\n - Range [0, 1]\n\nTanh:\n - Similar but almost always better than sigmoid\n - Range [-1, 1]\n - Smoothly differentiable\n - Maps outliers towards the mean\n\nReLU:\n - Simple, commonly used\n - Range [0, 1]\n - Nearly linear\n - No vanishing gradient problem (derivative is 1 for all high values of z)\n\nVanishing Gradient Problem: When values of z get too high, activation functions such as sigmoid and tanh tend to output values very close to 1, and thus derivatives very close to 0. When these derivatives are multiplied together, the value reaches near 0, and the gradient becomes too small for learning to continue. \n\n\n## SLP Chapter 7.2 The XOR problem\n\nNeural networks can solve problems (such as XOR) that are not linearly separable when they have nonlinear activation functions\n\n\n## SLP Chapter 7.3 Feedforward Neural Networks\n\nFeedforward networks consist of an input layer, hidden layer(s), and an output layer\n\n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n\nPretraining: Using another algorithm to learn an embedding representation for the input, which can then be passed into the next algorithm\n\n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n\nNeural Language Model compared to n-gram models:\n - Can handle longer word histories\n - Generalize better\n - More accurate word predictions\n - Much more complex\n - Slower\n - Less interpretable\n - Generally better for larger tasks, whereas n-gram are generally better for smaller tasks\n\n\n## SLP Chapter 7.6 Training Neural Nets\n\nHyperparameters:\n - Learning Rate\n - Mini-batch size\n - Model architecture\n - Loss function\n - Optimizer\n\n\n## SLP Chapter 7.7 Training the neural language model\n\nTraining neural language model\n\n\n## Module 13 Lecture Notes\n\n\n\n## Module 13 Definitions\n* **Concept** - definition\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n\nPerplexity: Intrinisic measure for language models, calculated as the inverse probability of the test set, normalized by the length of the test set\n\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n\nRecurrent Neural Network: Neural network that contains some sort of cycle within the connected layers, i.e. the input to a layer is directly or indirectly affected by the layer's own earlier outputs\n\n\n## SLP Chapter 9.3 RNNs as Language Models\n\nRNNs as language models:\n - Process input sequence a single word at a time\n - Predicts next word based on current word and previous hidden state\n - No limited context problem (such problem occurs in n-gram models) since hidden state can extend all the way to the start\n\n\n## SLP Chapter 6.1 Lexical Semantics\n\nLexical semantics\n\n\n## SLP Chapter 6.2 Vector Semantics\n\nVector semantics\n\n\n## SLP Chapter 6.3 Words and Vectors\n\nTerm-document matrix: rows of words, columns of documents, with each cell representing how many times a given word occurs in a given document\n\nTerm-term matrix: rows of words, columns of words, with each cell representing how often the row (target) word appears in some context with the column (context) word within some training corpus\n\n\n## SLP Chapter 6.4 Cosine for measuring similarity\n\nCosine similarity:\n - Calculates how similar two words are\n - Values between 0 and 1\n - Adjusts for word frequency\n\n\n## SLP Chapter 6.8 Word2vec\n\nSkip-gram with negative sampling (SGNS, word2vec) learns two embeddings for each word, one with the word as the target and another with the word as the context. The learned weights are used as the embeddings that are then passed into other language models\n\n\n## SLP Chapter 6.10 Semantic properties of embeddings\n\nFirst-order co-occurence (syntagmic association): Words that are typically near each other\nSecond-order co-occurence (paradigmatic association): Words that have similar neighbors\n\n\n## SLP Chapter 6.11 Bias and Embeddings\n\nEmbeddings can often amplify bias\n\n\n## SLP Chapter 6.12 Evaluating Vector Models\n\nSimilarity: Intrinsic evaluation method, compare word similarity output to human created ratings\n\n\n## Module 13 Lecture Notes\n\n\n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}