{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c3ad73fd-9236-4d9d-b713-e0b52176fe11"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5e52bb0d-9d13-4818-8787-9ad2eae048de"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = True"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" \n",
        "Based on the MEU, the agent will choose actions based on what action gives the most expected utility.\n",
        "By choosing the highest expected utility, there is a higher chance the agent will succeed.\n",
        "\n",
        "\"\"\" + sample_equation\n",
        "chapter16_2 =  \"\"\" \n",
        "A > B means A is preferred over B. A ~ B means that A or B is good, indifferent. A >= B means that\n",
        "either A is preferred over B, or either is good.\n",
        "\n",
        "Preference rules:\n",
        "\n",
        "1) Orderability: an agent must have a preferance or be indifferent between choices, must make a choice\n",
        "\n",
        "2) Transitivity: the transitivity property must hold, if agent has one preferance over another, and \n",
        "that other preference is preferred over a third, then the first is more preferred than the third\n",
        "\n",
        "3) Continutiy: the inbetween choice shares an indifference preference with the lower choice with \n",
        "chance p and the higher choice with chance 1-p\n",
        "\n",
        "4) Substitutability: if an agent has a preference, then that preference holds in other lotteries, \n",
        "given that everything else besides the preference stays the same in the lottery. \n",
        "\n",
        "5) Monotonicity: If two lotteries are the same, then the agent should always go for the choice that \n",
        "has a higher chance of getting the preferred lottery\n",
        "\n",
        "6) Decomposability: Can simplify more complex lotteries into simpler ones.\n",
        "\n",
        "\n",
        "Utility Functions:\n",
        "\n",
        "Existence of Utility Functions: If there is a preference, than the utility of that preference is \n",
        "greater than the utility of the other. If there is not a preference, then both choices have the \n",
        "same utility.\n",
        "\n",
        "Expected Utility: Sum of probability_i * utility of (outcome_i)\n",
        "\n",
        "\"\"\"\n",
        "chapter16_3 =  \"\"\" \n",
        "There are many utility scales in the real world such as VSL, micromort, and QALY.\n",
        "\n",
        "Monetary value is not directly proportional to utility. It's more of a log value.\n",
        "\n",
        "Rational agent theory is a popular economical theory based on how people should behave. Other types\n",
        "of theories are descriptive theories of how people actually behave. People might not always make\n",
        "the most rational choice, but they can still be reasonable choices. A lot of heuristics can affect\n",
        "decisions such as framing and anchoring effects.\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\"\n",
        "For stochastic outcomes, the probability of reaching the next state is based on the current state and\n",
        "the current action (P(s'|s,a)). Transition models can be written as T(s,a,s'). Reward functions can\n",
        "be written as R(s,a,s'). These types of problems are usually solved using DP and use recursion to\n",
        "remember the best solution so far.\n",
        "\n",
        "In finite horizon, time/the amount of moves left may factor into the optimal action. If time is not\n",
        "a factor, then it is infinite horizon. Stationary policies do not factor in time, while nonstationary\n",
        "policies as based on time.\n",
        "\n",
        "Reasons why additive discount-reward works:\n",
        "\n",
        "1): people tend to value the now over the future\n",
        "\n",
        "2): Sooner rather than later is better because investing can make more rewards\n",
        "\n",
        "3): Future rewards may never come, therefore it's better to take the current rewards\n",
        "\n",
        "4): Preferences change over time. Now is preferred more than whatever is in the future.\n",
        "\n",
        "5): No infinities\n",
        "\n",
        "5a): Discounted rewards confines infinite sequences to finite utility. Has to be less than maximum reward / (1 - discount rate)\n",
        "\n",
        "5b): If there exist a terminal state and the policy can get to the terminal state, then it will never be an infinite sequence because\n",
        "it will eventually get to the terminal state.\n",
        "\n",
        "5c): Average reward per move can be used to analyze infinite sequences. \n",
        "\n",
        "The optimal policy will work for any starting state.\n",
        "\n",
        "The optimal policy for a given state s can be found by finding the largest action corresponding to the largest Q-function utility value\n",
        "\"\"\"\n",
        "chapter17_2 =  \"\"\"\n",
        "There are four algorithms used in MDPS: value iteration, policy iteration, linear programming and Monte Carlo planning.\n",
        "\n",
        "1): Value Iteration: A bottom up iteration process that looks at the neighboring nodes all the way to the root node. Used to find the best policy for each state. (OFFLINE)\n",
        "\n",
        "2): Policy Iteration: 2 step process; policy evaluation: evalute the current policy for its expected utility of all states for the given policy; policy improvement: calculate a new improved policy based on the current policy. (OFFLINE)\n",
        "\n",
        "3): Linear Programming: General method to solve constrained optimization problems. (ONLINE)\n",
        "\n",
        "Online means heavy computation at each decision (expected-minmax) and offline means precomputed information\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\"\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"a function that gives the utility of a given state\",\n",
        "\"expected utility\" : \"the average utility based on probabilistic values\",\n",
        "\"maximum expected utility.\": \"theory that says always choose the action with the most expected utility\",\n",
        "\"lottery\": \"probability-outcome pairs for each action\",\n",
        "\"value function/ordinal utility function\": \"the preference on states in deterministic environments\",\n",
        "\"preference elicitation\": \"a way to figure out the utility function by showing a bunch of choices\",\n",
        "\"normalized utilities\": \"u-upsidedown-T (the worst) is set to 0 and u_T (the best) is set to 1\",\n",
        "\"standard lottery\": \"there's a p chance of getting the best, and 1-p chance of getting the worse\",\n",
        "\"value of a statistical life\": \"the value used to determine how much it would cost to save 1 life; around $10 million\",\n",
        "\"micromort\": \"the chance of death is equal to 1 / 1 million\",\n",
        "\"QALY (quality-adjusted-life-year)\": \"the tradeoff between a shorter life for a better life\",\n",
        "\"monotonic preferences\": \"an agent prefers more rather than less; more makes them happier\",\n",
        "\"expected monetary value\": \"basically expected value, but with money\",\n",
        "\"risk-averse\": \"don't like to take risk, prefer a sure-thing over an expect value gamble\",\n",
        "\"risk-seeking\": \"like to take risk, gambles\",\n",
        "\"certainty equivalent\": \"the amount of value needed to not take a gamble\",\n",
        "\"insurance premimum\": \"the value between the expected value and the certainity equivalent\",\n",
        "\"risk-neutral\": \"not risk-adverse nor risk-seeking, takes small risk\",\n",
        "\"unbiased\": \"no error; zero error\",\n",
        "\"order statistic\": \"statistics ordered based on the kth best value\",\n",
        "\"optimizer's curse\": \"people will overestimate the expected utility for the best choice\",\n",
        "\"normative theory\": \"theories on the most optimal ways to act\",\n",
        "\"descriptive theory\": \"theories on how people actually act\",\n",
        "\"certainty effect\": \"people like sure things, 100% events\",\n",
        "\"ambiguity aversion\": \"people like known probabilities\",\n",
        "\"framing effect\": \"the way a problem is asked can affect the person's choice\",\n",
        "\"anchoring effect\": \"people tend to make choices around other choices; ex: expensive items tend to make less expensive items more attractive\",\n",
        "\"evolutionary psychology\": \"humans did not evolve to make the best mathematical decisions\",\n",
        "\"sequential decision problems\": \"the overall utility based on all the decisions made in the sequence\",\n",
        "\"multi-armed bandit\": \"a slot machine with multi-arms and different probabilities for results\",\n",
        "\"Markovian\": \"the probability of reaching the next state depends only on the previous state\",\n",
        "\"environment history\": \"all states and actions that the agent took\",\n",
        "\"reward\": \"the value recieved when reaching a new state\",\n",
        "\"Markov decision process (MDP)\": \"problems that have markovian transition models, probablistic environments, and rewards. Usually solved using DP, recursive process.\",\n",
        "\"policy\": \"a list of actions that tells the agent what to do at every given state\",\n",
        "\"optimal policy\": \"the best policy that gives the highest expect utility based on environment history\",\n",
        "\"finite horizon\": \"there is a fixed amount of moves/time and then times up\",\n",
        "\"infinite horizon\": \"there is no fixed amount of moves/time, never ends\",\n",
        "\"nonstationary\": \"when one of the deciding factors of a policy is reliant on time\",\n",
        "\"stationary\": \"when the policy does not factor in time and only uses the information of the current state\",\n",
        "\"additive discounted-reward\": \"a method used to give bigger reward to near future moves rather than distant future moves, exponential decrease\",\n",
        "\"discount factor\": \"a value between 0-1; larger for near future rewards and smaller for distant future rewards\",\n",
        "\"additive rewards\": \"a special case when the discount value reaches 1, you get the full reward\",\n",
        "\"attribute\": \"part of the history\",\n",
        "\"proper policy\": \"a policy that has a guaranteed path to the end state\",\n",
        "\"Bellman equation\": \"the utility of a state based on the maximum value of the sum of the transition * reward + discount*utility of next\",\n",
        "\"action-utility function/Q-funtion\": \"how much utility is gained by taking an action a in a state s\",\n",
        "\"shapping theorem\": \"a specific transformation of reward + potential will not change the optimal policy for reward'\",\n",
        "\"potential\": \"the name given to discount * function(s') - function(s)\",\n",
        "\"sparse\": \"contains little information, lots of zeros\",\n",
        "\"dynamic decision networks\": \"a network using decisions, rewards, and utility nodes\",\n",
        "\"factored representation\": \"feature based\",\n",
        "\"contraction\": \"how to show that the iteration converges\",\n",
        "\"policy loss\": \"the expected utility loss from executing an unoptimal policy\",\n",
        "\"async policy iteration\": \"picking any subset of states to update with either steps of policy iteration\",\n",
        "\n",
        "}\n",
        "\n"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ec432b0b-ae18-4710-823c-df1bd34d756c"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n \nBased on the MEU, the agent will choose actions based on what action gives the most expected utility.\nBy choosing the highest expected utility, there is a higher chance the agent will succeed.\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n \nA > B means A is preferred over B. A ~ B means that A or B is good, indifferent. A >= B means that\neither A is preferred over B, or either is good.\n\nPreference rules:\n\n1) Orderability: an agent must have a preferance or be indifferent between choices, must make a choice\n\n2) Transitivity: the transitivity property must hold, if agent has one preferance over another, and \nthat other preference is preferred over a third, then the first is more preferred than the third\n\n3) Continutiy: the inbetween choice shares an indifference preference with the lower choice with \nchance p and the higher choice with chance 1-p\n\n4) Substitutability: if an agent has a preference, then that preference holds in other lotteries, \ngiven that everything else besides the preference stays the same in the lottery. \n\n5) Monotonicity: If two lotteries are the same, then the agent should always go for the choice that \nhas a higher chance of getting the preferred lottery\n\n6) Decomposability: Can simplify more complex lotteries into simpler ones.\n\n\nUtility Functions:\n\nExistence of Utility Functions: If there is a preference, than the utility of that preference is \ngreater than the utility of the other. If there is not a preference, then both choices have the \nsame utility.\n\nExpected Utility: Sum of probability_i * utility of (outcome_i)\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n \nThere are many utility scales in the real world such as VSL, micromort, and QALY.\n\nMonetary value is not directly proportional to utility. It's more of a log value.\n\nRational agent theory is a popular economical theory based on how people should behave. Other types\nof theories are descriptive theories of how people actually behave. People might not always make\nthe most rational choice, but they can still be reasonable choices. A lot of heuristics can affect\ndecisions such as framing and anchoring effects.\n\n\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\nFor stochastic outcomes, the probability of reaching the next state is based on the current state and\nthe current action (P(s'|s,a)). Transition models can be written as T(s,a,s'). Reward functions can\nbe written as R(s,a,s'). These types of problems are usually solved using DP and use recursion to\nremember the best solution so far.\n\nIn finite horizon, time/the amount of moves left may factor into the optimal action. If time is not\na factor, then it is infinite horizon. Stationary policies do not factor in time, while nonstationary\npolicies as based on time.\n\nReasons why additive discount-reward works:\n\n1): people tend to value the now over the future\n\n2): Sooner rather than later is better because investing can make more rewards\n\n3): Future rewards may never come, therefore it's better to take the current rewards\n\n4): Preferences change over time. Now is preferred more than whatever is in the future.\n\n5): No infinities\n\n5a): Discounted rewards confines infinite sequences to finite utility. Has to be less than maximum reward / (1 - discount rate)\n\n5b): If there exist a terminal state and the policy can get to the terminal state, then it will never be an infinite sequence because\nit will eventually get to the terminal state.\n\n5c): Average reward per move can be used to analyze infinite sequences. \n\nThe optimal policy will work for any starting state.\n\nThe optimal policy for a given state s can be found by finding the largest action corresponding to the largest Q-function utility value\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\nThere are four algorithms used in MDPS: value iteration, policy iteration, linear programming and Monte Carlo planning.\n\n1): Value Iteration: A bottom up iteration process that looks at the neighboring nodes all the way to the root node. Used to find the best policy for each state. (OFFLINE)\n\n2): Policy Iteration: 2 step process; policy evaluation: evalute the current policy for its expected utility of all states for the given policy; policy improvement: calculate a new improved policy based on the current policy. (OFFLINE)\n\n3): Linear Programming: General method to solve constrained optimization problems. (ONLINE)\n\nOnline means heavy computation at each decision (expected-minmax) and offline means precomputed information\n\n\n\n## Module 8 Lecture Notes\n\n\n\n## Module 8 Definitions\n* **Utility function** - a function that gives the utility of a given state\n* **Expected utility** - the average utility based on probabilistic values\n* **Maximum expected utility.** - theory that says always choose the action with the most expected utility\n* **Lottery** - probability-outcome pairs for each action\n* **Value function/ordinal utility function** - the preference on states in deterministic environments\n* **Preference elicitation** - a way to figure out the utility function by showing a bunch of choices\n* **Normalized utilities** - u-upsidedown-T (the worst) is set to 0 and u_T (the best) is set to 1\n* **Standard lottery** - there's a p chance of getting the best, and 1-p chance of getting the worse\n* **Value of a statistical life** - the value used to determine how much it would cost to save 1 life; around $10 million\n* **Micromort** - the chance of death is equal to 1 / 1 million\n* **Qaly (quality-adjusted-life-year)** - the tradeoff between a shorter life for a better life\n* **Monotonic preferences** - an agent prefers more rather than less; more makes them happier\n* **Expected monetary value** - basically expected value, but with money\n* **Risk-averse** - don't like to take risk, prefer a sure-thing over an expect value gamble\n* **Risk-seeking** - like to take risk, gambles\n* **Certainty equivalent** - the amount of value needed to not take a gamble\n* **Insurance premimum** - the value between the expected value and the certainity equivalent\n* **Risk-neutral** - not risk-adverse nor risk-seeking, takes small risk\n* **Unbiased** - no error; zero error\n* **Order statistic** - statistics ordered based on the kth best value\n* **Optimizer's curse** - people will overestimate the expected utility for the best choice\n* **Normative theory** - theories on the most optimal ways to act\n* **Descriptive theory** - theories on how people actually act\n* **Certainty effect** - people like sure things, 100% events\n* **Ambiguity aversion** - people like known probabilities\n* **Framing effect** - the way a problem is asked can affect the person's choice\n* **Anchoring effect** - people tend to make choices around other choices; ex: expensive items tend to make less expensive items more attractive\n* **Evolutionary psychology** - humans did not evolve to make the best mathematical decisions\n* **Sequential decision problems** - the overall utility based on all the decisions made in the sequence\n* **Multi-armed bandit** - a slot machine with multi-arms and different probabilities for results\n* **Markovian** - the probability of reaching the next state depends only on the previous state\n* **Environment history** - all states and actions that the agent took\n* **Reward** - the value recieved when reaching a new state\n* **Markov decision process (mdp)** - problems that have markovian transition models, probablistic environments, and rewards. Usually solved using DP, recursive process.\n* **Policy** - a list of actions that tells the agent what to do at every given state\n* **Optimal policy** - the best policy that gives the highest expect utility based on environment history\n* **Finite horizon** - there is a fixed amount of moves/time and then times up\n* **Infinite horizon** - there is no fixed amount of moves/time, never ends\n* **Nonstationary** - when one of the deciding factors of a policy is reliant on time\n* **Stationary** - when the policy does not factor in time and only uses the information of the current state\n* **Additive discounted-reward** - a method used to give bigger reward to near future moves rather than distant future moves, exponential decrease\n* **Discount factor** - a value between 0-1; larger for near future rewards and smaller for distant future rewards\n* **Additive rewards** - a special case when the discount value reaches 1, you get the full reward\n* **Attribute** - part of the history\n* **Proper policy** - a policy that has a guaranteed path to the end state\n* **Bellman equation** - the utility of a state based on the maximum value of the sum of the transition * reward + discount*utility of next\n* **Action-utility function/q-funtion** - how much utility is gained by taking an action a in a state s\n* **Shapping theorem** - a specific transformation of reward + potential will not change the optimal policy for reward'\n* **Potential** - the name given to discount * function(s') - function(s)\n* **Sparse** - contains little information, lots of zeros\n* **Dynamic decision networks** - a network using decisions, rewards, and utility nodes\n* **Factored representation** - feature based\n* **Contraction** - how to show that the iteration converges\n* **Policy loss** - the expected utility loss from executing an unoptimal policy\n* **Async policy iteration** - picking any subset of states to update with either steps of policy iteration\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\"Types of bandit problems:\n",
        "\n",
        "1): Markov reward process\n",
        "\n",
        "2): One-armed bandit (cost of pulling one over not pulling the other)\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\" \n",
        "RL:\n",
        "\n",
        "1): Model-based RL: uses transition models, can be useful for state estimation, \n",
        "\n",
        "2): Model-free RL: Learnes directly how to behave;\n",
        "\n",
        "2a) Action-utility Learning: Q-learning using the Q-function for utility\n",
        "\n",
        "2b) Policy Search: learns how to policies; reflex agent\n",
        "\"\"\"\n",
        "chapter22_2 =  \"\"\"\n",
        "Passive learning is like policy evaluation, except the agent does not have access to the transition model\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "chapter22_3 =  \"\"\"\n",
        "\n",
        "\"\"\"\n",
        "chapter22_4 =  \"\"\"\n",
        "\n",
        "\"\"\"\n",
        "chapter22_5 =  \"\"\"\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"n-armed bandit\": \"machine with n levers\",\n",
        "\"exploitation\": \"take advantage of the current situtation\",\n",
        "\"exploration\": \"explore unknown states for better or worse reward\",\n",
        "\"one-armed bandit\": \"a two-armed bandit problem condensed to 1; the choice of pulling one is the cost of not pulling the other, zero-sum\",\n",
        "\"Gittins index\": \"related to how much reward you can get from machine M\",\n",
        "\"Bernoulli bandit\": \"given reward of 0 or 1 with unknown probabilities\",\n",
        "\"exploration bonus\": \"higher bonus when trying out paths not yet explored\",\n",
        "\"upper confidence\": \"the upper bound on the range of confidence\",\n",
        "\"Thompson sampling\": \"randomly chooses an arm based on optimal probablities from sample\",\n",
        "\"selection problem\": \"a subset of problems that focus on selecting the best option\",\n",
        "\"bandit superprocess\": \"simplfies the MRP to a MDP for each arm\",\n",
        "\"opportunity cost\": \"the cost of not choosing the other option\",\n",
        "\"dominating policy\": \"a policy that converts an arm into a MRP, an upper bound\",\n",
        "\"reinforcement learning\": \"learning by interacting with the environment and receiving a reward based on the interaction\",\n",
        "\"passive reinforcement learning\": \"there is a fixed policy; agent learns the utility of the states\",\n",
        "\"active reinforcement learning\": \"learn the best policies through exploration\",\n",
        "\"apprenticeship learning\": \"the usage of demonstrations instead of rewards for learning\",\n",
        "\"passive learning agent\": \"an agent that is trying to learn the utility function of the policy\",\n",
        "\"direct utility estimation\": \"the expected utility from this state and every state after this state\",\n",
        "\"adaptive dynamic programming agent\": \"an agent solves problems by learning models and using DP\",\n",
        "\"temporal-difference\": \"the difference between successes\",\n",
        "\"prioritized sweeping\": \"more prioritzation for successors that gone through large adjustments\",\n",
        "\"active learning agent\": \"an agent that has to determine which actions to take\",\n",
        "\"greedy agent\": \"take the best action at each step; not always optimal\",\n",
        "\"GLIE\": \"greedy in the limit of infinite exploration; try each action in each state many times\",\n",
        "\"absorbing state\": \"actions and rewards have no effects\",\n",
        "\"Bayesian RL\": \"the optimal policy is the maximum of the sum of P(h|e)* utility of all start states\",\n",
        "\"robust control theory\": \"finding the best policy that gives the best outcome for the worse case from a set of models\",\n",
        "\"off-policy\": \"deciding whether to switch to another policy based on the expected value of the state-action\",\n",
        "\"on-policy\": \"follow the current policy and find the expected utility\",\n",
        "\"supervised learning\": \"put things into existing categories\",\n",
        "\"Widrow-Hoff rule/delta rule\": \"a rule that calculates the rate of difference between actual value and predicted value\",\n",
        "\"catastrophic forgetting\": \"the problem in which a model forgets what it learned when new information comes in\",\n",
        "\"experience replay\": \"replay the past events occasinally to keep the learning\",\n",
        "\"reward shaping\": \"reward the agent for each small goal they achieve; working towards the bigger goal\",\n",
        "\"hierarchical reinforcement learning\": \"the process of breaking up longer action sequences into smaller and easier ones\",\n",
        "\n",
        "}"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "41aebea2-7019-441e-b4b4-22cf9d76fd8b"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\nTypes of bandit problems:\n\n1): Markov reward process\n\n2): One-armed bandit (cost of pulling one over not pulling the other)\n\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n \nRL:\n\n1): Model-based RL: uses transition models, can be useful for state estimation, \n\n2): Model-free RL: Learnes directly how to behave;\n\n2a) Action-utility Learning: Q-learning using the Q-function for utility\n\n2b) Policy Search: learns how to policies; reflex agent\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\nPassive learning is like policy evaluation, except the agent does not have access to the transition model\n\n\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\n\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\n\n\n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **N-armed bandit** - machine with n levers\n* **Exploitation** - take advantage of the current situtation\n* **Exploration** - explore unknown states for better or worse reward\n* **One-armed bandit** - a two-armed bandit problem condensed to 1; the choice of pulling one is the cost of not pulling the other, zero-sum\n* **Gittins index** - related to how much reward you can get from machine M\n* **Bernoulli bandit** - given reward of 0 or 1 with unknown probabilities\n* **Exploration bonus** - higher bonus when trying out paths not yet explored\n* **Upper confidence** - the upper bound on the range of confidence\n* **Thompson sampling** - randomly chooses an arm based on optimal probablities from sample\n* **Selection problem** - a subset of problems that focus on selecting the best option\n* **Bandit superprocess** - simplfies the MRP to a MDP for each arm\n* **Opportunity cost** - the cost of not choosing the other option\n* **Dominating policy** - a policy that converts an arm into a MRP, an upper bound\n* **Reinforcement learning** - learning by interacting with the environment and receiving a reward based on the interaction\n* **Passive reinforcement learning** - there is a fixed policy; agent learns the utility of the states\n* **Active reinforcement learning** - learn the best policies through exploration\n* **Apprenticeship learning** - the usage of demonstrations instead of rewards for learning\n* **Passive learning agent** - an agent that is trying to learn the utility function of the policy\n* **Direct utility estimation** - the expected utility from this state and every state after this state\n* **Adaptive dynamic programming agent** - an agent solves problems by learning models and using DP\n* **Temporal-difference** - the difference between successes\n* **Prioritized sweeping** - more prioritzation for successors that gone through large adjustments\n* **Active learning agent** - an agent that has to determine which actions to take\n* **Greedy agent** - take the best action at each step; not always optimal\n* **Glie** - greedy in the limit of infinite exploration; try each action in each state many times\n* **Absorbing state** - actions and rewards have no effects\n* **Bayesian rl** - the optimal policy is the maximum of the sum of P(h|e)* utility of all start states\n* **Robust control theory** - finding the best policy that gives the best outcome for the worse case from a set of models\n* **Off-policy** - deciding whether to switch to another policy based on the expected value of the state-action\n* **On-policy** - follow the current policy and find the expected utility\n* **Supervised learning** - put things into existing categories\n* **Widrow-hoff rule/delta rule** - a rule that calculates the rate of difference between actual value and predicted value\n* **Catastrophic forgetting** - the problem in which a model forgets what it learned when new information comes in\n* **Experience replay** - replay the past events occasinally to keep the learning\n* **Reward shaping** - reward the agent for each small goal they achieve; working towards the bigger goal\n* **Hierarchical reinforcement learning** - the process of breaking up longer action sequences into smaller and easier ones\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\"\n",
        "Three reasons: Laziness, Theoretical Ignorance, Practical Ignorance\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "chapter12_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_5 = \"\"\"\n",
        "Bayes rule: P(b|a) = {P(a|b) * P(b)}   / P(a);\n",
        "\n",
        "In english: P(cause|effect) = {P(effect|cause) * P(cause)} / P(effect)\n",
        "\n",
        "Conditional Independent Probability: P(a AND b | c) = P(a|c) * P(b|c); P(a| b AND c) = P(a|c)\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "chapter12_6 = \"\"\"\n",
        "Naive Bayes Model: P(Cause) * all of (P(Effect_i | Cause))\n",
        "\"\"\"\n",
        "chapter12_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"decision theory\" : \"the combination of probability theory and utility theory\",\n",
        "\"product rule\": \"P(a AND b) = P(a|b)*P(b)\",\n",
        "\"independence\": \"two events that don't rely on each other; P(a|b) = a, P(a AND b) = P(a)*P(b)\",\n",
        "\"diagnostic direction\": \"given the effect, what is the cause?\",\n",
        "\"causal direction\": \"given the cause, what is the effect?\"\n",
        "}"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "d2c9f1fe-2b01-4092-da20-b46e7271be6f"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\nThree reasons: Laziness, Theoretical Ignorance, Practical Ignorance\n\n\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\nBayes rule: P(b|a) = {P(a|b) * P(b)}   / P(a);\n\nIn english: P(cause|effect) = {P(effect|cause) * P(cause)} / P(effect)\n\nConditional Independent Probability: P(a AND b | c) = P(a|c) * P(b|c); P(a| b AND c) = P(a|c)\n\n\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\nNaive Bayes Model: P(Cause) * all of (P(Effect_i | Cause))\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Decision theory** - the combination of probability theory and utility theory\n* **Product rule** - P(a AND b) = P(a|b)*P(b)\n* **Independence** - two events that don't rely on each other; P(a|b) = a, P(a AND b) = P(a)*P(b)\n* **Diagnostic direction** - given the effect, what is the cause?\n* **Causal direction** - given the cause, what is the effect?\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter13_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter13_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 411
        },
        "cellView": "form",
        "outputId": "a7756d60-6636-454e-96bd-af44b9a61434"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Your summary here \n\n## Module 11 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_6 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_7 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_9 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_10 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "59fb3d0f-eef6-44f7-8df1-7e24da1862b2"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Your summary here \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n Your summary here \n\n## Module 12 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_6 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "97d03ddd-949e-42ef-cd60-250e99537cd4"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Your summary here \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Your summary here \n\n## SLP Chapter 5.3 The cross-entropy loss function\n Your summary here \n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n Your summary here \n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 13 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_10 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_11 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 945
        },
        "outputId": "0ec028e1-8fd1-4639-8acc-414948ca5a48"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "T3cm1WV542_a",
        "outputId": "a1f4b0c0-c418-44fb-9f2f-027a4e20f763"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n \nBased on the MEU, the agent will choose actions based on what action gives the most expected utility.\nBy choosing the highest expected utility, there is a higher chance the agent will succeed.\n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n \nA > B means A is preferred over B. A ~ B means that A or B is good, indifferent. A >= B means that\neither A is preferred over B, or either is good.\n\nPreference rules:\n\n1) Orderability: an agent must have a preferance or be indifferent between choices, must make a choice\n\n2) Transitivity: the transitivity property must hold, if agent has one preferance over another, and \nthat other preference is preferred over a third, then the first is more preferred than the third\n\n3) Continutiy: the inbetween choice shares an indifference preference with the lower choice with \nchance p and the higher choice with chance 1-p\n\n4) Substitutability: if an agent has a preference, then that preference holds in other lotteries, \ngiven that everything else besides the preference stays the same in the lottery. \n\n5) Monotonicity: If two lotteries are the same, then the agent should always go for the choice that \nhas a higher chance of getting the preferred lottery\n\n6) Decomposability: Can simplify more complex lotteries into simpler ones.\n\n\nUtility Functions:\n\nExistence of Utility Functions: If there is a preference, than the utility of that preference is \ngreater than the utility of the other. If there is not a preference, then both choices have the \nsame utility.\n\nExpected Utility: Sum of probability_i * utility of (outcome_i)\n\n\n\n## AIMA Chapter 16.3 Utility Functions\n \nThere are many utility scales in the real world such as VSL, micromort, and QALY.\n\nMonetary value is not directly proportional to utility. It's more of a log value.\n\nRational agent theory is a popular economical theory based on how people should behave. Other types\nof theories are descriptive theories of how people actually behave. People might not always make\nthe most rational choice, but they can still be reasonable choices. A lot of heuristics can affect\ndecisions such as framing and anchoring effects.\n\n\n\n\n## AIMA Chapter 17.1 Sequential Decision Problems\n\nFor stochastic outcomes, the probability of reaching the next state is based on the current state and\nthe current action (P(s'|s,a)). Transition models can be written as T(s,a,s'). Reward functions can\nbe written as R(s,a,s'). These types of problems are usually solved using DP and use recursion to\nremember the best solution so far.\n\nIn finite horizon, time/the amount of moves left may factor into the optimal action. If time is not\na factor, then it is infinite horizon. Stationary policies do not factor in time, while nonstationary\npolicies as based on time.\n\nReasons why additive discount-reward works:\n\n1): people tend to value the now over the future\n\n2): Sooner rather than later is better because investing can make more rewards\n\n3): Future rewards may never come, therefore it's better to take the current rewards\n\n4): Preferences change over time. Now is preferred more than whatever is in the future.\n\n5): No infinities\n\n5a): Discounted rewards confines infinite sequences to finite utility. Has to be less than maximum reward / (1 - discount rate)\n\n5b): If there exist a terminal state and the policy can get to the terminal state, then it will never be an infinite sequence because\nit will eventually get to the terminal state.\n\n5c): Average reward per move can be used to analyze infinite sequences. \n\nThe optimal policy will work for any starting state.\n\nThe optimal policy for a given state s can be found by finding the largest action corresponding to the largest Q-function utility value\n\n\n## AIMA Chapter 17.2 Algorithms for MDPs\n\nThere are four algorithms used in MDPS: value iteration, policy iteration, linear programming and Monte Carlo planning.\n\n1): Value Iteration: A bottom up iteration process that looks at the neighboring nodes all the way to the root node. Used to find the best policy for each state. (OFFLINE)\n\n2): Policy Iteration: 2 step process; policy evaluation: evalute the current policy for its expected utility of all states for the given policy; policy improvement: calculate a new improved policy based on the current policy. (OFFLINE)\n\n3): Linear Programming: General method to solve constrained optimization problems. (ONLINE)\n\nOnline means heavy computation at each decision (expected-minmax) and offline means precomputed information\n\n\n\n## Module 8 Lecture Notes\n\n\n\n## Module 8 Definitions\n* **Utility function** - a function that gives the utility of a given state\n* **Expected utility** - the average utility based on probabilistic values\n* **Maximum expected utility.** - theory that says always choose the action with the most expected utility\n* **Lottery** - probability-outcome pairs for each action\n* **Value function/ordinal utility function** - the preference on states in deterministic environments\n* **Preference elicitation** - a way to figure out the utility function by showing a bunch of choices\n* **Normalized utilities** - u-upsidedown-T (the worst) is set to 0 and u_T (the best) is set to 1\n* **Standard lottery** - there's a p chance of getting the best, and 1-p chance of getting the worse\n* **Value of a statistical life** - the value used to determine how much it would cost to save 1 life; around $10 million\n* **Micromort** - the chance of death is equal to 1 / 1 million\n* **Qaly (quality-adjusted-life-year)** - the tradeoff between a shorter life for a better life\n* **Monotonic preferences** - an agent prefers more rather than less; more makes them happier\n* **Expected monetary value** - basically expected value, but with money\n* **Risk-averse** - don't like to take risk, prefer a sure-thing over an expect value gamble\n* **Risk-seeking** - like to take risk, gambles\n* **Certainty equivalent** - the amount of value needed to not take a gamble\n* **Insurance premimum** - the value between the expected value and the certainity equivalent\n* **Risk-neutral** - not risk-adverse nor risk-seeking, takes small risk\n* **Unbiased** - no error; zero error\n* **Order statistic** - statistics ordered based on the kth best value\n* **Optimizer's curse** - people will overestimate the expected utility for the best choice\n* **Normative theory** - theories on the most optimal ways to act\n* **Descriptive theory** - theories on how people actually act\n* **Certainty effect** - people like sure things, 100% events\n* **Ambiguity aversion** - people like known probabilities\n* **Framing effect** - the way a problem is asked can affect the person's choice\n* **Anchoring effect** - people tend to make choices around other choices; ex: expensive items tend to make less expensive items more attractive\n* **Evolutionary psychology** - humans did not evolve to make the best mathematical decisions\n* **Sequential decision problems** - the overall utility based on all the decisions made in the sequence\n* **Multi-armed bandit** - a slot machine with multi-arms and different probabilities for results\n* **Markovian** - the probability of reaching the next state depends only on the previous state\n* **Environment history** - all states and actions that the agent took\n* **Reward** - the value recieved when reaching a new state\n* **Markov decision process (mdp)** - problems that have markovian transition models, probablistic environments, and rewards. Usually solved using DP, recursive process.\n* **Policy** - a list of actions that tells the agent what to do at every given state\n* **Optimal policy** - the best policy that gives the highest expect utility based on environment history\n* **Finite horizon** - there is a fixed amount of moves/time and then times up\n* **Infinite horizon** - there is no fixed amount of moves/time, never ends\n* **Nonstationary** - when one of the deciding factors of a policy is reliant on time\n* **Stationary** - when the policy does not factor in time and only uses the information of the current state\n* **Additive discounted-reward** - a method used to give bigger reward to near future moves rather than distant future moves, exponential decrease\n* **Discount factor** - a value between 0-1; larger for near future rewards and smaller for distant future rewards\n* **Additive rewards** - a special case when the discount value reaches 1, you get the full reward\n* **Attribute** - part of the history\n* **Proper policy** - a policy that has a guaranteed path to the end state\n* **Bellman equation** - the utility of a state based on the maximum value of the sum of the transition * reward + discount*utility of next\n* **Action-utility function/q-funtion** - how much utility is gained by taking an action a in a state s\n* **Shapping theorem** - a specific transformation of reward + potential will not change the optimal policy for reward'\n* **Potential** - the name given to discount * function(s') - function(s)\n* **Sparse** - contains little information, lots of zeros\n* **Dynamic decision networks** - a network using decisions, rewards, and utility nodes\n* **Factored representation** - feature based\n* **Contraction** - how to show that the iteration converges\n* **Policy loss** - the expected utility loss from executing an unoptimal policy\n* **Async policy iteration** - picking any subset of states to update with either steps of policy iteration\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\nTypes of bandit problems:\n\n1): Markov reward process\n\n2): One-armed bandit (cost of pulling one over not pulling the other)\n\n\n\n## AIMA Chapter 22.1 Learning from Rewards\n \nRL:\n\n1): Model-based RL: uses transition models, can be useful for state estimation, \n\n2): Model-free RL: Learnes directly how to behave;\n\n2a) Action-utility Learning: Q-learning using the Q-function for utility\n\n2b) Policy Search: learns how to policies; reflex agent\n\n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n\nPassive learning is like policy evaluation, except the agent does not have access to the transition model\n\n\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n\n\n\n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n\n\n\n\n## Module 9 Lecture Notes\n Your summary here \n\n## Module 9 Definitions\n* **N-armed bandit** - machine with n levers\n* **Exploitation** - take advantage of the current situtation\n* **Exploration** - explore unknown states for better or worse reward\n* **One-armed bandit** - a two-armed bandit problem condensed to 1; the choice of pulling one is the cost of not pulling the other, zero-sum\n* **Gittins index** - related to how much reward you can get from machine M\n* **Bernoulli bandit** - given reward of 0 or 1 with unknown probabilities\n* **Exploration bonus** - higher bonus when trying out paths not yet explored\n* **Upper confidence** - the upper bound on the range of confidence\n* **Thompson sampling** - randomly chooses an arm based on optimal probablities from sample\n* **Selection problem** - a subset of problems that focus on selecting the best option\n* **Bandit superprocess** - simplfies the MRP to a MDP for each arm\n* **Opportunity cost** - the cost of not choosing the other option\n* **Dominating policy** - a policy that converts an arm into a MRP, an upper bound\n* **Reinforcement learning** - learning by interacting with the environment and receiving a reward based on the interaction\n* **Passive reinforcement learning** - there is a fixed policy; agent learns the utility of the states\n* **Active reinforcement learning** - learn the best policies through exploration\n* **Apprenticeship learning** - the usage of demonstrations instead of rewards for learning\n* **Passive learning agent** - an agent that is trying to learn the utility function of the policy\n* **Direct utility estimation** - the expected utility from this state and every state after this state\n* **Adaptive dynamic programming agent** - an agent solves problems by learning models and using DP\n* **Temporal-difference** - the difference between successes\n* **Prioritized sweeping** - more prioritzation for successors that gone through large adjustments\n* **Active learning agent** - an agent that has to determine which actions to take\n* **Greedy agent** - take the best action at each step; not always optimal\n* **Glie** - greedy in the limit of infinite exploration; try each action in each state many times\n* **Absorbing state** - actions and rewards have no effects\n* **Bayesian rl** - the optimal policy is the maximum of the sum of P(h|e)* utility of all start states\n* **Robust control theory** - finding the best policy that gives the best outcome for the worse case from a set of models\n* **Off-policy** - deciding whether to switch to another policy based on the expected value of the state-action\n* **On-policy** - follow the current policy and find the expected utility\n* **Supervised learning** - put things into existing categories\n* **Widrow-hoff rule/delta rule** - a rule that calculates the rate of difference between actual value and predicted value\n* **Catastrophic forgetting** - the problem in which a model forgets what it learned when new information comes in\n* **Experience replay** - replay the past events occasinally to keep the learning\n* **Reward shaping** - reward the agent for each small goal they achieve; working towards the bigger goal\n* **Hierarchical reinforcement learning** - the process of breaking up longer action sequences into smaller and easier ones\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n\nThree reasons: Laziness, Theoretical Ignorance, Practical Ignorance\n\n\n\n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n\nBayes rule: P(b|a) = {P(a|b) * P(b)}   / P(a);\n\nIn english: P(cause|effect) = {P(effect|cause) * P(cause)} / P(effect)\n\nConditional Independent Probability: P(a AND b | c) = P(a|c) * P(b|c); P(a| b AND c) = P(a|c)\n\n\n\n\n## AIMA Chapter 12.6 Naive Bayes Models\n\nNaive Bayes Model: P(Cause) * all of (P(Effect_i | Cause))\n\n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\n Your summary here \n\n## Module 10 Definitions\n* **Decision theory** - the combination of probability theory and utility theory\n* **Product rule** - P(a AND b) = P(a|b)*P(b)\n* **Independence** - two events that don't rely on each other; P(a|b) = a, P(a AND b) = P(a)*P(b)\n* **Diagnostic direction** - given the effect, what is the cause?\n* **Causal direction** - given the cause, what is the effect?\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Your summary here \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Your summary here \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Your summary here \n\n## Module 11 Definitions\n* **Concept** - definition\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Your summary here \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\n Your summary here \n\n## Module 12 Definitions\n* **Concept** - definition\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n Your summary here \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n Your summary here \n\n## SLP Chapter 5.3 The cross-entropy loss function\n Your summary here \n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n Your summary here \n\n## SLP Chapter 7.1 Units\n Your summary here \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 13 Definitions\n* **Concept** - definition\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\n Your summary here \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}