{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Template_for CIS521_midterm_2_notes.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "38c6f993-39b6-429e-d6b0-f970d2cf5d1c"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "ff524693-be32-42a2-8edf-5e686030e6c7"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"\"\n",
        "permission = True\n",
        "anonymous = False"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" Expected utility: avg utility value of outcomes wighted by prob that outcome occurs\n",
        "Utility function: assigns single number to express the desirability of a state. Maximum expected utility(MEU): rational agent should choose action that \n",
        "maxes agents expected utility. MEU is prescription for intelligent behavior. MEU formalizes but does not operationalize. If agent acts to max a utility\n",
        "function that reflects correctly performance measure then agent will get highest possible performance score (avg over all pos env). This is central\n",
        "justification for MEU. \n",
        "\n",
        "\"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\" each outcome of a lottery can be atomic or another lottery. reasonable preference relation axioms of utility theory:orderability(given 2 lotteries rational agent must\n",
        "either prefer one or equally; agent must decide)transitivity(if agent prefers A to B and prefers B to C then it prefers A to C)continuity(if some lottery B is between \n",
        "A and C preference then there is some probability for which agent will be indifferent between B and lottery that gets A with probability p and C with \n",
        "prob 1-p)substitutability(if agent is indifferent between 2 lotteries A and B then the agent is indifferent between 2+ complex that are the same\n",
        "except that B is substituted for A in one of them regardless of probabilities) monotonicity(suppose 2 lotteries have same outcomes, if A is prefered to B then agent\n",
        "must prefer lottery that has a higher probability for A)decomposability(compound lottery can be reduced to simpler ones using the laws of prob; no fun in gambling rule)\n",
        "existence of utility function, expected utility of lottery; theorem says utility functions exist not that they are unique; value function/ordinal utility: in deterministic\n",
        "env agent only needs preference on ranking not numbers\"\"\"\n",
        "chapter16_3 =  \"\"\" utility functions map lotteries to numbers; preference elicitation:work out human utility func by giving them choices and \n",
        "checking;micromort(value of own life);utility of money is propotional to log of amount almost; certainty equivalent:\n",
        "value agent will accept for lottery; risk-neutral: linear curve; optimizer's curse tendency for estimated \n",
        "expected utility to be too high; normative theory: should act; descriptive theory: actually act; certainty\n",
        "effect: attach to gains that are certain; framing effect: wording matters; anchoring effect: offer $200 and \n",
        "then offer $55 one to make it a good deal; \n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\" Finite horizon: after certain time nothing matters;\n",
        "W/finite horizon optimal action in state depends on time left non stationary policy; \n",
        "discount factor: preference of current reward over future reward if close to 0 insignificant if in distant future; proper policy: guaranteed to reach terminal state; infinite sequences compared by average reward; optimal policy is independent of starting state if using discounted utilities with infinite horizon; utility of a state is the expected reward for the next transition plus discounted utility of the next state assuming agent chooses optimal action: aka Bellman; q function is expected utility of taking a given action in a given state \n",
        " \"\"\"\n",
        "chapter17_2 =  \"\"\" Contraction: basic concept used in showing that value iteration converges, function of one argument that when applied to 2 different inputs produces 2 outputs that are closer together, only has 1 fixed point, repeated application always reaches fixed point in limit; max norm: measures length of vector by \n",
        "absolute value of its biggest component ; policy loss most you can loose by not taking opt policy; policy evaluation O(n^3); linear programming is a general approach for formulating constrained optimization problems; value iteration and policy iteration are offline algorithms \n",
        "Reinforcement learning: has underlying MDP, process of taking actions learning reward and then updating world, agent receives feedback thru rewards and agent utility is defined by reward function; exploration: unknown, exploitation: known; observing a sample:choosing action and seeing something that results; in reinforcement learning: just like MDP but we don’t know T or R (must try out actions and states to learn); model-based learning: learn approx model based on experiences, learn empirical MDP then solve (value iteration); model-free: take action then compare what happened to what you thought was going to happen then update values based on that, we don’t keep track of reward or transition functions; passive reinforcement learning: policy evaluation, don’t know R or T, goal learn state values, input fixed policy,; temporal difference learning: update V(s) every time we experience transition, learn from every experience, policy is still fixed; exponential moving average: increase alpha can give converging values; \n",
        "TD value learning is model free way to do policy evaluation mimick bellman but with running sample avg, learn Q Vals not values. \n",
        " \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\"Markovian: probability of arriving at s' only depends on s and not history of earlier state. Need to define initial state, terminal state,\n",
        "transition function(model), reward function, set of states, set of actions. MDP is sequential, fully observable, stochastic, additive rewards.\n",
        "MDP is nondeterministic search problems; one way of solving them is with expectimax. MDP solution is a policy. A policy takes in state and outputs \n",
        "action. Discounting: penalize actions taken in future (has less value); stationary preference even after adding equal discount same preference holds;\n",
        "infinite utility solution: fixed # steps/finite horizon, discounting, absorbing state(eventually will reach a terminal state); MDP cares about expectimax\n",
        "; max of all expectimax calculation; value iteration is guaranteed to converge to unique optimal values (policy may converge before values do) we know this\n",
        "because 1.if discount is less than 1 then as k increases the values will converge 2.if tree has max depth M then vm holds actual untruncated values\n",
        "How to calculate v's for fixed policy pi? make recursive bellman equations to updates time complexity=O(S^2); how to act: pick optimal qval\n",
        "alternative approach for opt vals: until converege(policy evaluation and policy improvement) this is called policy iteration;\n",
        "policy vs value iteration:both(compute same thing all opt vals), value iter(every iter updates both vals and policy; don't\n",
        "track policy but max over actions implicitly recomputes it), policy iter(do several passes that update utilities with\n",
        "fixed policy, after policy is eval new policy is picked,either new policy is better or done); worst case minimax\n",
        "terminal function scale doesn't matter(insensitivity to monotonic transformation); normalized utilitiy(between 0 and 1)\n",
        "money doesn't behave as utility func; \n",
        "\"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"Your definition here\",\n",
        "\"expected utility\" : \"your definition here\",\n",
        "\"etc.\": \"your definition here\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1495
        },
        "outputId": "e2cd0f94-809e-4a34-a1a5-0e49e17f50ab"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Expected utility: avg utility value of outcomes wighted by prob that outcome occurs\nUtility function: assigns single number to express the desirability of a state. Maximum expected utility(MEU): rational agent should choose action that \nmaxes agents expected utility. MEU is prescription for intelligent behavior. MEU formalizes but does not operationalize. If agent acts to max a utility\nfunction that reflects correctly performance measure then agent will get highest possible performance score (avg over all pos env). This is central\njustification for MEU. \n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n each outcome of a lottery can be atomic or another lottery. reasonable preference relation axioms of utility theory:orderability(given 2 lotteries rational agent must\neither prefer one or equally; agent must decide)transitivity(if agent prefers A to B and prefers B to C then it prefers A to C)continuity(if some lottery B is between \nA and C preference then there is some probability for which agent will be indifferent between B and lottery that gets A with probability p and C with \nprob 1-p)substitutability(if agent is indifferent between 2 lotteries A and B then the agent is indifferent between 2+ complex that are the same\nexcept that B is substituted for A in one of them regardless of probabilities) monotonicity(suppose 2 lotteries have same outcomes, if A is prefered to B then agent\nmust prefer lottery that has a higher probability for A)decomposability(compound lottery can be reduced to simpler ones using the laws of prob; no fun in gambling rule)\nexistence of utility function, expected utility of lottery; theorem says utility functions exist not that they are unique; value function/ordinal utility: in deterministic\nenv agent only needs preference on ranking not numbers\n\n## AIMA Chapter 16.3 Utility Functions\n utility functions map lotteries to numbers; preference elicitation:work out human utility func by giving them choices and \nchecking;micromort(value of own life);utility of money is propotional to log of amount almost; certainty equivalent:\nvalue agent will accept for lottery; risk-neutral: linear curve; optimizer's curse tendency for estimated \nexpected utility to be too high; normative theory: should act; descriptive theory: actually act; certainty\neffect: attach to gains that are certain; framing effect: wording matters; anchoring effect: offer $200 and \nthen offer $55 one to make it a good deal; \n \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n Finite horizon: after certain time nothing matters;\nW/finite horizon optimal action in state depends on time left non stationary policy; \ndiscount factor: preference of current reward over future reward if close to 0 insignificant if in distant future; proper policy: guaranteed to reach terminal state; infinite sequences compared by average reward; optimal policy is independent of starting state if using discounted utilities with infinite horizon; utility of a state is the expected reward for the next transition plus discounted utility of the next state assuming agent chooses optimal action: aka Bellman; q function is expected utility of taking a given action in a given state \n \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Contraction: basic concept used in showing that value iteration converges, function of one argument that when applied to 2 different inputs produces 2 outputs that are closer together, only has 1 fixed point, repeated application always reaches fixed point in limit; max norm: measures length of vector by \nabsolute value of its biggest component ; policy loss most you can loose by not taking opt policy; policy evaluation O(n^3); linear programming is a general approach for formulating constrained optimization problems; value iteration and policy iteration are offline algorithms \nReinforcement learning: has underlying MDP, process of taking actions learning reward and then updating world, agent receives feedback thru rewards and agent utility is defined by reward function; exploration: unknown, exploitation: known; observing a sample:choosing action and seeing something that results; in reinforcement learning: just like MDP but we don’t know T or R (must try out actions and states to learn); model-based learning: learn approx model based on experiences, learn empirical MDP then solve (value iteration); model-free: take action then compare what happened to what you thought was going to happen then update values based on that, we don’t keep track of reward or transition functions; passive reinforcement learning: policy evaluation, don’t know R or T, goal learn state values, input fixed policy,; temporal difference learning: update V(s) every time we experience transition, learn from every experience, policy is still fixed; exponential moving average: increase alpha can give converging values; \nTD value learning is model free way to do policy evaluation mimick bellman but with running sample avg, learn Q Vals not values. \n \n\n## Module 8 Lecture Notes\nMarkovian: probability of arriving at s' only depends on s and not history of earlier state. Need to define initial state, terminal state,\ntransition function(model), reward function, set of states, set of actions. MDP is sequential, fully observable, stochastic, additive rewards.\nMDP is nondeterministic search problems; one way of solving them is with expectimax. MDP solution is a policy. A policy takes in state and outputs \naction. Discounting: penalize actions taken in future (has less value); stationary preference even after adding equal discount same preference holds;\ninfinite utility solution: fixed # steps/finite horizon, discounting, absorbing state(eventually will reach a terminal state); MDP cares about expectimax\n; max of all expectimax calculation; value iteration is guaranteed to converge to unique optimal values (policy may converge before values do) we know this\nbecause 1.if discount is less than 1 then as k increases the values will converge 2.if tree has max depth M then vm holds actual untruncated values\nHow to calculate v's for fixed policy pi? make recursive bellman equations to updates time complexity=O(S^2); how to act: pick optimal qval\nalternative approach for opt vals: until converege(policy evaluation and policy improvement) this is called policy iteration;\npolicy vs value iteration:both(compute same thing all opt vals), value iter(every iter updates both vals and policy; don't\ntrack policy but max over actions implicitly recomputes it), policy iter(do several passes that update utilities with\nfixed policy, after policy is eval new policy is picked,either new policy is better or done); worst case minimax\nterminal function scale doesn't matter(insensitivity to monotonic transformation); normalized utilitiy(between 0 and 1)\nmoney doesn't behave as utility func; \n\n\n## Module 8 Definitions\n* **Utility function** - Your definition here\n* **Expected utility** - your definition here\n* **Etc.** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\" Bandit: arms are independent\n",
        "agent can only work on one arm at a time; one armed bandit\n",
        "the choice is pull again or no and you pull it T times (stopping time)\n",
        "Gittin's index: optimal policy for any bandit, pull arm with highest gittins \n",
        "then update values; first decision is O(n) then O(1);Bernoulli bandit: each arm \n",
        "produces reward of 1 or 0 with a fixed but unknown probability;\n",
        "exploration bonus: higher index for arms only tried a few times\n",
        ";how to calc gittin index: upper confidence bound, Thompson sampling; \n",
        "selection problems: choosing best option when goal is max total \n",
        "number of successes over time, no index func exists;\n",
        "optimal bandit algo explore less than opt selection probs;\n",
        "bandit superprocess(BSP): each arm is a full MDP in its own\n",
        "right rather than being MRP with only 1 possible action;\n",
        "globally optimal policy for bsp may include actions that\n",
        "are locally suboptimal from pov of mdp constituent where they are taken\n",
        "reason for this is availability of other mdp in which to act changes balance\n",
        "between short term and long term rewards in a component mdp.\n",
        "arm is unaffected by opportunity cost: dominating policy; by adding\n",
        "actions to states it is always possible to create a relaxed mdp version\n",
        "so that it has dominating policy\n",
        " \"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\"Reinforcement learning(RL): agent interacts with world \n",
        "and gets rewards as sign of how well it is doing; RL is dif than MDP because\n",
        "agent isn't given MDP as a problem to solve, agent is in MDP, it may not know transition model\n",
        "or reward function has to act to learn; sparse rewards: win/loose no more signals \n",
        ";model based RL: agent uses transition model of env to interpret\n",
        "reward signals and make decisions about how to act, state estimation:\n",
        "this is useful in partially observable; model free RL:\n",
        "agent either knows nor learns a transition model, it learns more direct 2 ways:\n",
        "action utility learning(q learning; agent learns q func, given this\n",
        "agent can choose what to do in state by taking action with highest qval)\n",
        "policy search (reflex agent: policy that maps states to actions);\n",
        "apprenticeship learning: training learning agent using  demonstrations\n",
        "rather than reward signals  \n",
        " \"\"\"\n",
        "chapter22_2 =  \"\"\" passive learning agent vs policy evaluation: \n",
        "passive learning does not know transition model which specifies prob of \n",
        "reach state s' from state s after doing action a; also doesn't know reward func;\n",
        "direct utility est: utility of state is defined as expected total reward\n",
        "from that state onward; utility of state is determined by\n",
        "reward and expected utility of successor states; temporal difference:\n",
        "difference in utilities between sucessive rates, temporal dif work by adjusting utility estimates towrd the ideal\n",
        "equilibrium; temporal difference (TD) doesn't need transition model to perform updates; adaptive dynamic programming (ADP) adjusts state\n",
        "to agree with all successors, TD adjusted states to agree w/observed successor, TD makes single adjustment per transition\n",
        "ADP makes as many as needed; we can make more efficient versions of ADP by directly approximating algos for value or policy \n",
        "iteration; approx algo can use min adjustment size that decreases as transition model become more accurate\n",
        "\"\"\"\n",
        "chapter22_3 =  \"\"\" greedy in the limit of infinite exploration(GLIE): must try each action in each state\n",
        "absorbing state: no actions have any effects and no rewards received; \n",
        " \"\"\"\n",
        "chapter22_4 =  \"\"\" Your summary here \"\"\"\n",
        "chapter22_5 =  \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" Temporal difference learning: policy fixed more values move toward value of whatever\n",
        "successor occurs:running average; interpolation: update one component by some amount and the other by 1-that alount\n",
        ";q learning is based on samples, as Alpha gets closer to one we put more weight on the sample, as it gets closer to zero, we put more weight \n",
        "on our previous estimates and so we're going to balance between those\n",
        "qlearning converges to optimal policy even if you're acting suboptimally \n",
        "(called off-policy learning) bad things: you have to explore sufficient, make\n",
        "learning rate small but not decrease too quickly; when to explore: badness is not yet \n",
        "established; epsilon greedy (if prob is small go rand else go policy)\n",
        "regret is measure of your total mistake cost( dif between \n",
        "your expected rewards and optimal expected rewards) it lets us quantify goodness of exploration\n",
        "technique, minimize regret: optimally\n",
        "learning to be optimal; intuitive interpretation: adjust weights of active features\n",
        "if something bad happens blame the features associated with it and \n",
        "later disprefer\n",
        "\"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"term\" : \"your definition here\",\n",
        "}"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1219
        },
        "outputId": "1cfaba1a-ba1a-4bfb-8ba1-8351e3cb3021"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"AIMA Chapter 22.5 Generalization in Reinforcement Learning\", chapter22_5),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Bandit: arms are independent\nagent can only work on one arm at a time; one armed bandit\nthe choice is pull again or no and you pull it T times (stopping time)\nGittin's index: optimal policy for any bandit, pull arm with highest gittins \nthen update values; first decision is O(n) then O(1);Bernoulli bandit: each arm \nproduces reward of 1 or 0 with a fixed but unknown probability;\nexploration bonus: higher index for arms only tried a few times\n;how to calc gittin index: upper confidence bound, Thompson sampling; \nselection problems: choosing best option when goal is max total \nnumber of successes over time, no index func exists;\noptimal bandit algo explore less than opt selection probs;\nbandit superprocess(BSP): each arm is a full MDP in its own\nright rather than being MRP with only 1 possible action;\nglobally optimal policy for bsp may include actions that\nare locally suboptimal from pov of mdp constituent where they are taken\nreason for this is availability of other mdp in which to act changes balance\nbetween short term and long term rewards in a component mdp.\narm is unaffected by opportunity cost: dominating policy; by adding\nactions to states it is always possible to create a relaxed mdp version\nso that it has dominating policy\n \n\n## AIMA Chapter 22.1 Learning from Rewards\nReinforcement learning(RL): agent interacts with world \nand gets rewards as sign of how well it is doing; RL is dif than MDP because\nagent isn't given MDP as a problem to solve, agent is in MDP, it may not know transition model\nor reward function has to act to learn; sparse rewards: win/loose no more signals \n;model based RL: agent uses transition model of env to interpret\nreward signals and make decisions about how to act, state estimation:\nthis is useful in partially observable; model free RL:\nagent either knows nor learns a transition model, it learns more direct 2 ways:\naction utility learning(q learning; agent learns q func, given this\nagent can choose what to do in state by taking action with highest qval)\npolicy search (reflex agent: policy that maps states to actions);\napprenticeship learning: training learning agent using  demonstrations\nrather than reward signals  \n \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n passive learning agent vs policy evaluation: \npassive learning does not know transition model which specifies prob of \nreach state s' from state s after doing action a; also doesn't know reward func;\ndirect utility est: utility of state is defined as expected total reward\nfrom that state onward; utility of state is determined by\nreward and expected utility of successor states; temporal difference:\ndifference in utilities between sucessive rates, temporal dif work by adjusting utility estimates towrd the ideal\nequilibrium; temporal difference (TD) doesn't need transition model to perform updates; adaptive dynamic programming (ADP) adjusts state\nto agree with all successors, TD adjusted states to agree w/observed successor, TD makes single adjustment per transition\nADP makes as many as needed; we can make more efficient versions of ADP by directly approximating algos for value or policy \niteration; approx algo can use min adjustment size that decreases as transition model become more accurate\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n greedy in the limit of infinite exploration(GLIE): must try each action in each state\nabsorbing state: no actions have any effects and no rewards received; \n \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.5 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n Temporal difference learning: policy fixed more values move toward value of whatever\nsuccessor occurs:running average; interpolation: update one component by some amount and the other by 1-that alount\n;q learning is based on samples, as Alpha gets closer to one we put more weight on the sample, as it gets closer to zero, we put more weight \non our previous estimates and so we're going to balance between those\nqlearning converges to optimal policy even if you're acting suboptimally \n(called off-policy learning) bad things: you have to explore sufficient, make\nlearning rate small but not decrease too quickly; when to explore: badness is not yet \nestablished; epsilon greedy (if prob is small go rand else go policy)\nregret is measure of your total mistake cost( dif between \nyour expected rewards and optimal expected rewards) it lets us quantify goodness of exploration\ntechnique, minimize regret: optimally\nlearning to be optimal; intuitive interpretation: adjust weights of active features\nif something bad happens blame the features associated with it and \nlater disprefer\n\n\n## Module 9 Definitions\n* **Term** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_6 = \"\"\" Your summary here \"\"\"\n",
        "chapter12_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" Your summary here \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\"probabilistic reasoning givs\n",
        "us framework for managing beliefs and knowledge; observed values (evidence)\n",
        "unobserved var(states), model(how known relates to unknown); probabilistic \n",
        "model is joint distribution over set of rand vals; probabilistic inference:\n",
        "compute a desired prob from other known prob (conditional from joint)\n",
        "inference by enumeration: evidence (given), query var(want to find out)\n",
        "hidden(not thinking but present); probabilistic lang model: goal compute\n",
        "prob of a sentence of words; markov assumption: prob depends only\n",
        "on previous k words not entire context\n",
        " \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"term\" : \"your definition here\",\n",
        "}"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1165
        },
        "cellView": "form",
        "outputId": "ab61d8f7-0221-40ac-d5bc-6da9173214de"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Your summary here \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\nprobabilistic reasoning givs\nus framework for managing beliefs and knowledge; observed values (evidence)\nunobserved var(states), model(how known relates to unknown); probabilistic \nmodel is joint distribution over set of rand vals; probabilistic inference:\ncompute a desired prob from other known prob (conditional from joint)\ninference by enumeration: evidence (given), query var(want to find out)\nhidden(not thinking but present); probabilistic lang model: goal compute\nprob of a sentence of words; markov assumption: prob depends only\non previous k words not entire context\n \n\n## Module 10 Definitions\n* **Term** - your definition here\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" Conditioning case: possible combination of values for parent nodes, each row sum must equal 1 \"\"\"\n",
        "chapter13_2 = \"\"\" Baysian network is correct representation of domain only if each\n",
        "node is conditionally independent of its other predecessors given parents;it is impossible for knowledge engineer to create\n",
        "Baysian that violates axioms of probability;compactness of Baysian is (locally structrued/spare); locally struct:\n",
        "each subcomponent interacts directly with limited num of components associated w/linear growth; Bayes: conditional prob table\n",
        "2^k nums, complete network (2^k) *n nums; casualmodel= fewer and easier nums; each variable is conditionally independent\n",
        "of its non-descendants given parents;  \"\"\"\n",
        "chapter13_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" Unconditional independence is rare and doesn't help\n",
        "us make inferences about other variables; conditional independence is our most basic and robust form of knowledge\n",
        "about uncertain environments; causal chains does not guarantee x independent\n",
        "of z; a path is active if each triple is active; grey circle means known/observed/ have evidence for;\n",
        "if all paths are inactive then independence is guaranteed; if 1 or more active then \n",
        "independence not guaranteed (remember path not triples); inference: calculate useful quantity\n",
        "from a joint probability distribution\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 567
        },
        "cellView": "form",
        "outputId": "8093c8db-5a46-4b27-d31e-ecb856e449c7"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Conditioning case: possible combination of values for parent nodes, each row sum must equal 1 \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Baysian network is correct representation of domain only if each\nnode is conditionally independent of its other predecessors given parents;it is impossible for knowledge engineer to create\nBaysian that violates axioms of probability;compactness of Baysian is (locally structrued/spare); locally struct:\neach subcomponent interacts directly with limited num of components associated w/linear growth; Bayes: conditional prob table\n2^k nums, complete network (2^k) *n nums; casualmodel= fewer and easier nums; each variable is conditionally independent\nof its non-descendants given parents;  \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Unconditional independence is rare and doesn't help\nus make inferences about other variables; conditional independence is our most basic and robust form of knowledge\nabout uncertain environments; causal chains does not guarantee x independent\nof z; a path is active if each triple is active; grey circle means known/observed/ have evidence for;\nif all paths are inactive then independence is guaranteed; if 1 or more active then \nindependence not guaranteed (remember path not triples); inference: calculate useful quantity\nfrom a joint probability distribution\n \n\n## Module 11 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_2 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_3 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_4 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_5 = \"\"\" Your summary here \"\"\"\n",
        "chapter19_6 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_7 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_9 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter4_10 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\"model-based classification:\n",
        "build model(bayes net) where label and features are rand variables, instantiate any ovserved features\n",
        "query for distribution of label condition on fetures; Naive Bayes:\n",
        "assume all features that we're using to represent inputs are independent\n",
        "and only depend on label, total num of parameters is linear in num of features\n",
        "inference for naive bayes: compute posterior distribution over label var Y\n",
        "estimates of local conditional probability tables (called parameters)\n",
        "relative freq parameters will overfit training data; we need to smooth\n",
        "or regularize estimates; empirical rate: estimate that max likelihood of data\n",
        "laplace performs poorly for large x or y; for real classification\n",
        "smoothing is critical; 2 unknowns: parameters(probabilities) and hyperparameters(amount\n",
        "/type of smoothing to do); learn parameters from training data, tune hyperparameters on different\n",
        "data, for each val of hyperparameter train and test held out data, \n",
        "choose best val and do final test on test data; multiclass decision rule:\n",
        "prediction highest score wins; properties of perceptrons:separability\n",
        "(true if some parameters get the training set perfectly correct),\n",
        "convergence(if training is separable, perceptron), mistake bound (max num of msitakes related to margin or degree of\n",
        "sparability;perceptron probs: noise(if data isn't separable weight might thrash; \n",
        "avg weight vectors over time can help), mediocre generalization(find a barely separating solution),\n",
        "overtraining(test/heldout accuracy usually rises then falls)\n",
        "max step size: bad to make updates that are too large, example may be\n",
        "labeled incorrectly, not enough features, solution:cap max pos val\n",
        "of the weird r with a constant C, corresponds to optimization that assumes non-separable\n",
        "data, usually converges faster than perceptron and better on noisy data;\n",
        "support vector: help find boundary the bounday between 2\n",
        "classed that maximizes the margin; naive bayes:builds model training data,\n",
        "gives prediction probabilities, strong assumptions about feature\n",
        "independence, once pass thru data; perceptrons/mira: less assumptions about\n",
        "data,usually more accurate\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1592
        },
        "cellView": "form",
        "outputId": "15b39fc6-6f2a-44cd-d983-412090792520"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Your summary here \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\nmodel-based classification:\nbuild model(bayes net) where label and features are rand variables, instantiate any ovserved features\nquery for distribution of label condition on fetures; Naive Bayes:\nassume all features that we're using to represent inputs are independent\nand only depend on label, total num of parameters is linear in num of features\ninference for naive bayes: compute posterior distribution over label var Y\nestimates of local conditional probability tables (called parameters)\nrelative freq parameters will overfit training data; we need to smooth\nor regularize estimates; empirical rate: estimate that max likelihood of data\nlaplace performs poorly for large x or y; for real classification\nsmoothing is critical; 2 unknowns: parameters(probabilities) and hyperparameters(amount\n/type of smoothing to do); learn parameters from training data, tune hyperparameters on different\ndata, for each val of hyperparameter train and test held out data, \nchoose best val and do final test on test data; multiclass decision rule:\nprediction highest score wins; properties of perceptrons:separability\n(true if some parameters get the training set perfectly correct),\nconvergence(if training is separable, perceptron), mistake bound (max num of msitakes related to margin or degree of\nsparability;perceptron probs: noise(if data isn't separable weight might thrash; \navg weight vectors over time can help), mediocre generalization(find a barely separating solution),\novertraining(test/heldout accuracy usually rises then falls)\nmax step size: bad to make updates that are too large, example may be\nlabeled incorrectly, not enough features, solution:cap max pos val\nof the weird r with a constant C, corresponds to optimization that assumes non-separable\ndata, usually converges faster than perceptron and better on noisy data;\nsupport vector: help find boundary the bounday between 2\nclassed that maximizes the margin; naive bayes:builds model training data,\ngives prediction probabilities, strong assumptions about feature\nindependence, once pass thru data; perceptrons/mira: less assumptions about\ndata,usually more accurate\n\n \n\n## Module 12 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\"dif between naive bayes and logistic regression:logisitic regression is descriminative(just learns to \n",
        "distinguish classes without learning much about it ) while bayes is\n",
        "generative(may generate pic of dog and see if it fits the features in order to separate, uses likelihood term); both are\n",
        "supervised ML; ML component:feature representation of input, classification(sigmod,softmax),objective func(cross-entrpy loss),\n",
        "algo to optimize objective func(stochastic gradient descent);sigmoid func takes real-valued number and maps it to range; \n",
        "naive bayes has overly strong conditional independence assumptions; logistic regression generally works best on large docs\n",
        "or dataset  \n",
        "\"\"\"\n",
        "slp_chapter5_2 = \"\"\" cost function/loss function distance between system output and gold output; \"\"\"\n",
        "slp_chapter5_3 = \"\"\" \n",
        "conditional maximum likelihood estimation\n",
        "loss function that prefers correct class labels\n",
        "of training examples\n",
        " \"\"\"\n",
        "slp_chapter5_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter5_6 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\"Tanh is smooth and \n",
        "differentiable and map outliers toward mean;;\n",
        "\n",
        " \"\"\"\n",
        "slp_chapter7_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_5 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter7_7 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\"Classifier Component:\n",
        "feature representation of input, classification function, \n",
        "objective function, algo for optimizing objective function; \n",
        "decision boundary=.5 (if more than output yes); learning in \n",
        "logistic regression: how to get weights of model we learn the \n",
        "parameters which requires 2 components: 1(loss function telling us\n",
        "distance between system output and gold output i.e cross\n",
        "entropy loss) 2(algo for optimizing objective function i.e. \n",
        "stochastic gradient descent to min loss func);cross entropy loss:\n",
        "negative log likelihood loss choose param that max the log prob\n",
        "of true labels in training data; cross entropy loss( why does minimizing\n",
        "negative log prob do what we want, we want loss to be smaller if the model's \n",
        "estimate is close to correct and we want the loss to be \n",
        "bigger if it is confused); gradient descent: used to find good\n",
        "settings for weights and bias by minimizing loss func, gradient\n",
        "descent is method that finds a min of a function by figuring out\n",
        "in which direction the function's slope is rising the most \n",
        "steeply and moving the other direction; convex func has\n",
        "only 1 min so gradient descent from any start will find min;\n",
        "magnitude of the amount to move in gradient descent is the value of slope\n",
        "weight by learning rate, higher/faster learning rate means that we should move w\n",
        "more on each step; stochasticgd chooses single rand example\n",
        "and updates so results can fluctuate; batch training is alt it computes\n",
        "gradient over entire dataset so give better est of which direction \n",
        "to move weight but is too long so we use mini batch(512/1024)\n",
        "loss is accumulated; overfitting: poor generalization and \n",
        "performance on unseen test set; regularization is way to penalize\n",
        "large weights, regularization term is added to loss function; lasso(L1)\n",
        "ridge(L2); neuron units apply nonlinear function (sigmoid,tanh, relu)\n",
        "Feed forward: multilayered all units connected no cycles, outputs\n",
        "from each layer are passed to units in the next layer and no outputs are passed\n",
        "to the sending layer; logistic regression: network with only one layer and a sigmoid\n",
        "output; for deeper networks computing gradient requires looking at earlier \n",
        "layers; backward differentiation:depends on computation graph\n",
        "importance of computation graph comes from backward pass\n",
        "used to compute derivatives for weight update it uses chain rule\n",
        "we need activation functions to be differentiable\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1513
        },
        "cellView": "form",
        "outputId": "e2f8d0a8-8bde-4001-82e6-813b8940ca0e"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\ndif between naive bayes and logistic regression:logisitic regression is descriminative(just learns to \ndistinguish classes without learning much about it ) while bayes is\ngenerative(may generate pic of dog and see if it fits the features in order to separate, uses likelihood term); both are\nsupervised ML; ML component:feature representation of input, classification(sigmod,softmax),objective func(cross-entrpy loss),\nalgo to optimize objective func(stochastic gradient descent);sigmoid func takes real-valued number and maps it to range; \nnaive bayes has overly strong conditional independence assumptions; logistic regression generally works best on large docs\nor dataset  \n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n cost function/loss function distance between system output and gold output; \n\n## SLP Chapter 5.3 The cross-entropy loss function\n \nconditional maximum likelihood estimation\nloss function that prefers correct class labels\nof training examples\n \n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n Your summary here \n\n## SLP Chapter 7.1 Units\nTanh is smooth and \ndifferentiable and map outliers toward mean;;\n\n \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\nClassifier Component:\nfeature representation of input, classification function, \nobjective function, algo for optimizing objective function; \ndecision boundary=.5 (if more than output yes); learning in \nlogistic regression: how to get weights of model we learn the \nparameters which requires 2 components: 1(loss function telling us\ndistance between system output and gold output i.e cross\nentropy loss) 2(algo for optimizing objective function i.e. \nstochastic gradient descent to min loss func);cross entropy loss:\nnegative log likelihood loss choose param that max the log prob\nof true labels in training data; cross entropy loss( why does minimizing\nnegative log prob do what we want, we want loss to be smaller if the model's \nestimate is close to correct and we want the loss to be \nbigger if it is confused); gradient descent: used to find good\nsettings for weights and bias by minimizing loss func, gradient\ndescent is method that finds a min of a function by figuring out\nin which direction the function's slope is rising the most \nsteeply and moving the other direction; convex func has\nonly 1 min so gradient descent from any start will find min;\nmagnitude of the amount to move in gradient descent is the value of slope\nweight by learning rate, higher/faster learning rate means that we should move w\nmore on each step; stochasticgd chooses single rand example\nand updates so results can fluctuate; batch training is alt it computes\ngradient over entire dataset so give better est of which direction \nto move weight but is too long so we use mini batch(512/1024)\nloss is accumulated; overfitting: poor generalization and \nperformance on unseen test set; regularization is way to penalize\nlarge weights, regularization term is added to loss function; lasso(L1)\nridge(L2); neuron units apply nonlinear function (sigmoid,tanh, relu)\nFeed forward: multilayered all units connected no cycles, outputs\nfrom each layer are passed to units in the next layer and no outputs are passed\nto the sending layer; logistic regression: network with only one layer and a sigmoid\noutput; for deeper networks computing gradient requires looking at earlier \nlayers; backward differentiation:depends on computation graph\nimportance of computation graph comes from backward pass\nused to compute derivatives for weight update it uses chain rule\nwe need activation functions to be differentiable\n \n\n## Module 13 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter9_3 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_2 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_4 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_8 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_10 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_11 = \"\"\" Your summary here \"\"\"\n",
        "slp_chapter6_12 = \"\"\" Your summary here \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\" \n",
        "neural language model advantage over ngram LM:\n",
        "they don't need smoothing, handle longer histories,\n",
        "generalize over contexts of similar words, neural LM\n",
        "have higher predictive accuracy than ngram LM;disadvantage:\n",
        "slower to train than ngram LM; probability func is expressed product\n",
        "of conditional prob of next word given previous word using multilayer feed forward neural \n",
        "network; neural LM input is kgram of words, output is prob distribution\n",
        "over next word. k context words are like window each word has \n",
        "embedding vector; we want word meaning to encode:synonyms,antonyms, connotations,semantic roles,\n",
        "entailment;vector space models:each doc in collection is represented as a \n",
        "point in space; cosine distance words with small angle more similar\n",
        "to each other; term term matrix or term context: square matrix each word is row and \n",
        "col vector length is size of vocab rather than doc;tf-idf:term freq* inverse doc freq\n",
        "sparse vector: most elements are 0, dense: most are non 0; how to get\n",
        "dense vector: train classifier; word2vec: fast, easy to train; embeddings\n",
        "capture relational meanings\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"concept\" : \"definition\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1292
        },
        "outputId": "b60e1039-1b66-4df0-af83-17c985a8aafe"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\nClassifier Component:\nfeature representation of input, classification function, \nobjective function, algo for optimizing objective function; \ndecision boundary=.5 (if more than output yes); learning in \nlogistic regression: how to get weights of model we learn the \nparameters which requires 2 components: 1(loss function telling us\ndistance between system output and gold output i.e cross\nentropy loss) 2(algo for optimizing objective function i.e. \nstochastic gradient descent to min loss func);cross entropy loss:\nnegative log likelihood loss choose param that max the log prob\nof true labels in training data; cross entropy loss( why does minimizing\nnegative log prob do what we want, we want loss to be smaller if the model's \nestimate is close to correct and we want the loss to be \nbigger if it is confused); gradient descent: used to find good\nsettings for weights and bias by minimizing loss func, gradient\ndescent is method that finds a min of a function by figuring out\nin which direction the function's slope is rising the most \nsteeply and moving the other direction; convex func has\nonly 1 min so gradient descent from any start will find min;\nmagnitude of the amount to move in gradient descent is the value of slope\nweight by learning rate, higher/faster learning rate means that we should move w\nmore on each step; stochasticgd chooses single rand example\nand updates so results can fluctuate; batch training is alt it computes\ngradient over entire dataset so give better est of which direction \nto move weight but is too long so we use mini batch(512/1024)\nloss is accumulated; overfitting: poor generalization and \nperformance on unseen test set; regularization is way to penalize\nlarge weights, regularization term is added to loss function; lasso(L1)\nridge(L2); neuron units apply nonlinear function (sigmoid,tanh, relu)\nFeed forward: multilayered all units connected no cycles, outputs\nfrom each layer are passed to units in the next layer and no outputs are passed\nto the sending layer; logistic regression: network with only one layer and a sigmoid\noutput; for deeper networks computing gradient requires looking at earlier \nlayers; backward differentiation:depends on computation graph\nimportance of computation graph comes from backward pass\nused to compute derivatives for weight update it uses chain rule\nwe need activation functions to be differentiable\n \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 8705
        },
        "id": "T3cm1WV542_a",
        "outputId": "48e0d6f8-88f9-420f-c1c6-a342175718ac"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Expected utility: avg utility value of outcomes wighted by prob that outcome occurs\nUtility function: assigns single number to express the desirability of a state. Maximum expected utility(MEU): rational agent should choose action that \nmaxes agents expected utility. MEU is prescription for intelligent behavior. MEU formalizes but does not operationalize. If agent acts to max a utility\nfunction that reflects correctly performance measure then agent will get highest possible performance score (avg over all pos env). This is central\njustification for MEU. \n\n $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n each outcome of a lottery can be atomic or another lottery. reasonable preference relation axioms of utility theory:orderability(given 2 lotteries rational agent must\neither prefer one or equally; agent must decide)transitivity(if agent prefers A to B and prefers B to C then it prefers A to C)continuity(if some lottery B is between \nA and C preference then there is some probability for which agent will be indifferent between B and lottery that gets A with probability p and C with \nprob 1-p)substitutability(if agent is indifferent between 2 lotteries A and B then the agent is indifferent between 2+ complex that are the same\nexcept that B is substituted for A in one of them regardless of probabilities) monotonicity(suppose 2 lotteries have same outcomes, if A is prefered to B then agent\nmust prefer lottery that has a higher probability for A)decomposability(compound lottery can be reduced to simpler ones using the laws of prob; no fun in gambling rule)\nexistence of utility function, expected utility of lottery; theorem says utility functions exist not that they are unique; value function/ordinal utility: in deterministic\nenv agent only needs preference on ranking not numbers\n\n## AIMA Chapter 16.3 Utility Functions\n utility functions map lotteries to numbers; preference elicitation:work out human utility func by giving them choices and \nchecking;micromort(value of own life);utility of money is propotional to log of amount almost; certainty equivalent:\nvalue agent will accept for lottery; risk-neutral: linear curve; optimizer's curse tendency for estimated \nexpected utility to be too high; normative theory: should act; descriptive theory: actually act; certainty\neffect: attach to gains that are certain; framing effect: wording matters; anchoring effect: offer $200 and \nthen offer $55 one to make it a good deal; \n \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n Finite horizon: after certain time nothing matters;\nW/finite horizon optimal action in state depends on time left non stationary policy; \ndiscount factor: preference of current reward over future reward if close to 0 insignificant if in distant future; proper policy: guaranteed to reach terminal state; infinite sequences compared by average reward; optimal policy is independent of starting state if using discounted utilities with infinite horizon; utility of a state is the expected reward for the next transition plus discounted utility of the next state assuming agent chooses optimal action: aka Bellman; q function is expected utility of taking a given action in a given state \n \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n Contraction: basic concept used in showing that value iteration converges, function of one argument that when applied to 2 different inputs produces 2 outputs that are closer together, only has 1 fixed point, repeated application always reaches fixed point in limit; max norm: measures length of vector by \nabsolute value of its biggest component ; policy loss most you can loose by not taking opt policy; policy evaluation O(n^3); linear programming is a general approach for formulating constrained optimization problems; value iteration and policy iteration are offline algorithms \nReinforcement learning: has underlying MDP, process of taking actions learning reward and then updating world, agent receives feedback thru rewards and agent utility is defined by reward function; exploration: unknown, exploitation: known; observing a sample:choosing action and seeing something that results; in reinforcement learning: just like MDP but we don’t know T or R (must try out actions and states to learn); model-based learning: learn approx model based on experiences, learn empirical MDP then solve (value iteration); model-free: take action then compare what happened to what you thought was going to happen then update values based on that, we don’t keep track of reward or transition functions; passive reinforcement learning: policy evaluation, don’t know R or T, goal learn state values, input fixed policy,; temporal difference learning: update V(s) every time we experience transition, learn from every experience, policy is still fixed; exponential moving average: increase alpha can give converging values; \nTD value learning is model free way to do policy evaluation mimick bellman but with running sample avg, learn Q Vals not values. \n \n\n## Module 8 Lecture Notes\nMarkovian: probability of arriving at s' only depends on s and not history of earlier state. Need to define initial state, terminal state,\ntransition function(model), reward function, set of states, set of actions. MDP is sequential, fully observable, stochastic, additive rewards.\nMDP is nondeterministic search problems; one way of solving them is with expectimax. MDP solution is a policy. A policy takes in state and outputs \naction. Discounting: penalize actions taken in future (has less value); stationary preference even after adding equal discount same preference holds;\ninfinite utility solution: fixed # steps/finite horizon, discounting, absorbing state(eventually will reach a terminal state); MDP cares about expectimax\n; max of all expectimax calculation; value iteration is guaranteed to converge to unique optimal values (policy may converge before values do) we know this\nbecause 1.if discount is less than 1 then as k increases the values will converge 2.if tree has max depth M then vm holds actual untruncated values\nHow to calculate v's for fixed policy pi? make recursive bellman equations to updates time complexity=O(S^2); how to act: pick optimal qval\nalternative approach for opt vals: until converege(policy evaluation and policy improvement) this is called policy iteration;\npolicy vs value iteration:both(compute same thing all opt vals), value iter(every iter updates both vals and policy; don't\ntrack policy but max over actions implicitly recomputes it), policy iter(do several passes that update utilities with\nfixed policy, after policy is eval new policy is picked,either new policy is better or done); worst case minimax\nterminal function scale doesn't matter(insensitivity to monotonic transformation); normalized utilitiy(between 0 and 1)\nmoney doesn't behave as utility func; \n\n\n## Module 8 Definitions\n* **Utility function** - Your definition here\n* **Expected utility** - your definition here\n* **Etc.** - your definition here\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n Bandit: arms are independent\nagent can only work on one arm at a time; one armed bandit\nthe choice is pull again or no and you pull it T times (stopping time)\nGittin's index: optimal policy for any bandit, pull arm with highest gittins \nthen update values; first decision is O(n) then O(1);Bernoulli bandit: each arm \nproduces reward of 1 or 0 with a fixed but unknown probability;\nexploration bonus: higher index for arms only tried a few times\n;how to calc gittin index: upper confidence bound, Thompson sampling; \nselection problems: choosing best option when goal is max total \nnumber of successes over time, no index func exists;\noptimal bandit algo explore less than opt selection probs;\nbandit superprocess(BSP): each arm is a full MDP in its own\nright rather than being MRP with only 1 possible action;\nglobally optimal policy for bsp may include actions that\nare locally suboptimal from pov of mdp constituent where they are taken\nreason for this is availability of other mdp in which to act changes balance\nbetween short term and long term rewards in a component mdp.\narm is unaffected by opportunity cost: dominating policy; by adding\nactions to states it is always possible to create a relaxed mdp version\nso that it has dominating policy\n \n\n## AIMA Chapter 22.1 Learning from Rewards\nReinforcement learning(RL): agent interacts with world \nand gets rewards as sign of how well it is doing; RL is dif than MDP because\nagent isn't given MDP as a problem to solve, agent is in MDP, it may not know transition model\nor reward function has to act to learn; sparse rewards: win/loose no more signals \n;model based RL: agent uses transition model of env to interpret\nreward signals and make decisions about how to act, state estimation:\nthis is useful in partially observable; model free RL:\nagent either knows nor learns a transition model, it learns more direct 2 ways:\naction utility learning(q learning; agent learns q func, given this\nagent can choose what to do in state by taking action with highest qval)\npolicy search (reflex agent: policy that maps states to actions);\napprenticeship learning: training learning agent using  demonstrations\nrather than reward signals  \n \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n passive learning agent vs policy evaluation: \npassive learning does not know transition model which specifies prob of \nreach state s' from state s after doing action a; also doesn't know reward func;\ndirect utility est: utility of state is defined as expected total reward\nfrom that state onward; utility of state is determined by\nreward and expected utility of successor states; temporal difference:\ndifference in utilities between sucessive rates, temporal dif work by adjusting utility estimates towrd the ideal\nequilibrium; temporal difference (TD) doesn't need transition model to perform updates; adaptive dynamic programming (ADP) adjusts state\nto agree with all successors, TD adjusted states to agree w/observed successor, TD makes single adjustment per transition\nADP makes as many as needed; we can make more efficient versions of ADP by directly approximating algos for value or policy \niteration; approx algo can use min adjustment size that decreases as transition model become more accurate\n\n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n greedy in the limit of infinite exploration(GLIE): must try each action in each state\nabsorbing state: no actions have any effects and no rewards received; \n \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n Your summary here \n\n## AIMA Chapter 22.5 Generalization in Reinforcement Learning\n Your summary here \n\n## Module 9 Lecture Notes\n Temporal difference learning: policy fixed more values move toward value of whatever\nsuccessor occurs:running average; interpolation: update one component by some amount and the other by 1-that alount\n;q learning is based on samples, as Alpha gets closer to one we put more weight on the sample, as it gets closer to zero, we put more weight \non our previous estimates and so we're going to balance between those\nqlearning converges to optimal policy even if you're acting suboptimally \n(called off-policy learning) bad things: you have to explore sufficient, make\nlearning rate small but not decrease too quickly; when to explore: badness is not yet \nestablished; epsilon greedy (if prob is small go rand else go policy)\nregret is measure of your total mistake cost( dif between \nyour expected rewards and optimal expected rewards) it lets us quantify goodness of exploration\ntechnique, minimize regret: optimally\nlearning to be optimal; intuitive interpretation: adjust weights of active features\nif something bad happens blame the features associated with it and \nlater disprefer\n\n\n## Module 9 Definitions\n* **Term** - your definition here\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Your summary here \n\n## AIMA Chapter 12.2 Basic Probability Notation\n Your summary here \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n Your summary here \n\n## AIMA Chapter 12.4 Independence\n Your summary here \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n Your summary here \n\n## AIMA Chapter 12.6 Naive Bayes Models\n Your summary here \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Your summary here \n\n## SLP Chapter 3.1 N-Grams\n Your summary here \n\n## SLP Chapter 3.2 Evaluating Language Models\n Your summary here \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n Your summary here \n\n## SLP Chapter 3.4 Generalization and Zeros\n Your summary here \n\n## SLP Chapter 3.5 Smoothing\n Your summary here \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n Your summary here \n\n## Module 10 Lecture Notes\nprobabilistic reasoning givs\nus framework for managing beliefs and knowledge; observed values (evidence)\nunobserved var(states), model(how known relates to unknown); probabilistic \nmodel is joint distribution over set of rand vals; probabilistic inference:\ncompute a desired prob from other known prob (conditional from joint)\ninference by enumeration: evidence (given), query var(want to find out)\nhidden(not thinking but present); probabilistic lang model: goal compute\nprob of a sentence of words; markov assumption: prob depends only\non previous k words not entire context\n \n\n## Module 10 Definitions\n* **Term** - your definition here\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n Conditioning case: possible combination of values for parent nodes, each row sum must equal 1 \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Baysian network is correct representation of domain only if each\nnode is conditionally independent of its other predecessors given parents;it is impossible for knowledge engineer to create\nBaysian that violates axioms of probability;compactness of Baysian is (locally structrued/spare); locally struct:\neach subcomponent interacts directly with limited num of components associated w/linear growth; Bayes: conditional prob table\n2^k nums, complete network (2^k) *n nums; casualmodel= fewer and easier nums; each variable is conditionally independent\nof its non-descendants given parents;  \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n Your summary here \n\n## Module 11 Lecture Notes\n Unconditional independence is rare and doesn't help\nus make inferences about other variables; conditional independence is our most basic and robust form of knowledge\nabout uncertain environments; causal chains does not guarantee x independent\nof z; a path is active if each triple is active; grey circle means known/observed/ have evidence for;\nif all paths are inactive then independence is guaranteed; if 1 or more active then \nindependence not guaranteed (remember path not triples); inference: calculate useful quantity\nfrom a joint probability distribution\n \n\n## Module 11 Definitions\n* **Concept** - definition\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n Your summary here \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n Your summary here \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n Your summary here \n\n## AIMA Chapter 5.5 Stochastic Games\n Your summary here \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n Your summary here \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n Your summary here \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n Your summary here \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n Your summary here \n\n## SLP Chapter 4.3 Worked Example\n Your summary here \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n Your summary here \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Your summary here \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n Your summary here \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n Your summary here \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n Your summary here \n\n## SLP Chapter 4.9 Statistical Significance Testing\n Your summary here \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n Your summary here \n\n## Module 12 Lecture Notes\nmodel-based classification:\nbuild model(bayes net) where label and features are rand variables, instantiate any ovserved features\nquery for distribution of label condition on fetures; Naive Bayes:\nassume all features that we're using to represent inputs are independent\nand only depend on label, total num of parameters is linear in num of features\ninference for naive bayes: compute posterior distribution over label var Y\nestimates of local conditional probability tables (called parameters)\nrelative freq parameters will overfit training data; we need to smooth\nor regularize estimates; empirical rate: estimate that max likelihood of data\nlaplace performs poorly for large x or y; for real classification\nsmoothing is critical; 2 unknowns: parameters(probabilities) and hyperparameters(amount\n/type of smoothing to do); learn parameters from training data, tune hyperparameters on different\ndata, for each val of hyperparameter train and test held out data, \nchoose best val and do final test on test data; multiclass decision rule:\nprediction highest score wins; properties of perceptrons:separability\n(true if some parameters get the training set perfectly correct),\nconvergence(if training is separable, perceptron), mistake bound (max num of msitakes related to margin or degree of\nsparability;perceptron probs: noise(if data isn't separable weight might thrash; \navg weight vectors over time can help), mediocre generalization(find a barely separating solution),\novertraining(test/heldout accuracy usually rises then falls)\nmax step size: bad to make updates that are too large, example may be\nlabeled incorrectly, not enough features, solution:cap max pos val\nof the weird r with a constant C, corresponds to optimization that assumes non-separable\ndata, usually converges faster than perceptron and better on noisy data;\nsupport vector: help find boundary the bounday between 2\nclassed that maximizes the margin; naive bayes:builds model training data,\ngives prediction probabilities, strong assumptions about feature\nindependence, once pass thru data; perceptrons/mira: less assumptions about\ndata,usually more accurate\n\n \n\n## Module 12 Definitions\n* **Concept** - definition\n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\ndif between naive bayes and logistic regression:logisitic regression is descriminative(just learns to \ndistinguish classes without learning much about it ) while bayes is\ngenerative(may generate pic of dog and see if it fits the features in order to separate, uses likelihood term); both are\nsupervised ML; ML component:feature representation of input, classification(sigmod,softmax),objective func(cross-entrpy loss),\nalgo to optimize objective func(stochastic gradient descent);sigmoid func takes real-valued number and maps it to range; \nnaive bayes has overly strong conditional independence assumptions; logistic regression generally works best on large docs\nor dataset  \n\n\n## SLP Chapter 5.2 Learning in Logistic Regression\n cost function/loss function distance between system output and gold output; \n\n## SLP Chapter 5.3 The cross-entropy loss function\n \nconditional maximum likelihood estimation\nloss function that prefers correct class labels\nof training examples\n \n\n## SLP Chapter 5.4 Gradient Descent\n Your summary here \n\n## SLP Chapter 5.5 Regularization\n Your summary here \n\n## SLP Chapter 5.6 Multinomial logistic regression\n Your summary here \n\n## SLP Chapter 7.1 Units\nTanh is smooth and \ndifferentiable and map outliers toward mean;;\n\n \n\n## SLP Chapter 7.2 The XOR problem\n Your summary here \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n Your summary here \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n Your summary here \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n Your summary here \n\n## SLP Chapter 7.6 Training Neural Nets\n Your summary here \n\n## SLP Chapter 7.7 Training the neural language model\n Your summary here \n\n## Module 13 Lecture Notes\nClassifier Component:\nfeature representation of input, classification function, \nobjective function, algo for optimizing objective function; \ndecision boundary=.5 (if more than output yes); learning in \nlogistic regression: how to get weights of model we learn the \nparameters which requires 2 components: 1(loss function telling us\ndistance between system output and gold output i.e cross\nentropy loss) 2(algo for optimizing objective function i.e. \nstochastic gradient descent to min loss func);cross entropy loss:\nnegative log likelihood loss choose param that max the log prob\nof true labels in training data; cross entropy loss( why does minimizing\nnegative log prob do what we want, we want loss to be smaller if the model's \nestimate is close to correct and we want the loss to be \nbigger if it is confused); gradient descent: used to find good\nsettings for weights and bias by minimizing loss func, gradient\ndescent is method that finds a min of a function by figuring out\nin which direction the function's slope is rising the most \nsteeply and moving the other direction; convex func has\nonly 1 min so gradient descent from any start will find min;\nmagnitude of the amount to move in gradient descent is the value of slope\nweight by learning rate, higher/faster learning rate means that we should move w\nmore on each step; stochasticgd chooses single rand example\nand updates so results can fluctuate; batch training is alt it computes\ngradient over entire dataset so give better est of which direction \nto move weight but is too long so we use mini batch(512/1024)\nloss is accumulated; overfitting: poor generalization and \nperformance on unseen test set; regularization is way to penalize\nlarge weights, regularization term is added to loss function; lasso(L1)\nridge(L2); neuron units apply nonlinear function (sigmoid,tanh, relu)\nFeed forward: multilayered all units connected no cycles, outputs\nfrom each layer are passed to units in the next layer and no outputs are passed\nto the sending layer; logistic regression: network with only one layer and a sigmoid\noutput; for deeper networks computing gradient requires looking at earlier \nlayers; backward differentiation:depends on computation graph\nimportance of computation graph comes from backward pass\nused to compute derivatives for weight update it uses chain rule\nwe need activation functions to be differentiable\n \n\n## Module 13 Definitions\n* **Concept** - definition\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Your summary here \n\n## SLP Chapter 9.2 Recurrent Neural Networks\n Your summary here \n\n## SLP Chapter 9.3 RNNs as Language Models\n Your summary here \n\n## SLP Chapter 6.1 Lexical Semantics\n Your summary here \n\n## SLP Chapter 6.2 Vector Semantics\n Your summary here \n\n## SLP Chapter 6.3 Words and Vectors\n Your summary here \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n Your summary here \n\n## SLP Chapter 6.8 Word2vec\n Your summary here \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n Your summary here \n\n## SLP Chapter 6.11 Bias and Embeddings\n Your summary here \n\n## SLP Chapter 6.12 Evaluating Vector Models\n Your summary here \n\n## Module 13 Lecture Notes\nClassifier Component:\nfeature representation of input, classification function, \nobjective function, algo for optimizing objective function; \ndecision boundary=.5 (if more than output yes); learning in \nlogistic regression: how to get weights of model we learn the \nparameters which requires 2 components: 1(loss function telling us\ndistance between system output and gold output i.e cross\nentropy loss) 2(algo for optimizing objective function i.e. \nstochastic gradient descent to min loss func);cross entropy loss:\nnegative log likelihood loss choose param that max the log prob\nof true labels in training data; cross entropy loss( why does minimizing\nnegative log prob do what we want, we want loss to be smaller if the model's \nestimate is close to correct and we want the loss to be \nbigger if it is confused); gradient descent: used to find good\nsettings for weights and bias by minimizing loss func, gradient\ndescent is method that finds a min of a function by figuring out\nin which direction the function's slope is rising the most \nsteeply and moving the other direction; convex func has\nonly 1 min so gradient descent from any start will find min;\nmagnitude of the amount to move in gradient descent is the value of slope\nweight by learning rate, higher/faster learning rate means that we should move w\nmore on each step; stochasticgd chooses single rand example\nand updates so results can fluctuate; batch training is alt it computes\ngradient over entire dataset so give better est of which direction \nto move weight but is too long so we use mini batch(512/1024)\nloss is accumulated; overfitting: poor generalization and \nperformance on unseen test set; regularization is way to penalize\nlarge weights, regularization term is added to loss function; lasso(L1)\nridge(L2); neuron units apply nonlinear function (sigmoid,tanh, relu)\nFeed forward: multilayered all units connected no cycles, outputs\nfrom each layer are passed to units in the next layer and no outputs are passed\nto the sending layer; logistic regression: network with only one layer and a sigmoid\noutput; for deeper networks computing gradient requires looking at earlier \nlayers; backward differentiation:depends on computation graph\nimportance of computation graph comes from backward pass\nused to compute derivatives for weight update it uses chain rule\nwe need activation functions to be differentiable\n \n\n## Module 14 Definitions\n* **Concept** - definition\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}