{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "“Template_for CIS521_midterm_2_notes.ipynb”的副本",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brNDGX-kvzds"
      },
      "source": [
        "# Template for Notes for CIS 521 Midterm 2\n",
        "\n",
        "The midterm exam for CIS 521 will be **open notes**. No other matierals are allowed for the exam - you may not use the textbook, the internet, a Python envionement, or communicate with anyone during the exam. \n",
        "\n",
        "This python notebook is where you should save all of your notes.  It provides a structured template that allows you to add summaries for each of the textbook sections that are part of the required reading, plus room for lecture notes and definitions.\n",
        "\n",
        "Your summaries must be written in your own words.  Do not copy and paste from the textbook.  Do not share your notes with classmates, or copy any of your classmates' notes.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPtM8g7R0Pds"
      },
      "source": [
        "# Instructions\n",
        "\n",
        "1. Make a copy of this Google Colab file. From the colab menu, pick **File > Save a copy in Drive**.  That will save a copy of this notebook in your Google drive.\n",
        "2. Go through each section of the notebook, and fill in all the summaries for each chapter and its corresponding sections.  Be sure to save your work frequently!\n",
        "3. Read the notes below about how you can format your summaries.\n",
        "4. In addition to these notes, you are allowed 1 page (both sides) for figures or handwritten notes.\n",
        "5. Submit your version of this notebook to Gradescope before you start the exam.\n",
        "6. If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print: \n",
        "* Run the last cell of this notebook\n",
        "* Click the \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n",
        "\n",
        "## Formatting Notes\n",
        "\n",
        "1. The max length for each section's summary is 500 words / 3000 characters.\n",
        "2. You can also write a 2000 word summary of the lectures for each module.\n",
        "3. You may also write definitions of concepts for each module.  The definitions should be written in your own words (not copy and pasted), and they should be no more than 20 words long for each concept. \n",
        "2. You can use Markdown formatting for your summaries. \n",
        "3. You can use Latex formatting for equations. Put two dollar signs before and after your equation like this: \n",
        "$$ \\epsilon = \\frac{1}{mc^2} $$\n",
        "4. When you write python code or psuedocode algorithms, you should surround it with triple backticks like this:\n",
        "```\n",
        "def function():\n",
        "      print('hello world')\n",
        "```\n",
        "5. Write \\ whenever you want to keep writing on the next line without a newline being inserted."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cw5IpDLCZa4W"
      },
      "source": [
        "## Example summary\n",
        "\n",
        "This is a sample summary of [Section 2.1.1](https://web.stanford.edu/~jurafsky/slp3/2.pdf ) of Jurfasky and Martin's Speech and Language Processing's book\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzaavW96tcxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ef114a6-3d81-4d48-83f4-f3998e9fa417"
      },
      "source": [
        "chapter_2_1_example = \"\"\"\n",
        "Regular expressions are an algebraic notation for characterizing \\\n",
        "a set of strings that are useful for searching for a pattern or through a corpus.\\\n",
        "Only the first matching result will be shown. The search string is case sensitive, \\\n",
        "but it can consist of a single character or a sequence of characters. \\\n",
        "While square braces specify a disjunction of characters, the dash can more easily look \\\n",
        "for characters over a range. The caret specifies the non-existence of a character. \\\n",
        "Additionally, the (?) is used to for flexibility on a single character, meaning \\\n",
        "\"the preceding character or nothing\". More importantly, the Kleene star means zero or \\\n",
        "more occurrences of the immediately previous character or regular expression in square braces.\\\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "length = len(chapter_2_1_example.split(\" \"))\n",
        "print(\"The example summary is {len} words long, so you could write an additional \\\n",
        "{remaining} words.\".format(len=length, remaining=(500-length)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The example summary is 116 words long, so you could write an additional 384 words.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3Uk9gQTevcj"
      },
      "source": [
        "## Example summary length\n",
        "\n",
        "The maximum summary length for each section is 500 words / 3000 characters.  Hopefully you'll find that to be quite generous. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DvZ57A-djo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b97e7ecf-2c48-4458-dc01-c57427e61768"
      },
      "source": [
        "# This is an example of a 500 word placeholder\n",
        "placeholder = \"\"\"ultrices vitae auctor eu augue ut lectus arcu bibendum at varius \\\n",
        " vel pharetra vel turpis nunc eget lorem dolor sed viverra ipsum nunc aliquet \\\n",
        " bibendum enim facilisis gravida neque convallis a cras semper auctor neque vitae \\\n",
        " tempus quam pellentesque nec nam aliquam sem et tortor consequat id porta nibh \\\n",
        " venenatis cras sed felis eget velit aliquet sagittis id consectetur purus ut \\\n",
        " faucibus pulvinar elementum integer enim neque volutpat ac tincidunt vitae semper \\ \n",
        " quis lectus nulla at volutpat diam ut venenatis tellus in metus vulputate eu \\\n",
        " scelerisque felis imperdiet proin fermentum leo vel orci porta non pulvinar neque \\ \n",
        " laoreet suspendisse interdum consectetur libero id faucibus nisl tincidunt eget \\\n",
        " nullam non nisi est sit amet facilisis magna etiam tempor orci eu lobortis \\\n",
        " elementum nibh tellus molestie nunc non blandit massa enim nec dui nunc mattis \\\n",
        " enim ut tellus elementum sagittis vitae et leo duis ut diam quam nulla porttitor \\\n",
        " massa id neque aliquam vestibulum morbi blandit cursus risus at ultrices mi tempus \\\n",
        " imperdiet nulla malesuada pellentesque elit eget gravida cum sociis natoque \\\n",
        " penatibus et magnis dis parturient montes nascetur ridiculus mus mauris vitae \\\n",
        " ultricies leo integer malesuada nunc vel risus commodo viverra maecenas accumsan \\\n",
        " lacus vel facilisis volutpat est velit egestas dui id ornare arcu odio ut sem nulla \\\n",
        " pharetra diam sit amet nisl suscipit adipiscing bibendum est ultricies integer \\\n",
        " quis auctor elit sed vulputate mi sit amet mauris commodo quis imperdiet massa \\\n",
        " tincidunt nunc pulvinar sapien et ligula ullamcorper malesuada proin libero nunc \\\n",
        " consequat interdum varius sit amet mattis vulputate enim nulla aliquet porttitor \\\n",
        " lacus luctus accumsan tortor posuere ac ut consequat semper viverra nam libero \\\n",
        " justo laoreet sit amet cursus sit amet dictum sit amet justo donec enim diam \\\n",
        " vulputate ut pharetra sit amet aliquam id diam maecenas ultricies mi eget mauris \\\n",
        " pharetra et ultrices neque ornare aenean euismod elementum nisi quis eleifend quam \\\n",
        " adipiscing vitae proin sagittis nisl rhoncus mattis rhoncus urna neque viverra justo \\\n",
        " nec ultrices dui sapien eget mi proin sed libero enim sed faucibus turpis in \\\n",
        " eu mi bibendum neque egestas congue quisque egestas diam in arcu cursus euismod \\\n",
        " quis viverra nibh cras pulvinar mattis nunc sed blandit libero volutpat sed cras \\\n",
        " ornare arcu dui vivamus arcu felis bibendum ut tristique et egestas quis ipsum \\\n",
        " suspendisse ultrices gravida dictum fusce ut placerat orci nulla pellentesque \\\n",
        " dignissim enim sit amet venenatis urna cursus eget nunc scelerisque viverra mauris \\\n",
        " in aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod in \\\n",
        " pellentesque massa placerat duis ultricies lacus sed turpis tincidunt id aliquet \\\n",
        " risus feugiat in ante metus dictum at tempor commodo ullamcorper a lacus vestibulum \\\n",
        " sed arcu non odio euismod lacinia at quis risus sed vulputate odio ut enim blandit \\\n",
        " volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in \\\n",
        " dictum non consectetur a erat nam at lectus urna\"\"\"\n",
        "\n",
        "print(\"The length in words is\", len(placeholder.split(' ')))\n",
        "print(\"The length in characters is\", len(placeholder))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The length in words is 500\n",
            "The length in characters is 3030\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5yG3-YncJfTB"
      },
      "source": [
        "# Permissions to use your summaries\n",
        "\n",
        "We'd like to use your summaries to help develop better educational resources like [remember.school](https://remember.school) or a question-answering system about the textbook.  \n",
        "\n",
        "1. Please enter your name.  \n",
        "\n",
        "2. If you grant permission for us to use your summaries under a [Creative Commons License](https://creativecommons.org/licenses/by/4.0/), please set ```permission = True```.  If you do not grant permission, please set it to false. Your answer won't affect your grade.\n",
        "\n",
        "3. If you grant us permission, and you'd like to remain anonymous, set ```anonymous = True```.  We'll remove your name before sharing your summaries with anyone. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RECnJArJew4"
      },
      "source": [
        "name = \"Minzhong Gao\"\n",
        "permission = False\n",
        "anonymous = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZZXyrV-xuAp"
      },
      "source": [
        "# Module 8 - Markov Decision Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NfE1CKxzS-"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 16, “Making Simple Decisions” (16.1-16.3)\n",
        "\n",
        "sample_equation = r\"\"\" $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \"\"\"\n",
        "\n",
        "chapter16_1 = \"\"\" This section reports combining beliefs and desires under uncertainty. The agent has available actions. There may be changes in the current state. We believe that the agent assigns a probability to each possible current state P(s). There may also be uncertainty about the action outcomes described by the transition model.\"\"\"  + sample_equation\n",
        "chapter16_2 =  \"\"\" An observer can learn about the utility function representing the agent's goal by following a rational agent's performance. \"\"\"\n",
        "chapter16_3 =  \"\"\" Utility functions map lotteries to real numbers. They need to accept the premises of order ability, transitivity, continuity, substitutability, monotonicity, and decomposability. An agent can have any choices he likes.  \"\"\"\n",
        "\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 17, “Making Complex Decision” (17.1-17.2)\n",
        "chapter17_1 = \"\"\" This section describes that the introduction of skepticism leads MDPs closer to the real world than deterministic search puzzles. Accordingly, MDPs have been studied in several fields, including AI, operations research, economics, and control theory. Several solution algorithms have been proposed. \"\"\"\n",
        "chapter17_2 =  \"\"\" This section presents four different algorithms for solving MDPs. The first three, value iteration, policy iteration, and linear programming, generate accurate solutions offline. The fourth is a group of online approximate algorithms that includes Monte Carlo planning. \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module8_lecture_notes = \"\"\" This module turns to stochastic or probabilistic task environments. We start by introducing a new kind of formalization called Markov Decision Processes, which are helpful for fully observable task environments. We have to make decisions, and the outcome for actions is non-deterministic. MDPs also look into new algorithms, such as Policy Iteration and Value Iteration, and the essential concepts and additive rewards. \"\"\"\n",
        "\n",
        "\n",
        "# Definitions of concepts from Module 8.\n",
        "# You can define any term in the textbook or lecture, \n",
        "# but you most use your own words for the definitions.\n",
        "# Each definition should be at most 25 words long.\n",
        "\n",
        "module8_definitions = {\n",
        "\"utility function\" : \"a utility function assigns values to certain actions that the AI system can take\",\n",
        "\"expected utility\" : \"A policy can be evaluated by determining its expected utility for an agent following the policy\",\n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elCCgY64spuq"
      },
      "source": [
        "## Preview of your notes for Module 8\n",
        "\n",
        "Here's a preview of your how your notes will be displayed.  At the end of this Python notebook, we'll aggregate your notes for all modules together into a single display.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYYeNcdM_maE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 852
        },
        "outputId": "2eee8188-d833-4a44-ec58-444aa6861c44"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "from IPython.display import display, Markdown, Latex\n",
        "\n",
        "module8_sections = [\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter16_1),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter16_2),\n",
        "  (\"AIMA Chapter 16.3 Utility Functions\", chapter16_3),\n",
        "  (\"AIMA Chapter 17.1 Sequential Decision Problems\", chapter17_1),\n",
        "  (\"AIMA Chapter 17.2 Algorithms for MDPs\", chapter17_2),\n",
        "  (\"Module 8 Lecture Notes\", module8_lecture_notes),\n",
        "]\n",
        "\n",
        "\n",
        "def length_check(sections):\n",
        "  warnings = \"\"\n",
        "  for section, summary in sections:\n",
        "    max_word_length = 500\n",
        "    max_character_length = 3000\n",
        "    if \"lecture\" in section.lower():\n",
        "      # It's OK to do longer summaries of the lectures\n",
        "      max_word_length = 2000\n",
        "      max_character_length = 12000  \n",
        "    if (len(summary) > max_character_length or len(summary.split(' ')) > max_word_length):\n",
        "      warnings += \"WARNING: Your summary of '{section}' is too long!\\n\".format(section=section)\n",
        "  return warnings\n",
        "\n",
        "def length_check_definitions(definitions):\n",
        "  warnings = \"\"\n",
        "  for concept, definition in definitions.items():\n",
        "    if(len(definition.split(' ')) > 25):\n",
        "      warnings += \"WARNING: Your definition of '{concept}' is too long!\\n\".format(concept=concept)\n",
        "  return warnings\n",
        "\n",
        "\n",
        "def format_module(module_num, title, sections, definitions, show_placeholder=False):\n",
        "  markdown = \"\"\n",
        "\n",
        "  # Check summary lengths\n",
        "  warnings = length_check(sections)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  # Check definition lengths\n",
        "  warnings = length_check_definitions(definitions)\n",
        "  if warnings != \"\":\n",
        "    markdown += \"## \" + warnings\n",
        "\n",
        "  markdown += \"# Module {num}: {title}\\n\".format(num=module_num, title=title)\n",
        "\n",
        "\n",
        "  for section, summary in sections:\n",
        "    if show_placeholder and summary == \" Your summary here \":\n",
        "      summary = placeholder\n",
        "    markdown += \"## {section}\\n{summary}\\n\\n\".format(section=section, summary=summary)\n",
        "  \n",
        "  markdown += \"## Module {num} Definitions\\n\".format(num=module_num)\n",
        "  for concept, definition in definitions.items():\n",
        "    markdown += \"* **{concept}** - {definition}\\n\".format(concept=concept.capitalize(), definition=definition)\n",
        "  return markdown\n",
        "\n",
        "\n",
        "module8_markdown = format_module(8, \"Markov Decision Processes\", module8_sections, module8_definitions)\n",
        "display(Markdown(module8_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n This section reports combining beliefs and desires under uncertainty. The agent has available actions. There may be changes in the current state. We believe that the agent assigns a probability to each possible current state P(s). There may also be uncertainty about the action outcomes described by the transition model. $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n An observer can learn about the utility function representing the agent's goal by following a rational agent's performance. \n\n## AIMA Chapter 16.3 Utility Functions\n Utility functions map lotteries to real numbers. They need to accept the premises of order ability, transitivity, continuity, substitutability, monotonicity, and decomposability. An agent can have any choices he likes.  \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n This section describes that the introduction of skepticism leads MDPs closer to the real world than deterministic search puzzles. Accordingly, MDPs have been studied in several fields, including AI, operations research, economics, and control theory. Several solution algorithms have been proposed. \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n This section presents four different algorithms for solving MDPs. The first three, value iteration, policy iteration, and linear programming, generate accurate solutions offline. The fourth is a group of online approximate algorithms that includes Monte Carlo planning. \n\n## Module 8 Lecture Notes\n This module turns to stochastic or probabilistic task environments. We start by introducing a new kind of formalization called Markov Decision Processes, which are helpful for fully observable task environments. We have to make decisions, and the outcome for actions is non-deterministic. MDPs also look into new algorithms, such as Policy Iteration and Value Iteration, and the essential concepts and additive rewards. \n\n## Module 8 Definitions\n* **Utility function** - a utility function assigns values to certain actions that the AI system can take\n* **Expected utility** - A policy can be evaluated by determining its expected utility for an agent following the policy\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xRbdqDox2zD"
      },
      "source": [
        "#Module 9 - Reinforcement Learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpBvu7EMx7sy"
      },
      "source": [
        "# Russell and Norvig, AIMA Section 17.3 “Bandit Problems”\n",
        "chapter17_3 =  \"\"\" The extraordinary thing about the Gittins index is that it provides a clear optimal policy for the bandit problem. We can pull the arm that has the highest Gittins index, then update the Gittins indices.  \"\"\"\n",
        "\n",
        "# Russell and Norvig, AIMA Chapter 22 “Reinforcement Learning” (Sections 22.1-22.5)\n",
        "chapter22_1 =  \"\"\" This section discusses passive reinforcement learning, where the policy of the agent is fixed. The task is to learn the utilities of states, which could also involve learning a scene model.   \"\"\"\n",
        "chapter22_2 =  \"\"\" The passive learning task is comparable to the policy evaluation task, part of the policy iteration algorithm. The distinction is that the passive learning agent does not know the reward function and transition model, which defines the probability of reaching state from the state after doing the action. \"\"\"\n",
        "chapter22_3 =  \"\"\" This section covers active reinforcement learning, where the agent must figure out what to do. The main issue is exploration, where an agent must undergo as much as possible of its environment to discover how to behave in it. \"\"\"\n",
        "chapter22_4 =  \"\"\" This section discusses functions approximation which makes it possible to represent utility functions for vast state spaces. Still, more importantly, it provides for inductive generalization, in which the agent can conclude from states it has visited to the states it has not yet seen. \"\"\"\n",
        "chapter22_5 =  \"\"\" This section discusses the final approach for reinforcement learning problems, which is called policy search. In some ways, policy search is the easiest of all the methods in this chapter. The idea of policy search is to sustain twiddling the policy as long as its performance upgrades, then stop.  \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module9_lecture_notes = \"\"\" This module investigates situations where a transition and a reward function are not given to us but must be acquired through interaction with the world. In reinforcement learning, every time an agent takes action in the world, it receives feedback in the form of a reward, and so the agent wants to maximize its expected rewards. We look at several different learning techniques for reinforcement learning&colon. \"\"\"\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module9_definitions = {\n",
        "\"passive learning agent\" : \"an agent already has a fixed policy that determines its actions\",\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydOK__lwWgdp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 753
        },
        "cellView": "form",
        "outputId": "2d7b45eb-11c3-4f6d-f665-4f46ede08566"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "\n",
        "module9_sections = [\n",
        "  (\"AIMA Chapter 17.3 Bandit Problems\", chapter17_3),\n",
        "  (\"AIMA Chapter 22.1 Learning from Rewards\", chapter22_1),\n",
        "  (\"AIMA Chapter 22.2 Passive Reinforcement Learning\", chapter22_2),\n",
        "  (\"AIMA Chapter 22.3 Active Reinforcement Learning\", chapter22_3),\n",
        "  (\"AIMA Chapter 22.4 Generalization in Reinforcement Learning\", chapter22_4),\n",
        "  (\"Module 9 Lecture Notes\", module9_lecture_notes),\n",
        "]\n",
        "\n",
        "module9_markdown = format_module(9, \"Reinforcement Learning\", module9_sections, module9_definitions)\n",
        "display(Markdown(module9_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n The extraordinary thing about the Gittins index is that it provides a clear optimal policy for the bandit problem. We can pull the arm that has the highest Gittins index, then update the Gittins indices.  \n\n## AIMA Chapter 22.1 Learning from Rewards\n This section discusses passive reinforcement learning, where the policy of the agent is fixed. The task is to learn the utilities of states, which could also involve learning a scene model.   \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n The passive learning task is comparable to the policy evaluation task, part of the policy iteration algorithm. The distinction is that the passive learning agent does not know the reward function and transition model, which defines the probability of reaching state from the state after doing the action. \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n This section covers active reinforcement learning, where the agent must figure out what to do. The main issue is exploration, where an agent must undergo as much as possible of its environment to discover how to behave in it. \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n This section discusses functions approximation which makes it possible to represent utility functions for vast state spaces. Still, more importantly, it provides for inductive generalization, in which the agent can conclude from states it has visited to the states it has not yet seen. \n\n## Module 9 Lecture Notes\n This module investigates situations where a transition and a reward function are not given to us but must be acquired through interaction with the world. In reinforcement learning, every time an agent takes action in the world, it receives feedback in the form of a reward, and so the agent wants to maximize its expected rewards. We look at several different learning techniques for reinforcement learning&colon. \n\n## Module 9 Definitions\n* **Passive learning agent** - an agent already has a fixed policy that determines its actions\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46MQLR-Gx8dZ"
      },
      "source": [
        "#Module 10 - Probabilities and Language Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BN0b1eVYx-nd"
      },
      "source": [
        "# Russell and Norvig, AIMA 4th Edition, AIMA Chapter 12 “Quantifying Uncertainty” (Sections 12.1-12.7)\n",
        "chapter12_1 = \"\"\" Therefore, the rational decision depends on the relative influence of various goals and the possibility of achieving them. This section sharpens these concepts to develop the general theories of hypothetical reasoning and rational decisions. \"\"\"\n",
        "chapter12_2 = \"\"\" This section introduces the language of probability theory, traditionally informal, written by mathematicians for other mathematicians. Here, we take a path more suited to the needs of AI and relate it with the ideas of formal logic. \"\"\"\n",
        "chapter12_3 = \"\"\" This section describes a manageable method for probabilistic inference—the computation of posterior probabilities for query propositions given experimental evidence. We use the full joint distribution as the “knowledge base” to obtain solutions to all questions. We also propose several valuable methods for manipulating equations involving probabilities.  \"\"\"\n",
        "chapter12_4 = \"\"\" This section introduces that independence assertions can help reduce the volume of the domain representation and the complexity of the inference puzzle when available.  \"\"\"\n",
        "chapter12_5 = \"\"\" This section introduces that conditional independence assertions can provide probabilistic systems to scale up. Furthermore, they are much more generally available than absolute independence assertions. \"\"\"\n",
        "chapter12_6 = \"\"\" The Naive Bayes model is often used in cases where the “effect” variables are not strictly independent given the cause variable. The naive Bayes systems often work very well in practice, even when the conditional independence assumption is false.  \"\"\"\n",
        "chapter12_7 = \"\"\" Uncertainty arises because the agent’s sensors give only partial information about the world. Pure logical inference can infer nothing about which square is most likely to be safe so that a logical agent might choose randomly. A probabilistic agent can do much better than a logical agent. \"\"\"\n",
        "\n",
        "\n",
        "# Jurafsky and Martin, Speech and Language Processing Chapter 3 “N-Gram Language Models”  (Sections 3.1-3.5,3.7)\n",
        "SLP_chapter3_1 = \"\"\" This section describes representing and calculating language model probabilities in log format as log probabilities. Multiplying sufficient n-grams together would end in numerical underflow. By adopting log probabilities instead of raw probabilities, we can get numbers that are not as small.  \"\"\"\n",
        "SLP_chapter3_2 = \"\"\" The best way to evaluate the production of a language model is to install it in an application and estimate how much the application develops. Such end-to-end evaluation is called extrinsic evaluation.  \"\"\"\n",
        "SLP_chapter3_3 = \"\"\" This section introduces that the n-gram model depends on the training corpus, like many statistical models. One indication of this is that the probabilities often encode specific facts about a given training corpus. \"\"\"\n",
        "SLP_chapter3_4 = \"\"\" This section introduces smoothing. To keep a language model from designating zero probability to these hidden events, we will have to shave off a bit of probability mass from some more joint events and give it to the events we’ve never seen.  \"\"\"\n",
        "SLP_chapter3_5 = \"\"\" This section introduces the interpolated Kneser-Ney algorithm, the most commonly used and best performing n-gram smoothing method. \"\"\"\n",
        "SLP_chapter3_7 = \"\"\" A better n-gram model assigns a greater likelihood to the test data, and perplexity is a normalized version of the possibility of the test set. \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module10_lecture_notes = \"\"\" This module reviews the basics of probability, an essential concept to master in artificial intelligence. We review probability theory basics, including the axioms of probability, an event, a probabilistic model, and much more. To show some practical applications of these concepts, we look at probabilistic language models. This model power thing like autocomplete on your smartphone. \"\"\"\n",
        "\n",
        "\n",
        "# Each definition should be at most 25 words long.\n",
        "module10_definitions = {\n",
        "\"Decision theory\" : \"Decision theory connects the agent’s beliefs and desires, setting the best action as the one that maximizes expected utility.\",\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvtN3p7ET3Tx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "d6b1d37d-34de-40f0-ec47-b594d8b254af"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module10_sections = [\n",
        "  (\"AIMA Chapter 12.1 Acting under Uncertainty\", chapter12_1),\n",
        "  (\"AIMA Chapter 12.2 Basic Probability Notation\", chapter12_2),\n",
        "  (\"AIMA Chapter 12.3 Inference Using Full Joint Distributions\", chapter12_3),\n",
        "  (\"AIMA Chapter 12.4 Independence\", chapter12_4),\n",
        "  (\"AIMA Chapter 12.5 Bayes’ Rule and Its Use\", chapter12_5),\n",
        "  (\"AIMA Chapter 12.6 Naive Bayes Models\", chapter12_6),\n",
        "  (\"AIMA Chapter 12.7 The Wumpus World Revisited\", chapter12_7),\n",
        "  (\"SLP Chapter 3.1 N-Grams\", SLP_chapter3_1),\n",
        "  (\"SLP Chapter 3.2 Evaluating Language Models\", SLP_chapter3_2),\n",
        "  (\"SLP Chapter 3.3 Sampling sentences from a language model\", SLP_chapter3_3),\n",
        "  (\"SLP Chapter 3.4 Generalization and Zeros\", SLP_chapter3_4),\n",
        "  (\"SLP Chapter 3.5 Smoothing\", SLP_chapter3_5),\n",
        "  (\"SLP Chapter 3.7 Huge Language Models and Stupid Backoff\", SLP_chapter3_7),\n",
        "  (\"Module 10 Lecture Notes\", module10_lecture_notes),\n",
        "]\n",
        "\n",
        "module10_markdown = format_module(10, \"Probabilies and Language Models\", module10_sections, module10_definitions)\n",
        "display(Markdown(module10_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Therefore, the rational decision depends on the relative influence of various goals and the possibility of achieving them. This section sharpens these concepts to develop the general theories of hypothetical reasoning and rational decisions. \n\n## AIMA Chapter 12.2 Basic Probability Notation\n This section introduces the language of probability theory, traditionally informal, written by mathematicians for other mathematicians. Here, we take a path more suited to the needs of AI and relate it with the ideas of formal logic. \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n This section describes a manageable method for probabilistic inference—the computation of posterior probabilities for query propositions given experimental evidence. We use the full joint distribution as the “knowledge base” to obtain solutions to all questions. We also propose several valuable methods for manipulating equations involving probabilities.  \n\n## AIMA Chapter 12.4 Independence\n This section introduces that independence assertions can help reduce the volume of the domain representation and the complexity of the inference puzzle when available.  \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n This section introduces that conditional independence assertions can provide probabilistic systems to scale up. Furthermore, they are much more generally available than absolute independence assertions. \n\n## AIMA Chapter 12.6 Naive Bayes Models\n The Naive Bayes model is often used in cases where the “effect” variables are not strictly independent given the cause variable. The naive Bayes systems often work very well in practice, even when the conditional independence assumption is false.  \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Uncertainty arises because the agent’s sensors give only partial information about the world. Pure logical inference can infer nothing about which square is most likely to be safe so that a logical agent might choose randomly. A probabilistic agent can do much better than a logical agent. \n\n## SLP Chapter 3.1 N-Grams\n This section describes representing and calculating language model probabilities in log format as log probabilities. Multiplying sufficient n-grams together would end in numerical underflow. By adopting log probabilities instead of raw probabilities, we can get numbers that are not as small.  \n\n## SLP Chapter 3.2 Evaluating Language Models\n The best way to evaluate the production of a language model is to install it in an application and estimate how much the application develops. Such end-to-end evaluation is called extrinsic evaluation.  \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n This section introduces that the n-gram model depends on the training corpus, like many statistical models. One indication of this is that the probabilities often encode specific facts about a given training corpus. \n\n## SLP Chapter 3.4 Generalization and Zeros\n This section introduces smoothing. To keep a language model from designating zero probability to these hidden events, we will have to shave off a bit of probability mass from some more joint events and give it to the events we’ve never seen.  \n\n## SLP Chapter 3.5 Smoothing\n This section introduces the interpolated Kneser-Ney algorithm, the most commonly used and best performing n-gram smoothing method. \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n A better n-gram model assigns a greater likelihood to the test data, and perplexity is a normalized version of the possibility of the test set. \n\n## Module 10 Lecture Notes\n This module reviews the basics of probability, an essential concept to master in artificial intelligence. We review probability theory basics, including the axioms of probability, an event, a probabilistic model, and much more. To show some practical applications of these concepts, we look at probabilistic language models. This model power thing like autocomplete on your smartphone. \n\n## Module 10 Definitions\n* **Decision theory** - Decision theory connects the agent’s beliefs and desires, setting the best action as the one that maximizes expected utility.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHREsHSSyA-h"
      },
      "source": [
        "#Module 11 - Probabilistic Reasoning and Bayes’ Nets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tuFC8w7CyCns"
      },
      "source": [
        "# Russell and Norvig, Russell and Norvig, AIMA Chapter 13 “Probabilistic Reasoning” (13.1, 13.2, 13.3)\n",
        "chapter13_1 = \"\"\" This section introduces a data structure called a Bayesian network to represent the dependencies among variables. Bayesian networks can represent any full joint probability distribution and can do so very concisely in many cases.  \"\"\"\n",
        "chapter13_2 = \"\"\" Continuous variables provide more accuracy, but they make accurate inference impossible except in a few exceptional cases. In practice, many variables would be persistent, and we would study the conditional distributions from historical data on applicants and their insurance claims. \"\"\"\n",
        "chapter13_3 = \"\"\" The essential task for any probabilistic inference scheme is to calculate the posterior probability distribution for a set of query variables, given some observed event. Typically, some assignment of values to a group of evidence variables. \"\"\"\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module11_lecture_notes = \"\"\" This module is built by reviewing a probabilistic model called a Bayes Net, where the nodes represent variables in the probability model. The arcs encode assumptions about how the events are related to each other. Bayes Nets are compact representations that allow independent encoding inferences between variables.  \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module11_definitions = {\n",
        "\"Bayesian network\" : \"A Bayesian network is a directed acyclic graph whose nodes correspond to random variables. Each node has a conditional distribution, given its parents.\",    \n",
        "}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqtEs0gHs9sO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 645
        },
        "cellView": "form",
        "outputId": "aa966892-767c-4676-89a7-03c7b29cf886"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module11_sections = [\n",
        "  (\"AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\", chapter13_1),\n",
        "  (\"AIMA Chapter 13.2 The Semantics of Bayesian Networks\", chapter13_2),\n",
        "  (\"AIMA 13.3 Exact Inference in Bayesian Networks\", chapter13_3),\n",
        "  (\"Module 11 Lecture Notes\", module11_lecture_notes),\n",
        "]\n",
        "\n",
        "module11_markdown = format_module(11, \"Probabilistic Reasoning and Bayes’ Nets\", \n",
        "                         module11_sections, \n",
        "                         module11_definitions)\n",
        "display(Markdown(module11_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n This section introduces a data structure called a Bayesian network to represent the dependencies among variables. Bayesian networks can represent any full joint probability distribution and can do so very concisely in many cases.  \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Continuous variables provide more accuracy, but they make accurate inference impossible except in a few exceptional cases. In practice, many variables would be persistent, and we would study the conditional distributions from historical data on applicants and their insurance claims. \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n The essential task for any probabilistic inference scheme is to calculate the posterior probability distribution for a set of query variables, given some observed event. Typically, some assignment of values to a group of evidence variables. \n\n## Module 11 Lecture Notes\n This module is built by reviewing a probabilistic model called a Bayes Net, where the nodes represent variables in the probability model. The arcs encode assumptions about how the events are related to each other. Bayes Nets are compact representations that allow independent encoding inferences between variables.  \n\n## Module 11 Definitions\n* **Bayesian network** - A Bayesian network is a directed acyclic graph whose nodes correspond to random variables. Each node has a conditional distribution, given its parents.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3FPMzG9yUls"
      },
      "source": [
        "#Module 12 - Naive Bayes and Perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEIOpL7vyWcc"
      },
      "source": [
        "# Russell and Norvig, AIMA Chapter 19 “Learning from Examples” (Sections 19.1-19.6)\n",
        "\n",
        "chapter19_1 = \"\"\" This chapter assumes limited prior knowledge on the agent's part. It starts from scratch and studies from the data. We consider transfer learning, in which knowledge from one domain is transported to a new environment so that learning can proceed faster with fewer data. \"\"\"\n",
        "chapter19_2 = \"\"\" The complexity tradeoff is not straightforward since an expressive language makes it likely for a simple hypothesis to fit the data, whereas limiting the expressiveness of the terminology means that any consistent theory must be complicated. \"\"\"\n",
        "chapter19_3 = \"\"\" This chapter describes the decision tree, which ends its decision by performing a sequence of tests, starting at the root and following the appropriate branch until a leaf is reached—each internal node in the tree matches to examining the value of one of the input attributes. The components from the node are labeled with the attribute's potential values, and the leaf nodes define what value is to be returned by the function. \"\"\"\n",
        "chapter19_4 = \"\"\" This chapter describes model selection as qualitative and subjective. We might pick polynomials preferably than decision trees based on something that we know about the problem.  \"\"\"\n",
        "chapter19_5 = \"\"\" The underlying principle of computational learning theory is that any severely faulty hypothesis will almost indeed be “found out” with high probability after a few examples because it will make an incorrect foresight. Thus, any idea compatible with a sufficiently large set of training examples is unlikely to be seriously wrong. \"\"\"\n",
        "chapter19_6 = \"\"\" This chapter describes linear functions of continuous-valued inputs. We start with the most manageable case regression with a univariate linear function, otherwise acknowledged as “fitting a straight line.” \"\"\"\n",
        "\n",
        "# Speech and Language Processing Chapter 4 “Naive Bayes and Sentiment Classification”\n",
        "slp_chapter4_1 = \"\"\" In this section, we introduce the multinomial naive Bayes classifier because it is a Bayesian classifier that makes a simplifying (naive) assumption about how the features interact. \"\"\"\n",
        "slp_chapter4_2 = \"\"\" In this section, we introduce how to train the Naive Bayes Classifier. \"\"\"\n",
        "slp_chapter4_3 = \"\"\" In this section, we introduce worked examples and test naive Bayes with add-one smoothing.  \"\"\"\n",
        "slp_chapter4_4 = \"\"\" While standard naive Bayes text classification can serve well for sentiment analysis, some minor changes are generally applied that improve performance. This variant is called binary, multinomial naive Bayes, or binary NB.  \"\"\"\n",
        "slp_chapter4_5 = \"\"\" Naive Bayes doesn’t expect our classifier to use all the training data words as features. Elements in naive Bayes can reveal any property of the input text we want. \"\"\"\n",
        "slp_chapter4_6 = \"\"\" The naive Bayes model can be surveyed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. \"\"\"\n",
        "slp_chapter4_7 = \"\"\" This chapter introduces the methods for evaluating text classification. \"\"\"\n",
        "slp_chapter4_8 = \"\"\" This chapter introduces test sets and cross-validation. The training and testing procedure for text classification follows the training set to train the model, then uses the development test set to tune some parameters. \"\"\"\n",
        "slp_chapter4_9 = \"\"\" There are two regular non-parametric tests used in NLP: approximate randomization and the bootstrap test. We represent estimated bootstrap randomization, showing the paired version of the trial, which is most prevalent in NLP. \"\"\"\n",
        "slp_chapter4_10 = \"\"\" This chapter introduces avoiding harm in classification. It is essential to prevent harms that may result from classifiers, both for naive Bayes classifiers and for the other classification algorithms. \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module12_lecture_notes = \"\"\" This module looks at two different machine learning algorithms, Naive Bayes and perceptrons, that use a set of labeled training data to learn how to perform classification tasks. Machine learning is an essential aspect of artificial intelligence.  \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module12_definitions = {\n",
        "\"Decision tree\" : \"a representation of a function that maps a vector of attribute values to a single output value—a decision \",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KUc_LgzDvWf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "5d7d1c0f-165b-4f78-dbcf-f72f74a617cb"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module12_sections = [\n",
        "  (\"AIMA Chapter 5.1 Game Theory\", chapter19_1),\n",
        "  (\"AIMA Chapter 5.2 Optimal Decisions in Games\", chapter19_2),\n",
        "  (\"AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\", chapter19_3),\n",
        "  (\"AIMA Chapter 5.5 Stochastic Games\", chapter19_4),\n",
        "  (\"AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\", chapter19_5),\n",
        "  (\"AIMA Chapter 16.2 The Basis of Utility Theory\", chapter19_6),\n",
        "  (\"SLP Chapter 4.1 Naive Bayes Classifiers\", slp_chapter4_1),\n",
        "  (\"SLP Chapter 4.2 Training the Naive Bayes Classifier\", slp_chapter4_2),\n",
        "  (\"SLP Chapter 4.3 Worked Example\", slp_chapter4_3),\n",
        "  (\"SLP Chapter 4.4 Optimizing for Sentiment Analysis\", slp_chapter4_4),\n",
        "  (\"SLP Chapter 4.5 Naive Bayes for other text classification tasks\", slp_chapter4_5),\n",
        "  (\"SLP Chapter 4.6 Naive Bayes as a Language Model\", slp_chapter4_6),\n",
        "  (\"SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\", slp_chapter4_7),\n",
        "  (\"SLP Chapter 4.8 Test sets and Cross-validation\", slp_chapter4_8),\n",
        "  (\"SLP Chapter 4.9 Statistical Significance Testing\", slp_chapter4_9),\n",
        "  (\"SLP Chapter 4.10 Avoiding Harms in Classification\", slp_chapter4_10),\n",
        "  (\"Module 12 Lecture Notes\", module12_lecture_notes),\n",
        "]\n",
        "\n",
        "module12_markdown = format_module(12, \"Naive Bayes and Perceptrons\", \n",
        "                         module12_sections, \n",
        "                         module12_definitions)\n",
        "display(Markdown(module12_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n This chapter assumes limited prior knowledge on the agent's part. It starts from scratch and studies from the data. We consider transfer learning, in which knowledge from one domain is transported to a new environment so that learning can proceed faster with fewer data. \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n The complexity tradeoff is not straightforward since an expressive language makes it likely for a simple hypothesis to fit the data, whereas limiting the expressiveness of the terminology means that any consistent theory must be complicated. \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n This chapter describes the decision tree, which ends its decision by performing a sequence of tests, starting at the root and following the appropriate branch until a leaf is reached—each internal node in the tree matches to examining the value of one of the input attributes. The components from the node are labeled with the attribute's potential values, and the leaf nodes define what value is to be returned by the function. \n\n## AIMA Chapter 5.5 Stochastic Games\n This chapter describes model selection as qualitative and subjective. We might pick polynomials preferably than decision trees based on something that we know about the problem.  \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n The underlying principle of computational learning theory is that any severely faulty hypothesis will almost indeed be “found out” with high probability after a few examples because it will make an incorrect foresight. Thus, any idea compatible with a sufficiently large set of training examples is unlikely to be seriously wrong. \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n This chapter describes linear functions of continuous-valued inputs. We start with the most manageable case regression with a univariate linear function, otherwise acknowledged as “fitting a straight line.” \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n In this section, we introduce the multinomial naive Bayes classifier because it is a Bayesian classifier that makes a simplifying (naive) assumption about how the features interact. \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n In this section, we introduce how to train the Naive Bayes Classifier. \n\n## SLP Chapter 4.3 Worked Example\n In this section, we introduce worked examples and test naive Bayes with add-one smoothing.  \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n While standard naive Bayes text classification can serve well for sentiment analysis, some minor changes are generally applied that improve performance. This variant is called binary, multinomial naive Bayes, or binary NB.  \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Naive Bayes doesn’t expect our classifier to use all the training data words as features. Elements in naive Bayes can reveal any property of the input text we want. \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n The naive Bayes model can be surveyed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n This chapter introduces the methods for evaluating text classification. \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n This chapter introduces test sets and cross-validation. The training and testing procedure for text classification follows the training set to train the model, then uses the development test set to tune some parameters. \n\n## SLP Chapter 4.9 Statistical Significance Testing\n There are two regular non-parametric tests used in NLP: approximate randomization and the bootstrap test. We represent estimated bootstrap randomization, showing the paired version of the trial, which is most prevalent in NLP. \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n This chapter introduces avoiding harm in classification. It is essential to prevent harms that may result from classifiers, both for naive Bayes classifiers and for the other classification algorithms. \n\n## Module 12 Lecture Notes\n This module looks at two different machine learning algorithms, Naive Bayes and perceptrons, that use a set of labeled training data to learn how to perform classification tasks. Machine learning is an essential aspect of artificial intelligence.  \n\n## Module 12 Definitions\n* **Decision tree** - a representation of a function that maps a vector of attribute values to a single output value—a decision \n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qVSQhTVSyYYO"
      },
      "source": [
        "# Module 13 - Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wm4mfW82yaRH"
      },
      "source": [
        "# Jurafsky and Martin, Speech and Language Processing Chapter 5 “Logistic Regression” (5.1-5.6)\n",
        "\n",
        "slp_chapter5_1 = \"\"\" This module looks at two different machine learning algorithms, Naive Bayes and perceptrons, using a set of labeled training data to discover how to perform classification tasks. Machine learning is a fundamental aspect of artificial intelligence.  \"\"\"\n",
        "slp_chapter5_2 = \"\"\" This section introduces that logistic regression is a case of supervised classification in which we know the exact label for each observation. \"\"\"\n",
        "slp_chapter5_3 = \"\"\" Conditional maximum likelihood estimation is the method to choose the parameters w,b that maximize the log probability of the accurate y labels in the training data given the observations x.  \"\"\"\n",
        "slp_chapter5_4 = \"\"\" This section introduces our object with gradient descent to obtain the optimal weights: decrease the loss function we have defined for the model. \"\"\"\n",
        "slp_chapter5_5 = \"\"\" This section introduces the problem called overfitting. A good model should conclude reasonably from the training data to the hidden test set, but a model that overfits will have lousy generalization. A new regularization term R(θ) is added to the objective function to avoid overfitting. \"\"\"\n",
        "slp_chapter5_6 = \"\"\" This section uses multinomial logistic regression, also named softmax regression. In multinomial logistic regression, the target y is a variable that ranges over more than two classes. \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 7 “Neural Networks and Neural Language Models”\n",
        "\n",
        "slp_chapter7_1 = \"\"\" This section introduces that the building block of a neural network is a single computational unit. A unit gets a set of real-valued numbers as input, calculates them, and generates an output. \"\"\"\n",
        "slp_chapter7_2 = \"\"\" This section introduces that a single neural unit cannot calculate some elementary functions of its input. Therefore we need to join these units into more extensive networks. \"\"\"\n",
        "slp_chapter7_3 = \"\"\" A feedforward network is a multilayer feedforward network. The units are connected with no cycles. The outputs in each layer are passed to units in the next higher layer, and no results are given back to lower layers.  \"\"\"\n",
        "slp_chapter7_4 = \"\"\" This section introduces that a feedforward neural net is an occurrence of supervised machine learning in which we know the exact output y for each observation x. \"\"\"\n",
        "slp_chapter7_5 = \"\"\" This section introduces that neural net-based language models have many benefits over the n-gram language models. Among these are that neural language models don’t need smoothing. They can manage much longer histories and induce over contexts of similar words. \"\"\"\n",
        "slp_chapter7_6 = \"\"\" Neural networks are built from neural units, initially inspired by human neurons but now utterly an abstract computational device. \"\"\"\n",
        "slp_chapter7_7 = \"\"\" In a fully-connected, feedforward network, each unit in layer i is connected to each unit in layer i+1, and there are no cycles. \"\"\"\n",
        "\n",
        "# Lecture notes \n",
        "module13_lecture_notes = \"\"\" This module investigates neural networks consisting of a set of neural units arranged in a network. We look at the neural networks’ training procedure, errors, attribution, and several neural architectures. We also look at how neural networks have been used as probabilistic language models. \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module13_definitions = {\n",
        "\"Neural networks\" : \"Neural networks are built from neural units, initially inspired by human neurons but now utterly an abstract computational device.\",    \n",
        "}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKWbCsbfxL3R",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "cellView": "form",
        "outputId": "d7438a56-091e-4823-fcac-11def0d6cfbb"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module13_sections = [\n",
        "  (\"SLP Chapter 5.1 Classification: the sigmoid\", slp_chapter5_1),\n",
        "  (\"SLP Chapter 5.2 Learning in Logistic Regression\", slp_chapter5_2),\n",
        "  (\"SLP Chapter 5.3 The cross-entropy loss function\", slp_chapter5_3),\n",
        "  (\"SLP Chapter 5.4 Gradient Descent\", slp_chapter5_4),\n",
        "  (\"SLP Chapter 5.5 Regularization\", slp_chapter5_5),\n",
        "  (\"SLP Chapter 5.6 Multinomial logistic regression\", slp_chapter5_6),\n",
        "\n",
        "  (\"SLP Chapter 7.1 Units\", slp_chapter7_1),\n",
        "  (\"SLP Chapter 7.2 The XOR problem\", slp_chapter7_2),\n",
        "  (\"SLP Chapter 7.3 Feedforward Neural Networks\", slp_chapter7_3),\n",
        "  (\"SLP Chapter 7.4 Feedforward networks for NLP: Classification\", slp_chapter7_4),\n",
        "  (\"SLP Chapter 7.5 Feedforward Neural Language Modeling\", slp_chapter7_5),\n",
        "  (\"SLP Chapter 7.6 Training Neural Nets\", slp_chapter7_6),\n",
        "  (\"SLP Chapter 7.7 Training the neural language model\", slp_chapter7_7),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module13_markdown = format_module(13, \"Neural Networks\", \n",
        "                         module13_sections, \n",
        "                         module13_definitions)\n",
        "display(Markdown(module13_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n This module looks at two different machine learning algorithms, Naive Bayes and perceptrons, using a set of labeled training data to discover how to perform classification tasks. Machine learning is a fundamental aspect of artificial intelligence.  \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n This section introduces that logistic regression is a case of supervised classification in which we know the exact label for each observation. \n\n## SLP Chapter 5.3 The cross-entropy loss function\n Conditional maximum likelihood estimation is the method to choose the parameters w,b that maximize the log probability of the accurate y labels in the training data given the observations x.  \n\n## SLP Chapter 5.4 Gradient Descent\n This section introduces our object with gradient descent to obtain the optimal weights: decrease the loss function we have defined for the model. \n\n## SLP Chapter 5.5 Regularization\n This section introduces the problem called overfitting. A good model should conclude reasonably from the training data to the hidden test set, but a model that overfits will have lousy generalization. A new regularization term R(θ) is added to the objective function to avoid overfitting. \n\n## SLP Chapter 5.6 Multinomial logistic regression\n This section uses multinomial logistic regression, also named softmax regression. In multinomial logistic regression, the target y is a variable that ranges over more than two classes. \n\n## SLP Chapter 7.1 Units\n This section introduces that the building block of a neural network is a single computational unit. A unit gets a set of real-valued numbers as input, calculates them, and generates an output. \n\n## SLP Chapter 7.2 The XOR problem\n This section introduces that a single neural unit cannot calculate some elementary functions of its input. Therefore we need to join these units into more extensive networks. \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n A feedforward network is a multilayer feedforward network. The units are connected with no cycles. The outputs in each layer are passed to units in the next higher layer, and no results are given back to lower layers.  \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n This section introduces that a feedforward neural net is an occurrence of supervised machine learning in which we know the exact output y for each observation x. \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n This section introduces that neural net-based language models have many benefits over the n-gram language models. Among these are that neural language models don’t need smoothing. They can manage much longer histories and induce over contexts of similar words. \n\n## SLP Chapter 7.6 Training Neural Nets\n Neural networks are built from neural units, initially inspired by human neurons but now utterly an abstract computational device. \n\n## SLP Chapter 7.7 Training the neural language model\n In a fully-connected, feedforward network, each unit in layer i is connected to each unit in layer i+1, and there are no cycles. \n\n## Module 13 Lecture Notes\n This module investigates neural networks consisting of a set of neural units arranged in a network. We look at the neural networks’ training procedure, errors, attribution, and several neural architectures. We also look at how neural networks have been used as probabilistic language models. \n\n## Module 13 Definitions\n* **Neural networks** - Neural networks are built from neural units, initially inspired by human neurons but now utterly an abstract computational device.\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYESi2kMybL8"
      },
      "source": [
        "# Module 14 - Natural Language Processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qIRJ4YYCzNHL"
      },
      "source": [
        "# Jurafsky and Martin, Chapter 9 “Sequence Processing with Recurrent Networks” (Sections 9.1-9.3)\n",
        "\n",
        "slp_chapter9_1 = \"\"\" Language models allow us to distribute conditional probability to every possible next word, spreading the entire vocabulary. We can assign probabilities to complete sequences by using these conditional probabilities in union with the chain rule.\"\"\"\n",
        "slp_chapter9_2 = \"\"\" This section introduces a recurrent neural network, which is any network that includes a cycle within its network connections. Every network where the value of a unit is directly or indirectly is conditioned on its own earlier outputs as an input.\"\"\"\n",
        "slp_chapter9_3 = \"\"\" In practice, it is pretty challenging to train RNNs for tasks demanding a network to use information distant from the current point of processing. \"\"\"\n",
        "\n",
        "# Jurafsky and Martin, Chapter 6 “Vector Semantics and Embeddings” (6.1-6.4, 6.8, 6.10-6.12)\n",
        "\n",
        "slp_chapter6_1 = \"\"\" Words have affective meanings or implications. The word connotation has various meanings in various fields.  \"\"\"\n",
        "slp_chapter6_2 = \"\"\" This section introduces the vectors semantics as the official way to represent word definition in NLP, helping us model many features of word sense we saw in the previous area. \"\"\"\n",
        "slp_chapter6_3 = \"\"\" Vector or distributional models of meaning are generally based on a co-occurrence matrix. It is a method of describing how frequently words co-occur. We look at two popular matrices: the term-document matrix and the term-term matrix. \"\"\"\n",
        "slp_chapter6_4 = \"\"\" This section introduces the cosine, which is based on dot product. The dot product operator from linear algebra is also called the inner product.  \"\"\"\n",
        "slp_chapter6_8 = \"\"\" We now introduce a more powerful word representation: embeddings, short, dense vectors. These d dimensions don’t have a clear interpretation. And the vectors are dense: instead of vector entries being sparse, mostly zero counts or functions of counts, the values will be real-valued numbers that can be negative. \"\"\"\n",
        "slp_chapter6_10 = \"\"\" In this section, we temporarily review some of the semantic properties of embeddings that have been studied. \"\"\"\n",
        "slp_chapter6_11 = \"\"\" In addition to their ability to learn word meaning from text, embeddings also represent the implicit biases and stereotypes latent in the text. As the prior section just showed, embeddings can roughly model relational similarity. \"\"\"\n",
        "slp_chapter6_12 = \"\"\" The most significant evaluation metric for vector models is extrinsic evaluation on tasks, using vectors in an NLP task and seeing whether this develops performance over some other model. \"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Lecture notes \n",
        "module14_lecture_notes = \"\"\" In this module, we looked into an application of AI-Natural Language Processing. The purpose of NLP is to allow computers to communicate in a human-like fashion. In this module, we focus on creating illustrations of words that capture their goals, using the idea of vector space models. \"\"\"\n",
        "\n",
        "# Definitions\n",
        "module14_definitions = {\n",
        "\"tf-idf\" : \"weights each cell by its term frequency and inverse document frequency\",    \n",
        "}\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXNcHuKjyNXa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "2ca94649-ed32-48dc-df99-25de6a9dd73a"
      },
      "source": [
        "#@title Preview of your summary\n",
        "#@markdown You don't need to modify anything in this cell. Just press play.\n",
        "module14_sections = [\n",
        "  (\"SLP Chapter 9.1 Language Models Revisited\", slp_chapter9_1),\n",
        "  (\"SLP Chapter 9.2 Recurrent Neural Networks\", slp_chapter9_2),\n",
        "  (\"SLP Chapter 9.3 RNNs as Language Models\", slp_chapter9_3),\n",
        "\n",
        "  (\"SLP Chapter 6.1 Lexical Semantics\", slp_chapter6_1),\n",
        "  (\"SLP Chapter 6.2 Vector Semantics\", slp_chapter6_2),\n",
        "  (\"SLP Chapter 6.3 Words and Vectors\", slp_chapter6_3),\n",
        "  (\"SLP Chapter 6.4 Cosine for measuring similarity\", slp_chapter6_4),\n",
        "  (\"SLP Chapter 6.8 Word2vec\", slp_chapter6_8),\n",
        "  (\"SLP Chapter 6.10 Semantic properties of embeddings\", slp_chapter6_10),\n",
        "  (\"SLP Chapter 6.11 Bias and Embeddings\", slp_chapter6_11),\n",
        "  (\"SLP Chapter 6.12 Evaluating Vector Models\", slp_chapter6_12),\n",
        "  (\"Module 13 Lecture Notes\", module13_lecture_notes),\n",
        "]\n",
        "\n",
        "module14_markdown = format_module(14, \"Natural Language Processing\", \n",
        "                         module14_sections, \n",
        "                         module14_definitions)\n",
        "display(Markdown(module14_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Language models allow us to distribute conditional probability to every possible next word, spreading the entire vocabulary. We can assign probabilities to complete sequences by using these conditional probabilities in union with the chain rule.\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n This section introduces a recurrent neural network, which is any network that includes a cycle within its network connections. Every network where the value of a unit is directly or indirectly is conditioned on its own earlier outputs as an input.\n\n## SLP Chapter 9.3 RNNs as Language Models\n In practice, it is pretty challenging to train RNNs for tasks demanding a network to use information distant from the current point of processing. \n\n## SLP Chapter 6.1 Lexical Semantics\n Words have affective meanings or implications. The word connotation has various meanings in various fields.  \n\n## SLP Chapter 6.2 Vector Semantics\n This section introduces the vectors semantics as the official way to represent word definition in NLP, helping us model many features of word sense we saw in the previous area. \n\n## SLP Chapter 6.3 Words and Vectors\n Vector or distributional models of meaning are generally based on a co-occurrence matrix. It is a method of describing how frequently words co-occur. We look at two popular matrices: the term-document matrix and the term-term matrix. \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n This section introduces the cosine, which is based on dot product. The dot product operator from linear algebra is also called the inner product.  \n\n## SLP Chapter 6.8 Word2vec\n We now introduce a more powerful word representation: embeddings, short, dense vectors. These d dimensions don’t have a clear interpretation. And the vectors are dense: instead of vector entries being sparse, mostly zero counts or functions of counts, the values will be real-valued numbers that can be negative. \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n In this section, we temporarily review some of the semantic properties of embeddings that have been studied. \n\n## SLP Chapter 6.11 Bias and Embeddings\n In addition to their ability to learn word meaning from text, embeddings also represent the implicit biases and stereotypes latent in the text. As the prior section just showed, embeddings can roughly model relational similarity. \n\n## SLP Chapter 6.12 Evaluating Vector Models\n The most significant evaluation metric for vector models is extrinsic evaluation on tasks, using vectors in an NLP task and seeing whether this develops performance over some other model. \n\n## Module 13 Lecture Notes\n This module investigates neural networks consisting of a set of neural units arranged in a network. We look at the neural networks’ training procedure, errors, attribution, and several neural architectures. We also look at how neural networks have been used as probabilistic language models. \n\n## Module 14 Definitions\n* **Tf-idf** - weights each cell by its term frequency and inverse document frequency\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX7lbnJd4sT5"
      },
      "source": [
        "# Your Exam Notes\n",
        "\n",
        "Here are your full set of exam notes that you can use on your midterm.  Choose **Runtime > Run all**, then your notes will be displayed.\n",
        "\n",
        "If you are taking the exam in person, then you must print a copy of your notes for yourself.   Here's how to print:\n",
        "* Run the cell below. \n",
        "* Click its \"Mirror cell in tab\" button (this is located in the upper right corner of the cell, immediately to the right of the trash can).  This will open the cell.  \n",
        "* In Chrome, you can then print your notes using the **File > Print** option from the browser's menu.  \n",
        "* In the print options, change the layout from Landscape to Portrait. \n",
        "* Clicking the Save, will save a PDF that you can then print.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "T3cm1WV542_a",
        "outputId": "62a57234-69b7-43e5-880e-f3123522d9ea"
      },
      "source": [
        "# This will generate the full set of notes that you can use for your exam.\n",
        "full_markdown = \"\\n\\n\".join([module8_markdown, \n",
        "                             module9_markdown, \n",
        "                             module10_markdown, \n",
        "                             module11_markdown, \n",
        "                             module12_markdown,\n",
        "                             module13_markdown,\n",
        "                             module14_markdown,]\n",
        "                       )\n",
        "display(Markdown(full_markdown))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/markdown": "# Module 8: Markov Decision Processes\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n This section reports combining beliefs and desires under uncertainty. The agent has available actions. There may be changes in the current state. We believe that the agent assigns a probability to each possible current state P(s). There may also be uncertainty about the action outcomes described by the transition model. $P(\\textnormal{RESULT}(a)=s') = \\sum_s P(s)P(s'|s,a)$ \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n An observer can learn about the utility function representing the agent's goal by following a rational agent's performance. \n\n## AIMA Chapter 16.3 Utility Functions\n Utility functions map lotteries to real numbers. They need to accept the premises of order ability, transitivity, continuity, substitutability, monotonicity, and decomposability. An agent can have any choices he likes.  \n\n## AIMA Chapter 17.1 Sequential Decision Problems\n This section describes that the introduction of skepticism leads MDPs closer to the real world than deterministic search puzzles. Accordingly, MDPs have been studied in several fields, including AI, operations research, economics, and control theory. Several solution algorithms have been proposed. \n\n## AIMA Chapter 17.2 Algorithms for MDPs\n This section presents four different algorithms for solving MDPs. The first three, value iteration, policy iteration, and linear programming, generate accurate solutions offline. The fourth is a group of online approximate algorithms that includes Monte Carlo planning. \n\n## Module 8 Lecture Notes\n This module turns to stochastic or probabilistic task environments. We start by introducing a new kind of formalization called Markov Decision Processes, which are helpful for fully observable task environments. We have to make decisions, and the outcome for actions is non-deterministic. MDPs also look into new algorithms, such as Policy Iteration and Value Iteration, and the essential concepts and additive rewards. \n\n## Module 8 Definitions\n* **Utility function** - a utility function assigns values to certain actions that the AI system can take\n* **Expected utility** - A policy can be evaluated by determining its expected utility for an agent following the policy\n\n\n# Module 9: Reinforcement Learning\n## AIMA Chapter 17.3 Bandit Problems\n The extraordinary thing about the Gittins index is that it provides a clear optimal policy for the bandit problem. We can pull the arm that has the highest Gittins index, then update the Gittins indices.  \n\n## AIMA Chapter 22.1 Learning from Rewards\n This section discusses passive reinforcement learning, where the policy of the agent is fixed. The task is to learn the utilities of states, which could also involve learning a scene model.   \n\n## AIMA Chapter 22.2 Passive Reinforcement Learning\n The passive learning task is comparable to the policy evaluation task, part of the policy iteration algorithm. The distinction is that the passive learning agent does not know the reward function and transition model, which defines the probability of reaching state from the state after doing the action. \n\n## AIMA Chapter 22.3 Active Reinforcement Learning\n This section covers active reinforcement learning, where the agent must figure out what to do. The main issue is exploration, where an agent must undergo as much as possible of its environment to discover how to behave in it. \n\n## AIMA Chapter 22.4 Generalization in Reinforcement Learning\n This section discusses functions approximation which makes it possible to represent utility functions for vast state spaces. Still, more importantly, it provides for inductive generalization, in which the agent can conclude from states it has visited to the states it has not yet seen. \n\n## Module 9 Lecture Notes\n This module investigates situations where a transition and a reward function are not given to us but must be acquired through interaction with the world. In reinforcement learning, every time an agent takes action in the world, it receives feedback in the form of a reward, and so the agent wants to maximize its expected rewards. We look at several different learning techniques for reinforcement learning&colon. \n\n## Module 9 Definitions\n* **Passive learning agent** - an agent already has a fixed policy that determines its actions\n\n\n# Module 10: Probabilies and Language Models\n## AIMA Chapter 12.1 Acting under Uncertainty\n Therefore, the rational decision depends on the relative influence of various goals and the possibility of achieving them. This section sharpens these concepts to develop the general theories of hypothetical reasoning and rational decisions. \n\n## AIMA Chapter 12.2 Basic Probability Notation\n This section introduces the language of probability theory, traditionally informal, written by mathematicians for other mathematicians. Here, we take a path more suited to the needs of AI and relate it with the ideas of formal logic. \n\n## AIMA Chapter 12.3 Inference Using Full Joint Distributions\n This section describes a manageable method for probabilistic inference—the computation of posterior probabilities for query propositions given experimental evidence. We use the full joint distribution as the “knowledge base” to obtain solutions to all questions. We also propose several valuable methods for manipulating equations involving probabilities.  \n\n## AIMA Chapter 12.4 Independence\n This section introduces that independence assertions can help reduce the volume of the domain representation and the complexity of the inference puzzle when available.  \n\n## AIMA Chapter 12.5 Bayes’ Rule and Its Use\n This section introduces that conditional independence assertions can provide probabilistic systems to scale up. Furthermore, they are much more generally available than absolute independence assertions. \n\n## AIMA Chapter 12.6 Naive Bayes Models\n The Naive Bayes model is often used in cases where the “effect” variables are not strictly independent given the cause variable. The naive Bayes systems often work very well in practice, even when the conditional independence assumption is false.  \n\n## AIMA Chapter 12.7 The Wumpus World Revisited\n Uncertainty arises because the agent’s sensors give only partial information about the world. Pure logical inference can infer nothing about which square is most likely to be safe so that a logical agent might choose randomly. A probabilistic agent can do much better than a logical agent. \n\n## SLP Chapter 3.1 N-Grams\n This section describes representing and calculating language model probabilities in log format as log probabilities. Multiplying sufficient n-grams together would end in numerical underflow. By adopting log probabilities instead of raw probabilities, we can get numbers that are not as small.  \n\n## SLP Chapter 3.2 Evaluating Language Models\n The best way to evaluate the production of a language model is to install it in an application and estimate how much the application develops. Such end-to-end evaluation is called extrinsic evaluation.  \n\n## SLP Chapter 3.3 Sampling sentences from a language model\n This section introduces that the n-gram model depends on the training corpus, like many statistical models. One indication of this is that the probabilities often encode specific facts about a given training corpus. \n\n## SLP Chapter 3.4 Generalization and Zeros\n This section introduces smoothing. To keep a language model from designating zero probability to these hidden events, we will have to shave off a bit of probability mass from some more joint events and give it to the events we’ve never seen.  \n\n## SLP Chapter 3.5 Smoothing\n This section introduces the interpolated Kneser-Ney algorithm, the most commonly used and best performing n-gram smoothing method. \n\n## SLP Chapter 3.7 Huge Language Models and Stupid Backoff\n A better n-gram model assigns a greater likelihood to the test data, and perplexity is a normalized version of the possibility of the test set. \n\n## Module 10 Lecture Notes\n This module reviews the basics of probability, an essential concept to master in artificial intelligence. We review probability theory basics, including the axioms of probability, an event, a probabilistic model, and much more. To show some practical applications of these concepts, we look at probabilistic language models. This model power thing like autocomplete on your smartphone. \n\n## Module 10 Definitions\n* **Decision theory** - Decision theory connects the agent’s beliefs and desires, setting the best action as the one that maximizes expected utility.\n\n\n# Module 11: Probabilistic Reasoning and Bayes’ Nets\n## AIMA Chapter 13.1 Representing Knowledge in an Uncertain Domain\n This section introduces a data structure called a Bayesian network to represent the dependencies among variables. Bayesian networks can represent any full joint probability distribution and can do so very concisely in many cases.  \n\n## AIMA Chapter 13.2 The Semantics of Bayesian Networks\n Continuous variables provide more accuracy, but they make accurate inference impossible except in a few exceptional cases. In practice, many variables would be persistent, and we would study the conditional distributions from historical data on applicants and their insurance claims. \n\n## AIMA 13.3 Exact Inference in Bayesian Networks\n The essential task for any probabilistic inference scheme is to calculate the posterior probability distribution for a set of query variables, given some observed event. Typically, some assignment of values to a group of evidence variables. \n\n## Module 11 Lecture Notes\n This module is built by reviewing a probabilistic model called a Bayes Net, where the nodes represent variables in the probability model. The arcs encode assumptions about how the events are related to each other. Bayes Nets are compact representations that allow independent encoding inferences between variables.  \n\n## Module 11 Definitions\n* **Bayesian network** - A Bayesian network is a directed acyclic graph whose nodes correspond to random variables. Each node has a conditional distribution, given its parents.\n\n\n# Module 12: Naive Bayes and Perceptrons\n## AIMA Chapter 5.1 Game Theory\n This chapter assumes limited prior knowledge on the agent's part. It starts from scratch and studies from the data. We consider transfer learning, in which knowledge from one domain is transported to a new environment so that learning can proceed faster with fewer data. \n\n## AIMA Chapter 5.2 Optimal Decisions in Games\n The complexity tradeoff is not straightforward since an expressive language makes it likely for a simple hypothesis to fit the data, whereas limiting the expressiveness of the terminology means that any consistent theory must be complicated. \n\n## AIMA Chapter 5.3 Heuristic Alpha–Beta Tree Search\n This chapter describes the decision tree, which ends its decision by performing a sequence of tests, starting at the root and following the appropriate branch until a leaf is reached—each internal node in the tree matches to examining the value of one of the input attributes. The components from the node are labeled with the attribute's potential values, and the leaf nodes define what value is to be returned by the function. \n\n## AIMA Chapter 5.5 Stochastic Games\n This chapter describes model selection as qualitative and subjective. We might pick polynomials preferably than decision trees based on something that we know about the problem.  \n\n## AIMA Chapter 16.1 Combining Beliefs and Desires under Uncertainty\n The underlying principle of computational learning theory is that any severely faulty hypothesis will almost indeed be “found out” with high probability after a few examples because it will make an incorrect foresight. Thus, any idea compatible with a sufficiently large set of training examples is unlikely to be seriously wrong. \n\n## AIMA Chapter 16.2 The Basis of Utility Theory\n This chapter describes linear functions of continuous-valued inputs. We start with the most manageable case regression with a univariate linear function, otherwise acknowledged as “fitting a straight line.” \n\n## SLP Chapter 4.1 Naive Bayes Classifiers\n In this section, we introduce the multinomial naive Bayes classifier because it is a Bayesian classifier that makes a simplifying (naive) assumption about how the features interact. \n\n## SLP Chapter 4.2 Training the Naive Bayes Classifier\n In this section, we introduce how to train the Naive Bayes Classifier. \n\n## SLP Chapter 4.3 Worked Example\n In this section, we introduce worked examples and test naive Bayes with add-one smoothing.  \n\n## SLP Chapter 4.4 Optimizing for Sentiment Analysis\n While standard naive Bayes text classification can serve well for sentiment analysis, some minor changes are generally applied that improve performance. This variant is called binary, multinomial naive Bayes, or binary NB.  \n\n## SLP Chapter 4.5 Naive Bayes for other text classification tasks\n Naive Bayes doesn’t expect our classifier to use all the training data words as features. Elements in naive Bayes can reveal any property of the input text we want. \n\n## SLP Chapter 4.6 Naive Bayes as a Language Model\n The naive Bayes model can be surveyed as a set of class-specific unigram language models, in which the model for each class instantiates a unigram language model. \n\n## SLP Chapter 4.7 Evaluation: Precision, Recall, F-measure\n This chapter introduces the methods for evaluating text classification. \n\n## SLP Chapter 4.8 Test sets and Cross-validation\n This chapter introduces test sets and cross-validation. The training and testing procedure for text classification follows the training set to train the model, then uses the development test set to tune some parameters. \n\n## SLP Chapter 4.9 Statistical Significance Testing\n There are two regular non-parametric tests used in NLP: approximate randomization and the bootstrap test. We represent estimated bootstrap randomization, showing the paired version of the trial, which is most prevalent in NLP. \n\n## SLP Chapter 4.10 Avoiding Harms in Classification\n This chapter introduces avoiding harm in classification. It is essential to prevent harms that may result from classifiers, both for naive Bayes classifiers and for the other classification algorithms. \n\n## Module 12 Lecture Notes\n This module looks at two different machine learning algorithms, Naive Bayes and perceptrons, that use a set of labeled training data to learn how to perform classification tasks. Machine learning is an essential aspect of artificial intelligence.  \n\n## Module 12 Definitions\n* **Decision tree** - a representation of a function that maps a vector of attribute values to a single output value—a decision \n\n\n# Module 13: Neural Networks\n## SLP Chapter 5.1 Classification: the sigmoid\n This module looks at two different machine learning algorithms, Naive Bayes and perceptrons, using a set of labeled training data to discover how to perform classification tasks. Machine learning is a fundamental aspect of artificial intelligence.  \n\n## SLP Chapter 5.2 Learning in Logistic Regression\n This section introduces that logistic regression is a case of supervised classification in which we know the exact label for each observation. \n\n## SLP Chapter 5.3 The cross-entropy loss function\n Conditional maximum likelihood estimation is the method to choose the parameters w,b that maximize the log probability of the accurate y labels in the training data given the observations x.  \n\n## SLP Chapter 5.4 Gradient Descent\n This section introduces our object with gradient descent to obtain the optimal weights: decrease the loss function we have defined for the model. \n\n## SLP Chapter 5.5 Regularization\n This section introduces the problem called overfitting. A good model should conclude reasonably from the training data to the hidden test set, but a model that overfits will have lousy generalization. A new regularization term R(θ) is added to the objective function to avoid overfitting. \n\n## SLP Chapter 5.6 Multinomial logistic regression\n This section uses multinomial logistic regression, also named softmax regression. In multinomial logistic regression, the target y is a variable that ranges over more than two classes. \n\n## SLP Chapter 7.1 Units\n This section introduces that the building block of a neural network is a single computational unit. A unit gets a set of real-valued numbers as input, calculates them, and generates an output. \n\n## SLP Chapter 7.2 The XOR problem\n This section introduces that a single neural unit cannot calculate some elementary functions of its input. Therefore we need to join these units into more extensive networks. \n\n## SLP Chapter 7.3 Feedforward Neural Networks\n A feedforward network is a multilayer feedforward network. The units are connected with no cycles. The outputs in each layer are passed to units in the next higher layer, and no results are given back to lower layers.  \n\n## SLP Chapter 7.4 Feedforward networks for NLP: Classification\n This section introduces that a feedforward neural net is an occurrence of supervised machine learning in which we know the exact output y for each observation x. \n\n## SLP Chapter 7.5 Feedforward Neural Language Modeling\n This section introduces that neural net-based language models have many benefits over the n-gram language models. Among these are that neural language models don’t need smoothing. They can manage much longer histories and induce over contexts of similar words. \n\n## SLP Chapter 7.6 Training Neural Nets\n Neural networks are built from neural units, initially inspired by human neurons but now utterly an abstract computational device. \n\n## SLP Chapter 7.7 Training the neural language model\n In a fully-connected, feedforward network, each unit in layer i is connected to each unit in layer i+1, and there are no cycles. \n\n## Module 13 Lecture Notes\n This module investigates neural networks consisting of a set of neural units arranged in a network. We look at the neural networks’ training procedure, errors, attribution, and several neural architectures. We also look at how neural networks have been used as probabilistic language models. \n\n## Module 13 Definitions\n* **Neural networks** - Neural networks are built from neural units, initially inspired by human neurons but now utterly an abstract computational device.\n\n\n# Module 14: Natural Language Processing\n## SLP Chapter 9.1 Language Models Revisited\n Language models allow us to distribute conditional probability to every possible next word, spreading the entire vocabulary. We can assign probabilities to complete sequences by using these conditional probabilities in union with the chain rule.\n\n## SLP Chapter 9.2 Recurrent Neural Networks\n This section introduces a recurrent neural network, which is any network that includes a cycle within its network connections. Every network where the value of a unit is directly or indirectly is conditioned on its own earlier outputs as an input.\n\n## SLP Chapter 9.3 RNNs as Language Models\n In practice, it is pretty challenging to train RNNs for tasks demanding a network to use information distant from the current point of processing. \n\n## SLP Chapter 6.1 Lexical Semantics\n Words have affective meanings or implications. The word connotation has various meanings in various fields.  \n\n## SLP Chapter 6.2 Vector Semantics\n This section introduces the vectors semantics as the official way to represent word definition in NLP, helping us model many features of word sense we saw in the previous area. \n\n## SLP Chapter 6.3 Words and Vectors\n Vector or distributional models of meaning are generally based on a co-occurrence matrix. It is a method of describing how frequently words co-occur. We look at two popular matrices: the term-document matrix and the term-term matrix. \n\n## SLP Chapter 6.4 Cosine for measuring similarity\n This section introduces the cosine, which is based on dot product. The dot product operator from linear algebra is also called the inner product.  \n\n## SLP Chapter 6.8 Word2vec\n We now introduce a more powerful word representation: embeddings, short, dense vectors. These d dimensions don’t have a clear interpretation. And the vectors are dense: instead of vector entries being sparse, mostly zero counts or functions of counts, the values will be real-valued numbers that can be negative. \n\n## SLP Chapter 6.10 Semantic properties of embeddings\n In this section, we temporarily review some of the semantic properties of embeddings that have been studied. \n\n## SLP Chapter 6.11 Bias and Embeddings\n In addition to their ability to learn word meaning from text, embeddings also represent the implicit biases and stereotypes latent in the text. As the prior section just showed, embeddings can roughly model relational similarity. \n\n## SLP Chapter 6.12 Evaluating Vector Models\n The most significant evaluation metric for vector models is extrinsic evaluation on tasks, using vectors in an NLP task and seeing whether this develops performance over some other model. \n\n## Module 13 Lecture Notes\n This module investigates neural networks consisting of a set of neural units arranged in a network. We look at the neural networks’ training procedure, errors, attribution, and several neural architectures. We also look at how neural networks have been used as probabilistic language models. \n\n## Module 14 Definitions\n* **Tf-idf** - weights each cell by its term frequency and inverse document frequency\n",
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}