From 40671c8432c0b947ed77a41be1048c2955018fc4 Mon Sep 17 00:00:00 2001
From: TerenceLiu <terenceliu1012@outlook.com>
Date: Sat, 5 Nov 2022 21:30:07 +0800
Subject: [PATCH] update

---
 algorithm/interactive.ipynb | 105 +--------
 algorithm/lecture.ipynb     | 444 +++++-------------------------------
 2 files changed, 61 insertions(+), 488 deletions(-)

diff --git a/algorithm/interactive.ipynb b/algorithm/interactive.ipynb
index c7a5307..219a184 100644
--- a/algorithm/interactive.ipynb
+++ b/algorithm/interactive.ipynb
@@ -1,59 +1,5 @@
 {
  "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "b705b060",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "slide"
-    }
-   },
-   "source": [
-    "# Steepest Descent Method\n",
-    "\n",
-    "Instructor: Dr.Yuhui Deng\n",
-    "\n",
-    "![level_gradient](figure/level_gradient.png)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f576c3bd",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "subslide"
-    }
-   },
-   "source": [
-    "## Problem Description\n",
-    "\n",
-    "Problem: \n",
-    "\n",
-    "$\\begin{align*} \n",
-    "min(fx) \\\\\n",
-    "s.t. \\  x \\in \\mathbb{R}^n\n",
-    "\\end{align*}$\n",
-    "where\n",
-    "\n",
-    "* $f: \\mathbb{R}^n \\rightarrow \\mathbb{R}$\n",
-    "* No constraints are placed on the variables $X$\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c6e94df1",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "slide"
-    }
-   },
-   "source": [
-    "# One Dimential Function\n",
-    "\n",
-    "$$min \\ f(x)=x^2 - 4x + 7$$\n",
-    "\n"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -120,7 +66,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "61e0d2b8-41c8-4790-8b25-2df6ba4c3625",
    "metadata": {},
    "outputs": [],
@@ -130,53 +76,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "id": "cc84e26c-7135-427b-9a5b-c39b6b495fe5",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "e2a6bc45f7354f529c8475f034416135",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "VBox(children=(HBox(children=(Text(value='-2 * x * sin(-(pi/4) * x)+10', description='Expression:', style=Text…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "c2c3118f55d241e68649a094a6bb3836",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "cdc830df33fe4705b448245a1d64c335",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
+   "outputs": [],
    "source": [
     "a = func_1d()"
    ]
@@ -207,7 +110,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.13"
+   "version": "3.7.13"
   }
  },
  "nbformat": 4,
diff --git a/algorithm/lecture.ipynb b/algorithm/lecture.ipynb
index a901c84..ca2c0ce 100644
--- a/algorithm/lecture.ipynb
+++ b/algorithm/lecture.ipynb
@@ -1,361 +1,103 @@
 {
  "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "9b492ee6",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "slide"
-    }
-   },
-   "source": [
-    "# Steepest Descent Method\n",
-    "\n",
-    "Instructor: Dr.Yuhui Deng"
-   ]
-  },
   {
    "cell_type": "code",
-   "execution_count": 6,
-   "id": "2472b8ca",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "slide"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "f7a7d3f014ca4f599b561ca1e4e5e9ff",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "VBox(children=(HBox(children=(Text(value='x1^2 + x2^2', description='Expression:', style=TextStyle(description…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "5739708475934a0aa0fc6172f98ec6be",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/plain": [
-       "<optimization.common.contourPlot2d at 0x147a89a30>"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "execution_count": null,
+   "id": "011c5f34-be40-4f0b-8146-73f66ba18672",
+   "metadata": {},
+   "outputs": [],
    "source": [
     "from optimization.common import *\n",
-    "contourPlot2d(environ=\"notebook\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1d34025b",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "slide"
-    }
-   },
-   "source": [
-    "## Problem Description\n",
-    "\n",
-    "$\\textbf{Problem:}\\left\\{\\begin{align*}\n",
-    "\\min \\ f(x) \\\\\n",
-    "s.t. \\ x \\in \\mathbb{R}^n\n",
-    "\\end{align*} \\right.$\n",
-    "\n",
-    "where: \n",
-    "* $f: \\mathbb{R}^n \\mapsto \\mathbb{R}$\n",
-    "* No constraints are placed on the variables $x$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6f867a65",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "slide"
-    }
-   },
-   "source": [
-    "### One Dimentional Function\n",
-    "\n",
-    "$$\\min \\ f(x) = x^2 - 4x + 7$$"
+    "from optimization.gd_new import*"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "id": "302d9c60",
    "metadata": {
     "slideshow": {
      "slide_type": "fragment"
     }
    },
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "cf33688620c54a4d85cb0f9dbce63fe3",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "VBox(children=(HBox(children=(Text(value='(x - 2)**2 + 3', description='Expression:', style=TextStyle(descript…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "aa0a66db4d33470fbe7a091588bb27a6",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "3a84ef2c3bd14854b81d4cb48005991f",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/plain": [
-       "<optimization.common.funcPlot1d at 0x1674d2eb0>"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
-    "funcPlot1d(environ=\"notebook\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e2ce7e95",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "fragment"
-    }
-   },
-   "source": [
-    "$$x_{k+1} = x_{k} + a \\cdot d \\quad  d = \\left\\{\\begin{align*} +1 &\\text{ left} \\\\ -1 &\\text{ right}\\end{align*}\\right. \\ a \\geq 0$$"
+    "funcPlot1d(environ=\"jupyterlab\")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "id": "a4827f0c",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "578d548bc16c45caa29f47f60b58acf0",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "VBox(children=(HBox(children=(VBox(children=(Text(value='sin(x) + sin((10.0 / 3.0) * x)', description='Express…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "ead520028b844ef78e0932d537ac7172",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "16060b8b5ae04f31833395b3b1f4259c",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/plain": [
-       "<optimization.gd_new.gd_1d at 0x1675023a0>"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
-    "gd_1d(environ=\"notebook\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f55e58dd",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "slide"
-    }
-   },
-   "source": [
-    "### Multi Dimentional Function\n",
-    "\n",
-    "$$(1 - 8 \\cdot x_1 + 7 \\cdot x_1^2 - \\frac{7x_{1}^{3}}{3} + \\frac{x^{4}}{4}) \\cdot x_2^2 \\cdot e^{(-x_2)}$$"
+    "a = gd_1d(environ=\"jupyterlab\")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": null,
    "id": "7c589589",
    "metadata": {
     "slideshow": {
      "slide_type": "fragment"
     }
    },
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "5d9c8974d7c04f4dbdb7aa333ae7d172",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "VBox(children=(HBox(children=(Text(value='(1 - 8 * x1 + 7 * x1^2 - (7/3) * x1^3 + (1/4) * x1^4) * x2^2 * E^(-x…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "909764ee6cd94053a17da64592bb11e5",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "ba448ff14b8f454b8f4b6d3cb8f92c9e",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/plain": [
-       "<optimization.common.funcPlot2d at 0x167583b50>"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
-    "funcPlot2d(environ=\"notebook\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1a5b2203",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "subslide"
-    }
-   },
-   "source": [
-    "Some Questions:\n",
-    "\n",
-    "1. How many choices for the searching direction?\n",
-    "2. Which direction will you choose?\n",
-    "3. How far will you walk?\n"
+    "funcPlot2d(environ=\"jupyterlab\")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": null,
    "id": "f28bd167",
    "metadata": {
     "slideshow": {
      "slide_type": "subslide"
     }
    },
+   "outputs": [],
+   "source": [
+    "gd_2d(environ=\"jupyterlab\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3c737f58-0d7a-4e55-b50e-ec1e538c3822",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "expr = \"(1 - 8 * x1 + 7 * x1**2 - (7/3) * x1**3 + (1/4) * x1**4) * x2**2 * E**(-x2)\"\n",
+    "xn = np.array([0, 2])\n",
+    "x1 = symbols('x1')\n",
+    "x2 = symbols('x2')\n",
+    "expr = sympify(expr)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8940dc84-f52a-42a3-bf60-b8d50dd19620",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gradient = np.array([diff(expr, x1).subs(x1, xn[0]).subs(x2, xn[1]), \n",
+    "                                     diff(expr, x2).subs(x1, xn[0]).subs(x2, xn[1])], dtype=float)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "d5651fc9-3fcd-4e91-8d3c-04897da1ea02",
+   "metadata": {},
    "outputs": [
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "bd08d4c9d0014062a1e0654177dfbd55",
+       "model_id": "21388df7b7514d58a0e413beff1985ce",
        "version_major": 2,
        "version_minor": 0
       },
@@ -369,7 +111,7 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "26123777d5214fa28d4bfd16d9afe3b8",
+       "model_id": "83d82890dadb4e2eafc36993ff5acbe8",
        "version_major": 2,
        "version_minor": 0
       },
@@ -383,7 +125,7 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "176568fccd1f49d79abc66fff51859d6",
+       "model_id": "e7df9fb42125432a884d262cf6a8e5b3",
        "version_major": 2,
        "version_minor": 0
       },
@@ -393,89 +135,17 @@
      },
      "metadata": {},
      "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "7f1d7ace2eef4a5480c72444f52fd4c1",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/plain": [
-       "<optimization.gd_new.gd_2d at 0x167c26b20>"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
     }
    ],
    "source": [
-    "gd_2d(environ=\"notebook\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d585d6de",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "subslide"
-    }
-   },
-   "source": [
-    "### Multi Dimentional Function\n",
-    "\n",
-    "**Descent Direction** If a direction vector $d$ has the property that at least for sufficiently small $\\alpha > 0$, \n",
-    "$$f(x + \\alpha d) < f(x)$$\n",
-    "then $d$ is said a **descent direction** for function $f$ at point $x$."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3adf3a40",
-   "metadata": {
-    "slideshow": {
-     "slide_type": "subslide"
-    }
-   },
-   "source": [
-    "### Multi Dimentional Function\n",
-    "\n",
-    "梯度取极值"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "49f800ef",
-   "metadata": {},
-   "source": [
-    "### Multi Dimentional Function\n",
-    "\n",
-    "**When exact line search is used**, the steepest descent method obtains\n",
-    "$$x^{k+1} = x^{k} - \\alpha_{k} \\nabla f(x^k)$$\n",
-    "where $\\alpha_k$ is the solution of the one-dimensional minimization problem: \n",
-    "$$\\min_{\\alpha > 0} F(\\alpha) \\equiv f(x^{k} - \\alpha \\nabla f(x^k))$$\n",
-    "So,\n",
-    "$$\\begin{align*}F^{\\prime} = 0 \\Rightarrow - \\nabla f(x^k - \\alpha_k \\nabla f(x^k))^T \\nabla f(x^k) = 0 \\\\\n",
-    "\\Rightarrow \\nabla f(x^{k+1})^T \\nabla f(x^k) = 0\n",
-    "\\end{align*}$$\n",
-    "\n",
-    "This means that **every pair of $\\nabla f(x^{k+1})$ and $\\nabla f(x^k)$ are vertical.** Hence $\\{x^k\\}$ usually takes a zigzag path to approach the solution $x^{*}$. By this fact, we may better understand the reason why steepest descent method progresses slowly when the contours of $f$ are flat, see Section 4.7 (Appendix 2)."
+    "from optimization.gradient import *\n",
+    "a = gd2d_compete(environ=\"jupyterlab\")"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1b5fb35c",
+   "id": "3ce3b0fa-5813-49dd-90de-b28c5d3faf46",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -498,7 +168,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.13"
+   "version": "3.7.13"
   }
  },
  "nbformat": 4,