TerenceLiu
/
optimization
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

update

This commit is contained in:
TerenceLiu 2022-11-05 21:30:07 +08:00
parent e995851232
commit 40671c8432
2 changed files with 61 additions and 488 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

105
algorithm/interactive.ipynb
View File

@ -1,59 +1,5 @@
{ {
"cells": [ "cells": [
{
"cell_type": "markdown",
"id": "b705b060",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Steepest Descent Method\n",
"\n",
"Instructor: Dr.Yuhui Deng\n",
"\n",
"![level_gradient](figure/level_gradient.png)"
]
},
{
"cell_type": "markdown",
"id": "f576c3bd",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Problem Description\n",
"\n",
"Problem: \n",
"\n",
"$\\begin{align*} \n",
"min(fx) \\\\\n",
"s.t. \\ x \\in \\mathbb{R}^n\n",
"\\end{align*}$\n",
"where\n",
"\n",
"* $f: \\mathbb{R}^n \\rightarrow \\mathbb{R}$\n",
"* No constraints are placed on the variables $X$\n"
]
},
{
"cell_type": "markdown",
"id": "c6e94df1",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# One Dimential Function\n",
"\n",
"$$min \\ f(x)=x^2 - 4x + 7$$\n",
"\n"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
@ -120,7 +66,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 1, "execution_count": null,
"id": "61e0d2b8-41c8-4790-8b25-2df6ba4c3625", "id": "61e0d2b8-41c8-4790-8b25-2df6ba4c3625",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@ -130,53 +76,10 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 2, "execution_count": null,
"id": "cc84e26c-7135-427b-9a5b-c39b6b495fe5", "id": "cc84e26c-7135-427b-9a5b-c39b6b495fe5",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [],
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e2a6bc45f7354f529c8475f034416135",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(HBox(children=(Text(value='-2 * x * sin(-(pi/4) * x)+10', description='Expression:', style=Text…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c2c3118f55d241e68649a094a6bb3836",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "cdc830df33fe4705b448245a1d64c335",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [ "source": [
"a = func_1d()" "a = func_1d()"
] ]
@ -207,7 +110,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.9.13" "version": "3.7.13"
} }
}, },
"nbformat": 4, "nbformat": 4,

442
algorithm/lecture.ipynb
View File

@ -1,361 +1,103 @@
{ {
"cells": [ "cells": [
{
"cell_type": "markdown",
"id": "9b492ee6",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Steepest Descent Method\n",
"\n",
"Instructor: Dr.Yuhui Deng"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 6, "execution_count": null,
"id": "2472b8ca", "id": "011c5f34-be40-4f0b-8146-73f66ba18672",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f7a7d3f014ca4f599b561ca1e4e5e9ff",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(HBox(children=(Text(value='x1^2 + x2^2', description='Expression:', style=TextStyle(description…"
]
},
"metadata": {}, "metadata": {},
"output_type": "display_data" "outputs": [],
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5739708475934a0aa0fc6172f98ec6be",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<optimization.common.contourPlot2d at 0x147a89a30>"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [ "source": [
"from optimization.common import *\n", "from optimization.common import *\n",
"contourPlot2d(environ=\"notebook\")" "from optimization.gd_new import*"
]
},
{
"cell_type": "markdown",
"id": "1d34025b",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Problem Description\n",
"\n",
"$\\textbf{Problem:}\\left\\{\\begin{align*}\n",
"\\min \\ f(x) \\\\\n",
"s.t. \\ x \\in \\mathbb{R}^n\n",
"\\end{align*} \\right.$\n",
"\n",
"where: \n",
"* $f: \\mathbb{R}^n \\mapsto \\mathbb{R}$\n",
"* No constraints are placed on the variables $x$"
]
},
{
"cell_type": "markdown",
"id": "6f867a65",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### One Dimentional Function\n",
"\n",
"$$\\min \\ f(x) = x^2 - 4x + 7$$"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 8, "execution_count": null,
"id": "302d9c60", "id": "302d9c60",
"metadata": { "metadata": {
"slideshow": { "slideshow": {
"slide_type": "fragment" "slide_type": "fragment"
} }
}, },
"outputs": [ "outputs": [],
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "cf33688620c54a4d85cb0f9dbce63fe3",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(HBox(children=(Text(value='(x - 2)**2 + 3', description='Expression:', style=TextStyle(descript…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "aa0a66db4d33470fbe7a091588bb27a6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3a84ef2c3bd14854b81d4cb48005991f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<optimization.common.funcPlot1d at 0x1674d2eb0>"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [ "source": [
"funcPlot1d(environ=\"notebook\")" "funcPlot1d(environ=\"jupyterlab\")"
]
},
{
"cell_type": "markdown",
"id": "e2ce7e95",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$x_{k+1} = x_{k} + a \\cdot d \\quad d = \\left\\{\\begin{align*} +1 &\\text{ left} \\\\ -1 &\\text{ right}\\end{align*}\\right. \\ a \\geq 0$$"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 10, "execution_count": null,
"id": "a4827f0c", "id": "a4827f0c",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [],
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "578d548bc16c45caa29f47f60b58acf0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(HBox(children=(VBox(children=(Text(value='sin(x) + sin((10.0 / 3.0) * x)', description='Express…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ead520028b844ef78e0932d537ac7172",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "16060b8b5ae04f31833395b3b1f4259c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<optimization.gd_new.gd_1d at 0x1675023a0>"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [ "source": [
"gd_1d(environ=\"notebook\")" "a = gd_1d(environ=\"jupyterlab\")"
]
},
{
"cell_type": "markdown",
"id": "f55e58dd",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Multi Dimentional Function\n",
"\n",
"$$(1 - 8 \\cdot x_1 + 7 \\cdot x_1^2 - \\frac{7x_{1}^{3}}{3} + \\frac{x^{4}}{4}) \\cdot x_2^2 \\cdot e^{(-x_2)}$$"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 11, "execution_count": null,
"id": "7c589589", "id": "7c589589",
"metadata": { "metadata": {
"slideshow": { "slideshow": {
"slide_type": "fragment" "slide_type": "fragment"
} }
}, },
"outputs": [ "outputs": [],
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5d9c8974d7c04f4dbdb7aa333ae7d172",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(HBox(children=(Text(value='(1 - 8 * x1 + 7 * x1^2 - (7/3) * x1^3 + (1/4) * x1^4) * x2^2 * E^(-x…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "909764ee6cd94053a17da64592bb11e5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ba448ff14b8f454b8f4b6d3cb8f92c9e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<optimization.common.funcPlot2d at 0x167583b50>"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [ "source": [
"funcPlot2d(environ=\"notebook\")" "funcPlot2d(environ=\"jupyterlab\")"
]
},
{
"cell_type": "markdown",
"id": "1a5b2203",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Some Questions:\n",
"\n",
"1. How many choices for the searching direction?\n",
"2. Which direction will you choose?\n",
"3. How far will you walk?\n"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 13, "execution_count": null,
"id": "f28bd167", "id": "f28bd167",
"metadata": { "metadata": {
"slideshow": { "slideshow": {
"slide_type": "subslide" "slide_type": "subslide"
} }
}, },
"outputs": [],
"source": [
"gd_2d(environ=\"jupyterlab\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3c737f58-0d7a-4e55-b50e-ec1e538c3822",
"metadata": {},
"outputs": [],
"source": [
"expr = \"(1 - 8 * x1 + 7 * x1**2 - (7/3) * x1**3 + (1/4) * x1**4) * x2**2 * E**(-x2)\"\n",
"xn = np.array([0, 2])\n",
"x1 = symbols('x1')\n",
"x2 = symbols('x2')\n",
"expr = sympify(expr)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8940dc84-f52a-42a3-bf60-b8d50dd19620",
"metadata": {},
"outputs": [],
"source": [
"gradient = np.array([diff(expr, x1).subs(x1, xn[0]).subs(x2, xn[1]), \n",
" diff(expr, x2).subs(x1, xn[0]).subs(x2, xn[1])], dtype=float)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "d5651fc9-3fcd-4e91-8d3c-04897da1ea02",
"metadata": {},
"outputs": [ "outputs": [
{ {
"data": { "data": {
"application/vnd.jupyter.widget-view+json": { "application/vnd.jupyter.widget-view+json": {
"model_id": "bd08d4c9d0014062a1e0654177dfbd55", "model_id": "21388df7b7514d58a0e413beff1985ce",
"version_major": 2, "version_major": 2,
"version_minor": 0 "version_minor": 0
}, },
@ -369,7 +111,7 @@
{ {
"data": { "data": {
"application/vnd.jupyter.widget-view+json": { "application/vnd.jupyter.widget-view+json": {
"model_id": "26123777d5214fa28d4bfd16d9afe3b8", "model_id": "83d82890dadb4e2eafc36993ff5acbe8",
"version_major": 2, "version_major": 2,
"version_minor": 0 "version_minor": 0
}, },
@ -383,7 +125,7 @@
{ {
"data": { "data": {
"application/vnd.jupyter.widget-view+json": { "application/vnd.jupyter.widget-view+json": {
"model_id": "176568fccd1f49d79abc66fff51859d6", "model_id": "e7df9fb42125432a884d262cf6a8e5b3",
"version_major": 2, "version_major": 2,
"version_minor": 0 "version_minor": 0
}, },
@ -393,89 +135,17 @@
}, },
"metadata": {}, "metadata": {},
"output_type": "display_data" "output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7f1d7ace2eef4a5480c72444f52fd4c1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<optimization.gd_new.gd_2d at 0x167c26b20>"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
} }
], ],
"source": [ "source": [
"gd_2d(environ=\"notebook\")" "from optimization.gradient import *\n",
] "a = gd2d_compete(environ=\"jupyterlab\")"
},
{
"cell_type": "markdown",
"id": "d585d6de",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Multi Dimentional Function\n",
"\n",
"**Descent Direction** If a direction vector $d$ has the property that at least for sufficiently small $\\alpha > 0$, \n",
"$$f(x + \\alpha d) < f(x)$$\n",
"then $d$ is said a **descent direction** for function $f$ at point $x$."
]
},
{
"cell_type": "markdown",
"id": "3adf3a40",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Multi Dimentional Function\n",
"\n",
"梯度取极值"
]
},
{
"cell_type": "markdown",
"id": "49f800ef",
"metadata": {},
"source": [
"### Multi Dimentional Function\n",
"\n",
"**When exact line search is used**, the steepest descent method obtains\n",
"$$x^{k+1} = x^{k} - \\alpha_{k} \\nabla f(x^k)$$\n",
"where $\\alpha_k$ is the solution of the one-dimensional minimization problem: \n",
"$$\\min_{\\alpha > 0} F(\\alpha) \\equiv f(x^{k} - \\alpha \\nabla f(x^k))$$\n",
"So,\n",
"$$\\begin{align*}F^{\\prime} = 0 \\Rightarrow - \\nabla f(x^k - \\alpha_k \\nabla f(x^k))^T \\nabla f(x^k) = 0 \\\\\n",
"\\Rightarrow \\nabla f(x^{k+1})^T \\nabla f(x^k) = 0\n",
"\\end{align*}$$\n",
"\n",
"This means that **every pair of $\\nabla f(x^{k+1})$ and $\\nabla f(x^k)$ are vertical.** Hence $\\{x^k\\}$ usually takes a zigzag path to approach the solution $x^{*}$. By this fact, we may better understand the reason why steepest descent method progresses slowly when the contours of $f$ are flat, see Section 4.7 (Appendix 2)."
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "1b5fb35c", "id": "3ce3b0fa-5813-49dd-90de-b28c5d3faf46",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [] "source": []
@ -498,7 +168,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.9.13" "version": "3.7.13"
} }
}, },
"nbformat": 4, "nbformat": 4,