{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vectors\n",
    "\n",
    "Informally, we think of a vector as an object that has magnitude and direction. More formally, we think of an $n$-dimensional vector as an ordered tuple of numbers $(x_1, x_2, \\ldots, x_n)$ that follows the rules of scalar multiplication and vector addition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector space\n",
    "\n",
    "A vector space is a collection of vectors which is closed under addition and scalar multiplication. \n",
    "\n",
    "Examples:\n",
    "\n",
    "- Euclidean plane $\\mathbb{R}^2$ is a familiar vector space\n",
    "- The vector $\\pmatrix{0 & 0}$ is a trivial vector space that is a **vector subspace** of Euclidean space.\n",
    "- Polynomial functions of order $k$ is a vector space\n",
    "\n",
    "Polynomials of order 3 have the form $ax^3 + bx^2 + cx + d$, and can be represented as the vector\n",
    "\n",
    "$$\n",
    "\\pmatrix{a \\\\ b \\\\ c \\\\ d}\n",
    "$$\n",
    "\n",
    "- The space of all continuous functions is a vector space\n",
    "\n",
    "Consider two continuous functions, say, $f(x) = x^2$ and $g(x) = x^3)$. Scalar multiplication $(2 f)(x) = 2x^2$ and addition  $(f + g)(x) = x^2 + x^3$ are well defined and the result is a continuous function, so the space of all continuous functions is also a vector space. In this case, it is an infinite-dimensional vector space. \n",
    "\n",
    "Vector spaces are important because the theorems of linear algebra apply to all vector spaces, not just Euclidean space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Column vectors\n",
    "\n",
    "When we describe a vector $x$, we mean the column vector.  The row vector is denoted $x^T$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.random.random((5,1))\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Length\n",
    "\n",
    "The length of a vector is the Euclidean norm (i.e. Pythagoras theorem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.norm(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sqrt(np.sum(x**2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Direction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = x/np.linalg.norm(x)\n",
    "n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.norm(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Norms and distances\n",
    "\n",
    "Recall that the 'norm' of a vector $v$, denoted $||v||$ is simply its length.  For a vector with components \n",
    "$$v = \\left(v_1,...,v_n\\right)$$\n",
    "the norm of $v$ is given by:\n",
    "    \n",
    "$$||v|| = \\sqrt{v_1^2+...+v_n^2}$$\n",
    "\n",
    "The distance between two vectors is the length of their difference:\n",
    "    \n",
    "$$d(v,w) = ||v-w||$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = np.array([3,0]).reshape((-1,1))\n",
    "v = np.array([0,4]).reshape((-1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.norm(u - v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.norm(v - u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sqrt(np.sum((u - v)**2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(3).reshape((-1,1))\n",
    "y = np.arange(3).reshape((-1,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scalar multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "3 * x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector addition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "3*x + 4*y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transposition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dot product\n",
    "\n",
    "The dot product of two vectors $u$ and $v$ is written as $u \\cdot v$ and its value is given by $u^Tv$. The dot product of two $n$ dimensional vectors $v$ and $w$ is given by:\n",
    "\n",
    "$$u \\cdot v = u_1v_1+...+u_nv_n$$\n",
    "\n",
    "I.e. the dot product is just the sum of the product of the components.\n",
    "\n",
    "The inner product $\\langle u,v \\rangle$ of two vectors is a generalization of the dot product. It is any function that takes two vectors, returns a scalar (here we just consider inner products that return real numbers), and obeys the following properties:\n",
    "\n",
    "- symmetry $\\langle u,v \\rangle = \\langle v,u \\rangle$\n",
    "- positive definite \n",
    "    - $\\langle v,v \\rangle \\ge 0$\n",
    "    - $\\langle v,v \\rangle = 0 \\implies v = 0$\n",
    "- bilinear\n",
    "    - $\\langle au,v \\rangle = a \\langle u,v \\rangle$\n",
    "    - $\\langle u + v,w \\rangle = \\langle u,w \\rangle + \\langle v,w \\rangle$\n",
    "    - Linearity also applies to second argument because of symmetry\n",
    "\n",
    "Any inner product determines a norm via:\n",
    "\n",
    "$$||v|| = \\langle v,v \\rangle^{\\frac12}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating dot products"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = np.array([3,3]).reshape((-1,1))\n",
    "v = np.array([2,0]).reshape((-1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(u.T, v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the `@` operator to do matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "u.T @ v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(u * v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Geometry of dot product\n",
    "\n",
    "Geometrically, the dot product is the product of the length of $v$ and the length of the projection of $u$ onto the unit vector $\\widehat{v}$. \n",
    "\n",
    "$$\n",
    "u \\cdot v = \\lvert u \\rvert \\lvert v \\rvert \\cos \\theta\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(*zip(np.zeros_like(u), u), 'b-')\n",
    "plt.text(1.8, 0.1, 'v', fontsize=14)\n",
    "plt.plot(*zip(np.zeros_like(v), v), 'b-')\n",
    "plt.text(2.8, 2.6, 'u', fontsize=14)\n",
    "plt.tight_layout()\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Angle between two vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cos_angle = np.dot(u.T, v)/(np.linalg.norm(u)*np.linalg.norm(v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cos_angle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = 180/np.pi*np.arccos(cos_angle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outer product\n",
    "\n",
    "Note that the inner product is just matrix multiplication of a $1\\times n$ vector with an $n\\times 1$ vector.  In fact, we may write:\n",
    "\n",
    "$$\\langle v,w \\rangle = v^Tw$$\n",
    "\n",
    "The *outer product* of two vectors is just the opposite. It is given by:\n",
    "\n",
    "$$v\\otimes w = vw^T$$\n",
    "\n",
    "Note that I am considering $v$ and $w$ as *column* vectors.  The result of the inner product is a *scalar*. The result of the outer product is a *matrix*.\n",
    "\n",
    "For example, if $v$ and $w$ are both in $\\mathbb{R}^3$\n",
    "\n",
    "$$\n",
    "v \\otimes w = \\pmatrix{v_1\\\\v_2\\\\v_3} \\pmatrix{w_1 & w_2 & w_3} = \\pmatrix{\n",
    "v_1w_1 & v_1w_2 & v_1w_3\\\\\n",
    "v_2w_1 & v_2w_2 & v_2w_3 \\\\ \n",
    "v_3w_1 & v_3w_2 & v_3w_3}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([1,2,3]).reshape((-1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v @ v.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.outer(v, v)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}