{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Setting up a local install of Jupyter with multiple kernels (Python 3.5, Python 2.7, R, Juila)\n", "====\n", "\n", "The only installation you are recommended to do is to install Anaconda 3.5, so that you have a backup when the OIT version is flaky. The other kernels and the Docker version are **not required** and you should only do so if you are comforatble with command line installs. Even the Anaconda 3.5 installation is optional if the OIT version works well for you.\n", "\n", "Note: I have only done this on OSX 10.11.2 (El Capitan) with XCode and command line compilers installed.\n", "\n", "To install Anaconda for Python 3.5\n", "---- \n", "\n", "Download and install Anaconda Python 3.5 from https://www.continuum.io/downloads\n", "\n", "Open a terminal\n", "```bash\n", "conda update conda\n", "conda update anaconda\n", "```\n", "\n", "(OPTIONAL) To install Python 2.7 as well\n", "----\n", "\n", "Open a terminal\n", "```bash\n", "conda create -n py27 python=2.7 anaconda\n", "source activate py27\n", "ipython kernel install\n", "source deactivate\n", "```\n", "\n", "(OPTIONAL) To install R\n", "----\n", "\n", "- If you want `conda` to manage your R packages\n", "\n", "```bash\n", "conda install -y -c r r-irkernel r-recommended r-essentials\n", "```\n", "\n", "> Note: The bug that required this appears to have been fixed\n", "```\n", "wget https://anaconda.org/r/ncurses/5.9/download/osx-64/ncurses-5.9-1.tar.bz2 \\\n", " https://anaconda.org/r/nlopt/2.4.2/download/osx-64/nlopt-2.4.2-1.tar.bz2 \\\n", " && conda install --yes ncurses-5.9-1.tar.bz2 nlopt-2.4.2-1.tar.bz2\n", "```\n", "\n", "- If you have an existing R installation that you want to use\n", "\n", "Start R\n", "```R\n", "install.packages(c('rzmq','repr','IRkernel','IRdisplay'),\n", " repos = c('http://irkernel.github.io/', getOption('repos')))\n", "IRkernel::installspec()\n", "```\n", "\n", "(OPTIONAL) To install Julia\n", "----\n", "\n", "Download and install Julia from http://julialang.org/downloads/\n", "\n", "Start Julia\n", "```julia\n", "Pkg.add(\"IJulia\")\n", "Pkg.build(\"IJulia\")\n", "```\n", "\n", "(OPTIONAL) To install `pyspark`\n", "----\n", "\n", "Open a terminal\n", "```bash\n", "conda install -y -c anaconda-cluster spark\n", "```\n", "\n", "Check\n", "----\n", "Open terminal\n", "```\n", "jupyter notebook\n", "```\n", "\n", "See if the installed kernels are found in the drop-down menu." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## (OPTIONAL) Installing Spark via Docker\n", "\n", "- Install Docker (https://docs.docker.com/engine/installation/)\n", "- Launch the Docker Quickstart Terminal\n", "\n", "Be patient - this can take a while the first time you do it\n", "\n", "When done, it shouuld show something like this\n", "```\n", " ## .\n", " ## ## ## ==\n", " ## ## ## ## ## ===\n", " /\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\\___/ ===\n", " ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~\n", " \\______ o __/\n", " \\ \\ __/\n", " \\____\\_______/\n", "\n", "\n", "docker is configured to use the default machine with IP 192.168.99.100\n", "For help getting started, check out the docs at https://docs.docker.com\n", "```\n", "\n", "**Note the IP address given - you will need this to access the notebook.**\n", "\n", "In the Docker terminal\n", "```\n", "docker run -d -p 8888:8888 jupyter/all-spark-notebook\n", "```\n", "\n", "For how to connect to a Spark cluster, see [official instructions](https://github.com/jupyter/docker-stacks/tree/master/all-spark-notebook)\n", "\n", "### Testing the Docker installation\n", "\n", "Check by typing in the Docker terminal\n", "```\n", "docker ps\n", "```\n", "\n", "Be patient - this can take a while the first time you do it.\n", "\n", "It shoudl show something like\n", "```bash\n", "CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n", "965a6a80bf44 jupyter/all-spark-notebook \"tini -- start-notebo\" 4 minutes ago Up 4 minutes 0.0.0.0:8888->8888/tcp big_kilby\n", "```\n", "\n", "**Note the machine name (mine is big_kilby, yours will likely be different).**\n", "\n", "Open your browser at the following URL http://192.168.99.100:8888 (Use the IP given above)\n", "\n", "This should bring you to a Jupyter notebook. Open a Python3 notebook from the drop dwon menu and test:\n", "\n", "```python\n", "import pyspark\n", "sc = pyspark.SparkContext('local[*]')\n", "\n", "# do something to prove it works\n", "rdd = sc.parallelize(range(1000))\n", "rdd.takeSample(False, 5)\n", "```\n", "\n", "If successful, you should get a list of 5 integers after a short delay.\n", "\n", "Save and exit the notebook.\n", "\n", "Cleap up in the Docker terminal\n", "```\n", "docker stop big_kilby\n", "exit\n", "```\n", "\n", "Use the machine name foudnd with `docker ps` in place of `big_kilby`.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }