![](\"IPBytesOverTime.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"slide_helper": "subslide_end",
"slide_type": "subslide"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Tools: IPython\n",
"==============\n",
"\n",
"* Powerful interactive shells (terminal and Qt-based).\n",
"* A browser-based notebook with support for code, text, mathematical expressions, inline plots and other rich media.\n",
"* Support for interactive data visualization and use of GUI toolkits.\n",
"* Flexible, embeddable interpreters to load into your own projects.\n",
"* Easy to use, high performance tools for parallel computing.\n",
"\n",
"Installation\n",
"------------\n",
"\n",
" > pip install ipython"
]
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"slide_helper": "subslide_end",
"slide_type": "subslide"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Tools: pandas - Python Data Analysis Library\n",
"=====================================\n",
"\n",
"pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language \n",
"\n",
" * **Series** - array of data with an optional index\n",
" * **DataFrame** - 2D array of data hierarchical row and column indexing\n",
" \n",
"Installation\n",
"------------\n",
"\n",
"Linux / Mac with dev tools?\n",
"\n",
" > pip install pandas\n",
"\n",
"Otherwise see [pandas.pydata.org](http://pandas.pydata.org/)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Tools: matlibplot - Python Plotting\n",
"===================================\n",
"matlibplot hooks in with IPython notebook to provide in browser graphs."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"\n",
"x = np.linspace(0, 2)\n",
"y = np.sin(4 * np.pi * x) * np.exp(-5 * x)\n",
"\n",
"plt.fill(x, y, 'r')\n",
"plt.grid(True)\n",
"plt.show()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEACAYAAAC08h1NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8XHW57/FP7pc2TaTdtDQtnZoWIaX0SuWASgC3G1Qo\n26On1IPHomyLxwpH3FCrgBzFrbBBEFFANh7BLYLbG1W5yMUKIgJFmraklyRt2ia90bQzaWZym8yc\nP9ZMGdJJMsn6zfzWTL7v12u9ZlZmzVpPH4YnK8/6zW+BiIiIiIiIiIiIiIiIiIiIiIiIiIhkuQuB\nrUAjsHqQbeqAN4DNwLqMRCUiImlRADQBPqAI2ACcNmCbKuBNYFpsfVKmghMRkePlu3z/EpzC3wL0\nAY8CSwds80ngV0BrbP2Qy2OKiIgLbgt/NbAnYb019rNEs4ETgD8B64FPuTymiIi4UOjy/dEUtikC\nFgIXAOXAy8DfcK4JiIhIhrkt/G3A9IT16bzd0onbg9Pe6YotLwDzGFD4p06dGt27d6/LcERExpRm\nYNZI3+S21bMep5XjA4qBZcDaAds8DrwP50JwOfBeoGHgjvbu3Us0GtViYPn6179uPYZcWpRP5dOr\nC1AzmsLt9ow/DKwCnsYp7A8CW4CVsdfvxxnq+RSwEYgAD5Ck8Is5LS0ttkPIKcqnWcqnfW4LP8CT\nsSXR/QPWb48tIiJimdtWj3jQihUrbIeQU5RPs5RP+/JsB5AgGutZiYhICvLy8mAUdVxn/Dlo3bp1\ntkPIKcqnWcqnfVlb+KPRKD09PbbDEBHJOlnb6vnd737Hqs99jpdee41p06YN/wYRkRwz5lo9G+vr\nKTp4kA+dcw7t7e22wxERyRpZW/gb6+tZE4lw8b59fKSujs7OTtsheYZ6qGYpn2Ypn/Zlb+HfsoXZ\nwHf6+qhtauLjF11Eb2+v7bBERDwva3v8J1ZUUN/ZyUk4Xx/+eFkZ5R/6EP/561+Tn5+1v89ERFI2\npnr8fr+frp4epsTWC4Gfd3XR9swzXLNyJfo+gIjI4LKy8Dc2NjK7rOwdv+bKgLWhEH955BG+eeON\ntkLzBPVQzVI+zVI+7cvewp/krL4SeCoU4uE77+TRRx7JfGAiIlkgOwv/tm3MHmQUz2TgxlCI3z/2\nWGaD8pC6ujrbIeQU5dMs5dO+7Cz8GzYkPeOPqwUaNm/OXEAiIlkkKwv/9q1bOWWI108Ftu3ZQ39/\nf6ZC8hT1UM1SPs1SPu3LusIfjUZp3LOH2UNsUwFMKi5m165dmQpLRCRrZN04/kOHDjF72jQO9/QM\nGfyFEyaw6mc/46Mf/ai5CEVEPGTMjONvbGxkdknJsP/S2lCILQ26w6OIyEDZWfgjkWG3qw2HaVi/\nPgMReY96qGYpn2Ypn/aZKPwX4txQvRFYneT1OiAAvBFbbnBzsO1btnBKMDjsdrVAw8aNbg4lIpKT\n3Pb4C4BtwAeBNuA1YDmwJWGbOuBa4JJh9pVSj3/ZRRex9Kmn+OQw2x0BZpSUEOjqivfBRERyiq0e\n/xKgCWgB+oBHgaVJtjNWeRu3bx9yRE/cu4Bx+fm0traaOrSISE5wW/irgT0J662xnyWKAmcD9cAT\nOF2YUYlGozS2taVU+AFqi4tpGIMXeNVDNUv5NEv5tM9t4U9lGsy/A9OBecD3gd+O9mD79++nNC+P\nqhS3r+3q0sgeEZEBCl2+vw2nqMdNxznrT3Q04fmTwA+BE4DDA3e2YsUKfD4fAFVVVcyfP//YvB7r\n1q2jvr6eU0pKoLubdbH31MUek60X9vbS8Prrx94PvGN/ubpeV1fnqXiyfV35VD69sh5/3tLSghtu\ne++FOBd3LwD2Aq9y/MXdycBBnL8OlgC/AHxJ9jXsxd0HH3yQF6++mp+EQikF92fga6efzl82bUpp\nexGRbGLr4m4YWAU8DTQAj+EU/ZWxBeDjwCZgA3AXcNloD9a4ZQuzUyz6AKcBDTt2jLkbsySeHYh7\nyqdZyqd9bls94LRvnhzws/sTnv8gtrjWWF/PshFs/w9AfiTCgQMHmDJlyrDbi4iMBV4a4D5sq2fu\njBk8vHs3C0aw0w9UVnLzr3/N+eef7y46ERGPyfm5eiKRCM379qU8lDOutqeHLVu2DL+hiMgYkTWF\nv62tjaqiIsaP8H213d3HRvaMFeqhmqV8mqV82pc1hb+xsZHZRUUjft9pQMOGDeYDEhHJUlnT47/v\nvvtYf+21/EdX14h22gYsrKjgQEeHy/BERLwl53v8jW++ySkjLPoAU4Hunh7a29vNByUikoWyp/Bv\n3DjiC7vg/CqsLSsbUxd41UM1S/k0S/m0L3sKf1PTqAo/QG1f35icrE1EJJms6PH39/czrrSUI+Ew\nZaPY8e1A61VXcde997oKUETES3K6x79r1y5OLC4eVdGH2N24/v53kyGJiGStrCj8jY2NnDKKoZxx\ntUBDY6O5gDxOPVSzlE+zlE/7sqbwz+7pGfX7Twb8nZ0EAgFzQYmIZKnsKPybNjG7u3vU788HTi0r\nY+vWreaC8rD4HN5ihvJplvJpX1YU/u2bNo16RE/caf39GtkjIkKWFP7G5mbXhb82GKShvt5IPF6n\nHqpZyqdZyqd9ni/8fX19tLa3826X+6kFGtavNxGSiEhW8/w4/u3bt3PRokU0d3a62nkj8KFJk9j5\n1luu9iMi4hU5O45/+/btzM53H+ZMYL/fTzAYdB+UiEgW83zhb9y+3dWInrhCYHZZGdu2bXMflMep\nh2qW8mmW8mmficJ/IbAVp5uyeojtzsS5OfvHRrLzxk2bmN3bO/roEtRGoxrZIyJjntvCXwDcg1P8\na4HlOPc+SbbdrcBTjLAf1bh5M6e4DDKuNhikYeNGQ3vzLo2TNkv5NEv5tM9t4V8CNAEtQB/wKLA0\nyXZfBH4JjPjKauPOna6HcsbVRqMa2SMiY57bwl8N7ElYb439bOA2S4H41JiD32ZrgO7ubvb7/cxw\nFeLbaoGGMTAvv3qoZimfZimf9hW6fH8qRfwu4CuxbfMYotWzYsUKfD4fAFVVVUycOJHqsjIKOztZ\nF9umLvY4mvU+YPdbb9Hd3c3f/vY35/XYn53xD6PWta51rXt1Pf68paUFN9yO4z8LuBmnxw+wBojg\n9PPjdiQcZxIQAv4FWDtgX8eN43/11Vf5woc+xGsGJ1c7raKCX7z0EnPnzjW2TxERG2yN418PzAZ8\nQDGwjOML+rtxhtHPxOnzfz7JNkn5/X6qXAY40GmgkT0iMqa5LfxhYBXwNNAAPAZsAVbGFlf8fj9V\nkYjb3bxDTSjEzh07jO7TaxL/LBT3lE+zlE/73Pb4AZ6MLYnuH2TbK0ay40AgQGV//6iCGoyvv59N\nY2R6ZhGRZDz9zV2/30+VoS9vxfmAlu3bje7Ta+IXhMQM5dMs5dM+bxf+9naqwmGj+5wJtOzebXSf\nIiLZxNOFP/DWW1Qa3ucMYNfBgySbCTRXqIdqlvJplvJpn6cLv//QIeOjesYBFYWFHDhwwPCeRUSy\ng7cL/+HDxgs/gK+khJ07d6Zhz96gHqpZyqdZyqd9ni78Ab/feKsHwBeJuP7mm4hItvJ04fd3dKTn\njD8UoiWHz/jVQzVL+TRL+bTP24X/6NH0FP7+flrGwGRtIiLJePqeuxNKS9nT02O83fME8L33vpen\nYxO1iYhko5y75244HCbY20tFGvbtQ2P5RWTs8mzh7+joYEJxcVoC9OFMzxwxPA+QV6iHapbyaZby\naZ9nC38gEKCy0MRUQscrByYUFWksv4iMSZ4t/H6/n6qCgrTt31dcnLNDOjVO2izl0yzl0z5vF/68\n9F171lh+ERmrPFv4A4EAlWmcT8eXw/Pyq4dqlvJplvJpn2cLv9/vp8rwXPyJNJZfRMYqbxf+vr60\n7d9H7s7Lrx6qWcqnWcqnfZ4t/AG/n0rDN2FJ5ANa9uxJ2/5FRLzKs4Xff/BgWqZriJsB7D50KCfH\n8quHapbyaZbyaZ+Jwn8hsBVoBFYneX0pUA+8AbwOnJ/KTv1vvZXWwl8OVBYVsX///jQeRUTEe9wW\n/gLgHpziXwssB04bsM2zwDxgAbAC+FEqOw4cPpyWKZkTzczRsfzqoZqlfJqlfNrntvAvAZqAFqAP\neBTnDD9RMOH5eOBQKjv2HzmS1jN+0Fh+ERmb3Bb+aiDxCmlr7GcDXQpsAZ4Erk5lx/5AIP2FP0fn\n5VcP1Szl0yzl0z63k+Gk+g2r38aW9wM/Bd6TbKMVK1bg8/kA2HXgAFuBRbHX1sUe6wyu9yaM5Y9/\nGON/hmpd61rXutfW48/ddirczolwFnAzTo8fYA0QAW4d4j3NOC2i9gE/f8d8/BPHjWNbKMQklwEO\n5SngjjPP5JlXX03jUURE0sPWfPzrgdk4w+KLgWXA2gHb1CQEtjD2OLDov0M0GiXQ3Z32i7s+NC+/\niIw9bgt/GFgFPA00AI/h9PJXxhaA/w5swhnO+T3gsuF2GgwGKS0ooMhlcMPJ1bH8iX8WinvKp1nK\np30mJrx/MrYkuj/h+W2xJWV+v5/KoiJI45QNAGXAu4qL2bdvH9XVya5Ji4jkHk9+czfdc/EnysV5\n+eMXhMQM5dMs5dM+Txb+QCBAVX5mQpvZ359zhV9EZCieLPx+vz+tc/EnysWx/OqhmqV8mqV82ufZ\nwl+VoQuuvkhE8/KLyJjiycIfCASoCoczciwfuTcvv3qoZimfZimf9nmy8PuPHEnrXPyJfGhefhEZ\nW7xZ+A8dylir52RgT3t7To3lVw/VLOXTLOXTPk8W/sChQ2mfoC2uDHhXURF79+7N0BFFROzyZOH3\nHzqU9ukaEuXaWH71UM1SPs1SPu3zZuE/fDhjZ/wAPo3lF5ExxJOFP5CBufgTzcyxsfzqoZqlfJql\nfNrnycLvP3o0s62eSISWhoYMHlFExB638/GbdGw+/imVlbzR0cFJGTrwH4HbFi3i2fXrM3REERH3\nbM3HnxaBUCizPX40ll9Exg7PFf7u7m76IxFKM3jM+Fj+/v7+DB41fdRDNUv5NEv5tM9zhT8QCFBV\nXJzRHlQpMLGkhH379mXwqCIidniz8Bel+95bx/MVFeXMkE6NkzZL+TRL+bTPc4Xf7/dTmaG5+BP5\n+vvZmUNDOkVEBuPJwp/JC7txvlCIlh07LBzZPPVQzVI+zVI+7TNR+C8EtgKNwOokr/9PoB7YCLwE\nnDHUzgKBQMYmaEukeflFZKxwW/gLgHtwin8tsBw4bcA2O4AP4BT8bwI/GmqHfr+fSguja2YCLY2N\nGT9uOqiHapbyaZbyaZ/bwr8EaAJagD7gUWDpgG1eBgKx568A04baod/vpypDc/En8gEtra0ZP66I\nSKa5LfzVQOI3n1pjPxvMZ4Enhtph4PDhjN19K9HJQGuOjOVXD9Us5dMs5dO+QpfvH8kd0c8DPgOc\nM9gGK1asoKG+nonAXcB8oC722rrYY7rWXwYqCgvZu3cv06dPP/bhjP9ZqnWta13rttfjz90OPXf7\nPamzgJtxevwAa4AIcOuA7c4Afh3brmmQfUWj0SiXL13KP61dy6dcBjYa76us5FuPP865555r4egi\nIiNja66e9cBsnBZ5MbAMWDtgm5Nxiv7lDF70jwkcOWJlOCdATThMc3OzpaOLiGSG28IfBlYBTwMN\nwGPAFmBlbAG4CXgXcC/wBvDqUDv0HzmS0SmZE9UEgzRv22bp6OYk/lko7imfZimf9rnt8QM8GVsS\n3Z/w/MrYkhJ/R4e9M35g7aZNlo4uIpIZnpuP/+SJE/nL4cOcbCGAV4AvzJrF+hwZzy8iuS1n5uP3\nB4P2Wj1Ac1ubpaOLiGSGpwp/f38/wd5eKiwdfyIQ6e/n8OHDliIwQz1Us5RPs5RP+zxV+Ds6OphQ\nXGwtqDygprSUpqZhBx+JiGQtTxV+v99PZaGJ682jNysazfohnfEvfYgZyqdZyqd9niv8VQUFVmOo\nCQZp1sVdEclhnir8gUCAqjy7A41qIhGaN2+2GoNb6qGapXyapXza56nC7/f7qYyOZPof82YBTVu3\nWo1BRCSdPFf4qyzPjlkDNO/ebTUGt9RDNUv5NEv5tM9ThT8QCFDV12c1hmrgSDBIKBSyGoeISLp4\nqvD7jxyh0sJNWBLlA76yMnZk8f131UM1S/k0S/m0z1uF/+BBa/P0JJqVl6ex/CKSszxV+AOHDnmi\n8Nd0ddGcxYVfPVSzlE+zlE/7PFX4/e3t1ubpSVTT10fzm2/aDkNEJC28Vfgt3oQlUQ3QvGWL7TBG\nTT1Us5RPs5RP+zxV+AMW5+JPNAto2rnTdhgiImnhqfn4333iiTx98CCzLAfSC1QUFNDZ1UVRUZHl\naEREksuJ+fj9nZ2eOOMvBk4qLWV3ln+RS0QkGU8V/o7ubk9c3AWoKSzM2lk61UM1S/k0S/m0z0Th\nvxDYCjQCq5O8firwMtANfHmoHRXn5+OVxsqsnh6N5ReRnOR28vsC4B7gg0Ab8BqwFkgcEtMOfBG4\ndLidVRUXQzjsMiQzarq7s3Zkj8ZJm6V8mqV82uf2jH8J0AS0AH3Ao8DSAdu8BayPvT6kKss3YUlU\nA1k/PbOISDJuC381sCdhvTX2s1GxPRd/ohqgOUvn61EP1Szl0yzl0z63p9hGJ89vCQa5Ofa8CpgP\n1MXW18UeM7W+F9je1kY0GiUvL+/YhzX+Z6rWta51rWd6Pf68paUFN9yeYp8F3IxzgRdgDRABbk2y\n7deBTuCOQfYVXT5+PI90droMyZzJZWW80dTE1KlTbYciInIcW+P41wOzAR/O8PdlOBd3kxk2uCqP\nXNiNqykuztohnSIig3Fb+MPAKuBpoAF4DGdEz8rYAjAF5zrAl4AbgN3A+GQ7sz0X/0CzwuGsLPyJ\nfxaKe8qnWcqnfSaG0TwZWxLdn/B8PzA9lR1VRSIGwjGnJhSiads222GIiBjlqW/uemG6hkQ10SjN\nmzbZDmPE4heExAzl0yzl0z5PFX6vTNcQVwM0NzbaDkNExChPFX6vnfHPAppbW22HMWLqoZqlfJql\nfNqnwj+ESUBfOMyRI0dshyIiYox3vioL0QbgNNtRDLBgwgQeeO45Fi9ebDsUEZF3yIn5+L12xg+x\nC7xZOKRTRGQwKvzDmBUK0Zxl0zOrh2qW8mmW8mmfpwp/qe0Akqjp76dp40bbYYiIGOOpHr/RGd8M\neR74v2ecwZ/r622HIiLyDqPt8XtnAnyPqgGa9+wZdrts1tnZSUdHR9Jl1qxZnHPOOfEPmIjkAC/9\n3+zJM/5+YHxhIYc7OigrK7MdTkrWrVuX0rcjOzs7+fynP80v166lqqiICYWFTMjPZwIwIRqlor+f\nV4HSyZNZtXo1n7z8csrLy9Mdvuekmk9JjfJpTk6M6vGiAmBGWRk7svSmLIPZvHkzZ9bWUvzEE7SH\nw+zr6mLb0aO8FgjwXCDAbzo6eDgYpCEY5LYdO3j82ms5+cQTue7qq9m5c6ft8EXEBZ3xp+AjEyaw\n8qc/5ZJLLrEdihH/78EHuf7qq7k9FOLTI3jfDuCHRUX8pKCAc84+m5tvv50FCxakK0wRGYbO+NNo\nVnd31g3pTCYYDLJi2TL+/eqrWTfCog/wbuD2vj52dXfzT88/z4XnnMP111xDKBRKR7gikiYq/Cmo\n6e2luaHBdhgpSzZOuqGhgSVz5hBZu5bXQiHmuNj/OOB/A5u6umh94AHmvvvdPPfccy726G0ad26W\n8mmfCn8KaoCmN9+0Hcao1dfXc+6SJVy7ezcPdXczztB+TwQe6eri7gMH+Mwll3DFsmW0t7cb2ruI\npIt6/CnYClw8eTKN+/fbDmXE+vr6eO/pp/PF7du5Io3HOQp8rbiY/yor485772XZZZdpCKhImqnH\nn0YzgT3t7fT19dkOZcRu//a3+YfWVlak+TgVwN29vfwmEOCWK6/kI3V1Gv0j4lEq/CkoAWaWldGQ\nJX3+eA91y5YtfPc73+FHoVDG/rQ7C/h7KMT7X3qJxXPmcNu//VtW/sJMpJ60WcqnfSYK/4U43ZBG\nYPUg29wde70eyMrxf4siEV5//XXbYaSsv7+fz152GTd3dzMjw8cuBtb09/NqVxfPfutbLD71VF55\n5ZUMRyEig3Fb+AuAe3CKfy2wnOOn1P8wzs2sZgOfA+51eUwrFgWDvP7SS7bDSEldXR333HUXhc3N\nfN7ilZMa4OlQiK/s2MGl553HqiuvJBAIWItntPQtU7OUT/vcdgD+G/B1nMIP8JXY43cStrkP+BPw\nWGx9K3AucGDAvjx7cRfgBeD6U0/lb1u22A5lWDt27GDJ3Lm8HAox23YwMUeA1SUl/L60lC9/9at8\n7vOfp6KiwnZYo9LT00N7ezuHDh16x9LR0UFfXx/hvj76enoI9/Y6j7FWV0lZGSXl5c5jaSklJSWU\nlJRQXl5ORUXFoEthoabUkuRsTdJWDSTOYNYKvDeFbaZxfOHnZpfBpNNR4JWtW7nhhhs8/T9iNBrl\nG9/4BuXAz2wHM8DUnh7O7OnhX1ev5l9XO13Bq666ismTJ1uO7J0ikQitra00NTXR1NTEvn37RryP\nIpz/ueKPUaAntvSbDDZBfn4+48aNo7y8nJKSEoqLi489xp8XFRWRn59PXl4e+fn5x5a8vDzy8vKI\nRCLvWKLRKJFIhP7+fsLh8KCPgy39/f3v2DYSiaTpX59e8fzEnyc+Dnw+mOiAU9vE9ZG8ZoLbCpZq\nRAOzkvR9v503j6oq53YspaWlTJkyBZ/PB0BLSwuAtfX2lhZ46CEavvUtzgBaYjH7Yo9eWX8m9vix\nyy+npaDAM/mLry/w+VgAvPTSSzz77LPcd999AHwQ5wxhpP9et+sTgI3AeiBIcoWFhVRXV1NdXc2Z\nZ55JZWUlu3btSvrvM70+Y8YMQqEQ27ZtIxgMUl5eTkdHB3v37qW7u5toNEowGKSzs5NwOMzRo0c5\nevToIP8S78jPz6egoIDCwsJjv3Dy8/MpLi4mLy+PcDhMXl4excXFgDMsOXG9p6cHgKKiIgB6e3uJ\nRqMUFRURjUbp7e0FoKCggGg0emyAQX5+PtFolHA4TDQaJS8v79gvt3iBTVZoo9FoWgqwLW4Lfxsw\nPWF9Os4Z/VDbTIv97DgbNmxwGU56hf1+Lnj88bSOh3ejFfhDWRmbXn2V008/3XY4KTl48CDf/+53\nue8HP2AccGJnJx/AuWBkcshZGNiOU+Q3FhRwaNw46sNhAv39zJ01i/+1ZAlnLFnCvHnzmDt3LuPH\njzd4dJH0GO13Zdz2+AuBbcAFwF7gVZwLvImN8A8Dq2KPZwF3xR4Hinr9N+qdd95J85o13BM72/Ca\nS8rLWfSlL/H1W26xHcqIHT16lN/85jf8+ckn+fOf/oQ/EOD9RUV84OhRzgXOYPizlC6cM4rW2OMe\nYGt5ORsLC9kaCjFt0iTmnn46Z5x9NnPnzWPevHnMnDmT/HyNapbsNNoev4nh3RfhFPMC4EHg28DK\n2Gv3xx7jI3+CwBXA35Psx/OF/4UXXmD1JZfwsgdHpvwF+PTkyWzZvZu//vWvWT9yoq2tjRdeeIEX\nnn6aPz/3HFvb2ijOz6e8sJDyggLK8vMpz8+nPC+PUDRKW28vR/v6qD7hBKZNmcK0GTOonjWL98yZ\nwxlnnMGcOXMYN250k1Vo/nizlE9zbN6B68nYkuj+AeurDBzHugULFrApFCKM925dduu4cVx3003H\neqDZrrq6muXLl7N8+XLA6bF2d3fT1dVFKBQiFArR1dVFMBiktLSUadOmMWnSJJ29i6TAS5OpeP6M\nH+DU6mr+a+9e5toOJMFm4B8rK9m5fz+lpV68Zb2IpIPm6smQRYsX47Xv795WVsY1112noi8iKVHh\nH6FF557L6yUltsM4Zhfwh7w8Pr/q7W6a5kIxS/k0S/m0T4V/hBYtWsTrHjqzvqOkhH9ZuZLKykrb\noYhIllCPf4Q6OjqYOmkS/r4+6xd43wLeU1rKmzt2cNJJJ1mORkQyTT3+DJkwYQLVkybhhRl7vl9Y\nyCc+8QkVfREZERX+UfDCBd6jwL2FhVx3003HvaYeqlnKp1nKp30q/KOw6Nxzed3yePkH8vM5/4IL\nmDVrltU4RCT7qMc/CuvWreOrS5fy144OK8fvBWrKy3n8xRdZuHChlRhExD71+DNowYIFbOzqImzp\n+D8DaufPV9EXkVFR4R+FyspKpk6cyFYLx47gTM+w+pvfHHQb9VDNUj7NUj7tU+EfpUWLFlm5wPs4\nMGH6dM477zwLRxeRXKDCP0o2LvBGgVvHj2f1LbcMOQ+3Zj40S/k0S/m0T4V/lBYtXpzxb/C+DLw1\nbhyXXnppRo8rIrlFhX+UFi5cSH1XV9run5rMneXlXLNmDQUFBUNupx6qWcqnWcqnfSr8o5TpC7wt\nwPPAFZ/5TIaOKCK5SoXfhUxe4P1+cTGfufJKKioqht1WPVSzlE+zlE/7VPhdyNQF3qPAT/Lz+eKX\nv5z2Y4lI7lPhd2HR4sW8XlaW9uP8OC+PD15wASeffHJK26uHapbyaZbyaZ+bwn8C8AywHfgjUDXI\ndj8GDgCbXBzLkxYuXEh9KJTWC7z9wPfKyvjSDTek8SgiMpa4mavnNuBQ7HE18C7gK0m2ez/QCTwM\nQ96qNmvm6kk0+6STeHz/fmrTtP9fA/8+Zw4vb96cpiOISLayMVfPJcBDsecPAYMNLn8ROOLiOJ62\naOFC1qdx/3eOH8+Xkky9LCIyWm4K/2ScFg6xx8nuw8k+6bzAux7YXVrKxz72sRG9Tz1Us5RPs5RP\n+4a7e+AzwJQkP//agPVobHFlxYoV+Hw+AKqqqpg/f/6xoV/xD4vX1hctXsxvS0tZ19vrvB77t6yL\nPbpZv6WkhC9edx2FhYWe+fdqXetat7cef97S0oIbbnr8W3Hq1H7gJOBPwKmDbOsDfkcO9viDwSAz\nJk/mtWCQmQb32wbMLStjx969VFUNdt1cRMYyGz3+tcCnY88/DfzWxb6y1rhx47jis5/lnmKz7Z57\nioq4/FOfUtEXEePcFP7vAP+IM5zz/Ng6wFTgDwnb/Rz4K3AKsAe4wsUxPWnVtdfyk/x8Og3tLwj8\nR0EB11wTE3ECAAAFKUlEQVR//ajen/hnobinfJqlfNo3XI9/KIeBDyb5+V7gIwnry10cIyvMmDGD\n8849l4f++Ee+YKBd9TBwzjnnUFNT4z44EZEBdM9dQ1588UU+e9FFbA0GXf0ZFQBqy8r41fPPc9ZZ\nZ5kKT0RykO65a9n73vc+xp90Ek+53M+NJSVc9M//rKIvImmjwm9IXl4e/+fGG7lr/PhR7+N14Bcl\nJdx6992uYlEP1Szl0yzl0z4VfoOWLVvGpoICGkbx3n7gqnHj+PaddzJx4kTToYmIHKMev2E333AD\n+++4g/u6u0f0vh/m5fHo/PmsW7+e/Hz9PhaR4Y22x6/Cb9j+/fs5beZMmru7OSHV9+B8WWvda68x\nZ86cdIYnIjlEF3c9YsqUKVzy0Y/ywAjO2r9cVsZnr7rKWNFXD9Us5dMs5dM+Ff40uGbNGu4pLaUv\nhW2fBV4aP54bv/nNdIclIgKo1ZM2H1iwgFUbNvA/htimGzijvJw7Hn2Uiy++OFOhiUiOUKvHY665\n4Qa+N8zQztsKC6k9+2wVfRHJKBX+NFm6dCltpaW8lvCzPuAt3r5X5d3Fxdz94IPGj60eqlnKp1nK\np31u5uqRIRQWFvLF66/n4ptuoqSggCN9fYTCYarKynjX+PGcUFXFHdddl/IN1EVETFGPP40ikQib\nN2+moqKCE044gYqKCo3RFxFjNI5fRGSM0cVdOUY9VLOUT7OUT/tU+EVExhi1ekREspRaPSIikhI3\nhf8E4BneHpae7K7g04E/AW8Cm4GrXRxPUqQeqlnKp1nKp31uCv9XcAr/KcBzsfWB+oAvAXOAs4Av\nAKe5OKakYMOGDbZDyCnKp1nKp31uCv8lwEOx5w8BlybZZj8Q/6/cCWwBpro4pqTA7/fbDiGnKJ9m\nKZ/2uSn8k4EDsecHYutD8QELgFdcHFNERFwabsqGZ4ApSX7+tQHr0dgymPHAL4FrcM78JY1aWlps\nh5BTlE+zlE/73Azn3ArU4bRzTsK5iHtqku2KgN8DTwJ3DbG/JqDGRTwiImNNMzArkwe8DVgde/4V\n4DtJtskDHgbuzFRQIiKSPifg3EBq4HDOqcAfYs/fB0RwLvC+EVsuzGyYIiIiIiKSMRfiXBto5O02\n0UB3x16vxxkFJIMbLp91QIC3/9q6IWORZZ8f44xO2zTENvpspm64fNahz2aqUv0irCc/nwU4F3B9\nOBd8N3D8l7k+DDwRe/5e4G+ZCi4LpZLPOmBtRqPKXu/H+Z9lsEKlz+bIDJfPOvTZTNUUYH7s+Xhg\nGy5rZybn6lmCU6hacL7R+yiwdMA2iV8KewXnusFw3w8Yq1LJJ3hrIj4vexE4MsTr+myOzHD5BH02\nU5XKF2FH9PnMZOGvBvYkrLfGfjbcNtPSHFe2SiWfUeBsnD/9ngBqMxNaTtJn0yx9NkfHR/Ivwo7o\n85nJe+6mOufywLMAzdWcXCp5+TtOfzAEXAT8FmduJRkdfTbN0Wdz5Ib7ImzKn89MnvG34fyHjpuO\n81tpqG2mxX4mx0sln0dx/scC5wt0RTjDcGXk9Nk0S5/NkSkCfgX8J84vyYE8+/ksxPmWmQ8oZviL\nu2ehC2hDSSWfk3n7LGAJzvUAGZyP1C7u6rOZGh+D51OfzdSl8kVYT38+L8K5It0ErIn9bGVsibsn\n9no9sDCj0WWf4fL5BZzhXxuAv+J8ICS5nwN7gV6cXuln0GfTjeHyqc9m6pJ9EfYi9PkUERERERER\nEREREREREREREREREREREREREfGe/w/Hhfue7vs2CgAAAABJRU5ErkJggg==\n",
"text": [
"![](\"appfwk-arch.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas\n",
"======\n",
"\n",
"Primary object types:\n",
"\n",
"* **Series** - 1 dimensional array of element\n",
"* **DataFrames** - 2 dimensional array of elements\n",
"\n",
"Data is stored is compact binary form for efficiency of operation.\n",
"\n",
"Supports both row and column indexing by name."
]
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Series\n",
"==============\n",
"\n",
"A **Series** is similar to a standard Python **list**, however much more efficient in memory and computation."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import pandas, numpy\n",
"s = pandas.Series([10, 23, 19, 15, 56, 15, 41])\n",
"print type(s)"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 12,
"text": [
" location ip pkts bytes rtt\n",
"0 Boston 10.1.1.1 10 2356 0.100\n",
"1 Boston 10.1.1.2 23 16600 0.112\n",
"2 Boston 10.1.1.15 19 22600 0.085\n",
"3 SanFran 10.38.5.1 15 10550 0.030\n",
"4 SanFran 10.38.8.2 56 35000 0.020\n",
"5 London 192.168.4.6 15 3400 0.130\n",
"6 London 192.168.5.72 41 55000 0.120\n",
"\n",
"[7 rows x 5 columns]"
]
}
],
"prompt_number": 12
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: DataFrame operations\n",
"============================\n",
"The DataFrame supports a number of operations directly"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df.mean()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 13,
"text": [
"pkts 25.571429\n",
"bytes 20786.571429\n",
"rtt 0.085286\n",
"dtype: float64"
]
}
],
"prompt_number": 13
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"source": [
"Notice that `location` and `ip` are string columns, thus there is no operation 'mean' for such columns."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df.sum()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 14,
"text": [
"location BostonBostonBostonSanFranSanFranLondonLondon\n",
"ip 10.1.1.110.1.1.210.1.1.1510.38.5.110.38.8.2192...\n",
"pkts 179\n",
"bytes 145506\n",
"rtt 0.597\n",
"dtype: object"
]
}
],
"prompt_number": 14
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"In the case of sum(), it is possible to add strings, so `location` and `ip` are included in the results.\n",
"*(Be careful, on large data sets this can backfire!)*"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Selecting a Column\n",
"==========================\n",
"Each column of data in a **DataFrame** can be extracted by the standard indexing operator:\n",
"\n",
" df[_colname_]\n",
" \n",
"The result is a **Series**."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df['location']"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 15,
"text": [
"0 Boston\n",
"1 Boston\n",
"2 Boston\n",
"3 SanFran\n",
"4 SanFran\n",
"5 London\n",
"6 London\n",
"Name: location, dtype: object"
]
}
],
"prompt_number": 15
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df['pkts']"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 16,
"text": [
"0 10\n",
"1 23\n",
"2 19\n",
"3 15\n",
"4 56\n",
"5 15\n",
"6 41\n",
"Name: pkts, dtype: int64"
]
}
],
"prompt_number": 16
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Selecting Rows\n",
"======================\n",
"Selecting a subset of rows based by index works just like array slicing:\n",
"\n",
" df[_start_:_end_]\n",
" \n",
"Note that the `:` is always required, but `| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "
| 1 | \n", "Boston | \n", "10.1.1.2 | \n", "23 | \n", "16600 | \n", "0.112 | \n", "
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "
| 3 | \n", "SanFran | \n", "10.38.5.1 | \n", "15 | \n", "10550 | \n", "0.030 | \n", "
| 4 | \n", "SanFran | \n", "10.38.8.2 | \n", "56 | \n", "35000 | \n", "0.020 | \n", "
| 5 | \n", "London | \n", "192.168.4.6 | \n", "15 | \n", "3400 | \n", "0.130 | \n", "
| 6 | \n", "London | \n", "192.168.5.72 | \n", "41 | \n", "55000 | \n", "0.120 | \n", "
7 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 17,
"text": [
" location ip pkts bytes rtt\n",
"0 Boston 10.1.1.1 10 2356 0.100\n",
"1 Boston 10.1.1.2 23 16600 0.112\n",
"\n",
"[2 rows x 5 columns]"
]
}
],
"prompt_number": 17
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df[2:4]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "
| 1 | \n", "Boston | \n", "10.1.1.2 | \n", "23 | \n", "16600 | \n", "0.112 | \n", "
2 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 18,
"text": [
" location ip pkts bytes rtt\n",
"2 Boston 10.1.1.15 19 22600 0.085\n",
"3 SanFran 10.38.5.1 15 10550 0.030\n",
"\n",
"[2 rows x 5 columns]"
]
}
],
"prompt_number": 18
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Filtering Rows\n",
"======================\n",
"\n",
"DataFrame rows can be filtered using boolean expressions\n",
"\n",
" df[_boolean_expression_]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df[df['location'] == 'Boston']"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "
| 3 | \n", "SanFran | \n", "10.38.5.1 | \n", "15 | \n", "10550 | \n", "0.030 | \n", "
2 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 19,
"text": [
" location ip pkts bytes rtt\n",
"0 Boston 10.1.1.1 10 2356 0.100\n",
"1 Boston 10.1.1.2 23 16600 0.112\n",
"2 Boston 10.1.1.15 19 22600 0.085\n",
"\n",
"[3 rows x 5 columns]"
]
}
],
"prompt_number": 19
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df[df['pkts'] < 20]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "subslide_end",
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "
| 1 | \n", "Boston | \n", "10.1.1.2 | \n", "23 | \n", "16600 | \n", "0.112 | \n", "
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "
3 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 20,
"text": [
" location ip pkts bytes rtt\n",
"0 Boston 10.1.1.1 10 2356 0.100\n",
"2 Boston 10.1.1.15 19 22600 0.085\n",
"3 SanFran 10.38.5.1 15 10550 0.030\n",
"5 London 192.168.4.6 15 3400 0.130\n",
"\n",
"[4 rows x 5 columns]"
]
}
],
"prompt_number": 20
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Expressions can be combined to provide more complex filtering:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df[(df['location'] == 'Boston') & (df['pkts'] < 20)]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "subslide_end",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "
| 3 | \n", "SanFran | \n", "10.38.5.1 | \n", "15 | \n", "10550 | \n", "0.030 | \n", "
| 5 | \n", "London | \n", "192.168.4.6 | \n", "15 | \n", "3400 | \n", "0.130 | \n", "
4 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 21,
"text": [
" location ip pkts bytes rtt\n",
"0 Boston 10.1.1.1 10 2356 0.100\n",
"2 Boston 10.1.1.15 19 22600 0.085\n",
"\n",
"[2 rows x 5 columns]"
]
}
],
"prompt_number": 21
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The boolean expression is actually a **Series** of True/False values of the same length as the number of rows, thus can actually be assigned as a series:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"bos = df['location'] == 'Boston'\n",
"print type(bos)\n",
"bos"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "
2 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 23,
"text": [
" location ip pkts bytes rtt\n",
"0 Boston 10.1.1.1 10 2356 0.100\n",
"2 Boston 10.1.1.15 19 22600 0.085\n",
"\n",
"[2 rows x 5 columns]"
]
}
],
"prompt_number": 23
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Use single \"`&`\" and \"`|`\" and parenthesis for constructing arbitrary boolean expressions. Use \"`~`\" for negation.\n",
"\n",
"The result of a filtering operation is a new **DataFrame** object"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df2 = df[bos & pkts_lt_20]\n",
"print 'Length:', len(df2)\n",
"df2"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Length: 2\n"
]
},
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "
2 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 24,
"text": [
" location ip pkts bytes rtt\n",
"0 Boston 10.1.1.1 10 2356 0.100\n",
"2 Boston 10.1.1.15 19 22600 0.085\n",
"\n",
"[2 rows x 5 columns]"
]
}
],
"prompt_number": 24
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Selecting Multiple Columns\n",
"==================================\n",
"\n",
"A new **DataFrame** can be constructed from a subset of another DataFrame using the ``ix[]`` operator.\n",
"\n",
" DataFrame.ix[_row_expr_,_column_lise_]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"locrtt = df.ix[:,['location', 'rtt']]\n",
"locrtt"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "subslide_end",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "
2 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 25,
"text": [
" location rtt\n",
"0 Boston 0.100\n",
"1 Boston 0.112\n",
"2 Boston 0.085\n",
"3 SanFran 0.030\n",
"4 SanFran 0.020\n",
"5 London 0.130\n",
"6 London 0.120\n",
"\n",
"[7 rows x 2 columns]"
]
}
],
"prompt_number": 25
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The first argument to the `ix[]` indexer is actually expected to be a boolean Series. This makes it possible to select rows and columns in a single operation. The use of \":\" above is short-hand for *all* rows."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"boslocrtt = df.ix[bos,['location', 'rtt']]\n",
"boslocrtt"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "rtt | \n", "
|---|---|---|
| 0 | \n", "Boston | \n", "0.100 | \n", "
| 1 | \n", "Boston | \n", "0.112 | \n", "
| 2 | \n", "Boston | \n", "0.085 | \n", "
| 3 | \n", "SanFran | \n", "0.030 | \n", "
| 4 | \n", "SanFran | \n", "0.020 | \n", "
| 5 | \n", "London | \n", "0.130 | \n", "
| 6 | \n", "London | \n", "0.120 | \n", "
7 rows \u00d7 2 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 26,
"text": [
" location rtt\n",
"0 Boston 0.100\n",
"1 Boston 0.112\n",
"2 Boston 0.085\n",
"\n",
"[3 rows x 2 columns]"
]
}
],
"prompt_number": 26
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Adding Columns\n",
"======================\n",
"\n",
"New columns (**Series**) be added to a **DataFrame** in one of three ways:\n",
"\n",
"* A constant value for all rows\n",
"* Supplying a `list` matching the number of rows\n",
"* Expression based on constants and **Series** objects of the same size"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df['co'] = 'RVBD'\n",
"df['proto'] = ['tcp', 'tcp', 'udp', 'udp', 'tcp', 'tcp', 'udp']\n",
"df['Bpp'] = df['bytes'] / df['pkts'] # Bytes per packet\n",
"df['bpp'] = 8 * df['bytes'] / df['pkts']\n",
"df['slow'] = df['rtt'] >= 0.1\n",
"df.ix[:,['ip', 'proto', 'Bpp', 'bpp', 'slow', 'delay', 'co']]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "rtt | \n", "
|---|---|---|
| 0 | \n", "Boston | \n", "0.100 | \n", "
| 1 | \n", "Boston | \n", "0.112 | \n", "
| 2 | \n", "Boston | \n", "0.085 | \n", "
3 rows \u00d7 2 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 27,
"text": [
" ip proto Bpp bpp slow delay co\n",
"0 10.1.1.1 tcp 235.600000 1884.800000 True NaN RVBD\n",
"1 10.1.1.2 tcp 721.739130 5773.913043 True NaN RVBD\n",
"2 10.1.1.15 udp 1189.473684 9515.789474 False NaN RVBD\n",
"3 10.38.5.1 udp 703.333333 5626.666667 False NaN RVBD\n",
"4 10.38.8.2 tcp 625.000000 5000.000000 False NaN RVBD\n",
"5 192.168.4.6 tcp 226.666667 1813.333333 True NaN RVBD\n",
"6 192.168.5.72 udp 1341.463415 10731.707317 True NaN RVBD\n",
"\n",
"[7 rows x 7 columns]"
]
}
],
"prompt_number": 27
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Grouping and Aggregating\n",
"================================\n",
"\n",
"A common operation is to group data by a key column or set of of key columns. The groups rows that share the same value for all key columns producing one row per unique key pair:\n",
"\n",
" df.groupby(_key_columns_]\n",
" \n",
"For example, to group by the `location` column:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"gb = df.groupby('location')\n",
"print type(gb)\n",
"gb.indices"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"| \n", " | ip | \n", "proto | \n", "Bpp | \n", "bpp | \n", "slow | \n", "delay | \n", "co | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "10.1.1.1 | \n", "tcp | \n", "235.600000 | \n", "1884.800000 | \n", "True | \n", "NaN | \n", "RVBD | \n", "
| 1 | \n", "10.1.1.2 | \n", "tcp | \n", "721.739130 | \n", "5773.913043 | \n", "True | \n", "NaN | \n", "RVBD | \n", "
| 2 | \n", "10.1.1.15 | \n", "udp | \n", "1189.473684 | \n", "9515.789474 | \n", "False | \n", "NaN | \n", "RVBD | \n", "
| 3 | \n", "10.38.5.1 | \n", "udp | \n", "703.333333 | \n", "5626.666667 | \n", "False | \n", "NaN | \n", "RVBD | \n", "
| 4 | \n", "10.38.8.2 | \n", "tcp | \n", "625.000000 | \n", "5000.000000 | \n", "False | \n", "NaN | \n", "RVBD | \n", "
| 5 | \n", "192.168.4.6 | \n", "tcp | \n", "226.666667 | \n", "1813.333333 | \n", "True | \n", "NaN | \n", "RVBD | \n", "
| 6 | \n", "192.168.5.72 | \n", "udp | \n", "1341.463415 | \n", "10731.707317 | \n", "True | \n", "NaN | \n", "RVBD | \n", "
7 rows \u00d7 7 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 29,
"text": [
" pkts rtt\n",
"location \n",
"Boston 52 0.099\n",
"London 56 0.125\n",
"SanFran 71 0.025\n",
"\n",
"[3 rows x 2 columns]"
]
}
],
"prompt_number": 29
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"By copying columns, it is possible to compute alternate operations on the same data, producing different aggregated results:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df2 = df.ix[:,['location', 'pkts', 'rtt']]\n",
"df2['peak_rtt'] = df['rtt']\n",
"df2['min_rtt'] = df['rtt']\n",
"df2"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | pkts | \n", "rtt | \n", "
|---|---|---|
| location | \n", "\n", " | \n", " |
| Boston | \n", "52 | \n", "0.099 | \n", "
| London | \n", "56 | \n", "0.125 | \n", "
| SanFran | \n", "71 | \n", "0.025 | \n", "
3 rows \u00d7 2 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 30,
"text": [
" location pkts rtt peak_rtt min_rtt\n",
"0 Boston 10 0.100 0.100 0.100\n",
"1 Boston 23 0.112 0.112 0.112\n",
"2 Boston 19 0.085 0.085 0.085\n",
"3 SanFran 15 0.030 0.030 0.030\n",
"4 SanFran 56 0.020 0.020 0.020\n",
"5 London 15 0.130 0.130 0.130\n",
"6 London 41 0.120 0.120 0.120\n",
"\n",
"[7 rows x 5 columns]"
]
}
],
"prompt_number": 30
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"agg = (df2.groupby('location')\n",
" .aggregate({'pkts': 'sum',\n",
" 'rtt': 'mean',\n",
" 'peak_rtt': 'max',\n",
" 'min_rtt': 'min'}))\n",
"agg"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "pkts | \n", "rtt | \n", "peak_rtt | \n", "min_rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10 | \n", "0.100 | \n", "0.100 | \n", "0.100 | \n", "
| 1 | \n", "Boston | \n", "23 | \n", "0.112 | \n", "0.112 | \n", "0.112 | \n", "
| 2 | \n", "Boston | \n", "19 | \n", "0.085 | \n", "0.085 | \n", "0.085 | \n", "
| 3 | \n", "SanFran | \n", "15 | \n", "0.030 | \n", "0.030 | \n", "0.030 | \n", "
| 4 | \n", "SanFran | \n", "56 | \n", "0.020 | \n", "0.020 | \n", "0.020 | \n", "
| 5 | \n", "London | \n", "15 | \n", "0.130 | \n", "0.130 | \n", "0.130 | \n", "
| 6 | \n", "London | \n", "41 | \n", "0.120 | \n", "0.120 | \n", "0.120 | \n", "
7 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 31,
"text": [
" pkts rtt min_rtt peak_rtt\n",
"location \n",
"Boston 52 0.099 0.085 0.112\n",
"London 56 0.125 0.120 0.130\n",
"SanFran 71 0.025 0.020 0.030\n",
"\n",
"[3 rows x 4 columns]"
]
}
],
"prompt_number": 31
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Indexing\n",
"================\n",
"\n",
"Notice that the output of the preview groupby/aggregate operation looked a bit different from previous data frames, the first column is in bold:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"agg"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | pkts | \n", "rtt | \n", "min_rtt | \n", "peak_rtt | \n", "
|---|---|---|---|---|
| location | \n", "\n", " | \n", " | \n", " | \n", " |
| Boston | \n", "52 | \n", "0.099 | \n", "0.085 | \n", "0.112 | \n", "
| London | \n", "56 | \n", "0.125 | \n", "0.120 | \n", "0.130 | \n", "
| SanFran | \n", "71 | \n", "0.025 | \n", "0.020 | \n", "0.030 | \n", "
3 rows \u00d7 4 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 32,
"text": [
" pkts rtt min_rtt peak_rtt\n",
"location \n",
"Boston 52 0.099 0.085 0.112\n",
"London 56 0.125 0.120 0.130\n",
"SanFran 71 0.025 0.020 0.030\n",
"\n",
"[3 rows x 4 columns]"
]
}
],
"prompt_number": 32
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"The bold rows/columns indicate that the **DataFrame** is *indexed*. "
]
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Let's look at the DataFrame without the index -- by calling `reset_index()`:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"agg.reset_index()"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | pkts | \n", "rtt | \n", "min_rtt | \n", "peak_rtt | \n", "
|---|---|---|---|---|
| location | \n", "\n", " | \n", " | \n", " | \n", " |
| Boston | \n", "52 | \n", "0.099 | \n", "0.085 | \n", "0.112 | \n", "
| London | \n", "56 | \n", "0.125 | \n", "0.120 | \n", "0.130 | \n", "
| SanFran | \n", "71 | \n", "0.025 | \n", "0.020 | \n", "0.030 | \n", "
3 rows \u00d7 4 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 33,
"text": [
" location pkts rtt min_rtt peak_rtt\n",
"0 Boston 52 0.099 0.085 0.112\n",
"1 London 56 0.125 0.120 0.130\n",
"2 SanFran 71 0.025 0.020 0.030\n",
"\n",
"[3 rows x 5 columns]"
]
}
],
"prompt_number": 33
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Indexing a **DataFrame** makes the data in the indexed column to meta-data associated with the object. This index is carried on to **Series** objects:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"agg['pkts']"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 34,
"text": [
"location\n",
"Boston 52\n",
"London 56\n",
"SanFran 71\n",
"Name: pkts, dtype: int64"
]
}
],
"prompt_number": 34
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The resulting series still acts like an array, but may be indexed by either a numeric row value or a location name:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print \"Boston:\", agg['pkts']['Boston']\n",
"print \"Item 0:\", agg['pkts'][0]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Boston: 52\n",
"Item 0: 52\n"
]
}
],
"prompt_number": 35
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df = df.ix[:,['location', 'ip', 'pkts', 'bytes', 'rtt', 'slow']]"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"prompt_number": 36
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Modifying a Subset of Rows\n",
"==================================\n",
"\n",
"Often it is useful to assign a subset of rows in a column. This is possible by assigning a value to the result of the `ix[]` indexer: \n",
"\n",
" DataFrame.ix[_row_expr_, _column_list_>] = | \n", " | location | \n", "pkts | \n", "rtt | \n", "min_rtt | \n", "peak_rtt | \n", "
|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "52 | \n", "0.099 | \n", "0.085 | \n", "0.112 | \n", "
| 1 | \n", "London | \n", "56 | \n", "0.125 | \n", "0.120 | \n", "0.130 | \n", "
| 2 | \n", "SanFran | \n", "71 | \n", "0.025 | \n", "0.020 | \n", "0.030 | \n", "
3 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 37,
"text": [
" location ip pkts bytes rtt slow delay\n",
"0 Boston 10.1.1.1 10 2356 0.100 True \n",
"1 Boston 10.1.1.2 23 16600 0.112 True \n",
"2 Boston 10.1.1.15 19 22600 0.085 False \n",
"3 SanFran 10.38.5.1 15 10550 0.030 False \n",
"4 SanFran 10.38.8.2 56 35000 0.020 False \n",
"5 London 192.168.4.6 15 3400 0.130 True \n",
"6 London 192.168.5.72 41 55000 0.120 True \n",
"\n",
"[7 rows x 7 columns]"
]
}
],
"prompt_number": 37
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Compute boolean **Series** for each range of rtt:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"slow_rows = (df['rtt'] > 0.110)\n",
"normal_rows = ((df['rtt'] > 0.050) & (~slow_rows))\n",
"fast_rows = ((~slow_rows) & (~normal_rows))"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"prompt_number": 38
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df.ix[slow_rows,'delay'] = 'slow'\n",
"df.ix[normal_rows, 'delay'] = 'normal'\n",
"df.ix[fast_rows, 'delay'] = 'fast'\n",
"df"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "slow | \n", "delay | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "True | \n", "\n", " |
| 1 | \n", "Boston | \n", "10.1.1.2 | \n", "23 | \n", "16600 | \n", "0.112 | \n", "True | \n", "\n", " |
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "False | \n", "\n", " |
| 3 | \n", "SanFran | \n", "10.38.5.1 | \n", "15 | \n", "10550 | \n", "0.030 | \n", "False | \n", "\n", " |
| 4 | \n", "SanFran | \n", "10.38.8.2 | \n", "56 | \n", "35000 | \n", "0.020 | \n", "False | \n", "\n", " |
| 5 | \n", "London | \n", "192.168.4.6 | \n", "15 | \n", "3400 | \n", "0.130 | \n", "True | \n", "\n", " |
| 6 | \n", "London | \n", "192.168.5.72 | \n", "41 | \n", "55000 | \n", "0.120 | \n", "True | \n", "\n", " |
7 rows \u00d7 7 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 39,
"text": [
" location ip pkts bytes rtt slow delay\n",
"0 Boston 10.1.1.1 10 2356 0.100 True normal\n",
"1 Boston 10.1.1.2 23 16600 0.112 True slow\n",
"2 Boston 10.1.1.15 19 22600 0.085 False normal\n",
"3 SanFran 10.38.5.1 15 10550 0.030 False fast\n",
"4 SanFran 10.38.8.2 56 35000 0.020 False fast\n",
"5 London 192.168.4.6 15 3400 0.130 True slow\n",
"6 London 192.168.5.72 41 55000 0.120 True slow\n",
"\n",
"[7 rows x 7 columns]"
]
}
],
"prompt_number": 39
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"pandas: Unstacking Data\n",
"=======================\n",
"\n",
"Unstacking data is about pivoting data based on row values. For example, let's say we want to compute the total bytes for each delay category by location:\n",
"\n",
" slow normal fast\n",
" Boston ? ? ?\n",
" SanFran ? ? ?\n",
" London ? ? ?\n",
"\n",
"Where the `?` in each cell is the total bytes for that combination.\n",
"\n",
"First we groupby both `location` and `delay`, aggregating the `bytes` column:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"loc_delay_bytes = (df.groupby(['location', 'delay'])\n",
" .aggregate({'bytes': 'sum'}))\n",
"loc_delay_bytes"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | location | \n", "ip | \n", "pkts | \n", "bytes | \n", "rtt | \n", "slow | \n", "delay | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "Boston | \n", "10.1.1.1 | \n", "10 | \n", "2356 | \n", "0.100 | \n", "True | \n", "normal | \n", "
| 1 | \n", "Boston | \n", "10.1.1.2 | \n", "23 | \n", "16600 | \n", "0.112 | \n", "True | \n", "slow | \n", "
| 2 | \n", "Boston | \n", "10.1.1.15 | \n", "19 | \n", "22600 | \n", "0.085 | \n", "False | \n", "normal | \n", "
| 3 | \n", "SanFran | \n", "10.38.5.1 | \n", "15 | \n", "10550 | \n", "0.030 | \n", "False | \n", "fast | \n", "
| 4 | \n", "SanFran | \n", "10.38.8.2 | \n", "56 | \n", "35000 | \n", "0.020 | \n", "False | \n", "fast | \n", "
| 5 | \n", "London | \n", "192.168.4.6 | \n", "15 | \n", "3400 | \n", "0.130 | \n", "True | \n", "slow | \n", "
| 6 | \n", "London | \n", "192.168.5.72 | \n", "41 | \n", "55000 | \n", "0.120 | \n", "True | \n", "slow | \n", "
7 rows \u00d7 7 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 40,
"text": [
" bytes\n",
"location delay \n",
"Boston normal 24956\n",
" slow 16600\n",
"London slow 58400\n",
"SanFran fast 45550\n",
"\n",
"[4 rows x 1 columns]"
]
}
],
"prompt_number": 40
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "subslide_end",
"slideshow": {
"slide_type": "-"
}
},
"source": [
"The resulting **DataFrame** has two index columns, `location` and `delay`. "
]
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Now the `unstack()` function is used to create a column for each unique value of `delay`:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"loc_delay_bytes = loc_delay_bytes.unstack('delay')"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"prompt_number": 41
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"source": [
"Let's fill in zeros for the combinations that were not present:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"loc_delay_bytes = loc_delay_bytes.fillna(0)\n",
"loc_delay_bytes"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_helper": "subslide_end"
},
"slide_helper": "slide_end",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | \n", " | bytes | \n", "
|---|---|---|
| location | \n", "delay | \n", "\n", " |
| Boston | \n", "normal | \n", "24956 | \n", "
| slow | \n", "16600 | \n", "|
| London | \n", "slow | \n", "58400 | \n", "
| SanFran | \n", "fast | \n", "45550 | \n", "
4 rows \u00d7 1 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 42,
"text": [
" bytes \n",
"delay fast normal slow\n",
"location \n",
"Boston 0 24956 16600\n",
"London 0 0 58400\n",
"SanFran 45550 0 0\n",
"\n",
"[3 rows x 3 columns]"
]
}
],
"prompt_number": 42
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6,
"slide_type": "subslide"
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"SteelScript for Python\n",
"======================\n",
"\n",
"SteelScript is a collection of **Python packages** that provide libraries for collecting, processing, and visualizing data from a variety of sources. Most packages focus on network performance.\n",
"\n",
"Getting SteelScript and the Wireshark extension\n",
"-----------------------------------------------\n",
"\n",
"Core SteelScript is available on PyPI:\n",
"\n",
" > pip install steelscript\n",
" > pip install steelscript.wireshark\n",
" \n",
"The bleeding edge is available on github:\n",
"\n",
"* http://github.com/riverbed/steelscript/steelscript\n",
"* http://github.com/riverbed/steelscript/steelscript.wireshark\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"SteelScript: Reading Packet Capture Files\n",
"========================================="
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from steelscript.wireshark.core.pcap import PcapFile\n",
"\n",
"pcap = PcapFile('/ws/traces/net-2009-11-18-17_35.pcap')\n",
"pcap.info()\n",
"print pcap.starttime\n",
"print pcap.endtime\n",
"print pcap.numpackets"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"2009-11-18 17:35:54-08:00\n",
"2009-11-19 09:47:05-08:00\n",
"302030\n"
]
}
],
"prompt_number": 43
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Perform a query on the PCAP file"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pdf = pcap.query(['frame.time_epoch', 'ip.src', 'ip.dst', 'ip.len', 'ip.proto'], \n",
" starttime = pcap.starttime, \n",
" duration='1min',\n",
" as_dataframe=True)\n",
"pdf = pdf[~(pdf['ip.len'].isnull())]\n",
"print len(pdf), \"packets loaded\""
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"5529 packets loaded\n"
]
}
],
"prompt_number": 44
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pdf[:10]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"html": [
"| \n", " | bytes | \n", "||
|---|---|---|---|
| delay | \n", "fast | \n", "normal | \n", "slow | \n", "
| location | \n", "\n", " | \n", " | \n", " |
| Boston | \n", "0 | \n", "24956 | \n", "16600 | \n", "
| London | \n", "0 | \n", "0 | \n", "58400 | \n", "
| SanFran | \n", "45550 | \n", "0 | \n", "0 | \n", "
3 rows \u00d7 3 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 45,
"text": [
" frame.time_epoch ip.src ip.dst ip.len \\\n",
"0 2009-11-18 17:35:54.369201-08:00 137.226.34.227 192.168.1.105 1420 \n",
"1 2009-11-18 17:35:54.369434-08:00 137.226.34.227 192.168.1.105 1420 \n",
"2 2009-11-18 17:35:54.369655-08:00 192.168.1.105 137.226.34.227 40 \n",
"3 2009-11-18 17:35:54.369702-08:00 137.226.34.227 192.168.1.105 1420 \n",
"4 2009-11-18 17:35:54.369713-08:00 137.226.34.227 192.168.1.105 1420 \n",
"5 2009-11-18 17:35:54.369723-08:00 137.226.34.227 192.168.1.105 1332 \n",
"6 2009-11-18 17:35:54.369733-08:00 137.226.34.227 192.168.1.105 1420 \n",
"7 2009-11-18 17:35:54.369896-08:00 192.168.1.105 137.226.34.227 40 \n",
"8 2009-11-18 17:35:54.369947-08:00 137.226.34.227 192.168.1.105 1420 \n",
"9 2009-11-18 17:35:54.369986-08:00 137.226.34.227 192.168.1.105 1420 \n",
"\n",
" ip.proto \n",
"0 6 \n",
"1 6 \n",
"2 6 \n",
"3 6 \n",
"4 6 \n",
"5 6 \n",
"6 6 \n",
"7 6 \n",
"8 6 \n",
"9 6 \n",
"\n",
"[10 rows x 5 columns]"
]
}
],
"prompt_number": 45
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Examine some characteristics of the data:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pdf['ip.proto'].unique()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 46,
"text": [
"array([ 6., 17.])"
]
}
],
"prompt_number": 46
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"tcpdf = pdf[pdf['ip.proto'] == 6]\n",
"len(tcpdf)"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 47,
"text": [
"5527"
]
}
],
"prompt_number": 47
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Unique source IPs, or dest IPs:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pdf['ip.src'].unique()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 48,
"text": [
"array(['137.226.34.227', '192.168.1.105', '192.168.1.1'], dtype=object)"
]
}
],
"prompt_number": 48
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pdf['ip.dst'].unique()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 49,
"text": [
"array(['192.168.1.105', '137.226.34.227', '224.0.0.1'], dtype=object)"
]
}
],
"prompt_number": 49
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Examine the `frame.time_epoch` column:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"s = pdf['frame.time_epoch']"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"prompt_number": 50
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"s.describe()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 51,
"text": [
"count 5529\n",
"unique 5529\n",
"top 2009-11-18 17:36:49.131404-08:00\n",
"freq 1\n",
"Name: frame.time_epoch, dtype: object"
]
}
],
"prompt_number": 51
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"SteelScript: Resampling\n",
"=======================\n",
"\n",
"Pandas supports a number of functions specifically designed for time-series data. Resampling is particularly useful. \n",
"\n",
"The current data set `pdf` contains one row per packet received over one minute. Let's compute the data rate in bits/sec at 1 second granularity.\n",
"\n",
"Step 1 - Time Index\n",
"-------------------\n",
"Resampling requires that **DataFrame** object have a datetime column as an index. The return value of the above ``pcap.query()`` autmatically converts the `frame.time_epoch` column into a datetime, now set it as the index:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print \"frame.time_epoch:\", pdf['frame.time_epoch'].dtype\n",
"pdf_indexed = pdf.set_index('frame.time_epoch')"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"frame.time_epoch: object\n"
]
}
],
"prompt_number": 52
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Step 2 - Resample\n",
"-----------------\n",
"Resample at 1second granularity, summing all ip.len values."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pdf_1sec = pdf_indexed.resample('1s', {'ip.len': 'sum'})"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"prompt_number": 53
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pdf_1sec.plot()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 54,
"text": [
"| \n", " | frame.time_epoch | \n", "ip.src | \n", "ip.dst | \n", "ip.len | \n", "ip.proto | \n", "
|---|---|---|---|---|---|
| 0 | \n", "2009-11-18 17:35:54.369201-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1420 | \n", "6 | \n", "
| 1 | \n", "2009-11-18 17:35:54.369434-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1420 | \n", "6 | \n", "
| 2 | \n", "2009-11-18 17:35:54.369655-08:00 | \n", "192.168.1.105 | \n", "137.226.34.227 | \n", "40 | \n", "6 | \n", "
| 3 | \n", "2009-11-18 17:35:54.369702-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1420 | \n", "6 | \n", "
| 4 | \n", "2009-11-18 17:35:54.369713-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1420 | \n", "6 | \n", "
| 5 | \n", "2009-11-18 17:35:54.369723-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1332 | \n", "6 | \n", "
| 6 | \n", "2009-11-18 17:35:54.369733-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1420 | \n", "6 | \n", "
| 7 | \n", "2009-11-18 17:35:54.369896-08:00 | \n", "192.168.1.105 | \n", "137.226.34.227 | \n", "40 | \n", "6 | \n", "
| 8 | \n", "2009-11-18 17:35:54.369947-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1420 | \n", "6 | \n", "
| 9 | \n", "2009-11-18 17:35:54.369986-08:00 | \n", "137.226.34.227 | \n", "192.168.1.105 | \n", "1420 | \n", "6 | \n", "
10 rows \u00d7 5 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 70,
"text": [
" frame.time_epoch ip.cli ip.srv \\\n",
"0 2009-11-18 17:35:54.369201-08:00 192.168.1.105 137.226.34.227 \n",
"1 2009-11-18 17:35:54.369434-08:00 192.168.1.105 137.226.34.227 \n",
"2 2009-11-18 17:35:54.369655-08:00 192.168.1.105 137.226.34.227 \n",
"3 2009-11-18 17:35:54.369702-08:00 192.168.1.105 137.226.34.227 \n",
"4 2009-11-18 17:35:54.369713-08:00 192.168.1.105 137.226.34.227 \n",
"\n",
" tcp.srvport ip.len c2s s2c \n",
"0 80 1420 0 1420 \n",
"1 80 1420 0 1420 \n",
"2 80 40 40 0 \n",
"3 80 1420 0 1420 \n",
"4 80 1420 0 1420 \n",
"\n",
"[5 rows x 7 columns]"
]
}
],
"prompt_number": 70
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Now, we can compute metrics for each unique host-pair:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cs = (pdf.groupby(['ip.cli', 'ip.srv', 'tcp.srvport'])\n",
" .aggregate({'c2s': 'sum',\n",
" 's2c': 'sum', \n",
" 'ip.len': 'sum'}))"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"prompt_number": 71
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cs.sort('ip.len', ascending=False)[:10]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"html": [
"| \n", " | frame.time_epoch | \n", "ip.cli | \n", "ip.srv | \n", "tcp.srvport | \n", "ip.len | \n", "c2s | \n", "s2c | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "2009-11-18 17:35:54.369201-08:00 | \n", "192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "1420 | \n", "0 | \n", "1420 | \n", "
| 1 | \n", "2009-11-18 17:35:54.369434-08:00 | \n", "192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "1420 | \n", "0 | \n", "1420 | \n", "
| 2 | \n", "2009-11-18 17:35:54.369655-08:00 | \n", "192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "40 | \n", "40 | \n", "0 | \n", "
| 3 | \n", "2009-11-18 17:35:54.369702-08:00 | \n", "192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "1420 | \n", "0 | \n", "1420 | \n", "
| 4 | \n", "2009-11-18 17:35:54.369713-08:00 | \n", "192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "1420 | \n", "0 | \n", "1420 | \n", "
5 rows \u00d7 7 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 72,
"text": [
" c2s ip.len s2c\n",
"ip.cli ip.srv tcp.srvport \n",
"192.168.1.105 137.226.34.227 80 1888380 127030608 125142228\n",
" 65.54.95.7 80 647104 79579509 78932405\n",
" 65.54.95.201 80 194283 27386115 27191832\n",
" 68.142.123.21 80 165452 14803514 14638062\n",
" 65.54.95.209 80 73449 9929531 9856082\n",
" 208.111.129.62 80 72190 5422145 5349955\n",
" 68.142.123.31 80 64617 5085691 5021074\n",
"192.168.1.102 207.171.185.129 80 25447 3322205 3296758\n",
"192.168.1.104 87.106.1.47 80 30364 1468823 1438459\n",
"192.168.1.105 65.54.95.185 80 12475 948700 936225\n",
"\n",
"[10 rows x 3 columns]"
]
}
],
"prompt_number": 72
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Pick the top 10 conversations, this will be used later for filtering:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"top = cs.sort('ip.len', ascending=False)[:3]\n",
"top"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | \n", " | \n", " | c2s | \n", "ip.len | \n", "s2c | \n", "
|---|---|---|---|---|---|
| ip.cli | \n", "ip.srv | \n", "tcp.srvport | \n", "\n", " | \n", " | \n", " |
| 192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "1888380 | \n", "127030608 | \n", "125142228 | \n", "
| 65.54.95.7 | \n", "80 | \n", "647104 | \n", "79579509 | \n", "78932405 | \n", "|
| 65.54.95.201 | \n", "80 | \n", "194283 | \n", "27386115 | \n", "27191832 | \n", "|
| 68.142.123.21 | \n", "80 | \n", "165452 | \n", "14803514 | \n", "14638062 | \n", "|
| 65.54.95.209 | \n", "80 | \n", "73449 | \n", "9929531 | \n", "9856082 | \n", "|
| 208.111.129.62 | \n", "80 | \n", "72190 | \n", "5422145 | \n", "5349955 | \n", "|
| 68.142.123.31 | \n", "80 | \n", "64617 | \n", "5085691 | \n", "5021074 | \n", "|
| 192.168.1.102 | \n", "207.171.185.129 | \n", "80 | \n", "25447 | \n", "3322205 | \n", "3296758 | \n", "
| 192.168.1.104 | \n", "87.106.1.47 | \n", "80 | \n", "30364 | \n", "1468823 | \n", "1438459 | \n", "
| 192.168.1.105 | \n", "65.54.95.185 | \n", "80 | \n", "12475 | \n", "948700 | \n", "936225 | \n", "
10 rows \u00d7 3 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 73,
"text": [
" c2s ip.len s2c\n",
"ip.cli ip.srv tcp.srvport \n",
"192.168.1.105 137.226.34.227 80 1888380 127030608 125142228\n",
" 65.54.95.7 80 647104 79579509 78932405\n",
" 65.54.95.201 80 194283 27386115 27191832\n",
"\n",
"[3 rows x 3 columns]"
]
}
],
"prompt_number": 73
},
{
"cell_type": "markdown",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Now, filter the original choosing only rows in the top 3:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cst = pdf.set_index(['ip.cli', 'ip.srv', 'tcp.srvport'])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 74
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cst_top = (cst[cst.index.isin(top.index)]\n",
" .ix[:,['frame.time_epoch', 'ip.len']])\n",
"cst_top[:10]"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"html": [
"| \n", " | \n", " | \n", " | c2s | \n", "ip.len | \n", "s2c | \n", "
|---|---|---|---|---|---|
| ip.cli | \n", "ip.srv | \n", "tcp.srvport | \n", "\n", " | \n", " | \n", " |
| 192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "1888380 | \n", "127030608 | \n", "125142228 | \n", "
| 65.54.95.7 | \n", "80 | \n", "647104 | \n", "79579509 | \n", "78932405 | \n", "|
| 65.54.95.201 | \n", "80 | \n", "194283 | \n", "27386115 | \n", "27191832 | \n", "
3 rows \u00d7 3 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 75,
"text": [
" frame.time_epoch \\\n",
"ip.cli ip.srv tcp.srvport \n",
"192.168.1.105 137.226.34.227 80 2009-11-18 17:35:54.369201-08:00 \n",
" 80 2009-11-18 17:35:54.369434-08:00 \n",
" 80 2009-11-18 17:35:54.369655-08:00 \n",
" 80 2009-11-18 17:35:54.369702-08:00 \n",
" 80 2009-11-18 17:35:54.369713-08:00 \n",
" 80 2009-11-18 17:35:54.369723-08:00 \n",
" 80 2009-11-18 17:35:54.369733-08:00 \n",
" 80 2009-11-18 17:35:54.369896-08:00 \n",
" 80 2009-11-18 17:35:54.369947-08:00 \n",
" 80 2009-11-18 17:35:54.369986-08:00 \n",
"\n",
" ip.len \n",
"ip.cli ip.srv tcp.srvport \n",
"192.168.1.105 137.226.34.227 80 1420 \n",
" 80 1420 \n",
" 80 40 \n",
" 80 1420 \n",
" 80 1420 \n",
" 80 1332 \n",
" 80 1420 \n",
" 80 40 \n",
" 80 1420 \n",
" 80 1420 \n",
"\n",
"[10 rows x 2 columns]"
]
}
],
"prompt_number": 75
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Rollup the results into 1 second intervals"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cst_top_time = (cst_top.reset_index()\n",
" .set_index(['frame.time_epoch', 'ip.cli', 'ip.srv', 'tcp.srvport'])\n",
" .unstack(['ip.cli', 'ip.srv', 'tcp.srvport'])\n",
" .resample('1s','sum')\n",
" .fillna(0))"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"prompt_number": 76
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cst_top_time.plot()"
],
"language": "python",
"metadata": {
"internals": {
"frag_helper": "fragment_end",
"frag_number": 6
},
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 77,
"text": [
"| \n", " | \n", " | \n", " | frame.time_epoch | \n", "ip.len | \n", "
|---|---|---|---|---|
| ip.cli | \n", "ip.srv | \n", "tcp.srvport | \n", "\n", " | \n", " |
| 192.168.1.105 | \n", "137.226.34.227 | \n", "80 | \n", "2009-11-18 17:35:54.369201-08:00 | \n", "1420 | \n", "
| 80 | \n", "2009-11-18 17:35:54.369434-08:00 | \n", "1420 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369655-08:00 | \n", "40 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369702-08:00 | \n", "1420 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369713-08:00 | \n", "1420 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369723-08:00 | \n", "1332 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369733-08:00 | \n", "1420 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369896-08:00 | \n", "40 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369947-08:00 | \n", "1420 | \n", "||
| 80 | \n", "2009-11-18 17:35:54.369986-08:00 | \n", "1420 | \n", "
10 rows \u00d7 2 columns
\n", "