{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "[Link to this jupyter notebook](./Files/0204-Web_Scraping.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# In-Class Assignment: Web Scraping\n", "\n", "Today we will be exploring some of the extensive datasets available at the National Oceanic and Atmospheric Administration (NOAA). Work as a team to try to get as many of todays activities done. We will meet as a class again around 3:30pm to discuss what you learned. \n", "\n", "\n", "\n", "Image From: https://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/us-climate-reference-network-uscrn" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Agenda for today's class (80 minutes)\n", "\n", "\n", "1. [(10 minutes) NOAA Example](#NOAA_Example)\n", "1. [(5 minutes) Installing Beautiful Soup](#Installing_Beautiful_Soup)\n", "2. [(20 minutes) Presidential data example](#Presidential_data_example)\n", "4. [(20 minutes) Dynamic Website example](#DynamicWebsites)\n", "5. [(25 minutes) wrap-up Discussion](#Wrapup)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "\n", "# 1. NOAA Example and Coding Standards.\n", "\n", "We are going to start today's activity by doing a code review of a **_web spider_** program. \n", "\n", "✅ **DO THIS:** Download the [noaa_scrapper.py](./Files/noaa_scrapper.py) and [this jupyter notebook](./Files/0204-Web_Scraping.ipynb') annd put them in the same directory. Run the file via the following command:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "running as noaa_scraper\n" ] } ], "source": [ "%matplotlib inline \n", "import matplotlib.pyplot as plt\n", "\n", "from noaa_scraper import get_noaa_temperatures" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **DO THIS:** Run the ```get_noaa_temperatures``` function as follows:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "691ca32a38264410b08bf84dcfc82a97", "version_major": 2, "version_minor": 0 }, "text/plain": [ "FloatProgress(value=0.0)" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2007/CRNS0101-05-2007-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2007-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2008/CRNS0101-05-2008-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2008-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2009/CRNS0101-05-2009-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2009-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2010/CRNS0101-05-2010-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2010-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2011/CRNS0101-05-2011-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2011-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2012/CRNS0101-05-2012-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2012-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2013/CRNS0101-05-2013-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2013-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2014/CRNS0101-05-2014-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2014-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2015/CRNS0101-05-2015-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2015-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2016/CRNS0101-05-2016-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2016-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2017/CRNS0101-05-2017-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2017-MI_Gaylord_9_SSW.txt\n", "FOUND http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2018/CRNS0101-05-2018-MI_Gaylord_9_SSW.txt\n", "downloading... ./data/CRNS0101-05-2018-MI_Gaylord_9_SSW.txt\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "04e5ba406f9145a4ac3b3c790f63d37a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "FloatProgress(value=0.0, max=12.0)" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "reading... ./data/CRNS0101-05-2007-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2008-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2009-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2010-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2011-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2012-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2013-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2014-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2015-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2016-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2017-MI_Gaylord_9_SSW.txt\n", "reading... ./data/CRNS0101-05-2018-MI_Gaylord_9_SSW.txt\n" ] }, { "data": { "text/plain": [ "[]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEFCAYAAAAG45eHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA7GElEQVR4nO2dd5gURfrHv+8GcmYXWElLzoK4EiSriICKWe68UzF7nuHOU1HPU089udOfp57pOM+cz6xgRBBEMpJBCZIkLDksYZfd+v0xM8swOzNd3V1VHfb9PA8POzPdXdUzXW+99dYbSAgBhmEYJpxkeN0BhmEYRh8s5BmGYUIMC3mGYZgQw0KeYRgmxLCQZxiGCTFZXncgnpycHJGfn+91NxiGYQLFvHnztgshcpN95ishn5+fj7lz53rdDYZhmEBBROtSfcbmGoZhmBDDQp5hGCbEsJBnGIYJMSzkGYZhQgwLeYZhmBDDQp5hGCbEsJBnGIYJMSzkGYZhFLGzqBgTF2/2uhvHwEKeYRhGEde+Ohe/e30+Cvcd8ror5bCQZxiGUcSGnQcBAIdLyjzuyVFYyPuA0jKBklL/PBSqKSktQ2kZVyBjws+WvREN/olJKz3uyVFYyPuAc56ejnZ3f+Z1N7TR7u7P0OOvX3rdDYYxxsqt+7zuQjks5H3A4l/2eN0F5azfcQAHi0ux91AJAGDfoSMe94hhzEFEXnehHBbyjBYGPjIZ17w6F6Wl3phpSssExn22AoV79WyAlZUJ/Gr8TKzdXqTl+pWJoY99i/yxE3AkRCbLBRt2e92FcljIM9qYtnK70fbmrduFDTsP4JOFm3Dtq3Px3Lercft7i7S09fXyrZixZgcGPzoF63YUIX/sBHy5dIuWtkyzde8h3PzWDzhUUmqkvZWF+wEAXy8vNNJejLIygZ4PfIXFG9WspPcf9udq1Vf55BnGDec/+32F96b8uE1LWwfjBODCqJB4ZspqDO7QCFWygq07PfDpMny6aDNO7dQYZ3c/zli7pjfnX5mxFjuLinHWU99h7biRrq93+7sLFfRKPcF+GhlpBj0yGW/MWu91N0JDZsZRm6sQEeG0YMNu3PjmfK+6pAyv/KAOFJvVhHcUFSu93s/bD5T/nRV9Pn7css9zMxQL+UrCuh0HcNcHi5N+9tQ3K/H8tDXa+7DnYInS623ZcwjDn5jmiXtmZtzGWtHho1r9F0u3Gu+LcqJfp+mtw8+XhMPcBQAZGYSvlm3FsMen4tEvf/K2L562HjCW/LIHt76zEGUh8/l+9Muf8OCE5drbufzF2Uqv1+fhSVi+eS8e+HQZtu8/nPI4HbbSjDhNPtXkGVREVMqbdhCZtKIQ1782z9hegE6Kj5Th6lcipUyf+3a1p31hIW+Dq1+Zi/fmbywPeAg6xUfKcPZT3ym/7uEjRwfpobi/VW1wJfLpos24493UG6yql8uvz1qHa1+dp/SafoQ06/K7iorx1uxjTYifLdmibR8lkUkKN3r3HCzBXsUrVVWEauN1/Y4DqFM9C/VqVPG6K75m0COT8ZveLXFKp0ZYpEHwXv/aUbv0sk17y//WpRlu338Yk1aY88y4+4MlxtrygugWA254Yz7mrM1Hr1YNMKJbnvJ2bnl7Ab79KZlAN7NSXrZ5r/VBknS/37/BfqHR5F+ftQ4DH5mMU//vW+1tqXgEhRCY/GNh+aadSdbtOICHJuozz3wTJ3DjBTuBcN/HS9Hu7ona2k6GKo20cO8hLDEUuLbnYAnmrduJpyevQv7YCUafk7K4tl76fi1+97r6zWQhBLbtS25i82BIuMLv5tvQaPIx7Ur1jvmUHwvRKa8OGteplvKhdMKHC37BH95eiAfP6Vrhs5em/4wjZQJXDWitrD0/UFxahpe+X+v4/OWb96KktAzHN6unrE8yLN20B2VlwFkaTFupuOyF2Z4F1CQTsks37UGX4+oquf7qbfuNKGMAMGvNDjSuUw35OTW1tbFkk/XEX3ykDCWlZahZ1bzIDY2Q18XlL85B03rV8fa1fXBE4Yy9afeh6P8HK3x23yfLAEC7kL87YBuGw5+YBgD2fZpdKvIjnzQn3GMs2rhbexvz1u3Ebf9bhE9v6o8aVdKLgj0H3NubD5WUYuST09Apr07a41SMsn2HSnD4SBkuHj8TgINnxgYyK48Ln/seCzfu0dqPVLCQT8Kqwv3IrV0VdatnAwB+2X0Q/f8+ufxzFUtnk8vv+LbihcfMNTu1t+2HnDXfr9qO4RpsyiYRQv2ext8mrsCa7UVYtmkvCvIblL//5bKKbqAqcrH8vL0Iq7dF/ulm4D8mY5eCiUkGma9moSanAxlCY5NXyWmPfYtzn56uVRDHLm3aTe3sp6Zrvf7CBBPDzW8t0NqeDNdrsCnrJvHJW7O9CKsK9WQ2lHnKfZRvS4pEAT9v3U68MmMt8sdOwEcLfvGoV97AmnwK1mwvgon9FN1uaoDZjaxRT8tNIj9t3Yf2jWtr7k1wSfzNTnssYsNWudyPPXnxbfkh7/+GnQesD7LJ+c/OKP/79ZnrMapHU2XXNjGG3cCafBrKdGry0f9NaEjeD9uKnP7Pqdi856Bvkzr5lV4PfY0daQK/7BB79uJXrFtTxIBkKHhQZS9RVKw5GErhmCsrE1izfb+6C2pAiZAnoj8Q0VIiWkJEbxJRNSJqQERfEdHK6P/1VbSVDF0RclqFfIrQ8eUKfXcBYOOuA/gqiY3VD/R9+BuMMuixEgYK9x1O4Vtun5gGevH4mZbJtfxWnNoN63eoWym0vmuiLZPksk17y2ssmMK1kCeipgBuAlAghOgKIBPAaABjAUwSQrQDMCn6WguptA+3qIyIS0Qk0a/X7ziAhz9bobSd/n+fjOte8290polNOCYFcRrGO3M3AgDmrtuV9FA3rq8xVhfK/da6F7db9h5C/tgJmltJzognp+GyF9Sm97BClU0+C0B1IioBUAPAJgB3Ahgc/fxlAFMA3KGoPSOkCgJRoeCLJPaagY9MTn6wAx6euBx5daspux5jDlOeV8mE6U1v/qCtPVm30B1F6uJR/MgP63cbbc+1Ji+E+AXAowDWA9gMYI8Q4ksAjYUQm6PHbAbQKNn5RHQNEc0lornbtpnJWaGCz5dsQf7YCa5NRem0lg07DzhOv/rvqWvK/e1l+c/UNdh9QG0wGWMfq73PzXvUrFxNe8zITl2vzVyPjbvUb75WVlSYa+oDGAWgFYDjANQkot/Ini+EGC+EKBBCFOTm5rrtjjFiJpA1Ds0NMg/8gH9MxqX/Nbe0e2jictz5frACpMLIqzPWpv18iyIhn0h8Yjmv0XWPlREVG6+nAfhZCLFNCFEC4H0AJwPYSkR5ABD932xtL0M43ZxdtyMyOXxjkVgrlY1UF34IXkqGFzl+vGKRR4XdH/96pdbr+9vRMLyoEPLrAfQhohoUCYs7FcByAB8DuCx6zGUAPlLQVuARQuDN2esxfdUOAMBijwZ00DgYoBzj89a5jCS2mM9enblOSRWlRP/u/0zVXzhGljBP6abNZCps8rMAvAtgPoDF0WuOBzAOwFAiWglgaPR1KEhWS1SWGWt24M73F6ctcsFUJEiKfHzgjRNkbrXzX75w1QZQUdjoFj4B+gm1YnpFo8S7RghxL4B7E94+jIhWHygmLLL2By6My0ZpV/gcOBwcjdRPqBAQhXsPYdeBEnRo4u9IW1OmqUS7d0mpnnb/N3cDalfLwpy1+nMlBQEVgWW22jPamk/Zsf8wpvwYsY3f8Ebw8pxUBlQIvt4PT8Kwx6cq6I01j3/tvK6n7J3mj52Atnc5z82/Zrs9p4H4AjB2uO3dRbjutfm2XAf3G94bGvHENKy1+X04JXDmGj/gNnfEb/87G5e/OMdX3gXMsagp1KLgIpK42cT8aMEm6WNVpr+2YsST04y1NealOVKr6kRKywRemv6z7fOWbd6LwY9OsX2eE1Rk9LRDKIS8W1Zvi+SecCIEkkWuqmbNNnO5Mb5btR3zDHv0yCDUlmn1LXt8Wic0xm3/S5/+QCXfrbIfN/PhD7/Yjg8xjWmbfCiEvAlB6yXnudjodYKbjWVdhP03jvHB/I1edyEt/5tnsn/2xaHOspaqOHzErMYSCiGfyGqDmq8JE4BffddNEiTvGjesMvjsho3DR0qxU3H5zzAQCiG/PiH/tNP6kU4Eid1gKCfmuIwARJHo3rSqJDLe97nJTWLXXFNZFAG7hELIr3WZOjS2fPpsiT/Tqfp94H+8cBMGPzql3ENJB27TPgclJ08QKjB1+cvneGPWeu3tbNhZsf4xY59QCHlV/PEd+5tKK7fu167F+n3gL4lG7f64RU95OsC9kGeTlzqKiktx3ydLve5GBZ6f5p+IXT8RivJ/XsrA299bBEBvNXjTGzV2MRK8w0txX+FHvePJb1Z53QVfwpo8Ewgqi4z3o/BMht9Xl8xRWMgbZpziyk+VBbfmmr9+6m/f6RimA2Wc4st9osqiCdgkFEI+IOMCALCyMLwuck5z68uweOPRbJ1z1+5E4T57+cb9Wuc2qJgac0Vc6N01oRDy7DrlD96eu0HbtV+IC1W/4LkZGPkkFwD3ElN61Yt2UhQESNkzSSiEfJA0ecYZK7ceuwLati+cqZpNPctuPaFMmZU++OEX+Y19VvaSEg4hz1O4p5hYSWVnhuJR9QWTVxS6zsZpasSt3laEN2bL+eRXltQXduGRwwQCv+eADxIrC93HM+wzaCtfKpnimM22yWEhzwSCyqLJ86q0IvyNuKNyjBxGK2YUqPCraau37cfbc/SnCwgar89arzWaOuyEQsjzxmv4MVgbwzNGPDENRcVcuCYZpip6hZFwCHmvO8AEiv0+9b32e/oK5ljKAqJ5hELIB4HCfYdwqMSdlvbct6tx6QuzFfUoWKjMj9P13i+wYae7zKVBJqwblKZvKwgFSgAW8sbo9dAkXPnyHMfn/7B+F8Z9tgJTf7JfEs0JpWWiPLukH1A9gBNrEAQZu3b8kMp4lBrWrN/zeRWvGCzkDTJ91Q7H5+pMGZCMx7/+CWf+6zssk3RfCytGMmy65I73FnvdBcbHhFbIf7l0i9ddUIrpzeWF0VwxMjlijGQaVtyGX+3yTHAIyl5gKIR8MgH4xdJwJaQyLeRNmYVkUT2PXPvqPKnjZv+8U3HLDGMWJUKeiOoR0btEtIKIlhNRXyJqQERfEdHK6P/1VbTFmEUmR8n2/frzyHhlNrnvk2CkKPYbV740Bze9+YPX3WCgTpN/AsDnQoiOALoDWA5gLIBJQoh2ACZFX2shWZSgrOY7cbE/67omEn+P71hke5y+arvCdq0pdun69968jfhk4SZX19BFEGzydjFxS5NWFOJjn/6mlQ3XQp6I6gAYCOC/ACCEKBZC7AYwCsDL0cNeBnCO27Zs9UvyuNdmrtPaD1XET1ovfJc6/eqnizbhkudnGejRUdwW9Lj1fwtxI2t9xlCVyCuME6AdglLgRYUm3xrANgAvEtEPRPQ8EdUE0FgIsRkAov83SnYyEV1DRHOJaO62bc7swJUt+1y6h+v3b6gVljLPsYlv/5fdBw20wthh/FQunB0EVAj5LAA9ATwrhDgBQBFsmGaEEOOFEAVCiILc3FxHHUhmrskIyCwry4q43B0m70wmYZaJyD8dLqQyAVH7Djn3wtm85yB+3LIPew6WOL6GLM9PMy9wJ60oNN6mbvYcKMFPW8OVJ0eFkN8IYKMQImYjeBcRob+ViPIAIPq/viciiRwKmYzHs1NWe92FlJQGdNk+4B+TLY9xs4Io3HsYwx6fiov/PcPxNWR5cIJ89KWbiSseGc8jt/s1pjnv2ek4/Z9yeXLcmilN4VrICyG2ANhARB2ib50KYBmAjwFcFn3vMgAfuW0rFSGT55aYjOyTMtcE41k3zqinpwM4dhXmBzYbNH0FRRDGWG1jxbj7gP4VmgqyFF3nRgCvE1EVAGsAjEFkAnmHiK4EsB7AhYraksLEEtkrSkrNaUfvzd+Ifm1z0h4TtIFc2dmwy5yQ1/1obNlzCNWrZKJu9Wy9DQUYJUJeCLEAQEGSj05VcX0rcmtXrfDeZ0vkIl79pmXJYFKkvj//Fzx2UY+0x7CMDxbz1u0y1pZup4g+D09CnWpZWHTfMK3tBJlQRLw2ql3N8bk7i4oV9qTy8e1P27D7oJrvMMyrr8qKCcviXkV7DGElFEK+suEX/+SdRcW47IXZWPKLmiRmL01fq+Q6jH/wy7NamWEhH0D8MmwOH1Fbxei5b1ezUHCB1Xf3+ZItKHjwK6Vtdrv3C4yfmtrzi39N72EhH0DW7QhPLvR4DpaUYuaaypkQzETu/r9+shTb96s1T+47fAR/m7gCa7btT/o5z9nJ2XfInGmShbxGSssEDhaXKtd4/YKOAWzSc8hPPDFppetr7LJw6dMpb7ftS56kbq+hfZbJigOz3px9bCGWQyWlyB87AU9PXqXk+te8IpcFVQWhEPJ+DXz6/Rvz0ekvn+PA4XAKeUYdXy1znxr76lfmpv3cC6365e/XGmlnzEvOq64l4873jy3EEpusHvniRyXXn7HGeQEhu4RCyKd6eE2XA0sk5sbpx0mooKU/Mz/z6t45qUwmMbbstS4A45RUv9vzaZLpBYkgP5ehEPKp+Nc37pfAKpDJ/2IXP2xQet8DJh4vdRoTj+PnkrEvfqJni3pedyEcQj5VwIWfClH7DT+uLgBvJq9DJaU4VOKNSa34SBkOFKvx8z7i4X6G26Cn7EzrB/L1Wd6lBXf6WI4d3kltRxygKq2Bp6T6AXyg7AIASMNUKoT3gtoPqwkVdLznczSsWQXz7hlqvO0Ln/u+vJ6uW4qKPdz7cfkoRLLGpr9IYortvQY9VJzSq1WDlJ/NW7cTJ7ZM/bkqQqHJpyI+p8qyTXs909b8qDSbXtr/4bT2UsfNWG1uQyqeHR5FPqsS8JWBxMRqMz16VlSx35BDRqiFfEyO7dh/GCOenIY73lvk6DqZGfJiOn/sBOSPnXDMezoEqttLzl9vLn8JAGRJLMcBYGVh+s1DK+b++TRX5zMRTu/cGDee0lb6+PjnUdcKz+2z4YTSMgEhhJYcPKaUv1AL+Vit0wPRZezctc4EW15d57lxAGCphr0BtwPJ7Tj8dNEm6Vwzl5+cL33db1YUWnqJpCOnVsVkdYx9Lu2bj+7N6jk6t/v9X9o+R9b0+OEPvyjbw7BCCIE2d03E/QnF3EtKy6RMRS0a1Ej7uSlzayiEfCpNu6RUHPO5U5fKE126G/72hdmuzvcba7cX4fdv/IBb3logdbwQAqd3bix9fSt/byve/93Jrs5ngP7t0qeXTiSmNKzcuk9rwrBb3l6A+z5equ368cTu6aUEX//rXp2H4++znsjaN66loVf2CYWQt9qZLxfyDtXXu0a42yHX4a9vestz3rqdeGfOBgDAoWgEr2zVpLaNa6Nd49rSbdkp3JCMni38GQMQNFo2TK+JxhMzZxxwsPl75vF5ttyMt+5NHl1rxaL7Tsd3dwyRPj42xhI1blVlD1duNWN+CoWQt5LdsXqvTmuRZtmwyQeJatlyP//ctTtx/rMzcLvDPY2aVTIdnffz9iL85aMlRmrIAsCjNqIZuzWtq7En/qBd49qY/KfBts5xYoKoWSULo3s1t3+iTepUy0az+jYmrqhgIQC7iux78ljJpb9+uiz9AYoIhZDXTYOaVSwF1Q+GNzJV7G3JarwXPJe8RqlsH2IDv3VuTbkTANz1wWIMeXQKXpmxDsu3yKUy/vyWAdLXT8ZTCXlJtu9PrTF64b7aKkf++zPdZuxZcBL4RwTcM7IzVjxwhvTxAFCk2TZ/VJMnXOrA5OoXB+NKJeR3O0yWRER44JyuaY/50/8WOrq2U+asdZ+tcfylBXjmkp62z9u+L+JueFDSJbV3q4YA7K2I3ph1NEGUrOBQKQS37j2Egge/Vna9619zn5DK6zQd6XDTMyIgI4NQLVtuxRebUGaulhsD713fF69c0ct2v45OXOknfL9TKYT8rJ8j/rRuBsl5PZuhY5PUduXEQA03rB030vKYpZvce+zUqpqFrsfZNztMXblN+ti140biuHrVbbfhJc9PW4Pef5uk9Jqy5SjTkVOriuUxphKCpcLJMIgfO/NtBKQN6ZgrddwJzetjYHu5Y+OJ7TM4Hdp2PODmr9+lLY6nUgj5g0k2g75ftT3t7HzvWZ0BADcMaVP+3ue3DEx5vCoRL/swqopgP66effdQpw+9jhw+OnhwwnLLYxrWtBa4qnnwnG6Wx9xryPMkETcuvXafCrvPn3MhHfk/5qWniw07D+C8Z77HPR8u0XL9SiHkkz1/v35+FkaPn5nynDH9WmHtuJG4bVhHqTZUKfKX9W0pdZyqogNZmRlSK4d4Mr3Op+AD7ATIqaJ2NbksJF7ULU7lieIHVK6yZXjjqt4AgItPkttMjsWbLN2kpoxmIpVCyKdynVylMIJOlZZ6aic5f/KZBvNRJ+JUwN1yWjvFPTmWqlnOvHic4EXanmb15cxePR/4Ci9O93+K37+O6gLAnxMD4Ow3rl8jGye3zcHacSNxRtc8ANZpvW9/N+K1put7CIWQt/ot4m3xT3y9UkvYtdsfqHerBhh3nvVyPIaXW3BONaPh3fIwWlK7cctjF3VHU0N7Ad/cOshIO3a+9/s/WYaftzuLN/jzSAdxIQ68a5J55MiePWP1Du1mFCepDLIyK4pUq59t2WY9GnyMUAj5dAgh8NGCX8pf//Prn7D3oB7XKzdl/t6+ti9G92ohfbzbecqNtcGNuaa+IVv2eT2bYdrt8oEvMZwoAK1z/RHZmMhlNt3+cmtHUkKky5yYCieblI3rRNpr3uDoZJydZS2SlvyyB7/6z0yM+2yFvU7a5FCJ/Y0vNzE1rMk75KMFmzAnIWeNlmRDROjw58+VXzcVO4rcuXS5ieLNz5EPKEnEpCk7w0FjTifPznl1nJ2oEbv1ck/t2Air/zYCxzvIWVOuldv4yod1aYIXLz8JV/ZvXf5erapZeOfavmnPixUjl424dopMjvtEkqVAkc0BpMsxQZmQJ6JMIvqBiD6Nvm5ARF8R0cro/57EmifzoNHhbrxc85IrkQ07nT/gH97QD1f2b+X4/PFT1zg+N8NnBtjCfc5K4l01oDVOiKv6c9sZHRT1KElb/Vuhe/N6lse55fYzOjreb3EyORIRhnRsVKFNJysJO+TbSNdgl2RJye4Y3hEnt2mY9HgThetVavI3A4j3PRsLYJIQoh2ASdHXxilL8vQle68y0aN5PVceB268APwl4oFeDx3rDy/zZIzsloe+bRrijav6YPZdpwIAhnRopKF3Ef58Zmd8dEM/AMDMO0/FFJupBmSpVdV9DaEguMnaTb5mh2TDKjszA11TpMFod/dn2voSQ4mQJ6JmAEYCeD7u7VEAXo7+/TKAc1S0ZZdk8vzmt35Q3k7Xpt4t1/PHTsD5z35vedwlvVvgjat7G+hRhF8lyUdi2p3NLjI2+VjirupVMtGojnycQf7YCa6fvSZ1qyFfU3oDNz+NQCS197DHpyrrjy6Gdm4idZxpVdDvNvnHAdwOIH7t0VgIsRkAov8nVXWI6BoimktEc7dtk4+klCXZDzV9lXr3Q68XB/PWWefOeejcbji5jT4tJpFfJdlIbm8jG6UKdOwBNKrtPGf9Rws2KeyJWtx8VUIIXPL8LGV90cmg9rm2PNns4MYcqUv9cS3kiehMAIVCCEfJOYQQ44UQBUKIgtxc+6HH1tdXe714TwCd7YSBZJkaRx6fZ7QPdrVemZ/R76sRp4T1vpJRu1q25TGO9hkc9EU3KjT5fgDOJqK1AN4CcAoRvQZgKxHlAUD0fzVJmJOQ7sdQ7UkzqnvTFO14g5+LaftBaHSy6fViKglYaZmoUCYyyHyjKMe6KYZ3bYIHosFYKunVKvkGq5e4FvJCiDuFEM2EEPkARgP4RgjxGwAfA7gsethlAD5y25YT5q/brfR6fxzaHhec2KzC+14J22emrPakXZO4mqhtntrxHjNusH7MKOlmSn4rWlAmKGRkEH7bNz/9QTZ/oiX3D0u5qSv13WpSinT6yY8DMJSIVgIYGn1tnFmKw/8zMgg1HBbB0MFbc9ZbHxRwRj75neNzdcREuOXsp76TGs/XDmrtyuNl8x5n7qGMM9x6J+la97r3mYpDCDEFwJTo3zsAnKry+k5wWvLPLwxol4NpK7d73Y3A4keXvkUb5dJE3zm8E+4c7q705LJNe9H5OGuT1emdG6cMHps+9hQcLC7FaY9966ovMTrn1cGFBRVXw35jwuLN6i4m8Rj63bvGt2ipr5rkkm7mknTBH9lJcmHEKD5SpkyIveygqEI67oumavYaHdWUVAx+U7rHiCenSY2BdBvUTetVR93q1huVsky8eQDG9HMejGeHc3oc5/jcuz5YrKwf/Qx6tSUSeiFvSo93YxZIJ6bT2frb//kzrN95oPz1Xhfphwe1z03q1+6Uyw0N4hg/Ppi8dNzNGjJfboj7zuPxIse8DDL7RX7ewHfD46NP8LoLACJ1Is7oIuefr5rQC/niI/rDht2SbHi9flVv/N+F3W1NHcff96WrfrTO8WeiLRlSpRlOtxJySipf6G80RaK6ReYZspLxKjV5P1G/hrn7qpad/ln0rZ+8HzC9uZasPdWKUL+2OTj/xGbsfy+Bk2yTbkhlO61bPVs65YDbZ/bVK3vhrWv6yLUl0ZTVIVUkskMGkSm3mX120qHL5Ticv5wHFCckGrKTsOz0zqkLhZiU8TKCx4/L+uZJkkLF88HvTlbaXqpkU4D94CunDGiXiz6t1flk++1ntSq0oYq61bOV5OzprTmpmhtYyCti3Y5j7bTTV8l7xJydZnPIpFDdIVE27ojCjeyfHhyu7FrpSJUcyilWAS+mipXIIjV5+8zV9G2LdMN+o34N6/0Yq2+YzTU+wm9ajyo++sE6r4rKew+qCcAqH86LY06yvIbJZ+jAYetiNn57pk3W0FXRkg+Cu1MSzFHmMTUllneqBo3JwSdT2cpvGp8MqnPYWwkgHW6bqTi7u7WL4JUvzzHQk+Ci4vGQuUb/tundKNlP3kfIZFK0IwzT+bqrEqoyRaCvkHB7lJl0rh3U2vogg6jWCq02yGQ8evYcdO7uGo+V4ACA+et3Wx7jx70WU5jKsXRhgZn6xomwkPcB6Z4xVWNPpvZkrkQKXZn+uI3S9Dsq5oxNqkrXKZJPlVfEq0FFUKLvy/95iWklREZ78pu5Zu2O5AE88cgoNH4z13z9x0HG2/RTqgRVPTGd599PqJi0x/TLd38RTYRCyJumSd1q+PdvT1R2vZpVlKYQcoyM8LKadK6SqB076dZBePaSnrLdSkvbRmYDuFrl1JSa5IPG8K7eRGP6ARXmmoJ8BS6UbJP3F4Papy9wIqvvLr7vdFRPk9XyoXO72uiVOxrViZhrBqa5N6v7qi8R2t8mtxaGdzNbPEQVk/80GHUVREmqsgOruo6/1mfhpV0apWT2zzu1tBkqId8m15xXQ7Xs9OmGZVMcW1WoaZ1rTlMd3KERXrmiF64f1CblMVv2KLIlM75Cpri9mw31FhYBa8mQ2SNKR51qcivkvgqDyqyQ8cxTTaiE/G3DOmDlQ+4CbK4b1AYn5buPtpu71rrm6gkt6rluRzUD2+em9UYpkvC5NoVJX2q/YvIbuO30Dph40wBHGUurZWdgxp2nYM7dp0mfM+fu08qLpjshp5bcJPF/F3V33IZdmtgo/K4KfxiDFZKdmYEbT2mLf32zytH5Y4d3VNKPfYePpP386gGtcOvpHZS0ZRKr/Pw5tcxlYqxpo3jLXSM6YvqqHfj2J2fF4p/81Qno1ER+c7JlwxoVoqDjSSecieQ33GVyxcuQnWGt72VlZjhur0wAeXXNRAJf0a8VXpj+M87rmbxUZyJWq3KVeOG4ECpNPkYQhGf9mlWMPlyqsPKnPr+nvWIQbrIbXtlf3nxwzcA2rnLmn9KxEdrZ8EBxo2F/9YdBWPFA8tTJiXTKq4O7R1i7rB4oTq50/P38bnjx8pOk9lLc4NQP38n3mJ0ZOStTYuIyjRfhCP77FioJVTSkwE1Gx6j2KRMZGSPdgPzLR0vTnptl877c7BvqyBWfCrvdtJq80q2I2jaqZUsBqCVhe+78ly+Sn1s1G0M6NpJuy+88MbpH+d92ni2ZOJKgEgoh71Ww3rAuqbNHWnGpVRHhOD68oZ/jdvLqVsOkWwfhHxccL33Ogg27U362dJN8dk0ZvCqkYBfVQZHvz9+o7FpuuhbTenXTsKa7TVRZRvVoWu79VMfCqSEeNyLEbwnpEgmFkD+K2dl4cAfnGpCd5Fw9mtfDrLuclcu9qKA52uTa0wyHGRS8D57T1dZmnFeoDoDafUBNWgPA3QQkswpQwR0O97qcuIhePaA1HjynKy4+ST6NgJu0DnasQl7ooyET8mYxOYM3drgrf0on+xORqZzoQMS848RV7neDU7t5BoFPF6krEu2nCNxUVDe4/5SdmYHf9Glpy/vq90PaOm6vzOfF51jIx3FaJ3vml4Htc/G2ZHUeL3j9qt4py+IFmTev7oPbz1DjBSWLn1PJhhlTX/sfA+Cs4RQW8lG+/uNAPH9Zge3zehsMpLCLjvqm6Zh9tzOTkl2yDNmR3WB0We7/r8Mx95zV2esuWCITSBbjhiFtjRd8ZyEfpW0jcwmaTnNgQgGAu0d0wk2nmvMosUuj2mYCPbzYaGdN3huGuNj3MoWd57FH83qYd89QfZ1JQuiCofxOw5pV8PjoExyde/XAiF/4k5NWquwSI4Gf7d4lpS6Mwpy0xjV+jFyPx7UmT0TNiWgyES0noqVEdHP0/QZE9BURrYz+r60yr9/S36ajX9scJYWDKzNuPCFUZb+0op5EzU9V1LAR+cuo57GLenjdhbSoMNccAXCrEKITgD4AbiCizgDGApgkhGgHYFL0tVaCsKQ2OR1V5mo/qRjeLQ8XnmgvKhew/2w9fnEP2204xU15Q5ni7UxqiJA2i6wfcC3khRCbhRDzo3/vA7AcQFMAowC8HD3sZQDnuG2LYQCgZUN3Lp5Opj67YrRBzSoY2tl5sJwdslyE75t0bQwj6VIH+wWlG69ElA/gBACzADQWQmwGIhMBgKQ7KER0DRHNJaK527Y5Sx4VJKrZCIKqTHx722Cp41b/bQSa1DWfyc8JXy3baqSdqg6fqZfGnITTDE1EYaVb03ped8ESZRKHiGoBeA/ALUII6dh3IcR4IUSBEKIgNzd9IY4w8Ocz/e8S5gWy2rmK9MLN69tPX2uq2LMTnOaecROxbZcg7ZvZ4Y4z/O9fr0TIE1E2IgL+dSHE+9G3txJRXvTzPACFKtoKOm6yLjJquGFIG9cFKfxEZgbhzav9G5QHeJdfSjdWRX/8gArvGgLwXwDLhRCPxX30MYDLon9fBuAjt20x9gjpuHJNVmYG7rG5ovKvHh+hQEGhG52EVcj7fdMVUKPJ9wPwWwCnENGC6L8RAMYBGEpEKwEMjb7WQlgfIC+Z/KfBGNxB3nxmKpuhKuz21sfWmkBgJyqUUYtrh20hxHdIPWbMxLlH4XGojlY5NXHARqk/9v2vSKe8Oli+WW1q5lT4/dlnEX8sQzs3NrYxz64eAeScHvIFQNzQxoZ7mN1iIUHDycbrw+d109ATf3FRgVzMgRtN/tMb+zs+168ki1LWVbgk3CMzpJgqa/bAqC7Sx27bd1hjT9RjQrPs0byegVYimPb+OS7qxlqjitwKzk1gXtemdR2f61ceGNXVWFss5AOI7Izv1gwaZu3cr9HAfR1mNTVtrjkrupqU3Xj06dftGc0bVHTj1TVPh3cUh5gMCyHfvrH/o/CSMaBdjtddSMrfzjVjdnnz6j747+X20117iWyJPZbx1uhKgsdCPoBYKdhe+O42b+C+SlabXHOTkx3N8te9W+jrSBytcmpKmz8SsasFXn5yvqN2EpENcmrsMg11XkCinF3BmjwTw02uEl2M6Jrn+ho3G8yV77ScYljo0MRd/QS7Wmez+u6UgNqGatGGEf9JCw+4qn8rr7tgC6si4LHhZzSUXIEWUt9gxZy+bRrinWv7GmtPN3Y3XlXZyGWvY2VitMKUs4GX6KoYFapvzqmHwQWSbmB+wephULmBUy1b7hFxk+42niE2ArDc0qtVA2NthQ3TwWG63AuTkU7p63JcHS1tjjuvmzalI1RCvrJgUuM9VCJXdUjVEHRqk3bKDxal2KoY9DAKUlRt/7aRTfLehiZKk6a8dB5D3TW5xY7u1SKpx40KWMiHkEt6twQAtDWY61qVgLrXcOFmqwnz29sHm+kI/B+1Gk+/tjn48cEzUJDfACaU7JYN9QjAZJSUhssXKLRCvoENbdfP9TuTYdXbs7ofh7XjRhorrA2o+w4b+WhD9NK+LZFX173XUFipmhXReE0EYsmkmFblMZSO4wMYmBVaIR/mijdWg0rlkLtAslRekEwNMuTVrWY7U6VrQvYdqiRbwmzWqI6a9NHpUjBcfFJzTLp1kJJ2TBFaIW8HUwKqmyItwKQs+PPITlLHhU0+Na9fQ0qwBBWVnlcmfnsZTV6Vx1BpWeoLEZHReA4VhOIpTvbj2glbNyWgruifr+Q6VpOSyknLyl1TS6OVFLcmr6BpmHbIkkhlrSr2IZ2QDyKhEPIx4h+DQQZLmzHBNo/5JU2y23nSbxqmyhoDMgGAJ7Sop6QtmYyZBS3r49K+LZW0p5tQCfl4/jqqC6aPPcXrbhyDquWktSavbnDJapcD2/sz74wMvAhxR6rvr22j2qiSlaHE+0ZGk1eVdE5Gk3/3+pPxV0WZJE/XXEzdHyqMBrIzM9C0nr88I1QJ+Xo1zPnJy6IqGMoLkvU9rIWndRBRBCp+X+0a1cIHvztZSRsywVCqrCymq1iNv1RvUrrQCnk7mJJPqh6dji7zjthB9rsJsIyX2tQzgcleNKuvzu+8OEkBDCBiwqumyIwno0SoEs5skw8laoaX1SalquVkw5rJXcUa1VbjQlbZSCZAbjxFTYRlVdmNa5gv/KGbu8+U88ySQUbIt1AUMZpizgosoRfyr1/VW3sbk/80GP/+7YnoapHXQpV+kGoy+eCGfvjXr05Q1Io9ghZQFk+iIj+gXQ4GtleTQyfIZiy3yOaal0FmtaUqJcb5PZsquY5fCIWQT2c/7dc2x9Lzw+04bJVTE8O6NLEW4ppXgU3rVcdZ3c3Uf00kyC7lHfOOnZxVatR2SgCanA5UZjx873o1dvd0mLSo5YZsRRzgoVkRr5UmK1Oeys28FwxVEJL5Tm8Y0kap+55pG/ldIzoe81pl6+MvPVHh1eSw2rN565o+SuumntiyPlrn1FR2vWSYNGWFyyIfMiGfiofPM1O+zcp9Zs7aXcqaOqWjXrerRFJ5N3Q5rg5uG9ZR6SA0XX81MdOkSnlSu1o2rugnV69ARbtTbxuCdy006z4O68ja4eQ2+tuIR5V5DQhfPdpKIeTPOaEpXhxzkvZ2rJ6NmpJFj/1EvK29bvWKNtZk77nF6zGm2o5+w5A2Usep2Ndo0bCGJ8Fdib+ZSqGbiidG98BTv47sQan8xcLmPlsphDwAlKXLR6GoDSsNINH2q5LrB8sJEqcQAW9f26fC+27LyCXDtCaV2Jxqw4CdjKhBJdF98eoBrZW38fSvex7zelSPpuUTmnT6DR8x7rxuePTC7trb0T7lE9EZAJ4AkAngeSHEON1tJuOIRdIhFZgOoojx7W2D0bKhHptovFaT6Lr57nV9cXyzelraNYnuny1srpEy6NhXGXl8Hm54I/L3Sfn1AQAD2uXi2kGtcVV/95PKu9f1xY9b9xlTMkb3MlMgXuv0R0SZAJ4GMBxAZwC/IiLD+VsjmAhwsHo4dGi9ALQJ+HgIVMFmXJDfQIsG9c+L9Ws3x3LsD+eZUA7wXGBav+nWtB6AyGRy5/BOSjxiCvIb4JLeLW3FNgQB3XfTC8AqIcQaIUQxgLcAjFLdyOY9hyyPSavJK+qH1XPes0V9RS15gymf73NPqJjD/ryeTSss11WRKKDCrHjrujfTq9iDJaXart3aZ4ne3KLbXNMUwIa41xsBHBOdRETXALgGAFq0cLZ8aVCzCmpVzUobqp3OJq8K014hJqiSmYHrBrXBmcfneaJofnfHEOwsKjZqFvJKxpuYXJr4qPKWG0rL9IalFrSsj7nr1HnDeYluTT7ZY3uMJBRCjBdCFAghCnJzne3In5TfAEvuH5bWHJJOk1dFCGU8iAhjh3dE16Z1PdFwm9WvoV3AV9h4DYEmP6ZfvtH2TD/7ByULzDulXeOKsqR7s+CV/gP0C/mNAJrHvW4GYJPmNpMypEPFCeTkNg3RKqcm8uqp0W7C5nqVSFg3ECuYazzS5VW2Gqu/GjZyakU8lXSn503m539p33ytbepCt5CfA6AdEbUioioARgP4WHObSWlYq+LGzKD2uZj8p8HKBoRsPdSgYlLGPzCqC246VU2SMCsSJ2eJ+hRMAqZMlW9c3QeD2udiqGYhn8w7qE2jYNrqtdrkhRBHiOj3AL5AxIXyBSHEUp1t2kG10Lp6QGtc2b812tw1Ue2FfYLJZFu/Nag1xcunS3q3wM2nmZlcElGVljcdun7Bni3rY9OizZqufpT2jWvj5St6aW/HJ9mnlaBdZxFCTBRCtBdCtBFCPKS7vXTUTogEVL0sJyLf5CbXQfydzb7rVM/6oZOHzu2GRrXNb072a9vQSOFwXfr2oxd2x8SbBmi6unmS1YsN6siuVAvTfm2PLVFn0vzQIclGjipMLZXjv69GIfHSAMxkHbxtWIe0n3sxsaikWnYmOluk2g4SqtIW+4FKJeQfufB43Dm8IzprTC+QCpkalX4nrLnRc5Ls15jm96e0VXq9VD9VOH9B9QQ5dXYiIboVa2pXy8a1g9qgd+sGxtvWqWybdl9LzNrIWGO12pKpYWqHdgHdJPQL8QrNcXUjq6yg6jiVcrTGMieqrFzjBbGHzpSMP9peuF1FdWBZa0DxV3ruCeGqbmSaqnGb4DHPvKC6pYbH8GSD6we3Qb3q2TjfoMujqvqT8RAiAj6iJepXM2Ib1WEM+tKN6e8srDENpmhar3r53/+8uAfmrdupLfeUbiqlkK+alYnLJQs5uCUrg/DMJT1xcsKmrwpOaFEf8wyGXsfkhlfZNoOM1erH1DfKv5x96lbPxsUnmckYqYNKKeR18+mN/TFpeSH++fVPAIDTuzTR0s6LY07Cmm1FyDJkI4/phiwo7MPzYnAJek6qSmmT103XpnW1F/EAInsKdgpFuyVmAgj4M5+Ubk3r4s7hHa0PdIhVBS0dFbaSwUacygcLeUaaMAuIT27sj2sH6ZuYL+3bMuVntatlGaseFcL5WTtB/85YyDPS8F6ec9KZ1HQFY917lif1eUJDnWoRa3bQ96BYyGsmTIKRPTb0oOtbHdOvFb78w0BNVw8/MbfXgMt4FvIM4zU6J8/2Cek07j+7i7a2wkZsn4Q1eSYpHDDEJPKfSwu87oI2T68wUu4yrLc+iXZYyGvGqwIUOjHp0RMmhnZujEcvjBQpP6fHceXv60xexzgnllE26Jo8+8lrIjOqBgxOUpEqyHx+ywAcFxcNyDgjgwgdm9TGii37jLjbMvaJ5a9hIc8kJSszA9NuH2Ikja1JOjYJTzpZT6Gjtnjez/YnYYnwZiGvkeYa8tUwwSbo0ZOVicxyTd7jjriEbfIMY5CYvIjfq2G570/CosmzkGcYg9SoEklXW7d6No6UmnHbiLXJ2KPcJs/eNQzDyDKiax7uObMzbhvWASsL9wMAtuw5pLXNvLrBLi3oFbzxyjCMbTIyCFf2PzbN9e6DJR71hklHRlQFDriMZ02eYbyidjQ3CvvJ+5OwaPIs5BnGI3RUC2PU8ceh7VElKwNtAl4vl801DOMRpvzjObGcMwZ3aISfHhzudTdcw5o8w3jE6GhJuab1OYKY0YcrTZ6IHgFwFoBiAKsBjBFC7I5+dieAKwGUArhJCPGFu64yTLj4TZ+W+E2f1MVEgsh71/fFoo17vO4GE4dbTf4rAF2FEMcD+AnAnQBARJ0BjAbQBcAZAJ4hInbWZRgPMGmsObFlA4zp18r6QMYYroS8EOJLIcSR6MuZAJpF/x4F4C0hxGEhxM8AVgHo5aYthmEYxj4qbfJXAPgs+ndTABviPtsYfa8CRHQNEc0lornbtm1T2B2GYYD0pQeZ8GNpkyeirwEkqzRwtxDio+gxdwM4AuD12GlJjk/qbCqEGA9gPAAUFBQE2yGVYXxI9eyIkH9pzEke94TxAkshL4Q4Ld3nRHQZgDMBnCqOptjbCKB53GHNAGxy2kmGYZwTc6GsVZU9pisjrtZxRHQGgDsAnC2EOBD30ccARhNRVSJqBaAdgNlu2mIYhmHs43ZqfwpAVQBfRbWFmUKI64QQS4noHQDLEDHj3CCEKHXZFsMwDqgWNddwUFTlxJWQF0K0TfPZQwAecnN9hmHc88+LeuDVmevQs0U9r7vCeAAb6Rgm5DSqUw23nt7B624wHsG+VQzDMCGGhTzDMEyIYSHPMAwTYljIMwzDhBgW8gzDMCGGhTzDMEyIYSHPMAwTYljIMwzDhBgSPqpETkTbAKxzcYkcANsVdcdr+F78SZjuBQjX/VTme2kphMhN9oGvhLxbiGiuEKLA636ogO/Fn4TpXoBw3Q/fS3LYXMMwDBNiWMgzDMOEmLAJ+fFed0AhfC/+JEz3AoTrfvhekhAqmzzDMAxzLGHT5BmGYZg4WMgzDMOEmMAJeSI6g4h+JKJVRDQ2yedERE9GP19ERD296KcMEvdySfQeFhHR90TU3Yt+ymJ1P3HHnUREpUR0gcn+2UHmXohoMBEtIKKlRPSt6T7KIvGc1SWiT4hoYfRexnjRTxmI6AUiKiSiJSk+D9L4t7oXNeNfCBGYfwAyAawG0BpAFQALAXROOGYEgM8AEIA+AGZ53W8X93IygPrRv4f79V5k7yfuuG8ATARwgdf9dvHb1EOkhnGL6OtGXvfbxb3cBeDv0b9zAewEUMXrvqe4n4EAegJYkuLzQIx/yXtRMv6Dpsn3ArBKCLFGCFEM4C0AoxKOGQXgFRFhJoB6RJRnuqMSWN6LEOJ7IcSu6MuZAJoZ7qMdZH4bALgRwHsACk12ziYy9/JrAO8LIdYDgBDCr/cjcy8CQG2KVPquhYiQP2K2m3IIIaYi0r9UBGX8W96LqvEfNCHfFMCGuNcbo+/ZPcYP2O3nlYhoKH7F8n6IqCmAcwE8Z7BfTpD5bdoDqE9EU4hoHhFdaqx39pC5l6cAdAKwCcBiADcLIcrMdE85QRn/dnE8/oNWyJuSvJfoAypzjB+Q7icRDUHkR+6vtUfukLmfxwHcIYQojSiNvkXmXrIAnAjgVADVAcwgoplCiJ90d84mMvcyDMACAKcAaAPgKyKaJoTYq7lvOgjK+JfG7fgPmpDfCKB53OtmiGgfdo/xA1L9JKLjATwPYLgQYoehvjlB5n4KALwVFfA5AEYQ0REhxIdGeiiP7HO2XQhRBKCIiKYC6A7Ab0Je5l7GABgnIsbfVUT0M4COAGab6aJSgjL+pVAy/r3efLC5UZEFYA2AVji6idQl4ZiROHbjZbbX/XZxLy0ArAJwstf9VXE/Cce/BP9uvMr8Np0ATIoeWwPAEgBdve67w3t5FsB90b8bA/gFQI7XfU9zT/lIvVkZiPEveS9Kxn+gNHkhxBEi+j2ALxDxGnhBCLGUiK6Lfv4cIl4bIxD5cg4goqX4Dsl7+QuAhgCeiWq/R4RPs+xJ3k8gkLkXIcRyIvocwCIAZQCeF0IkdYXzEsnf5QEALxHRYkSE4x1CCF+m7CWiNwEMBpBDRBsB3AsgGwjW+Aek7kXJ+Oe0BgzDMCEmaN41DMMwjA1YyDMMw4QYFvIMwzAhhoU8wzBMiGEhzzAM4xFWScqSHH8RES2LJpJ7Q+oc9q5hGIbxBiIaCGA/Ivl2uloc2w7AOwBOEULsIqJGQiJnEmvyDMMwHiGSJCkjojZE9Hk0J9I0IuoY/ehqAE+LaNIyGQEPsJBnGIbxG+MB3CiEOBHAnwA8E32/PYD2RDSdiGYS0RkyFwtUxCvDMEyYIaJaiOSR/19cEr+q0f+zALRDJEq2GYBpRNRVCLE73TVZyDMMw/iHDAC7hRA9kny2EcBMIUQJgJ+J6EdEhP4cqwsyDMMwPkBE0jv/TEQXAuXlDGNl/z4EMCT6fg4i5ps1VtdkIc8wDOMR0SRlMwB0IKKNRHQlgEsAXElECwEsxdFKXl8A2EFEywBMBnCbkEg/zC6UDMMwIYY1eYZhmBDDQp5hGCbEsJBnGIYJMSzkGYZhQgwLeYZhmBDDQp5hGCbEsJBnGIYJMf8P6UYRfC451GwAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "air_temperatures = get_noaa_temperatures('http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/', 'Gaylord', 100)\n", "plt.plot(air_temperatures)\n", "# plt.axis([0,1000,-20,80])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **DO THIS:** With your group, do a code review of the contents of the **noaa_scraper.py** file and figure out what it does. What are the main part of this module and what do they do? Be prepared to discuss this with the class. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Put your notes here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "\n", "# 2. Installing Beautiful Soup\n", "\n", "For this class we will be trying out [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/), a Python web parsing module. \n", "\n", "✅ **DO THIS:** Install the ```beautifulsoup4``` library on your computer (the following will work on jupyterhub but should work anywhere). When you are done, help your neighbor and raise your hand if you need help." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "#!mkdir packages" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "#!pip install -t ./packages/ beautifulsoup4" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "import sys\n", "sys.path.append('./packages/')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "# 3. Presidential data example\n", "This second example is a **_web scraper_** program. Found this idea by reading the following blog post: https://blog.exploratory.io/scraping-us-presidents-list-from-web-and-transforming-it-to-be-useful-fff534470bb6\n", "\n", "✅ **DO THIS:** Click on the following link and review the page source with your teams. Discuss which tags you need to look for to try and isolate the table data only. Ideally we want to create a ```pandas table``` of this data:\n", "https://www.loc.gov/rr/print/list/057_chron.html\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Put notes on what you find here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download and view html\n", "\n", "The following code should download the above website and parse read it into a ```beautifulsoup``` object:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "#The following library downloads the data and stores it in a page variable\n", "import requests\n", "page = requests.get(\"https://www.loc.gov/rr/print/list/057_chron.html\")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "#Import and run beautifule should html.parser\n", "from bs4 import BeautifulSoup\n", "soup = BeautifulSoup(page.content, 'html.parser')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **DO THIS:** explore the ```soup``` variable using python functions such as; ```type```, ```dir``` and ```help```.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Put your answer to the above here" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "##ANSWER##\n", "type(soup)\n", "##ANSWER##" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "##ANSWER##\n", "dir(soup)\n", "##ANSWER##" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "##ANSWER##\n", "help(soup)\n", "##ANSWER##" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Print out the raw html using \"pretty print\" \n", "print(soup.prettify())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Find the Tables\n", "\n", "Next, the following code finds all of the ```table``` sections in the website:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tables = soup.find_all('table')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "type(tables)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "len(tables)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "According to the above the results show that there are 9 ```table``` objects in the document. We are just looking for the one that has our data in it. \n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "✅ **DO THIS:** Find the table from the nine tables that has only the data we want. Make a variable ```table``` that only includes the information we want. Hint, it is not the first table which we can see by using the following code. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "table = tables[0]\n", "print(table.prettify())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "##ANSWER##\n", "table = tables[3]\n", "print(table.prettify())\n", "##ANSWER##" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parse out all the rows\n", "\n", "The rows of a table are determined by the ```tr``` (table row) tag and the columns are determined by the ```td```. The following code can find all of the rows in the table:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rows = table.find_all('tr')\n", "rows" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Get the column labels\n", "\n", "The first row is the column header row as can be seen by running the following code:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rows[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels = []\n", "for c in rows[0].find_all('th'):\n", " labels.append(c.get_text())\n", "labels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parse Rows\n", "\n", "✅ **DO THIS:** The next step is to loop though the remaining rows and save the data as a list of lists" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#put your code here" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "##ANSWER##\n", "\n", "data = [] \n", "for row in rows[1:]:\n", " myrow = []\n", " for c in row.find_all('td'):\n", " myrow.append(c.get_text())\n", " data.append(myrow)\n", "\n", "##ANSWER##" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convert list of list to Pandas Dataframe\n", "\n", "Assuming the above works, we can convert the list of lists and labels to a Pandas Dataframe" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd \n", " \n", "# Create the pandas DataFrame \n", "df = pd.DataFrame(data, columns=labels) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "\n", "\n", "# 4. Dynamic Website example\n", "\n", "The two above examples were fairly simple. However, sometimes websites get a lot more complex. This is especially true when the website includes \"client side\" code. This code (typically javascript) runs on the web browser in your local computer and not the web server. It makes the problem difficult because to pull the data out you often need to either figure out what the code is doing and mimic it in your python program or \"render\" the program using a javascript client and then figure out the output. \n", "\n", "\n", "Fortunatly there are tools that can help. Have your team do a google search and see if you can find some python tools specifically designed to help render dynamic websites. See if you can download/install and test the code.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "\n", "\n", "\n", "# 5. Wrap-up Discussion" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Written by Dr. Dirk Colbry, Michigan State University\n", "\"Creative
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" } }, "nbformat": 4, "nbformat_minor": 2 }