Vector_Dictionary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scrapping the page for a given URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import _pickle as pickle\n",
    "from os import walk\n",
    "import numpy as np\n",
    "import nltk\n",
    "import nltk, re, pprint\n",
    "from nltk import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "import gensim\n",
    "from gensim.models import Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/ranvir28singh/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the pickle file containing the articles extracted from web"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 314,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#os.chdir(\"/Users/ranvir28singh/Documents\")\n",
    "\n",
    "with open('700K_articles.p', 'rb') as fp:\n",
    "    e= pickle.load(fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 315,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'VETERANS saluted Worcester\\'s first ever breakfast club for ex-soldiers which won over hearts, minds and bellies. \\n \\nThe Worcester Breakfast Club for HM Forces Veterans met at the Postal Order in Foregate Street at 10am on Saturday. \\n \\nThe club is designed to allow veterans a place to meet, socialise, eat and drink, giving hunger and loneliness their marching orders. \\n \\nFather-of-two Dave Carney, aged 43, of Merrimans Hill, Worcester, set up the club after being inspired by other similar clubs across the country. \\n \\nHe said: \"As you can see from the picture, we had a good response. Five out of the 10 that attended said they saw the article in the newspaper and turned up. \\n \\n\"We even had an old chap travel from Droitwich and he was late on parade by three hours. \\n \\n\"It\\'s generated a lot of interest and I estimate (from other veterans who saw the article) that next month\\'s meeting will attract about 20 people. Onwards and upwards.\" \\n \\nHe said the management at the pub had been extremely hospitable to them. \\n \\nMr Carney said: \"They bent over backwards for us. They really looked after us well. That is the best choice of venue I could have made. They even put \\'reserved for the armed forces\\'. \\n   Promoted stories   \\nThe reserve veteran with the Royal Engineers wanted to go to a breakfast club but found the nearest ones were in Bromsgrove and Gloucester so he decided to set up his own, closer to home. \\n \\nHe was influenced by Derek Hardman who set up a breakfast club for veterans in Hull and Andy Wilson who set one up in Newcastle. He said the idea has snowballed and there were now 70 similar clubs across the country and even some in Germany. \\n \\nMr Carney said with many Royal British Legion clubs closing he wanted veterans and serving personnel to feel they had somewhere they could go for good grub, beer and banter to recapture the comradery of being in the forces. \\n \\nThe Postal Order was chosen because of its central location and its proximity to the railway station and hotels and reasonably priced food and drink. \\n \\nThe management of the pub have even given the veterans a designated area within the pub. \\n   \\n Share article  \\n   \\nThe next meeting is at the Postal Order on Saturday, October 3 at 10am. \\n \\nThe breakfast club meets on the first Saturday of each month for those who want to attend in future.'"
      ]
     },
     "execution_count": 315,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e['desc'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenising (only taking first 50 tokens) and building vocabulary dictionary(with unique words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "alldesc=[]\n",
    "allhead=[]\n",
    "#allword=[]\n",
    "uniquewords_set =set()\n",
    "for i in range(700000):\n",
    "    print('prop='+str(i/700000))\n",
    "    temp=word_tokenize(e['head'][i].translate ({ord(c): \"\" for c in \"—!@#$%^&*()[]{};.:,/'<>?\\|`\\\"~-=_+\"}))\n",
    "    count_word=0\n",
    "    temp_prop=''\n",
    "    for j in range(len(temp)):\n",
    "        if temp[j] in model :\n",
    "            uniquewords_set.add(temp[j])\n",
    "            temp_prop=temp_prop+temp[j]+' '\n",
    "            count_word=count_word+1\n",
    "        elif temp[j].lower() in model:\n",
    "            uniquewords_set.add(temp[j].lower())\n",
    "            temp_prop=temp_prop+temp[j]+' '\n",
    "            count_word=count_word+1\n",
    "        # max length of input or output=50    \n",
    "        if count_word>49:\n",
    "            break\n",
    "    #print(temp_prop)\n",
    "    \n",
    "    allhead.append(word_tokenize(temp_prop))\n",
    "    emp=word_tokenize(e['desc'][i].translate ({ord(c): \"\" for c in \"—!@#$%^&*()[]{};:.,/'<>?\\|`\\\"~-=_+\"}))\n",
    "    count_word=0\n",
    "    emp_prop=''\n",
    "    for j in range(len(emp)):\n",
    "        if emp[j] in model :\n",
    "            uniquewords_set.add(emp[j])\n",
    "            emp_prop=emp_prop+emp[j]+' '\n",
    "            count_word=count_word+1\n",
    "        elif emp[j].lower() in model:\n",
    "            uniquewords_set.add(emp[j].lower())\n",
    "            emp_prop=emp_prop+emp[j]+' '\n",
    "            count_word=count_word+1\n",
    "        # max length of input or output=50    \n",
    "        if count_word>49:\n",
    "            break\n",
    "    #print(emp_prop)             \n",
    "    alldesc.append(word_tokenize(emp_prop))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 318,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "uniquewords=list(uniquewords_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 319,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 319,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(alldesc[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 320,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Jumpshot',\n",
       " 'Gives',\n",
       " 'Marketers',\n",
       " 'Renewed',\n",
       " 'Visibility',\n",
       " 'Into',\n",
       " 'Paid',\n",
       " 'Organic',\n",
       " 'Keywords',\n",
       " 'With',\n",
       " 'Launch',\n",
       " 'Jumpshot',\n",
       " 'Elite']"
      ]
     },
     "execution_count": 320,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allhead[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 321,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "298948"
      ]
     },
     "execution_count": 321,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(uniquewords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating dump of the tokensied articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "aFile=open('final_dump_700k.p', 'wb')\n",
    "aFile.write(pickle.dumps({\"head\":allhead,\"desc\":alldesc}))\n",
    "aFile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Gensim's Word2Vec Model(https://radimrehurek.com/gensim/models/word2vec.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin.gz', binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3000000"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabg = list(model.vocab.keys())\n",
    "len(vocabg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['</s>', 'in', 'for', 'that', 'is', 'on', '##', 'The', 'with', 'said', 'was', 'the', 'at', 'not', 'as', 'it', 'be', 'from', 'by', 'are', 'I', 'have', 'he', 'will', 'has', '####', 'his', 'an', 'this', 'or', 'their', 'who', 'they', 'but', '$', 'had', 'year', 'were', 'we', 'more', '###', 'up', 'been', 'you', 'its', 'one', 'about', 'would', 'which', 'out']\n"
     ]
    }
   ],
   "source": [
    "print(vocabg[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assigning index to each word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "empty = 0 # RNN mask of no data\n",
    "eos = 1  # end of sentence\n",
    "start_idx = eos+1 \n",
    "def get_idx(vocabg):\n",
    "    #word2idx={}\n",
    "    \n",
    "    word2idx = dict((word, idx+start_idx) for idx,word in enumerate(vocabg))\n",
    "    word2idx['<empty>'] = empty\n",
    "    word2idx['<EOS>'] = eos\n",
    "    \n",
    "    \n",
    "    idx2word = dict((idx,word) for word,idx in word2idx.items())\n",
    "\n",
    "    return word2idx, idx2word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 323,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word2idx, idx2word = get_idx(uniquewords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 324,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "298950"
      ]
     },
     "execution_count": 324,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(word2idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "990524"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a vector dictionary for all words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 325,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vecs=[]\n",
    "count_boba=0\n",
    "for w in word2idx:\n",
    "    if w in model:\n",
    "        g =model[w]\n",
    "    elif w in model:\n",
    "        g = model[w.lower()]\n",
    "    else:\n",
    "        count_boba=count_boba+1\n",
    "    vecs.append(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting number of words not availible in gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "print(count_boba)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 327,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vecs=np.asarray(vecs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a dump of the vector dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\n",
    "with open('google_dict_700k.p','wb') as fp:\n",
    "    pickle.dump((vecs, idx2word, word2idx),fp,-1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}