01_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment and run this cell if you're on Colab or Kaggle\n",
    "# !git clone https://github.com/nlp-with-transformers/notebooks.git\n",
    "# %cd notebooks\n",
    "# from install import *\n",
    "# install_requirements()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using transformers v4.16.2\n",
      "Using datasets v2.1.0\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "from utils import *\n",
    "setup_chapter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hello Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"transformer-timeline\" caption=\"The transformers timeline\" src=\"images/chapter01_timeline.png\" id=\"transformer-timeline\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Encoder-Decoder Framework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"rnn\" caption=\"Unrolling an RNN in time.\" src=\"images/chapter01_rnn.png\" id=\"rnn\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"enc-dec\" caption=\"Encoder-decoder architecture with a pair of RNNs. In general, there are many more recurrent layers than those shown.\" src=\"images/chapter01_enc-dec.png\" id=\"enc-dec\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Mechanisms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"enc-dec-attn\" caption=\"Encoder-decoder architecture with an attention mechanism for a pair of RNNs.\" src=\"images/chapter01_enc-dec-attn.png\" id=\"enc-dec-attn\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"attention-alignment\" width=\"500\" caption=\"RNN encoder-decoder alignment of words in English and the generated translation in French (courtesy of Dzmitry Bahdanau).\" src=\"images/chapter02_attention-alignment.png\" id=\"attention-alignment\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"transformer-self-attn\" caption=\"Encoder-decoder architecture of the original Transformer.\" src=\"images/chapter01_self-attention.png\" id=\"transformer-self-attn\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer Learning in NLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"transfer-learning\" caption=\"Comparison of traditional supervised learning (left) and transfer learning (right).\" src=\"images/chapter01_transfer-learning.png\" id=\"transfer-learning\"/>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"ulmfit\" width=\"500\" caption=\"The ULMFiT process (courtesy of Jeremy Howard).\" src=\"images/chapter01_ulmfit.png\" id=\"ulmfit\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hugging Face Transformers: Bridging the Gap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Tour of Transformer Applications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"Dear Amazon, last week I ordered an Optimus Prime action figure \\\n",
    "from your online store in Germany. Unfortunately, when I opened the package, \\\n",
    "I discovered to my horror that I had been sent an action figure of Megatron \\\n",
    "instead! As a lifelong enemy of the Decepticons, I hope you can understand my \\\n",
    "dilemma. To resolve the issue, I demand an exchange of Megatron for the \\\n",
    "Optimus Prime figure I ordered. Enclosed are copies of my records concerning \\\n",
    "this purchase. I expect to hear from you soon. Sincerely, Bumblebee.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No model was supplied, defaulted to distilbert-base-uncased-finetuned-sst-2-english (https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n"
     ]
    }
   ],
   "source": [
    "#hide_output\n",
    "from transformers import pipeline\n",
    "\n",
    "classifier = pipeline(\"text-classification\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NEGATIVE</td>\n",
       "      <td>0.901546</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      label     score\n",
       "0  NEGATIVE  0.901546"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "outputs = classifier(text)\n",
    "pd.DataFrame(outputs)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No model was supplied, defaulted to dbmdz/bert-large-cased-finetuned-conll03-english (https://huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ignored unknown kwarg option direction\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity_group</th>\n",
       "      <th>score</th>\n",
       "      <th>word</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ORG</td>\n",
       "      <td>0.879010</td>\n",
       "      <td>Amazon</td>\n",
       "      <td>5</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MISC</td>\n",
       "      <td>0.990859</td>\n",
       "      <td>Optimus Prime</td>\n",
       "      <td>36</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>LOC</td>\n",
       "      <td>0.999755</td>\n",
       "      <td>Germany</td>\n",
       "      <td>90</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MISC</td>\n",
       "      <td>0.556571</td>\n",
       "      <td>Mega</td>\n",
       "      <td>208</td>\n",
       "      <td>212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PER</td>\n",
       "      <td>0.590255</td>\n",
       "      <td>##tron</td>\n",
       "      <td>212</td>\n",
       "      <td>216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ORG</td>\n",
       "      <td>0.669692</td>\n",
       "      <td>Decept</td>\n",
       "      <td>253</td>\n",
       "      <td>259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>MISC</td>\n",
       "      <td>0.498350</td>\n",
       "      <td>##icons</td>\n",
       "      <td>259</td>\n",
       "      <td>264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>MISC</td>\n",
       "      <td>0.775362</td>\n",
       "      <td>Megatron</td>\n",
       "      <td>350</td>\n",
       "      <td>358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>MISC</td>\n",
       "      <td>0.987854</td>\n",
       "      <td>Optimus Prime</td>\n",
       "      <td>367</td>\n",
       "      <td>380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PER</td>\n",
       "      <td>0.812096</td>\n",
       "      <td>Bumblebee</td>\n",
       "      <td>502</td>\n",
       "      <td>511</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  entity_group     score           word  start  end\n",
       "0          ORG  0.879010         Amazon      5   11\n",
       "1         MISC  0.990859  Optimus Prime     36   49\n",
       "2          LOC  0.999755        Germany     90   97\n",
       "3         MISC  0.556571           Mega    208  212\n",
       "4          PER  0.590255         ##tron    212  216\n",
       "5          ORG  0.669692         Decept    253  259\n",
       "6         MISC  0.498350        ##icons    259  264\n",
       "7         MISC  0.775362       Megatron    350  358\n",
       "8         MISC  0.987854  Optimus Prime    367  380\n",
       "9          PER  0.812096      Bumblebee    502  511"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner_tagger = pipeline(\"ner\", aggregation_strategy=\"simple\")\n",
    "outputs = ner_tagger(text)\n",
    "pd.DataFrame(outputs)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question Answering "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No model was supplied, defaulted to distilbert-base-cased-distilled-squad (https://huggingface.co/distilbert-base-cased-distilled-squad)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ignored unknown kwarg option direction\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.631291</td>\n",
       "      <td>335</td>\n",
       "      <td>358</td>\n",
       "      <td>an exchange of Megatron</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      score  start  end                   answer\n",
       "0  0.631291    335  358  an exchange of Megatron"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reader = pipeline(\"question-answering\")\n",
    "question = \"What does the customer want?\"\n",
    "outputs = reader(question=question, context=text)\n",
    "pd.DataFrame([outputs])    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No model was supplied, defaulted to sshleifer/distilbart-cnn-12-6 (https://huggingface.co/sshleifer/distilbart-cnn-12-6)\n",
      "Your min_length=56 must be inferior than your max_length=45.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Bumblebee ordered an Optimus Prime action figure from your online store in Germany. Unfortunately, when I opened the package, I discovered to my horror that I had been sent an action figure of Megatron instead.\n"
     ]
    }
   ],
   "source": [
    "summarizer = pipeline(\"summarization\")\n",
    "outputs = summarizer(text, max_length=45, clean_up_tokenization_spaces=True)\n",
    "print(outputs[0]['summary_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "599be06888b547eaacd399a129e784c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/750k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77f2f52b9a7c4a7eb2677e7e3b5c5bfc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/778k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2a5a3ffe96e74c978845c71980a90c23",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.21M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sehr geehrter Amazon, letzte Woche habe ich eine Optimus Prime Action Figur aus Ihrem Online-Shop in Deutschland bestellt. Leider, als ich das Paket öffnete, entdeckte ich zu meinem Entsetzen, dass ich stattdessen eine Action Figur von Megatron geschickt worden war! Als lebenslanger Feind der Decepticons, Ich hoffe, Sie können mein Dilemma verstehen. Um das Problem zu lösen, Ich fordere einen Austausch von Megatron für die Optimus Prime Figur habe ich bestellt. Anbei sind Kopien meiner Aufzeichnungen über diesen Kauf. Ich erwarte, bald von Ihnen zu hören. Aufrichtig, Bumblebee.\n"
     ]
    }
   ],
   "source": [
    "translator = pipeline(\"translation_en_to_de\", \n",
    "                      model=\"Helsinki-NLP/opus-mt-en-de\")\n",
    "outputs = translator(text, clean_up_tokenization_spaces=True, min_length=100)\n",
    "print(outputs[0]['translation_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from transformers import set_seed\n",
    "set_seed(42) # Set the seed to get reproducible results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dear Amazon, last week I ordered an Optimus Prime action figure from your online\n",
      "store in Germany. Unfortunately, when I opened the package, I discovered to my\n",
      "horror that I had been sent an action figure of Megatron instead! As a lifelong\n",
      "enemy of the Decepticons, I hope you can understand my dilemma. To resolve the\n",
      "issue, I demand an exchange of Megatron for the Optimus Prime figure I ordered.\n",
      "Enclosed are copies of my records concerning this purchase. I expect to hear\n",
      "from you soon. Sincerely, Bumblebee.\n",
      "\n",
      "Customer service response:\n",
      "Dear Bumblebee, I am sorry to hear that your order was mixed up. The order was\n",
      "completely mislabeled, which is very common in our online store, but I can\n",
      "appreciate it because it was my understanding from this site and our customer\n",
      "service of the previous day that your order was not made correct in our mind and\n",
      "that we are in a process of resolving this matter. We can assure you that your\n",
      "order\n"
     ]
    }
   ],
   "source": [
    "generator = pipeline(\"text-generation\")\n",
    "response = \"Dear Bumblebee, I am sorry to hear that your order was mixed up.\"\n",
    "prompt = text + \"\\n\\nCustomer service response:\\n\" + response\n",
    "outputs = generator(prompt, max_length=200)\n",
    "print(outputs[0]['generated_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Hugging Face Ecosystem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"ecosystem\" width=\"500\" caption=\"An overview of the Hugging Face ecosystem of libraries and the Hub.\" src=\"images/chapter01_hf-ecosystem.png\" id=\"ecosystem\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Hugging Face Hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"hub-overview\" width=\"1000\" caption=\"The models page of the Hugging Face Hub, showing filters on the left and a list of models on the right.\" src=\"images/chapter01_hub-overview.png\" id=\"hub-overview\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"hub-model-card\" width=\"1000\" caption=\"A example model card from the Hugging Face Hub. The inference widget is shown on the right, where you can interact with the model.\" src=\"images/chapter01_hub-model-card.png\" id=\"hub-model-card\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hugging Face Tokenizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hugging Face Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hugging Face Accelerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main Challenges with Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}